1137424 - Implement v2 HTTP Edge

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P1)

Description

[Mark Reid [:mreid]]

Implement the full HTTP Edge Spec here:
https://wiki.mozilla.org/CloudServices/DataPipeline/HTTPEdgeServerSpecification

Using a SandboxInput, we'll have full control over the parts that couldn't easily be added to the existing Heka HttpListenInput, primarily the server's response codes.

Comment 1

[Mark Reid [:mreid]]

We will need this before cutting the "tee" server over to the Heka-based HTTP edge server. The tee is currently returning the HTTP response from the old server.

Comment 2

[Mike Trinkala [:trink]]

In the recent weeks the current edge server has caused two minor outages/data loss.  The redesign is over due and should consist of a few days of work.  Getting it in test now makes deployment viable early in Q1.  For simplicity it will be an Nginx upstream module that produces a Heka message and pushes it to a Kafka topic (no decoding/validation, that will be moved from the edge back to the data warehouse loader)

Comment 3

[Mike Trinkala [:trink]]

After triage bumping the priority down 1 more

Comment 4

[Mark Reid [:mreid]]

Re: the soon-to-be-ingested SSL reports, we need to update the spec (and the code) to make the "id" field in the URL optional.

Comment 5

[Mike Trinkala [:trink]]

https://github.com/mozilla-services/nginx_moz_ingest