1299840 - Log requests against mozaggregator library

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, enhancement, P1)

Description

[Frank Bertsch [:frank]]

While working on Bug 1294694, it became difficult to understand how various groups are using t.m.o. without specifically going around and asking everyone. If we logged requests against aggregates.t.m.o., it would give us a better idea of how end users are using t.m.o. (and telemetry.js), which we could use to inform our decisions about expanding/pruning/migrating the aggregates dataset in the future.

Comment 2

[Frank Bertsch [:frank]]

Initially, I am essentially just going to store a log with timestamp and request. If there's other metadata we have about requests I might include that as well, for example, where it came from (could separate out TMO & telemetry.js users).

I'm hoping to just write these to a log, and upload that log to S3 after X time, X bytes, X rows, or shutdown. Currently looking at tools so I don't have to roll my own for this, there's a few nasty corner cases.

Comment 3

[Frank Bertsch [:frank]]

It doesn't look like New Relic will work with this. It's limited to 7 days retention on transactions, which will not be nearly enough. New Relic will be nice for the performance monitoring and error reporting, though.

Comment 4

[Frank Bertsch [:frank]]

Decided to use Papertrail. I have an account I setup, and it is backing up daily to an s3 bucket: s3://net-mozaws-dev-us-west-2-telemetry-aggregates-logs. PR 26 logs requests to this service.

Comment 5

[Frank Bertsch [:frank]]

Cloudwatch Logs is running in Prod now.