1370529 - Refactor Budget Dashboard ETL

Status: RESOLVED WONTFIX

(Data Platform and Tools :: General, enhancement, P3)

Description

[Mark Reid [:mreid]]

The Budget Dashboard[1] is powered by an ETL job that is running as a Jupyter notebook scheduled on ATMO.

It currently uses a dedicated data source in heka-framed protobuf form, processes data and incrementally updates a summary file.

The code should be updated to be idempotent.

We should determine the best path forward for this job, which could be:
- port the code to python_mozetl and add tests
- migrate to telemetry-batch-view
- change the data source to use direct-to-parquet output and either use SparkSQL for the job or move the dashboard into re:dash.
- other possibilities?

I believe the notebook has some embedded assumptions about user counts, so it may not be able to be publicly shared. I can provide the current notebook separately as needed.

[1] https://metrics.services.mozilla.com/telemetry-budget-dashboard/

Comment 1

[Mark Reid [:mreid]]

Attachment: budget_report.ipynb

I had a look and the notebook fetches the targets as metadata so the code seems fine to be public.

Comment 2

[Frank Bertsch [:frank]]

I don't want to generally take over this bug, but I am taking over the d2p for payload-size messages.

Comment 3

[Mark Reid [:mreid]]

direct-to-parquet output added here:
https://github.com/mozilla-services/mozilla-pipeline-schemas/pull/103

Thanks Frank!