1357875 - Add `topline_dashboard` to python_etl

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement, P1)

Description

[Anthony Miyaguchi [:amiyaguchi]]

`topline_dashboard` is reformats the Topline Summary view to accommodate the topline/executive report. This does the following things:

1. Marginalize the dataset to a limited set of countries and Other/ROW
2. Compute `ALL` rows
3. Collect and upload the csv to the dashboard view

Comment 1

[Anthony Miyaguchi [:amiyaguchi]]

This script will replace both run.sh and v4_reformat.py in the original reporting pipeline. The general approach will be to take the union of historical data and reformatted topline_summary data. For reference, run.sh appends this week/month's data to the end the csv file.

I plan to put 'v4-monthly.csv' and 'v4-weekly.csv' in a new, read-only location, such as `net-mozaws-prod-us-west-2-pipeline-analysis/topline/historical`. Are there any restrictions on the raw data that would prevent it from being put here?

Alternatively, this data could live next to the new data under a v0 tag if it were imported to telemetry-parquet.

Comment 2

[Mark Reid [:mreid]]

There's nothing in the data that would prevent it from being put into the 'analysis' bucket you mentioned.

Comment 3

[Anthony Miyaguchi [:amiyaguchi]]

Attachment: Bug 1357875 - Add script for generating topline dashboard data #18

Comment 4

[Anthony Miyaguchi [:amiyaguchi]]

This has been merged into python_mozetl.

Comment 5

[Anthony Miyaguchi [:amiyaguchi]]

Attachment: Bug 1357875 - Use collect instead of writing directly to `file://` via spark #27

Comment 6

[Anthony Miyaguchi [:amiyaguchi]]

The `file://` protocol doesn't work on EMR, due to permission issues. The workaround it to collect the dataframe and write directly to disk. For some unknown reason, this causes tests to fail, despite functionally being the same thing. In practice, the above patch sucessfully collects the dataframe and uploads it.