1361186 - txp mau dau to airflow

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement, P1)

Description

[Ilana]

Convert ATMO job txpmaudaudaily to airflow

Comment 1

[bugzilla]

Starting to work on this, targeting getting it into airflow tomorrow

Comment 2

[bugzilla]

Didn't finish this on Friday since I ended up making fairly significant changes, including simplifying the schema. Since I suspect this might be a fairly long-lived job, I think it makes sense to invest in changing the schema now and modifying the queries that rely on it. The schema change is large enough and would be disruptive enough that I think it might make more sense to do a complete backfill on this job to a whole new location and give query authors using this dataset a bit of lag time to change things before turning off the ATMO schedule. (I thought there was only one major query using this dataset but a quick search revealed this was not the case.)

Provided I don't run into any issues during testing I should have a PR for this job by the end of tomorrow and can start running the backfill (which should go quickly since this job is parquet -> parquet.)

Comment 4

[bugzilla]

Fixed back in May with https://github.com/mozilla/telemetry-airflow/pull/116