1392246 - Main Summary DAG hasn't run for 2 days

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement)

Description

[Mark Reid [:mreid]]

I got an alert from this re:dash query:
https://sql.telemetry.mozilla.org/queries/4920

I initially thought it was the hive partition import step, but I saw that the data is actually missing since 08/18. Airflow doesn't show any failures or successes (no runs at all) for 08/19 or 08/20.

Comment 1

[Mark Reid [:mreid]]

Jason, can you take a look at the Airflow server and see if you see anything that would prevent the Main Summary DAG from being run?

Comment 2

[bugzilla]

Starting from the 8/18 run these dags stopped running:
main_summary
bugzilla_dataset
mobile_clients
crash_summary
sync_view
socorro_import
txp_pulse
tab_spinner_severity
Additionally, longitudinal should have run over the weekend.

The dags were never triggered at all. There isn't much in the way of commonality between these that I can see.

Comment 3

[Mark Reid [:mreid]]

Jason appears to be PTO. Whd, can you take a look?

Also, it appears the last Churn job didn't run either.

Comment 4

[Wesley Dawson [:whd]]

I filed bug #1392310 for the underlying issue. Airflow is running dags now, but it looks like we may need to re-run jobs for 8/19 manually since the scheduler started running jobs from 8/20.

Comment 5

[Mark Reid [:mreid]]

Sunah took care of running the missed day's Main Summary DAG. Thanks!