1286825 - Airflow scheduler stopped working silently

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect)

Description

[Roberto Agostino Vitillo (:rvitillo)]

Apparently we are not the only ones experiencing this issue [1]. A workaround is to restart the scheduler "frequently".

[1] https://medium.com/handy-tech/airflow-tips-tricks-and-pitfalls-9ba53fba14eb#.q8na2qbpu "The scheduler should be restarted frequently"

Comment 1

[Roberto Agostino Vitillo (:rvitillo)]

See also https://github.com/apache/incubator-airflow/commit/34b585fec38b6ee7273116a1aa69ab76e1d4abe1

Comment 2

[Mark Reid [:mreid]]

It looks like the way this is typically handled is to set a limit on the number of runs the scheduler will process before stopping, then have some supervisor keep restarting it.

I'm currently testing to see if docker / ecs will handle the restarting for us if we just add a "-n 5" argument to the scheduler task.

If so, that should be all we need to do to work around the scheduler getting jammed up.

Comment 3

[Mark Reid [:mreid]]

Attachment: Add "num_runs" param to scheduler

Comment 4

[Mark Reid [:mreid]]

This worked for local testing, but failed on ECS deployment.

Next up: apply the hack mentioned in Comment 1

Comment 5

[Mark Reid [:mreid]]

Attachment: wrap the scheduler in a retry loop

Comment 6

[Mark Reid [:mreid]]

Roberto has deployed this change. We should keep an eye on this for a few days and make sure the scheduler no longer stops.

Comment 7

[Mark Reid [:mreid]]

I've confirmed that the scheduler is being restarted as expected in ECS:

$ cat /tmp/airflow_scheduler_errors.txt
...snip...
Tue Jul 19 16:11:09 UTC 2016
Tue Jul 19 16:11:35 UTC 2016
Tue Jul 19 16:12:01 UTC 2016
Tue Jul 19 16:12:27 UTC 2016
Tue Jul 19 16:12:53 UTC 2016
Tue Jul 19 16:13:19 UTC 2016