Closed Bug 1283486 Opened 9 years ago Closed 9 years ago

EMR Airflow operator should support a timeout parameter.

Categories

(Cloud Services Graveyard :: Metrics: Pipeline, defect)

Product:
Cloud Services Graveyard
Component:
Metrics: Pipeline
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: rvitillo, Unassigned)

References

Details

The timeout parameter is needed to kill clusters after a certain period of time in case they end up in e.g. an infinite loop.
Airflow's BaseOperator[1] has a parameter for "execution_timeout" which can be used for this. It's already in use for the main_summary scheduled job[2] and appears to work. Do we need something more than this? [1] http://pythonhosted.org/airflow/code.html#baseoperator [2] https://github.com/mozilla/telemetry-airflow/blob/master/dags/main_summary.py#L26
Flags: needinfo?(rvitillo)
Since it works as advertised we can close this Bug.
Status: NEW → RESOLVED
Closed: 9 years ago
Flags: needinfo?(rvitillo)
Resolution: --- → FIXED
I double checked one of the clusters (EMR id j-WQRQLAES4Y3X) that got automatically shutdown during testing - the cluster was terminated after 6 hours 2 minutes (with an execution timeout of 6 hours).
Product: Cloud Services → Cloud Services Graveyard
You need to log in before you can comment on or make changes to this bug.