1309622 - Move Airflow deployment away from ECS

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect, P1)

Description

[bugzilla]

- Investigate alternative deployment options (Heroku? AWS-based deployment services? Remove some steps [e.g. Ansible]?)
- Migrate repo & production deployment to the best alternative

Comment 1

[Jannis Leidel [:jezdez]]

Looking into this a notch, Airflow can be run on Heroku as shown in https://github.com/heroku/heroku-airflow

Alternatively I think following Cloudops' dockerflow (https://github.com/mozilla-services/Dockerflow) makes sense if we decide to continue to run Airflow on our own infrastructure.

For the matter, the Python library that implements dockerflow patterns for Python project (https://github.com/mozilla-services/python-dockerflow) would need to get a Flask extension. Ironically the heroku-airflow shows how to implement a custom flask extension as a Airflow plugin: https://github.com/heroku/heroku-airflow/blob/b87a5a83198c1b00072c41f5f4ff0532e9534b19/plugins/ssl_everything.py

Comment 2

[GitHub Autolander Bot]

Attachment: [telemetry-airflow] mozilla:bug1309622 > mozilla:master

Comment 3

[Jannis Leidel [:jezdez]]

Jason has reviewed and approved my PR that changes our Airflow instance to use Dockerflow patterns instead of Ansible/Beanstalk.

I think it would make sense to hand this over to ops now to set up the WTMO (Airflow) instance similarly to ATMO.

I'll be working on adding Flask support to python-dockerflow ASAP and update the Airflow code but I don't think we should block on this since we need to let the move to Dockerflow happen first before so we can update Airflow to use spot instances.

Can you think of something that is missing from the Airflow side?

Comment 4

[Jason Thomas [:jason]]

(In reply to Jannis Leidel [:jezdez] from comment #3)
> 
> Can you think of something that is missing from the Airflow side?

As of right now things look good, but if we run into issues we will reach out to you. Bug 1336975 for the actual migration into JenkinsV2.

Comment 5

[Wesley Dawson [:whd]]

https://github.com/mozilla/telemetry-airflow/pull/76 and https://github.com/mozilla/telemetry-airflow/pull/110.