This morning Bug 1344639 caused our ETL jobs to fail. After the mirror came back online I tried to rerun the failed jobs but nothing happened as Airflow was in an inconsistent state, probably due to some Celery failures. 

I tried redeploying Airflow but I was getting some errors due to my ecs-cli version being outdated. I updated ecs-cli to 0.4.6 and redeployed. I tried to re-run the jobs and I was greeted with the following error:

An error occurred (UnrecognizedClientException) when calling the RunJobFlow operation: The security token included in the request is invalid

It turns out that the new version of ecs-cli, when used in combination with a docker compose file, sets environment variables also when they are empty unlike older versions.

To fix the issue I pushed a patch to telemetry-airflow to unset empty AWS credentials defined in the environment.