1274968 - Configure Airflow's alerting system.

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P1)

Description

[Roberto Agostino Vitillo (:rvitillo)]

Airflow can send an e-mail to the owner of a job when it fails. We should configure it to use Amazon's SES SMTP interface [1].

[1] https://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-email-smtp.html

Comment 1

[Roberto Agostino Vitillo (:rvitillo)]

Blake, I need to create SMTP credentials for Airflow [1] but I am unable to do so because I don't have privileges to create IAM users. Could you please create the credentials?

[1] https://docs.aws.amazon.com/ses/latest/DeveloperGuide/smtp-credentials.html

Comment 2

[Blake Imsland [:robotblake]]

SES account has been created and information provided to Roberto.

Comment 3

[Mark Reid [:mreid]]

Where should we keep secrets such as these credentials? I thought maybe we could use the Airflow "Variables" support, but it doesn't look like you can reference those values from the configuration.

We could store them in the metadata bucket and fetch them from `entrypoint.sh` where we do the other variable substitution in airflow.cfg.

Comment 4

[Mark Reid [:mreid]]

Attachment: Configure SMTP