1329779 - Store spark application and history logs on EMR

Status: RESOLVED WONTFIX

(Data Platform and Tools Graveyard :: Operations, defect, P2)

Description

[Anthony Miyaguchi [:amiyaguchi]]

The applications logs for spark jobs scheduled on airflow should be stored for profiling. These application logs can be replayed by the spark history server to obtain metrics about a specific run in detail.

Currently these logs are not being archived. The application logs are being stored in `file:/tmp/spark-events` determined by the property `spark.history.fs.logDirectory`. The EMR operator [1] is a good place to look for uploading this file.

Another way to store these logs are to use the history server REST api at localhost:18080/api/v1 [2]. We can upload these zip files to a location in s3. An example of using this in shell is as follows:

> #!/bin/bash
> # Download the first application id
> app_id=`curl -s 'localhost:18080/api/v1/applications' | \
>         python -c "import sys, json; print json.load(sys.stdin)[0]['id']"`
> curl -o my_filename.zip localhost:18080/api/v1/applications/$app_id/logs


The spark api gives useful metrics for understanding the operation of a spark job and it would therefore be useful to keep this around.

[1] https://github.com/mozilla/telemetry-airflow/blob/master/dags/operators/emr_spark_operator.py#L114
[2] http://spark.apache.org/docs/latest/monitoring.html#rest-api

Comment 1

[Anthony Miyaguchi [:amiyaguchi]]

Attachment: spark-history-server.out.gz

We don't collect any history logs that are launched via EMR because they don't have access to the logging directory. This should probably be set via `spark.history.fs.logDirectory`. This would be very useful diagnostic material for failing, scheduled jobs.

Comment 2

[Anthony Miyaguchi [:amiyaguchi]]

I've encountered this bug a few times, I'll go ahead an assign this to myself.

Comment 3

[Anthony Miyaguchi [:amiyaguchi]]

Attachment: Bug 1335228 - Change owner of Spark log to spark instead of hadoop user. #61

This needs to be tested.

Comment 4

[Anthony Miyaguchi [:amiyaguchi]]

This is not a worthwhile effort now that we're migrating away from EMR. Logs are currently stored on Databricks.