1321316 - Backfill: Reprocess 1 month of main_summary to add engagement scalars

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P1)

Description

[Mark Reid [:mreid]]

Target time period:
20161029 to 20161129

Comment 1

["Saptarshi Guha[:joy]"]

Can we make this 20161001 to 20161129 (since last 15 days is always wonky)

Comment 2

["Saptarshi Guha[:joy]"]

Can we make this 20161001 to 20161129 (since last 15 days is always wonky)

Comment 3

[Mark Reid [:mreid]]

I'm just about done with 20161029 to 20161129 (part way though swapping in the updated files for 20161128). Please take a look at that period first, and if it looks like we'll need more data I will backfill further.

Note that if you are loading this data in Spark, you may need to set the 'mergeSchema' flag per:
https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.DataFrameReader.parquet

I will update again when this period is completely finished, should be within the hour.

Comment 4

[Mark Reid [:mreid]]

Ok, import of the original period is done.

Saptarshi, please take a look and let me know if you'll need more backfill.

Comment 5

[Mark Reid [:mreid]]

Calling this done for now. Please re-open if we need more backfill.