1159312 - Roles in moz-svc-dev have been completely wiped out

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P1)

Description

[Roberto Agostino Vitillo (:rvitillo)]

The telemetry analysis infrastructure is in an unusable state as all roles have been wiped out in moz-svc-dev. 

Can the EMR_DefaultRole, telemetry-spark-emr and lambda_telemetry_index_ping be restored?

Also my personal credentials have been rendered useless.

Comment 1

[Wesley Dawson [:whd]]

:ckolos and I are working on this presently.

Comment 2

[Chris Kolosiwsky [:ckolos] (ckolos has left the building)]

The following roles were deleted and have been recovered:

telemetry-spark-emr
telemetry-public-analysis-worker
telemetry-private-analysis-worker

stacks utilizing those roles will need to be updated. please test and advise.

Comment 3

[Chris Kolosiwsky [:ckolos] (ckolos has left the building)]

All other roles in the telemetry infrastructure are believed to be generated via cfn and should be recreated on stack update/creation

Comment 4

[Roberto Agostino Vitillo (:rvitillo)]

The EMR_DefaultRole, telemetry-spark-emr and lambda_telemetry_index_ping roles have been restored and tested. Once https://telemetry-dash.mozilla.org/ is working again it should be possible to start Spark clusters again.

Comment 5

[Mark Reid [:mreid]]

I spun up a new cloudformation stack for telemetry-dash.m.o and all appears to be working now.