1254547 - Parquet datasets are no longer accessible from Spark clusters

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P1)

Description

[Roberto Agostino Vitillo (:rvitillo)]

[hadoop@ip-172-31-20-15 ~]$ aws s3 ls s3://telemetry-parquet/longitudinal/

A client error (AccessDenied) occurred when calling the ListObjects operation: Access Denied

Comment 1

[Wesley Dawson [:whd]]

This didn't happen as part of the Spark 1.6 deploy. This happened because during that deploy I did a manual diff and noticed that somebody had added the parquet IAM permissions manually to the spark role. I made a note of this in https://bugzilla.mozilla.org/show_bug.cgi?id=1253392#c1 and was going to file a PR for it today (still am). As a consequence I did the CFN portion of the deploy manually to add the permissions for https://bugzilla.mozilla.org/show_bug.cgi?id=1253392 without losing the parquet permissions.

Later :rvitillo and :mreid attempted to deploy spark bootstrap updates with ansible for https://github.com/mozilla/emr-bootstrap-spark/pull/15 which aside from failing due to other IAM permissions issues wiped out the permissions not captured in version control.

I had a copy of the old permissions from when I did a diff so I've filed the PR to fix this: https://github.com/mozilla/emr-bootstrap-spark/pull/20

I'll close this once I've deployed it.

Comment 2

[Wesley Dawson [:whd]]

Deployed.