1279147 - Move parquet2hive jobs off of the Presto EMR master

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P2)

Description

[Roberto Agostino Vitillo (:rvitillo)]

I had to restart Presto on the master node as the JVM was using nearly all the available memory which was causing other things to fail (like parquet2hive). Blake, could you please limit [1] the total memory size that the Presto processes are allowed to consume?

[1] https://github.com/vitillo/emr-bootstrap-presto/blob/master/ansible/files/telemetry.sh#L57

Comment 1

[Blake Imsland [:robotblake]]

This brings up a bit of a meta question, we're already running hive and presto on these instances and it seems like a waste to limit their memory all the time to deal with the case of needing to run something like parquet2hive.  I'm happy to make the change in this case but would it make sense to run those jobs somewhere else?

Comment 2

[Roberto Agostino Vitillo (:rvitillo)]

Blake, we can't run parquet2hive somewhere else until the centralized metastore lands. This would be a temporary fix and I don't expect to reduce the amount of memory dedicated to Presto by more than a few GB. What's the ETA for the centralized metastore? And once it lands, how long would it take you to setup a remote metastore update process? Do you have an alternate suggestion on how to avoid this situation from happening again until then?

Comment 3

[Blake Imsland [:robotblake]]

I've got a tentative way to handle these outside of the presto machines that I'm currently investigating, assuming everything looks good this can land very soon after we verify that the new Presto EMR cluster is working as intended.

Comment 4

[Frank Bertsch [:frank]]

Isn't this obsolete now that the centralized metastore has landed?

Comment 5

[Blake Imsland [:robotblake]]

This will be taken care of by https://github.com/mozilla/emr-bootstrap-presto/pull/8