1402322 - Remove HBase Main Summary

Status: RESOLVED FIXED

(Data Platform and Tools Graveyard :: HBase, enhancement, P1)

Description

[Frank Bertsch [:frank]]

This job is failing over and over. Given the failures, it's hard to imagine anyone using it reliably.

We should:

1. Determine if anyone is using it
2. Come up with alternatives for their use case
3. Deprecate the job, HBase cluster, and related code (t-b-v and python_moztelemetry)

Comment 1

[Jason Thomas [:jason]]

CC'ing Mauro since he said he would take a look at the job timeout. +1 for getting rid of this cluster if we are not really using it.

Comment 2

[Mark Reid [:mreid]]

ni(rrayborn) since he has used the HBase API in the past.

Comment 3

[Frank Bertsch [:frank]]

This job has failed 26 out of the last 37 days. I think we can safely say that no one is using this cluster for anything.

Comment 4

[Jason Thomas [:jason]]

I am going to disable the job in airflow for now.

Comment 5

[Jason Thomas [:jason]]

https://github.com/mozilla/telemetry-airflow/pull/187

Comment 6

[Mark Reid [:mreid]]

Frank followed up with Rob separately and confirmed he doesn't need this functionality anymore.

Comment 7

[Frank Bertsch [:frank]]

Removed the HBase code from python_moztelemetry[0] and emailed fx-data-dev about the removal. We can probably stop the job on Monday and also remove the code from t-b-v.

[0] https://github.com/mozilla/python_moztelemetry/commit/e5dbc486855b1e76e7507f043d4e88fa56929774

Comment 8

[Frank Bertsch [:frank]]

Put in the PR for removing the job [0]. Flagged :mreid for review.

:jason, can you decommission the cluster?

[0] https://github.com/mozilla/telemetry-batch-view/pull/329

Comment 9

[Jason Thomas [:jason]]

Cluster has been decommissioned. Waiting a few days before I remove the S3 Bucket that contains the data.

Comment 10

[Jason Thomas [:jason]]

telemetry-hbase s3 bucket removed.