1309543 - Terminated clusters should not be visible from the UI

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P1)

Comment 1

[Jannis Leidel [:jezdez]]

Makes sense, this is somewhat related to bug 1309536. It doesn't make sense to show terminated clusters in the list other than for archive reasons. So I suggest to fix this by doing two things:

1) Order clusters by date and show active clusters first
2) Add a user facing feature to archive clusters that hides them from the users (but doesn't delete it)

For 2) we'd need a new ticket if you agree

Comment 2

[Roberto Agostino Vitillo (:rvitillo)]

(In reply to Jannis Leidel [:jezdez] from comment #1)
> Makes sense, this is somewhat related to bug 1309536. It doesn't make sense
> to show terminated clusters in the list other than for archive reasons. So I
> suggest to fix this by doing two things:
> 
> 1) Order clusters by date and show active clusters first
> 2) Add a user facing feature to archive clusters that hides them from the
> users (but doesn't delete it)
> 
> For 2) we'd need a new ticket if you agree

What about auto-archiving terminated clusters and have a separate page with the history of all the spawned clusters?

Comment 3

[Jannis Leidel [:jezdez]]

(In reply to Roberto Agostino Vitillo (:rvitillo) from comment #2)
> (In reply to Jannis Leidel [:jezdez] from comment #1)
> > Makes sense, this is somewhat related to bug 1309536. It doesn't make sense
> > to show terminated clusters in the list other than for archive reasons. So I
> > suggest to fix this by doing two things:
> > 
> > 1) Order clusters by date and show active clusters first
> > 2) Add a user facing feature to archive clusters that hides them from the
> > users (but doesn't delete it)
> > 
> > For 2) we'd need a new ticket if you agree
> 
> What about auto-archiving terminated clusters and have a separate page with
> the history of all the spawned clusters?

Sure sounds equally good. Would that be a per-user-archive? Or something only admins can see?

The latter case can be handled by Django's admin in which we can add a list filter for terminated clusters.

Comment 4

[Roberto Agostino Vitillo (:rvitillo)]

(In reply to Jannis Leidel [:jezdez] from comment #3)

> > What about auto-archiving terminated clusters and have a separate page with
> > the history of all the spawned clusters?
> 
> Sure sounds equally good. Would that be a per-user-archive? Or something
> only admins can see?
> 
> The latter case can be handled by Django's admin in which we can add a list
> filter for terminated clusters.

I think it makes sense to have a per-user-archive if in the cluster summary page there is a link to the logs of the cluster (Bug 1307528). That would come in handy if users want to report failures to us.

Comment 5

[Jannis Leidel [:jezdez]]

(In reply to Roberto Agostino Vitillo (:rvitillo) from comment #4)
> (In reply to Jannis Leidel [:jezdez] from comment #3)
> 
> > > What about auto-archiving terminated clusters and have a separate page with
> > > the history of all the spawned clusters?
> > 
> > Sure sounds equally good. Would that be a per-user-archive? Or something
> > only admins can see?
> > 
> > The latter case can be handled by Django's admin in which we can add a list
> > filter for terminated clusters.
> 
> I think it makes sense to have a per-user-archive if in the cluster summary
> page there is a link to the logs of the cluster (Bug 1307528). That would
> come in handy if users want to report failures to us.

Excellent

Comment 6

[Frank Bertsch [:frank]]

Well now that we've got terminated clusters being kept around, do we want to also have their logs available to see from the web ui?

Comment 7

[Roberto Agostino Vitillo (:rvitillo)]

(In reply to Frank Bertsch [:frank] from comment #6)
> Well now that we've got terminated clusters being kept around, do we want to
> also have their logs available to see from the web ui?

Ideally yes.

Comment 8

[Jannis Leidel [:jezdez]]

(In reply to Roberto Agostino Vitillo (:rvitillo) from comment #7)
> (In reply to Frank Bertsch [:frank] from comment #6)
> > Well now that we've got terminated clusters being kept around, do we want to
> > also have their logs available to see from the web ui?
> 
> Ideally yes.

Should the logs automatically be fetch by the cluster detail page, or on request only? How big are cluster logs generally?

Comment 9

[Jannis Leidel [:jezdez]]

I've filed bug 1310157 to track the log showing feature

Comment 10

[Jannis Leidel [:jezdez]]

Attachment: Current status with activity filter in dashboard

This is being worked on in https://github.com/mozilla/telemetry-analysis-service/pull/37 via bug 1309227

Comment 11

[Roberto Agostino Vitillo (:rvitillo)]

(In reply to Jannis Leidel [:jezdez] from comment #8)
> (In reply to Roberto Agostino Vitillo (:rvitillo) from comment #7)
> > (In reply to Frank Bertsch [:frank] from comment #6)
> > > Well now that we've got terminated clusters being kept around, do we want to
> > > also have their logs available to see from the web ui?
> > 
> > Ideally yes.
> 
> Should the logs automatically be fetch by the cluster detail page, or on
> request only? How big are cluster logs generally?

That really depends on the job. I don't think we necessarily have to retrieve the logs but we should to the very least have a pointer to them so that users can easily retrieve them if needed, as we did in atmo v1.

Comment 12

[Roberto Agostino Vitillo (:rvitillo)]

s/as we did in atmo v1/as we do with airflow [1]

[1] https://github.com/mozilla/telemetry-airflow/blob/master/dags/operators/emr_spark_operator.py#L136

With this approach users without AWS credentials cannot access the logs. That said, pointers to logs for ad-hoc jobs aren't that useful for users as they can inspect logs directly on their clusters. Links to the logs are useful for the Data Platform team for post-mortem debugging sessions though.

Comment 13

[Roberto Agostino Vitillo (:rvitillo)]

You can find an example of all the logs generated for an EMR cluster here: s3://telemetry-airflow/logs/[e-mail]/Crash Aggregate View/j-15GLG06TV5GQ4

Comment 14

[Jannis Leidel [:jezdez]]

Closing this bug as fixed since we hide terminated clusters by default, let's use bug 1310157 for further discussion about the log files