1335732 - Detect when we have an abnormal number of users on an old version

Status: REOPENED

(Release Engineering :: Release Automation, defect, P2)

Description

[Sylvestre Ledru [:Sylvestre]]

In bug 1277925, we identified that we had a bunch of users stuck on an old beta version.
As this was not on our radar for months, we should probably have a way to inform stakeholder that it is happening.

Email notification, dashboard, etc?

Comment 1

[Rail Aliiev [:rail]]

Bug 1334419 has some details on how we should test this kind of things.

Comment 2

[Sylvestre Ledru [:Sylvestre]]

Following a private irc chat with Rail:
I was more thinking about automation to say "we have an abnormal number users on the version XXX"
Not about this specific technical issue.

Maybe we should move that into telemetry or under some release management/engineer tooling

Comment 3

[Marco Castelluccio [:marco]]

I've built this query to see the share of users per version and per channel: https://sql.telemetry.mozilla.org/queries/3238.

We could write a simple script executed every day that warns us for weird percentages from that query (e.g. if a version, which is not the latest and not the latest - 1, is > 5%).