1546443 - add histogram PWMGR_NUM_SAVED_PASSWORDS to every main ping?

Status: RESOLVED WONTFIX

(Toolkit :: Password Manager, task, P3)

Description

[Josh Gaunt [:jgaunt]]

https://telemetry.mozilla.org/probe-dictionary/?detailView=histogram%2FPWMGR_NUM_SAVED_PASSWORDS

Looking at my own archived pings, it appears this histogram is only added to my main pings that are timestamped at midnight.

We (User Research) were hoping to use that probe's value to target normandy Heartbeat recipes to clients for in-product recruitment, but normandy only knows to look at the most recent archived ping, which of course isn't always the one made at midnight. If we included filtering criteria for that probe in a recipe, we would erroneously exclude clients who didn't have that probe in the archived ping that was most recent but otherwise satisfied the criteria.

As a pragmatic solution we can oversample clients without filtering on this histogram and then exclude them from analysis post-hoc after querying each clients' values with the Dataset api, but that's more work for more waste. We don't want to force 1000 people to take a survey to only include 100 if we could just have 100 people take a survey to begin with.

Ideally I could rely on this histogram being in every archived ping for normandy to check recipes against. Would that be intractably difficult, or an attainable request?

Comment 1

[Jan-Erik Rediger [:janerik]]

The Telemetry team doesn't own these probes.

It looks like they rely on a legacy feature of Telemetry, the gather-telemetry event.
That one is only (unreliably) invoked on idle-daily.

However, the main ping is sent on a more regular schedule, at shutdown and during environment changes. That's probably the reason why the data is not included in all main pings.
Unfortunately I don't really have a solution for you short of re-architecting how the PWMGR_ probes are handled.

Comment 2

[Matthew N. [:MattN]]

(In reply to Jan-Erik Rediger [:janerik] from comment #1)

It looks like they rely on a legacy feature of Telemetry, the gather-telemetry event.
That one is only (unreliably) invoked on idle-daily.

Is there a better way to record this telemetry that doesn't add extra work at component initialization/startup?

It looks like bug 1122066 was closed without a solution so idle-daily still looks like the best tool for the job to avoid doing unnecessary disk I/O.

Comment 3

[Jan-Erik Rediger [:janerik]]

(In reply to Matthew N. [:MattN] (PM me if requests are blocking you) from comment #2)

Is there a better way to record this telemetry that doesn't add extra work at component initialization/startup?

No automatic way I'm afraid. For now we keep supporting gather-telemetry exactly because we don't have an alternative yet.

I'll remove the ni?, so it shows up in triage tomorrow and I can discuss it with the team.

Comment 4

[Georg Fritzsche [:gfritzsche]]

This seems like something on the Password Manager side, moving it to a more suitable component.

Let us know if we can help, e.g. by having conversations on the recording mechanism here.

Comment 5

[Matthew N. [:MattN]]

(In reply to Georg Fritzsche [:gfritzsche] from comment #4)

Let us know if we can help, e.g. by having conversations on the recording mechanism here.

Did you read comment 2 and comment 3? Pointers to a better way to record this data are what's needed. e.g. could we move this data to the environment?

Comment 6

[Georg Fritzsche [:gfritzsche]]

Yes, i'm afraid the answer is that there is no mechanism that does what you want right now.

The constraints i'm aware of here are:

• PWMGR_ probes cause disk i/o, which (AFAICT here) shouldn't be triggered at arbitrary times.
• Telemetry collection/packaging has a fixed schedule at which it has to happen.
• Telemetry follows a push model, where you push your data onto the ping when you have it.

Thus gather-telemetry is not a mechanism that fits into the main ping model. Moving the data to the environment doesn't help with that AFAICT? (it doesn't solve the problem that the first main ping may not contain the PWMGR probes)

From that i think there are trade-off conversations needed about how to record/collect/cache the data on the password managers side?

Comment 7

[Matthew N. [:MattN]]

(In reply to Josh Gaunt [:jgaunt] from comment #0)

Ideally I could rely on this histogram being in every archived ping for normandy to check recipes against. Would that be intractably difficult, or an attainable request?

It wouldn't be too hard to gather this data when the password manager first initializes… it wouldn't guarantee we would always have the data but I think it would mean it would be included in a much larger percentage of pings. Would that be sufficient for this study? If not, I think we should WONTFIX.

Comment 8

[Josh Gaunt [:jgaunt]]

We were planning on trying this method regardless because although it would miss some clients without data it wouldn't falsely target anyone who didn't.

If it were present in a larger percentage of pings it would definitely boost our hit rate! That'd be great.

Comment 9

[Matthew N. [:MattN]]

OK, is it too late for your current study? We have other things going on right now so I'd rather wait to change this.

Comment 10

[Josh Gaunt [:jgaunt]]

In the worst case with the current status we can check telemetry post-hoc to verify the number of saved passwords before selecting a participant for recruitment.

That's to say, time isn't that essential and I can't make an argument to prioritize this over anything else. Being able to target up-front is nice mainly because it saves the time invested in checking telemetry afterward.

Comment 11

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Just a note for future travellers that the histogram is included in the ping after an idle-daily event fires, and not just in daily pings. idle-daily events fire approximately every 24 hours. If the browser is closed when an idle-daily event should have fired, an event will fire a few minutes after the next startup.

Whether the histogram is included in main pings, by ping reason: https://sql.telemetry.mozilla.org/queries/65266/source -- daily pings are the most likely to have them but they also appear in shutdown and other pings.