1350907 - Validate telemetry experiment annotations using SHIELD

Status: RESOLVED INVALID

(Toolkit :: Telemetry, enhancement, P3)

Description

[Alessio Placitelli [:Dexter]]

We should validate the Telemetry experiment annotation using a limited experiment, in order to make sure that the API works and that the data we collect looks good.

Comment 1

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

Experiments plan for evaluation:


Experiment 1 :: Really do nothing, test collection and sampling

- in NIGHTYLY|AURORA|BETA (as far as uplifted),  ALL LOCALES
- sample the 'smallest' sample O(100)
- for a NON-EXISTENT PREF
- set it to (null) in both branches
- VERIFY
  - data collection


Experiment 2 :: Test Value Setting

- in NIGHTYLY|AURORA|BETA (as far as uplifted),  ALL LOCALES
- sample the 'smallest' sample O(100)
- for a NON-EXISTENT PREF
* NEW!  set it to ARBITRARY VALUES in both branches
- VERIFY
  - data collection
  - settings

Comment 2

[Georg Fritzsche [:gfritzsche]]

This should be available in all pre-release channels, up to Beta 53 from Thu, 3/30.

Comment 3

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

Per:  https://github.com/mozilla/normandy/issues/734

We are doing a test as soon as release-drivers says yes, which we expect by monday 8 May.

Comment 4

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Can you confirm that the first round of experiment will be on Nightly55 only?

Comment 5

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

:Ritu, updated test plan below

1. I confirm this is in Nightly55 only.  If the test is successful, we can discuss next test in beta and release.
2. This pref doesn't exist.  Normandy can set prefs that do not already exist.  There is No Need for any additional patching or uplift to the tree to do this test.

Updated test plan (replaces comment 1):

----

see:  https://github.com/mozilla/normandy/issues/734

Description: After POSITIVE OKAY ("explicit yes") from RELEASE-DRIVERS we intend to ship a TEST of the Shield (Normandy) Pref-flip experiments system.

Population Affected: 1% of Nightly Channel (only), all locales and user configurations.

Expected Effect:  the default pref  ` test.normandy.prefflip.defaultpref` will be set to one of 3 values [0,1,2], with unequal probability.  This preference has no known user facing effect.  This preference doesn't otherwise exist.

Cleanup:  after we determine data is coming in correctly (within 2 days), will close/end the test.  Closing the test will reset the preference.  Within 24 hours (when profiles next contact the server), all running profiles should be reset.

Goals:

    Per  https://bugzilla.mozilla.org/show_bug.cgi?id=1350907, CONFIRM that experiment annotations are working in a real population
    Per https://github.com/mozilla/normandy/issues/734, CONFIRM that these work
        sampling ratio
        pref-flip experiments
        rollback/cancellation
    Confirm/iterate on template language for notifying Release-Drivers about pref-flips.


Uplift needed:  None.  Pref doesn't exist in tree, which is okay with Shield/Normandy. 'experiment tagging' code is already in release.  Shield updates as a system addon.

GL
Fx S+I team.

Comment 6

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

The plan in comment 5 looks good. Let's do it!

Comment 7

[Alessio Placitelli [:Dexter]]

SHIELD is no longer a thing