1196796 - Extend local validation script with totalTime & browser defaults

Status: RESOLVED WORKSFORME

(Cloud Services Graveyard :: Metrics: Pipeline, defect)

Description

[Georg Fritzsche [:gfritzsche]]

We should extend the v2/v4 comparison in this script:
https://gist.github.com/georgf/1b0831a6b81b6c9fe240

... with totalTime & browser defaults.

Comment 1

[Georg Fritzsche [:gfritzsche]]

Any other fields?

Comment 2

[Georg Fritzsche [:gfritzsche]]

Updated the script:
* added comparisons of v2 totalTime to v4 totalTime, sessionLength, subsessionLength
* if on a "newish" profile on 42+, don't match up v2 or v4 dates if they cover roughly the same historic dates, use all the data instead
* show the cutoff date used for date matching (and when it wasn't used)

This gives fine totalTime comparisons on my first run on a new profile, but looks rather broken on my profile with long history.

Caveats:
* sessionLength was introduced relatively recently (Nightly >= 2015-07-30)
* when not on a newish profile on 42+, we have to try date matching, so certain discrepancies are expected

Comment 3

[Georg Fritzsche [:gfritzsche]]

So we can approach this from two angles:
* trying to reproduce big mismatches on a fresh profile on 42+
* for bigger mismatches in our clients, manually dig into the historic data to check this (potentially time consuming)

Comment 4

[Georg Fritzsche [:gfritzsche]]

Mismatch data from Katie:
https://docs.google.com/spreadsheets/d/1XCi_yYTb6RVO89lfnAAf1MTqBV7F2YXPXJYY-6fmDU8/edit#gid=0

Comment 5

[Georg Fritzsche [:gfritzsche]]

(In reply to Georg Fritzsche [:gfritzsche] from comment #4)
> Mismatch data from Katie:
> https://docs.google.com/spreadsheets/d/1XCi_yYTb6RVO89lfnAAf1MTqBV7F2YXPXJYY-
> 6fmDU8/edit#gid=0

The sesssion chaining problems there went away after waiting for 60sec after startup (i.e. after letting Telemetry load the session data from disk).
The mismatches persisted.

Comment 6

[Georg Fritzsche [:gfritzsche]]

Updated:
* added the current browser default
* consolidated the output in one tab
* v2 history is now shown in table form too for manual inspection

Comment 7

[Georg Fritzsche [:gfritzsche]]

Updated, mostly added the historical browser default validation.

Comment 8

[Georg Fritzsche [:gfritzsche]]

This part seems to work fine, lets collect any suspicious data and file new bugs for investigations.