1271961 - Telemetry aggregation job is failing

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P1)

Description

[Roberto Agostino Vitillo (:rvitillo)]

The aggregation job has been failing in the past days for several reasons:

- Bug 1
  File "/home/hadoop/anaconda2/lib/python2.7/site-packages/mozaggregator/db.py", line 63, in <lambda>
    map(lambda x: (x[0][:4], _aggregate_to_sql(x))).\
  File "/home/hadoop/anaconda2/lib/python2.7/site-packages/mozaggregator/db.py", line 120, in _aggregate_to_sql
    histogram = _get_complete_histogram(channel, metric, payload["histogram"]) + [payload["sum"], payload["count"]]
  File "/home/hadoop/anaconda2/lib/python2.7/site-packages/mozaggregator/db.py", line 93, in _get_complete_histogram
    histogram = Histogram(metric, {"values": values}, revision=revision).get_value(autocast=False).values
  File "/home/hadoop/anaconda2/lib/python2.7/site-packages/moztelemetry/histogram.py", line 117, in __init__
    self.definition = histogram_tools.Histogram(name, histograms_definition[proper_name])
  File "/home/hadoop/anaconda2/lib/python2.7/site-packages/moztelemetry/histogram_tools.py", line 102, in __init__
    self.verify_attributes(name, definition)
  File "/home/hadoop/anaconda2/lib/python2.7/site-packages/moztelemetry/histogram_tools.py", line 225, in verify_attributes
    self.check_field_types(name, definition)
  File "/home/hadoop/anaconda2/lib/python2.7/site-packages/moztelemetry/histogram_tools.py", line 300, in check_field_types
    'should be {2}').format(key, name, type_name)
ValueError: value for key "keyed" in Histogram "ADDON_SHIM_USAGE" should be bool

- Bug 2:
  File "/home/hadoop/anaconda2/lib/python2.7/site-packages/mozaggregator/aggregator.py", line 57, in _sample_clients
    if not ping.get("environment", {}).get("settings", {}).get("telemetryEnabled", False):
AttributeError: 'float' object has no attribute 'get'

- Bug 3:
ImportError: No module named moztelemetry.spark

- Bug 4:
  File "/home/hadoop/anaconda2/lib/python2.7/site-packages/mozaggregator/db.py", line 65, in <lambda>
    map(lambda x: _upsert_build_id_aggregates(x[0], x[1], connection_string, dry_run=dry_run)).\
  File "/home/hadoop/anaconda2/lib/python2.7/site-packages/mozaggregator/db.py", line 158, in _upsert_build_id_aggregates
    cursor.execute("select merge_table(%s, %s, %s, %s, %s)", ('build_id', channel, version, build_id, stage_table_name))
ProgrammingError: relation "build_id_nightly_0_alpha" already exists
CONTEXT:  SQL statement "create table build_id_nightly_0_ALPHA as table staging_build_id_nightly_0_alpha"
PL/pgSQL function merge_table(text,text,text,text,regclass) line 11 at EXECUTE statement

Comment 1

[Roberto Agostino Vitillo (:rvitillo)]

Bug 2 and 4 are handled in [1]. Mark, could you please review it?

Bug 3 is due to Bug 1271207. 

Bug 1 is caused by histogram_tools. Georg, could we have seen this error before Bug 1245514 landed as well?

[1] https://github.com/mozilla/python_mozaggregator/pull/14

Comment 2

[Mark Reid [:mreid]]

Review feedback added.

Comment 3

[Georg Fritzsche [:gfritzsche]]

(In reply to Roberto Agostino Vitillo (:rvitillo) from comment #1)
> Bug 1 is caused by histogram_tools. Georg, could we have seen this error
> before Bug 1245514 landed as well?

If this is for "true" vs. true, then its new per this:
https://hg.mozilla.org/mozilla-central/diff/b35e412b5449/toolkit/components/telemetry/Histograms.json#l1.22
https://hg.mozilla.org/mozilla-central/diff/b35e412b5449/toolkit/components/telemetry/histogram_tools.py#l1.127

This probably means we need a follow-up bug on coercing "keyed" from "string" to "bool" in the pipeline in too.