Closed Bug 1341566 Opened 9 years ago Closed 7 years ago

python_moztelemetry: update protobuf to 3.2 and regenerate protobuf classes

Categories

(Data Platform and Tools :: General, defect, P1)

Product:
Data Platform and Tools
Component:
General
Type:
defect
Points:
1

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: rvitillo, Assigned: klukas)

References

Details

(Whiteboard: [DataPlatform])

User Story

Our aggregation job was failing with the error below. It appears that there is a new version of the protobuf library (3.2) which doesn't like classes [1] generated with an older version of the library. Regenerating those classes should fix the issue. It would be convenient to automate this so that this won't happen again next time the protobuf library is updated.

vagrant@vagrant-ubuntu-trusty-64:/vagrant$ ~/miniconda2/bin/python ./mozaggregator/service.py
./mozaggregator/service.py:8: ExtDeprecationWarning: Importing flask.ext.cors is deprecated, use flask_cors instead.
  from flask.ext.cors import CORS
./mozaggregator/service.py:9: ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use flask_cache instead.
  from flask.ext.cache import Cache
Traceback (most recent call last):
  File "./mozaggregator/service.py", line 11, in <module>
    from moztelemetry.histogram import Histogram
  File "/home/vagrant/miniconda2/lib/python2.7/site-packages/moztelemetry/__init__.py", line 1, in <module>
    from spark import *
  File "/home/vagrant/miniconda2/lib/python2.7/site-packages/moztelemetry/spark.py", line 11, in <module>
    from .dataset import Dataset
  File "/home/vagrant/miniconda2/lib/python2.7/site-packages/moztelemetry/dataset.py", line 15, in <module>
    from .heka import message_parser
  File "/home/vagrant/miniconda2/lib/python2.7/site-packages/moztelemetry/heka/message_parser.py", line 15, in <module>
    from .message import Message, Header
  File "/home/vagrant/miniconda2/lib/python2.7/site-packages/moztelemetry/heka/message.py", line 26, in <module>
    type=None),
  File "/home/vagrant/miniconda2/lib/python2.7/site-packages/google/protobuf/descriptor.py", line 649, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors should not be created directly, but only retrieved from their parent.

[1] https://github.com/mozilla/python_moztelemetry/blob/master/moztelemetry/heka/message.py

Attachments

(2 files)

Link to GitHub pull-request: https://github.com/mozilla/python_moztelemetry/pull/235
55 bytes, text/x-github-pull-request
klukas
: review+
Details | Review
Link to GitHub pull-request: https://github.com/mozilla/emr-bootstrap-spark/pull/583
55 bytes, text/x-github-pull-request
klukas
: review+
Details | Review
No description provided.
Points: --- → 1
Priority: -- → P2
What's the process to regenerate these classes? Is it documented anywhere?
Flags: needinfo?(mreid)
+1ing this, because `message.py` appears to be generated from proto2 which blocks updating moztelemetry to support python 3.
Blocks: 1268053
Did some code archaeology, and found a link to the source: https://github.com/mozilla/telemetry-server/blob/master/telemetry/util/heka_message.py#L8 The protobuf definition itself is here: https://github.com/mozilla-services/heka/blob/dev/message/message.proto I believe I just used "protoc" to generate python source originally, though I may have had to tweak a couple of the go-specific options in the definition above.
Flags: needinfo?(mreid)
Component: Metrics: Pipeline → Telemetry APIs for Analysis
Product: Cloud Services → Data Platform and Tools
Summary: python_moztelemetry doesn't work with protobuf 3.2 → python_moztelemetry: update protobuf to 3.2 and regenerate protobuf classes
Whiteboard: [DataPlatform]
No longer blocks: 1268053
Assignee: nobody → jklukas
Priority: P2 → P1
Most of the references I can find make it sound like the original reported error: TypeError: Descriptors should not be created directly, but only retrieved from their parent. is due to having generated python files without the _pb2 extension introduced for version 2 files when protobuf 3.0.0 came out, so we may already be fine. For good measure, though, I'm going ahead and PRing the regenerated classes with latest protoc.
Attachment #9021309 - Flags: review?(fbertsch)
Attachment #9021309 - Flags: review?(fbertsch) → review+
Attachment #9021527 - Flags: review?(fbertsch)
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Attachment #9021527 - Flags: review?(fbertsch) → review+
Component: Telemetry APIs for Analysis → General
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: