1347283 - Dataset API fails on improper utf8 string

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect)

Description

[Frank Bertsch [:frank]]

Attachment: failing_dataset_example.ipynb

> Dataset.from_source('telemetry').where(docType='OTHER').records(sc, sample=.01).count()

Errors with:
> UnicodeDecodeError: 'utf8' codec can't decode byte 0xe1 in position 74109: invalid continuation byte

Example is attached.

Comment 1

[Eric Rescorla (:ekr)]

mdoglio, mreid, thoughts on fixing this?

Comment 2

[Mauro Doglio [:mdoglio]]

It looks like we have some bogus data that the heka message parser cant'read. I'll ni whd to see if he can find where it is. We could catch the exception and move on, but I would prefer to know what's the root cause first.

Comment 3

[Mauro Doglio [:mdoglio]]

Actually, let me write a quick fix while Wes investigates.

Comment 4

[Mauro Doglio [:mdoglio]]

Attachment: PR 116 on python_moztelemetry

Comment 5

[Wesley Dawson [:whd]]

Comment on attachment 8847665 [details] [review]
PR 116 on python_moztelemetry

I'll dig through the data to find the bad UTF8. The new pipeline until https://github.com/mozilla-services/lua_sandbox_extensions/pull/84/commits/3f8c5bf040ec9ff8dce10430c86536b398e77a03 was deployed produced client-generated binary strings that could have a specific kind of UTF8 corruption (these are considered errors now). As long as the corruption happened within a string in the json (not part of the structure of the json document) and not in a string used as part of schema validation for the particular docType, the new pipeline would consider it valid and produce it as-is. If the corruption happened as part of the structure of the document then it would simply be sent to the errors stream.

We "fixed" this in the scala bindings by removing the valid UTF8 check and observing that the UTF8 decode can be done lossily with bad data. Unless we have a lossy decode option in python :mdoglio's "try except" may be the best we can do here.

Another thing we might want to do is take whatever data is going to OTHER and reprocess it into its own doc type, as OTHER pings are probably the ones most likely to have issues like this.

Leaving the NI until I've found the data.

Comment 6

[Wesley Dawson [:whd]]

https://docs.python.org/2/library/codecs.html#codec-base-classes makes it look like we can do lossy decoding, so I'll file a PR that does that.

Comment 7

[Wesley Dawson [:whd]]

I've filed https://github.com/mozilla/python_moztelemetry/pull/117 with a fix and a test case including some of the data form the OTHER bucket that failed. It now replaces the bad utf8 characters with the replacement character per the unicode spec.

Comment 8

[Mark Reid [:mreid]]

Looks good to me.

Comment 9

[Wesley Dawson [:whd]]

I've merged the above PR. Once :mreid or :rvitillo pushes an updated version to pypi we should be able to mark this as fixed.

Comment 10

[Mauro Doglio [:mdoglio]]

I deployed the last version to pypi. New clusters will get it automatically.

Comment 11

[Mauro Doglio [:mdoglio]]

This was fixed by https://github.com/mozilla/python_moztelemetry/pull/117