1353965 - Dataset API returning empty RDD

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P1)

Description

[Frank Bertsch [:frank]]

No matter what I put into get_pings, the result is now empty. All the mobile jobs failed because of this.

Comment 1

[Frank Bertsch [:frank]]

I've realized the Dataset API as a whole is broken:

>> Dataset.from_source('telemetry').where(docType = 'main').records(sc).count()
0

Comment 2

[Frank Bertsch [:frank]]

Just to be clear, the summaries() method is returning no files.

Comment 3

[Roberto Agostino Vitillo (:rvitillo)]

William, could you please verify if this is caused by the new version of python_moztelemetry that we released yesterday?

Comment 4

[Roberto Agostino Vitillo (:rvitillo)]

Frank, I can't seem to replicate this:

```
len(Dataset.from_source('telemetry').where(docType='main', submissionDate="20170401").summaries(sc)) 
143469
```

Comment 5

[William Lachance (:wlach)]

Frank, could you double check this is still broken for you per comment 4? Happy to look into this if so.

Comment 6

[Roberto Agostino Vitillo (:rvitillo)]

(In reply to Roberto Agostino Vitillo (:rvitillo) from comment #4)
> Frank, I can't seem to replicate this:
> 
> ```
> len(Dataset.from_source('telemetry').where(docType='main',
> submissionDate="20170401").summaries(sc)) 
> 143469
> ```

Please ignore this; turns out I tried that query on a cluster that was spawned before the new version of python_moztelemetry was released.

Comment 7

[William Lachance (:wlach)]

I've removed release 0.6.7 from pypi while I investigate this.

Comment 8

[William Lachance (:wlach)]

Fixed with: https://github.com/mozilla/python_moztelemetry/pull/140