1306049 - Sanitize arguments to "get_pings" before applying filtering

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect, P3)

Description

[Mark Reid [:mreid]]

String arguments to "get_pings" should be sanitized before being used to filter S3 object names.

For example, filtering on docType = "saved-session" should actually match "saved_session".

Path components are sanitized[1] when raw data is stored on S3, so anything with a hyphen etc will never match.

This may not be needed in the Dataset API (though might save some surprises), but should at least be included in 'get_pings'.

[1] https://github.com/mozilla-services/data-pipeline/blob/master/heka/plugins/s3splitfile/s3splitfile_common.go#L167

Comment 1

[Mark Reid [:mreid]]

Alternatively, we could modify Dataset to list prefixes available at each level in the tree to make it easy to discover what values can be used for filtering.

Comment 2

[Mark Reid [:mreid]]

https://github.com/mozilla/python_moztelemetry/pull/161