1572033 - Decommission Longitudinal dataset

Status: RESOLVED FIXED

(Data Platform and Tools Graveyard :: Datasets: Longitudinal, task)

Description

[Arkadiusz Komarzewski [:akomar]]

We won't need Longitudinal dataset on GCP (we can answer all relevant questions with main ping and clients_last_seen tables).
Because Longitudinal job uses Dataset API we'd like deprecate it to avoid moving it to GCP.

Comment 1

[Chris H-C :chutten]

:tdsmith mentioned on Slack that we probably want to highlight that the Hardware Report uses the longitudinal dataset. Or at least the one here does. Not sure if that's the same one that's on data.firefox.com

Comment 2

[Arkadiusz Komarzewski [:akomar]]

(In reply to Chris H-C :chutten from comment #1)

:tdsmith mentioned on Slack that we probably want to highlight that the Hardware Report uses the longitudinal dataset. Or at least the one here does. Not sure if that's the same one that's on data.firefox.com

I think we don't need to worry about Hardware Report:

• mozilla/firefox-hardware-report repo is archived
• data.firefox.com is powered by https://github.com/mozilla/Fx_Usage_Report which does not use longitudinal

Pinging :frank in case he knows about any current longitudinal usages related to hardware report.

Comment 3

[Frank Bertsch [:frank]]

We will need to transfer the hardware report, it is still in use and updating every week. I recommend we take the same approach we're taking with the Update Orphaning job, where we create a subset of longitudinal before the job and keep it as-is. The job is defined here: https://github.com/mozilla/python_mozetl/tree/master/mozetl/hardware_report

Comment 4

[Arkadiusz Komarzewski [:akomar]]

(In reply to Frank Bertsch [:frank] from comment #3)

We will need to transfer the hardware report, it is still in use and updating every week. I recommend we take the same approach we're taking with the Update Orphaning job, where we create a subset of longitudinal before the job and keep it as-is. The job is defined here: https://github.com/mozilla/python_mozetl/tree/master/mozetl/hardware_report

This will be handled in Bug 1582110.

Comment 5

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/telemetry-airflow/pull/693