1301048 - Setup rq on ATMO V2

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P1)

Description

[Mauro Doglio [:mdoglio]]

The current implementation only works on a single node deployment. The application installs a job in the crontab that simply checks every hour if there is something that needs to run. In a scenario with multiple web nodes multiple jobs will run, one per web node.

Comment 1

[Mauro Doglio [:mdoglio]]

A possible solution is to have a node in the cluster dedicated to the scheduler.

Comment 2

[Mauro Doglio [:mdoglio]]

On a second thought, it may be sufficient to have a single python command running every minute in a cron job. :robotblake is that doable in the dockerflow infrastructure?

Comment 3

[Blake Imsland [:robotblake]]

That's do-able, but I feel like the right way to do it would be to use celery / huey / rq and run workers either as separate containers on each node or using supervisor (or similar) inside the containers.

Comment 4

[Jannis Leidel [:jezdez]]

:mdoglio and me talked about this a bit and found that based on our experience Celery would be way too complex for our use case. Instead I pitched RQ as a replacement since it's a nice compromise between the quality of the developer API and operational complexity (e.g. using Redis ElastiCache). We'd be using RQ-Scheduler (https://github.com/ui/rq-scheduler) for the periodic scheduling on top of RQ.

Comment 5

[Mauro Doglio [:mdoglio]]

:jezdez can you please split the work required to configure rq into a set of bugs that block this one?

Comment 6

[GitHub Autolander Bot]

Attachment: [telemetry-analysis-service] mozilla:bug-1302777-add-redis-instance > mozilla:master

Comment 8

[Mauro Doglio [:mdoglio]]

This is on master now

Comment 9

[Jannis Leidel [:jezdez]]

Comment on attachment 8791539 [details] [review]
[telemetry-analysis-service] mozilla:bug-1302777-add-redis-instance > mozilla:master

This was reviewed on GitHub