1307087 - ATMO V2: scheduled jobs are failing

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P1)

Description

[Mauro Doglio [:mdoglio]]

This is what I found in the logs:

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/rq/worker.py", line 588, in perform_job
    rv = job.perform()
  File "/usr/local/lib/python2.7/dist-packages/rq/job.py", line 498, in perform
    self._result = self.func(*self.args, **self.kwargs)
  File "/app/atmo/jobs/jobs.py", line 8, in launch_jobs
    SparkJob.step_all()
  File "/app/atmo/jobs/models.py", line 136, in step_all
    if spark_join.should_run(now):
  File "/app/atmo/jobs/models.py", line 89, in should_run
    active = self.start_date <= at_time <= self.end_date
TypeError: can't compare offset-naive and offset-aware datetimes

Comment 1

[GitHub Autolander Bot]

Attachment: [telemetry-analysis-service] mozilla:bug-1307087-fix-scheduled-jobs > mozilla:master

Comment 2

[Mauro Doglio [:mdoglio]]

Another error similar to the one in comment 0:
Traceback (most recent call last):
  File "/app/.heroku/python/lib/python2.7/site-packages/rq/worker.py", line 588, in perform_job
    rv = job.perform()
  File "/app/.heroku/python/lib/python2.7/site-packages/rq/job.py", line 498, in perform
    self._result = self.func(*self.args, **self.kwargs)
  File "/app/atmo/jobs/jobs.py", line 8, in launch_jobs
    SparkJob.step_all()
  File "/app/atmo/jobs/models.py", line 139, in step_all
    if spark_join.should_run(now):
  File "/app/atmo/jobs/models.py", line 90, in should_run
    active = self.start_date <= at_time
TypeError: can't compare offset-naive and offset-aware datetimes

Comment 3

[Mauro Doglio [:mdoglio]]

I can't reproduce it anymore. I think it was fixed as a side-effect of fixing bug 1309227

Comment 4

[Roberto Agostino Vitillo (:rvitillo)]

I am still unable to successfuly run a scheduled job. It looks like jobs are not started at all on EMR.

Comment 5

[Jannis Leidel [:jezdez]]

:rvitillo  I dug a bit and found that spark job clusters aren’t launched with VisibleToAllUsers=True but False
which is why I wasn’t able to see the cluster in the AWS console. Should I add that to the spark jobs?

Comment 6

[GitHub Autolander Bot]

Attachment: [telemetry-analysis-service] mozilla:bug1307087 > mozilla:master

Comment 7

[Roberto Agostino Vitillo (:rvitillo)]

(In reply to Jannis Leidel [:jezdez] from comment #5)
> :rvitillo  I dug a bit and found that spark job clusters aren’t launched
> with VisibleToAllUsers=True but False
> which is why I wasn’t able to see the cluster in the AWS console. Should I
> add that to the spark jobs?

Go ahead.

Comment 8

[Mauro Doglio [:mdoglio]]

This also explains why I wasn't able to reproduce this issue locally where the jobs are spawned with my own credentials. Good spot!

Comment 9

[Roberto Agostino Vitillo (:rvitillo)]

Scheduled jobs appear to be still failing (tested on stage). I scheduled this simple notebook [1] which should not fail but it still did. As there are also no logs (filed Bug 1312749 for that one) I don't know what happened.

[1] https://raw.githubusercontent.com/mozilla/telemetry-airflow/master/examples/spark/example_date.ipynb

Comment 10

[Mauro Doglio [:mdoglio]]

I can reproduce this locally.

Comment 11

[GitHub Autolander Bot]

Attachment: [telemetry-analysis-service] mozilla:bug-1307087-run-uploaded-notebook > mozilla:master

Comment 12

[Jannis Leidel [:jezdez]]

Closing this as this was fixed in staging.

Comment 13

[Marco Castelluccio [:marco]]

Jannis, did this land on prod?

Comment 14

[Jannis Leidel [:jezdez]]

:marco We just landed it (with an unplaned delay) in the atmo-prod.herokuapp.com environment.

Comment 15

[Roberto Agostino Vitillo (:rvitillo)]

This is still not working for me on stage. I don't see the jobs being scheduled from the AWS console and the UI doesn't give me any indication that it tried to run my jobs.

Comment 16

[GitHub Autolander Bot]

Attachment: [telemetry-analysis-service] mozilla:job-status-refactor > mozilla:master

Comment 17

[Mauro Doglio [:mdoglio]]

Now that bug 1316623 is solved we can close this as well.

Comment 18

[GitHub Autolander Bot]

Attachment: [telemetry-analysis-service] mozilla:bug1307087-2 > mozilla:master