1796753 - Linux webrender tsan opt xpcshell frequent retries that end up as exception

Status: RESOLVED FIXED

(Core :: Sanitizers, defect)

Description

[Cosmin Sabou [:CosminS]]

There are frequent retries on tsan xpcshell tests like that often end up as an exception.

These Test groups are ran when these occur:

    browser/components/tests/unit/xpcshell.ini
    browser/extensions/formautofill/test/unit/heuristics/third_party/xpcshell.ini
    chrome/test/unit/xpcshell.ini
    devtools/client/application/test/xpcshell/xpcshell.ini
    devtools/client/shared/remote-debugging/test/xpcshell/xpcshell.ini
    devtools/client/webconsole/test/xpcshell/xpcshell.ini
    dom/encoding/test/unit/xpcshell.ini
    dom/indexedDB/test/unit/xpcshell-child-process.ini
    dom/media/webvtt/test/xpcshell/xpcshell.ini
    dom/promise/tests/unit/xpcshell.ini
    intl/strres/tests/unit/xpcshell.ini
    modules/libmar/tests/unit/xpcshell.ini
    modules/libpref/test/unit_ipc/xpcshell.ini
    netwerk/dns/tests/unit/xpcshell.ini
    remote/shared/messagehandler/test/xpcshell/xpcshell.ini
    services/sync/tests/unit/xpcshell.ini
    storage/test/unit/xpcshell.ini
    toolkit/components/autocomplete/tests/unit/xpcshell.ini
    toolkit/components/backgroundtasks/tests/xpcshell/xpcshell.ini
    toolkit/components/contentprefs/tests/unit_cps2/xpcshell.ini
    toolkit/components/contextualidentity/tests/unit/xpcshell.ini
    toolkit/components/crashes/tests/xpcshell/xpcshell.ini
    toolkit/components/crashmonitor/test/unit/xpcshell.ini
    toolkit/components/ctypes/tests/unit/xpcshell.ini
    toolkit/components/messaging-system/targeting/test/unit/xpcshell.ini
    toolkit/components/places/tests/queries/xpcshell.ini
    toolkit/components/places/tests/sync/xpcshell.ini
    toolkit/components/thumbnails/test/xpcshell.ini
    toolkit/components/url-classifier/tests/unit/xpcshell.ini
    toolkit/components/windowcreator/tests/unit/xpcshell.ini
    toolkit/content/tests/unit/xpcshell.ini
    toolkit/crashreporter/test/unit_ipc/xpcshell-phc.ini
    toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/xpcshell.ini
    toolkit/mozapps/update/tests/unit_background_update/xpcshell.ini
    widget/tests/unit/xpcshell.ini

decoder, is this something you could have a look over? Thank you.

Comment 1

[Intermittent Failures Robot]

28 failures in 2793 pushes (0.01 failures/push) were associated with this bug in the last 7 days.

This is the #43 most frequent failure this week.

Repository breakdown:

• autoland: 23
• mozilla-central: 5

Platform and build breakdown:

• linux1804-64-tsan-qr: 26
  • opt: 26
• linux1804-64-asan-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1796753&startday=2022-10-17&endday=2022-10-23&tree=all

Comment 2

[Intermittent Failures Robot]

22 failures in 3743 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 17
• mozilla-central: 5

Platform and build breakdown:

• linux1804-64-tsan-qr: 22
  • opt: 22

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1796753&startday=2022-10-24&endday=2022-10-30&tree=all

Comment 3

[Intermittent Failures Robot]

31 failures in 3626 pushes (0.009 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 22
• mozilla-central: 9

Platform and build breakdown:

• linux1804-64-tsan-qr: 31
  • opt: 31

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1796753&startday=2022-10-31&endday=2022-11-06&tree=all

Comment 4

[Intermittent Failures Robot]

22 failures in 3445 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 14
• mozilla-central: 7
• try: 1

Platform and build breakdown:

• linux1804-64-tsan-qr: 22
  • opt: 22

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1796753&startday=2022-11-07&endday=2022-11-13&tree=all

Comment 5

[Christian Holler (:decoder)]

When did this start? Since we still don't upload log artifacts for jobs that fail with an exception, this is difficult to debug. The most likely cause is that the machine swaps/runs OOM for some reason and it didn't before. Did we change anything about machine configurations? Are the failing test groups new? What did change around the time when this started?

Comment 6

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Started with this push on October 14th. The test groups are the same for the X6 task on the previous push for which it succeeded on the first attempt.

Comment 7

[Christian Holler (:decoder)]

That push has a lot of AWS -> GCP commits in it. Did this job move as well? Did the machine configuration change in any way?

Comment 8

[Intermittent Failures Robot]

15 failures in 3589 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 4
• autoland: 11

Platform and build breakdown:

• linux1804-64-tsan-qr: 15
  • opt: 15

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1796753&startday=2022-11-14&endday=2022-11-20&tree=all

Comment 9

[Intermittent Failures Robot]

15 failures in 3888 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 10
• mozilla-central: 5

Platform and build breakdown:

• linux1804-64-tsan-qr: 15
  • opt: 15

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1796753&startday=2022-11-21&endday=2022-11-27&tree=all

Comment 10

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

(In reply to Christian Holler (:decoder) from comment #7)

That push has a lot of AWS -> GCP commits in it. Did this job move as well? Did the machine configuration change in any way?

These jobs were not part of the changes. I will bisect on Try.

Comment 11

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Backfills point to bug 1774462 as the first push affected with frequent automatic retries of Linux TSan xpcshell failures because the tasks encounter issues. The affected task runs the tests in dom/indexedDB/test/unit/xpcshell-child-process.ini, among others.

Comment 12

[Jens Stutte [:jstutte]]

Hmm, it is not easy to look at anything here, as all the tasks I looked at did not finish the log parsing and trying to access the log through the task itself gives me a network error like this. In comment 0 I see dom/indexedDB/test/unit/xpcshell-child-process.ini, too, and that runs dom/indexedDB/test/test_keys.html, IIUC.
I wonder if we just end up with an OOM here given that test_keys.html allocates a huge array which bites us for some reason only in this constellation in tsan. We could try to avoid running this test in tsan (and other memory sensitive constellations).

Comment 13

[Christian Holler (:decoder)]

(In reply to Jens Stutte [:jstutte] from comment #12)

I wonder if we just end up with an OOM here given that test_keys.html allocates a huge array which bites us for some reason only in this constellation in tsan. We could try to avoid running this test in tsan (and other memory sensitive constellations).

I think this is exactly what happens and I agree, we should not run this test in configurations that require more memory (e.g. sanitizers).

Comment 14

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1774462

Comment 15

[Jens Stutte [:jstutte]]

(In reply to Christian Holler (:decoder) from comment #13)

I think this is exactly what happens and I agree, we should not run this test in configurations that require more memory (e.g. sanitizers).

I assume we do some different OOM handling in those builds? As the test wants to account for a fallible allocation of that array, mostly to exclude 32Bit systems, but that seems not to help here.

Comment 16

[Jens Stutte [:jstutte]]

Hmm, could that specific test just check for AppConstants.TSAN || AppConstants.ASAN ? We could make us just skip the one key with large allocation, somehow.

Comment 17

[Christian Holler (:decoder)]

(In reply to Jens Stutte [:jstutte] from comment #15)

(In reply to Christian Holler (:decoder) from comment #13)

I think this is exactly what happens and I agree, we should not run this test in configurations that require more memory (e.g. sanitizers).

I assume we do some different OOM handling in those builds? As the test wants to account for a fallible allocation of that array, mostly to exclude 32Bit systems, but that seems not to help here.

Our sanitizers are configured to allow for fallible allocations (for TSan in particular here). But it is possible that this fails in edge cases. We've seen this happen in fuzzing with ASan many times that if we hit the exact right spot to OOM, we might hit an infallible allocation in the ASan internals.

(In reply to Jens Stutte [:jstutte] from comment #16)

Hmm, could that specific test just check for AppConstants.TSAN || AppConstants.ASAN ? We could make us just skip the one key with large allocation, somehow.

I don't know if this is available inside this this particular test, but if it is, then I'd also prefer that option.

Comment 18

[Jens Stutte [:jstutte]]

Let me try that then.

Comment 19

[Jens Stutte [:jstutte]]

Attachment: Bug 1796753 - Disable test_keys.js for xpcshell tests under TSAN and ASAN. r?decoder

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/470093aeb138
Exlude keys known to consume a huge amount of memory for TSAN and ASAN. r=decoder

Comment 21

[Atila Butkovits]

Backed out for causing failures at test_keys.html.

Backout link: https://hg.mozilla.org/integration/autoland/rev/42a01ff3077c18e976e57d973e20f91ab412ae3a

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&selectedTaskRun=YUQhURHJS3O_d56EDApWTQ.0&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&revision=470093aeb138bc2065b0e8d7fb1a95f4353d64ad

Failure log: https://treeherder.mozilla.org/logviewer?job_id=398331914&repo=autoland&lineNumber=2067

Comment 22

[Jens Stutte [:jstutte]]

Ah, it also runs as mochitest and there we do not have AppConstants ?

Comment 23

[Jan Varga [:janv]]

I think the problem is that ChromeUtils is not available in mochitests.
Maybe you can get AppConstants using SpecialPowers in that case.
https://searchfox.org/mozilla-central/rev/2fc2ccf960c2f7c419262ac7215715c5235948db/dom/animation/test/document-timeline/test_document-timeline.html#30

Comment 24

[Jens Stutte [:jstutte]]

OK, so the same test_keys.js is run in three different contextes, two times in the mochitest test_keys.html (one normal, one as worker) and one time standalone as xpcshell test. Unfortunately all three environments need different ways of accessing AppConstants. But as we have that triple coverage, and after talking with :janv and :asuth, we think we can just disable the entire test for xpcshell with ASAN/TSAN - unless we know, that we should expect problems also from mochitests with similar OOMs. :decoder?

Comment 25

[Christian Holler (:decoder)]

In general it would be favorable to somewhat separate OOM-like tests from regular tests, but I am not aware of other tests right now causing problems and if the quickest way to move forward is to disable this test, we should just do that :)

Comment 26

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/499c02c864ff
Disable test_keys.js for xpcshell tests under TSAN and ASAN. r=decoder

Comment 27

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/499c02c864ff

Comment 28

[Intermittent Failures Robot]

14 failures in 3938 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 10
• mozilla-central: 4

Platform and build breakdown:

• linux1804-64-tsan-qr: 14
  • opt: 14

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1796753&startday=2022-11-28&endday=2022-12-04&tree=all

Comment 29

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:jstutte, is this bug important enough to require an uplift?

• If yes, please nominate the patch for beta approval.
• If no, please set status-firefox108 to wontfix.

For more information, please visit auto_nag documentation.

Comment 30

[Jens Stutte [:jstutte]]

Do we run TSAN tests frequently on beta/release versions?

Comment 31

[Christian Holler (:decoder)]

I believe we run them as often as other tests, there is nothing special about sanitizers there afaik.

Comment 32

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

TSan XPCshell doesn't run on beta and release.