1403075 - Cannot install a signed extension over a TLS connection with a custom CA

Status: RESOLVED DUPLICATE of bug 1308251

(Toolkit :: Add-ons Manager, defect, P2)

Description

[Andrei Petcu]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:58.0) Gecko/20100101 Firefox/58.0
Build ID: 20170925220207

Steps to reproduce:

Set up a MITM proxy (for testing purpose you can install something like ZAP https://www.owasp.org/index.php/OWASP_Zed_Attack_Proxy_Project)

Import the root CA of the proxy

Try to install a self signed extension for example this one: https://github.com/andreicristianpetcu/google_translate_this/releases/download/v1.2.7/google_translate_this_page-1.2.7-an.fx.xpi


Actual results:

I got a failed error message saying there was a connection issue


Expected results:

If I have a corporate proxy which does MITM and I have imported it's Root CA, I expect it to let me install unlisted signed extensions. If the browser trusts the CA for browsing, it should trust also for installing extensions. The same issue is with updating the extension. Please see https://discourse.mozilla.org/t/add-on-update-certificate-issuer-is-not-built-in/19457/9

Comment 1

[Andrei Petcu]

If there is some fast workaround for this, please suggest it.

I imported the Root CA like this https://vimeo.com/231286386

Comment 2

[Cosmin Muntean [:cmuntean], Ecosystem QA]

Hi Andrei,

I have not managed yet to set up the described environment to test this issue. But, first, can you please try something?
If you navigate to "about: config" page and set the "xpinstall.signatures.required" boolean pref to "false" and "extensions.legacy.enabled" to "true", the extension can be installed?

If the extension cannot be installed, can you please verify if you have set the proxy correctly? I think that the "connection issue" occurs because the experiment failed to download before installing it.

However, it seems that the experiment can be successfully installed on latest Nightly (58.0a1) build.

Comment 3

[Andrei Petcu]

My extension is an unlisted, signed WebExtension.

I flipped the flags as you said but it still reproduces on the latest Nightly.

It might be that the proxy is messing with the files that are transferred.

Comment 4

[Cosmin Muntean [:cmuntean], Ecosystem QA]

I am not an expert in web extensions, but I will assign a component to this bug and maybe someone with more experience can help with this. 

Also, I will also try in the next days to set up the environment in order to test this.

Comment 5

[Andrew Swan [:aswan]]

There seems to be some confusion in this report about the CA used to verify extension signing and the CA(s) used to establish TLS channels.  The xpi file in comment 0 is signed by Mozilla so I assume this is all about TLS.  Can you be more explicit about how you tried to install the extension?  Was it just by entering a URL to the xpi file in the location bar?  Or something else?

Comment 6

[Andrei Petcu]

I think yhe proxy messes up the xpi file in transit. No CA problem.

Comment 7

[Andrew Swan [:aswan]]

Closing based on comment 6

Comment 8

[Andrei Petcu]

For posterity here is the source of the bug (Discourse) https://discourse.mozilla.org/t/add-on-update-certificate-issuer-is-not-built-in/19457/13?u=andrei

Comment 9

[Kumar McMillan [:kumar]]

> If the browser trusts the CA for browsing, it should trust also for installing extensions.

I'm re-opening the bug for this reason ^

Using a custom root CA is not uncommon practice. In fact, it's how pretty much all anti-virus software works.

After launching the new addons.mozilla.org site with a button that serves a link to the XPI (as opposed to using InstallTrigger.install() like on the old site), anti-virus users could no longer install add-ons. See https://github.com/mozilla/addons/issues/525 (there is also a reddit thread somewhere).

When anti-virus users click the install button, they get a message like this in the console:

"1510156312761 addons.xpi WARN Download of https://addons.cdn.mozilla.net/user-media/addons/832020/addon-1.1.0-an+fx-windows.xpi?filehash=sha256%3A61bf622b2cac2749de8a43a3fada3597eda7863800fe146e20c04eafa2adacaa failed:
 [Exception... "Certificate issuer is not built-in." nsresult: "0x80004004 (NS_ERROR_ABORT)" 
location: "JS frame :: resource://gre/modules/CertUtils.jsm :: checkCert :: line 169" data: no] 
Stack trace: checkCert()@resource://gre/modules/CertUtils.jsm:169 < onStopRequest()@resource://gre/modules/addons/XPIInstall.jsm:2405"

In addition to the steps shown in the description of this bug, this issue also shows you how to reproduce it: https://github.com/mozilla/addons-frontend/issues/3867

Comment 10

[Andrew Swan [:aswan]]

(In reply to Kumar McMillan [:kumar] (needinfo all the things) from comment #9)
> > If the browser trusts the CA for browsing, it should trust also for installing extensions.
> 
> I'm re-opening the bug for this reason ^

I think this particular bug ended up being something else.  But since we're discussing this here, this is a request to change behavior that we have had for some time.  There is a hidden preference called extensions.install.requireBuiltInCerts that can be used to change this behavior but if we want to relax the default, we need to get approval from security (and I assume they will want some specific reason to evaluate)

Comment 11

[Andrew Swan [:aswan]]

(In reply to Kumar McMillan [:kumar] (needinfo all the things) from comment #9)
> After launching the new addons.mozilla.org site with a button that serves a
> link to the XPI (as opposed to using InstallTrigger.install() like on the
> old site), anti-virus users could no longer install add-ons. See
> https://github.com/mozilla/addons/issues/525 (there is also a reddit thread
> somewhere).

Whoops, meant to also respond to this (I also responded on the github issue).  With InstallTrigger, the web page initiating the install can pass a checksum of the xpi to be installed.  If the page does that, the checksum ensures the integrity of the thing being downloaded so we skip the check that the download channel must be validated to a built-in root CA.

Comment 12

[Kumar McMillan [:kumar]]

> I think this particular bug ended up being something else.

The description of this bug reads like the exact same thing our anti-virus users are encountering. Setting up a MITM proxy and then importing the root CA of the proxy simulates what common anti-virus software does. The OP's case of using an internal root CA blessed for corporate use also makes sense.

> If the browser trusts the CA for browsing, it should trust also for installing extensions.

So the extension installer is ignoring a non built-in (but trusted) root CA by design? It seems surprising to me that these users should be able do everything else in the browser except install extensions.

As far as the addons.mozilla.org case goes, we are going to work around this by using InstallTrigger until we fix our mozAddonManager implementation. Because of that, if security wants to close it as wontfix then that shouldn't impact us.

Comment 13

[Andrew Swan [:aswan]]

Paul, you want to weigh in?

Comment 14

[Andrew Swan [:aswan]]

Upon further reflection, this feature is kind of broken.  A page that uses IntallTrigger or mozAddonManager could still be modified by a MITM to simply change the hash.  We can also get the hash from an X-Target-Digest header on an HTTP redirect, but we don't require TLS and a built-in root on the channel we get that redirect on.

Comment 15

[Andrei Petcu]

The way I see it is that there are 2 certs involved: the proxy(or AV) and Mozilla's.

The TLS connection follows standard TLS rules: is the proxy/AV CA in the truststore? If yes then the XPI file gets downloaded, if not then it shows connection error.

After the download, the XPI file's signature is checked. If it was signed by Mozilla and the signature is valid then the extension is installed, if no then you get a "Extension is corrupted" error. The proxy has nothing to say in this, it's CA is not involved in the XPI signature verification.

If you read my comment https://bugzilla.mozilla.org/show_bug.cgi?id=1403075#c3 I think I may found the issue for my specific case using ZAP: the proxy messes up the file somehow. I am not 100% certain of this. Maybe the AV does something similar.

Comment 16

[Kumar McMillan [:kumar]]

> A page that uses IntallTrigger ... could still be modified by a MITM to simply change the hash

I was thinking about this too. This means a rogue anti-virus company configured to spy for a state (hypothetical only! :)) would need to change the contents of the add-on file and *also* change the AMO API responses to return the correct hash for it. With the current approach of requiring a built-in root CA, this adds one extra step to the attack process. I suppose that's some defense in depth protection? Is it worth it?

Comment 17

[Kumar McMillan [:kumar]]

@andrei: by the way, you could make yourself a working install button for an XPI file served over TLS via your own root CA using the InstallTrigger API https://developer.mozilla.org/en-US/docs/Web/API/InstallTrigger/install (this is an old Firefox API but it's still supported).

Comment 18

[Paul Theriault [:pauljt] (no longer reading bugmail)]

(In reply to Andrew Swan [:aswan] from comment #14)
> Upon further reflection, this feature is kind of broken.  A page that uses
> IntallTrigger or mozAddonManager could still be modified by a MITM to simply
> change the hash.  We can also get the hash from an X-Target-Digest header on
> an HTTP redirect, but we don't require TLS and a built-in root on the
> channel we get that redirect on.

Sorry for the long delay in commenting here. Truth is that I don't really have the background to be able provide much insight here. I don't know why we had this in the first place. Has this always been an issue? (has it always been impossible for users going through corporate certs to install addons). This seems hard to believe that it hasnt come up as an issue right now, and I dont see what changed. But let me try to muddle about this:

Addons now have to be signed, but there is no bar to getting an add-on signed, we provide an API for getting an addon signed. 
So while a MITM can't modify the contents of an XPI, I guess a MITM could just replace it? (You could even modify the addon and re-sign on the fly using the API). So in theory this restriction prevents corporations or state actors replacing your XPIs via MITM. 

But in the "malicious actor trying to get someone to install the wrong addon case", comment 14 seems to indicate that this control is bypassable in any case by simply using the installTrigger API or X-Target-Digest header on a redirect to get Firefox to skip the cert-pinning check. So it seems somewhat ineffective.

An futhermore, if you are getting MITM, a bigger threat to your security is probably your downloads being tampered with which we have no control over.

My gut feel here is that this was probably a bigger issue for legacy addons and this control is no longer needed (and is bypassable due to install trigger anyways). Bottom line is that if you can intercept SSL connections you can probably get full control over extensions anyways, due to remote resource loads or similar. But I'd like a second opinion - Dan?

Comment 19

[Daniel Veditz [:dveditz]]

(In reply to Paul Theriault [:pauljt] from comment #18)
> Truth is that I don't really have the background [...]
> I don't know why we had this in the first place.

This is a really old check, and was always a poor substitute for what we really wanted. We now have cert pinning for AMO (which includes add-on updates), cert pinning is relaxed for user-added roots (by default), and add-ons themselves are required to be signed. We shouldn't need the requireBuiltIns check anymore.

> Has this always been an issue? (has it always been impossible for users
> going through corporate certs to install addons). This seems hard to
> believe that it hasnt come up as an issue right now,

We used to discourage people from installing add-ons from sites other than AMO.[1] AMO always used the InstallTrigger+hash bypass so it worked. We needed the hash bit for InstallTrigger because early on the addons themselves were served from a CDN over insecure http: for cost reasons. 

> Addons now have to be signed, but there is no bar to getting an add-on
> signed, we provide an API for getting an addon signed. 

Yup. The only control we have now is blocklisting things after the fact. If someone's installing from somewhere other than AMO the initial extension is as likely to be malicious as whatever a MITM tries to replace it with.

> My gut feel here is that this was probably a bigger issue for legacy addons
> and this control is no longer needed

Yes, I think that's right. The only control we have is what extensions we sign and blocklist (and the blocklist can itself be blocked by a MITM until the next Firefox update--if that's allowed).

[1] I wish that were still the case: installing an extension is too easy and looks relatively harmless to an average user. Although less powerful than they used to be they can still ruin people's experience (ad injection, coin mining) and lives (stealing bank passwords).

Comment 20

[Andrew Swan [:aswan]]

(In reply to Daniel Veditz [:dveditz] from comment #19)
> (In reply to Paul Theriault [:pauljt] from comment #18)
> > My gut feel here is that this was probably a bigger issue for legacy addons
> > and this control is no longer needed
> 
> Yes, I think that's right. The only control we have is what extensions we
> sign and blocklist (and the blocklist can itself be blocked by a MITM until
> the next Firefox update--if that's allowed).

Can we make that statement more concrete?  ie, is it okay from a security perspective if we remove the requireBuiltInChecks logic for extension installs?  And same for extension updates?  (note that unlike xpis, extension update manifests are not signed in any way, but the new updated version must be a signed xpi for the same addon id with a version number that is greater than the installed version, so its not obvious how a man-in-the-middle who can modify an update manifest would exploit that ability)

Comment 22

[Daniel Veditz [:dveditz]]

(In reply to Andrew Swan [:aswan] from comment #20)
> (note that unlike xpis, extension update manifests are not signed in any way,

What do you mean? I thought both used the same update server and the same format?

Comment 23

[Daniel Veditz [:dveditz]]

I misunderstood your comment -- thanks for the clarification via IRC.

We certainly don't need built-in checks for the initial installation: we've got preloaded pins for AMO, HPKP already allows for locally-installed monitoring software to avoid this bug's problem, and if you're installing from another site the fact that it chains to a "builtin" root is no guarantee it's not foisting dodgy extensions.

Is this bug a dupe of bug 1308251? I'll mark that "depends on" for now since fixing this bug would fix all or most of that one.

On the update path most parts are signed except the update metadata response, and as you note there are additional sanity checks to prevent the worst abuses of modifying the metadata in transit. We have bug 1303183 and bug 1303186 (client and server respectively) asking that we use our content-signing mechanism for that data which will close the remaining gaps. We can remove the builtin checks for updates without having to wait for that.

Comment 24

[Paul Theriault [:pauljt] (no longer reading bugmail)]

What dan said.

Comment 25

[budau.adi]

Sorry for ressurecting this bug, but I've run into this issue at work. As other people have said, when you have an AV that does MITM the extension downloaded will not be validated because the root CA is not builtin if the extension is not downloaded from AMO. One example is the Lockwise which is developed by Firefox, but served from outside AMO.

I understand requieBuildInCerts is there to "protect" the user in case of MITM, but if the MITM is done by an AV company, they can already do much more than a Firefox Extension can. And if it done by a malicious third-party, well they can already replace downloaded files. But the argument supporting requireBuiltInCerts seems a bit moot when you can still bypass this check by intercepting the request and triggering a redirect to a user controlled domain where any signed extension could be returned.

If the whole idea is to serve all extensions from AMO, should requireBuiltInCert either be removed, or at some point all extensions outside AMO banned?

I work at an AV Company and we are trying to find a work-around to let users Install extensions from outside AMO and there just doesn't seem to be a solution, by the time a MITM notices you are requesting an XPI file, the connection has already been established.

Comment 27

[Shane Caraveo (:mixedpuppy)]

from what I've read here, this should be fixed by bug 1308251 so closing