crypto/x509: add SetFallbackRoots and golang.org/x/crypto/x509roots/fallback package

[breml]

Proposal

Go programs currently rely on the operating system to provide CA root certificate information in some sort of certificate system store. There are situations, where no such up-to-date CA root certificates are available like (Docker) containers built FROM scratch or out of date environments like poorly maintained or no longer updatable systems (e.g. older hardware appliances).

For some environments it is possible for the user of a Go program to add additional CA root certificates via SSL_CERT_FILE or SSL_CERT_DIR, but this is not the case for all supported environments (e.g. Windows) and it is definitively not user friendly.

Therefore, it is desirable for Go programs to have some mechanism to directly embed CA root certificate information into the program itself, so that they don't have to rely on system store to provide CA root certificates that may be absent (or out of date).

I make the following assumptions, which I think are reasonable:

• adding the CA root certificates into a program should be optional, as it increases the size of the program by approximately 250K (uncompressed), and most programs run in properly maintained systems with an up-to-date certificate system store
• it should be possible to embed CA root certificates when building an arbitrary program
• it should be possible for a program to always embed CA root certificates without requiring any special build step
• the program should always prefer data from the system if available, as it is likely to be more up to date than that included in the program
• the program should allow to override the preference and force the usage of the embeded data (controled by an environment variable, e.g. GO_ROOTCERTS_ENABLE)

Given those assumptions, I propose adding a new package crypto/x509/rootcerts. Importing this package (as import _ "crypto/x509/rootcerts") will cause CA root certificates to be embedded in the program. Also, building with the build tag rootcerts will force the package to be imported, and therefore will cause CA root certificates to be embedded in the program. The embedded CA root certificates will be used if and when no certificate system store is available (or the user forces the usage of the embedded data).

Source for the CA Root Certificates

I propose to use the Mozilla Included CA Certificate List, more specifically the PEM of Root Certificates in Mozilla's Root Store with the Websites (TLS/SSL) Trust Bit Enabled as the source for the CA root certificates.

The Mozilla Included CA Certificate List is the source for the CA root certificates embeded in the well known products of the Mozilla Foundation like for example Firefox (web browser) or Thunderbird (email client).

In contrast to most of the other software vendors, Mozilla maintains its Included CA Certificate List publicly and distributes it under an open source license (Mozilla Public License Version 2). This is also the reason why most of the Linux distributions, as well as
other free unix derivates and wide spread tools, use this list of CA root certificates as part of their distribution.

Some examples:

• Debian (and its derivates): ca-certificates
• Red Hat / Fedora / CentOS: ca-certificates / ca-certificates
• Alpine Linux: ca-certificates
• FreeBSD: ca_root_nss
• NetBSD: ca-certificates
• curl: cacert.pem

In summary in my opinion it is safe to say that the Mozilla Included CA Certificate List is well established and widely used.
In fact, if a Go program is run on Linux or an other free Unix derivate, chances are high that the root certificates used by the program are already provided by the Mozilla Included CA Certificate List.

Why include into the Go Standard Library

As the sample implementation (link in Annex below) clearly demostrates, that it is possible to write a 3rd party Go package, which achieves the same goal as the proposed package crypto/x509/rootcerts would. The main difference between a package in the standard library and a 3rd party package is: TRUST.

The root certificates are the top-most certificates in the trust chain and used to ensure the trustworthiness of the certificates signed by them either directly (intermediate certificates) or indirectly (through intermediate certificates). Therefore for a package containing and replacing the root certificates, trust is essential.

The same way, most users of Linux trust the CA root certificates provided by their distribution, it is very likely, that user would trust the CA root certificates provided by a package included in the Go standard library.

Additionally, the possibility to include the CA root certificates during build time, without altering the source code, is not possible with a 3rd party package but only if this package is included into the Go standard library and the build tag is implemented in to Go tool chain.

Update of the CA Root Certificates

The CA Root Certificates included in the standard library are updated with every release of Go (with the current schedule every 6 months). This would work the same way as it currently does for the package time/tzdata. The update frequency of the Included CA Certificate List is roughly every few months (2020: 5 times, 2019: 4 times, according to curl ca extract), which seems to be similar to the update frequency of the time zone data information.

In regards to updating the CA root certificates compiled into a Go binary, the same limitations apply as for the time/tzdata package. The information compiled into a binary is not updated. That being said, for the situations, this package is intended for, it is still an improvement because containers built FROM scratch are also not updated by default and out of date / not updatable systems obviously also do not get updates for the CA root certificates.

Annex

There is a sample implementation of this approach at github.com/breml/rootcerts with some additional reasoning about when to use such a package and what to keep in mind.

This proposal as well as the sample implementation are highly influenced by the proposal #38017 - time/tzdata and the implementation of the package time/tzdata by @ianlancetaylor

cc: @FiloSottile, @katiehockman, @mvdan

[mvdan]

At least from my personal experience, I've used scratch images quite a lot for deployments of "pure Go" services, and I needed ca-certificates more often than I needed tzdata. They both fit the same bill in terms of system files which the standard library might depend on, depending on what the program is doing. A significant amount of what I've written in the past ends up needing ca-certificates in the end (e.g. because of outgoing HTTPS requests), so it's almost a default that I make sure it's available beforehand.

This is all to say - if tzdata was included, I think rootcerts should too, given how I think it's at least just as important.

[seankhliao]

examples of other third party packges that provide this:
https://github.com/certifi/gocertifi
https://github.com/gwatts/rootcerts
https://github.com/alexflint/stdroots

I'm not particularly convinced that having it in std is a good idea, since out of date tzdata is usually just a minor bug but out of date ca certs is a security issue. Maybe somewhere under x/crypto.

[mvdan]

I agree with what the proposal says, though. The current solutions people are using are also potentially vulnerable to not getting new CA roots for a long time. This includes cases where the binary runs on a system that's simply not being updated regularly.

At the end of the day, it's up to the developer to keep up to date with Go versions and rebuild/redeploy their binaries as needed. If that maintenance is not happening, the result is the same whether or not this proposal is accepted and used by them.

[breml]

examples of other third party packges that provide this:
https://github.com/certifi/gocertifi
https://github.com/gwatts/rootcerts
https://github.com/alexflint/stdroots

One small difference between https://github.com/breml/rootcerts and the other 3rd party packages mentioned above is, that it can be used in a "none intrusive" way, because no application code needs to be changed. A simple blank import (import _ "crypto/x509/rootcerts") in package main is enough.

This is implemented the same way it is done in time/tzdata (see: https://github.com/breml/rootcerts/blob/master/rootcerts.go#L28-L34).

[rolandshoemaker]

I think there are two interesting parts to this proposal: implementing an API to set the default root pool, and adding a standard library package that contains some set of roots. While I am generally in favor of the first part, I am somewhat opposed to the second.

I think adding an API to crypto/x509 that allows setting the default root pool (rather than always using the system pool when no roots are specified) could be quite valuable, and would more cleanly and safely provide the functionality you (and some other existing packages) are getting with go:linkname tricks. I think the major downside of adding this kind of functionality though is allowing the user(/developer) to understand where their root pool is being set, and opens up the rabbit hole of dependencies setting a malicious set of roots (although it could be argued that this is already the case with the go:linkname trick, so we'd just be adding a more explicit easy to understand API to replace this inherently unsafe approach).

Speaking as a member of the security team, who would likely end up being responsible for maintaining it, I'm hesitant to implement a root store in the standard library. Even if we are just copying some existing root program, there are a number of somewhat complicated policy enforcement decisions we'd become embroiled in. I'm also not sure what the update process for the store would be, for instance if a root was to become distrusted in the NSS program, would we need to issue a security release of Go in order to provide an updated root store to users? I think if we were to do something like this it'd most likely need to live in a x/ repository as a module so that updates could be made in a less rigorous cadence. (Side note: just using the NSS trust store is not as simple on its face as it seems, while it is possible to extract a list of 'trusted' certificates there are somewhat complex policies around what certificates may be used for what, and a chunk of policy that is implemented in code, meaning that just pulling their list and taking it at face value is likely to result in trusting things we may not always want to trust.)

[breml]

@rolandshoemaker Thanks for your thoughtful feedback.

The main intend of my proposal is a standard library package that contains some set of roots.

What I care about is, a simple, save and trusted way to embed a trusted set of root certificates into a Go binary.
I more than once observed inexperienced developers handling the missing ca-certificates.crt problem in awful ways (e.g. committing the file together with the code, where it gets forgotten and therefore is never updated). To be able to import the root certs in the same simple and straight forward way as the time zone data is very compelling.

Go positions it self as the language of choice for the cloud and cloud native applications. To cite @spf13 (Steve Francia in https://www.infoq.com/articles/go-language-13-years/):

Go’s strength is still the cloud/server applications that Go is such a good fit for, ...

The second, much more significant phase, will be the industry shifting to take advantage of the unique cloud offerings, increasingly moving to cloud-native application development. In these cases, Go is the clear choice.

Lots of these cloud native applications will run in container environments and lots of them will need to talk to external APIs, which then will need root certificates to establish the secure connection. There I see a clear benefit in the Go standard library providing a simple way of including the CA root certificates into the Go binary. It is maybe a little bit of a stretch, but if a Go application is run in a container FROM scratch, then the Go application sort of becomes the operating system and operating systems normally provide a set of root certificates.

I clearly understand your concern for you as a member of the security team and I completely agree, that is likely your team, who will end up being responsible for the maintenance. But if we look at this from users point of view, then we are definitively in a better place, when the Go security team takes care of this, than if a random open source contributor publishes a package with this functionality. If this proposal gets accepted, I will happily remove/deprecate my package.
With the still rapid growing number of Go developers around the world I feel the maintenance of these root certificates inside of the Go standard library is worth the effort to increase the overall security and trust of the Go and especially the cloud native / containerized eco system.

For the update process as well as the location of such a package, it is my belief, that the package should life in the standard library (and not in x/) for the following reasons:

• My observation in projects in the industry is, that updating the Go tool chain (and with this the standard library) is a well understood process. Updating the Go tool chain is (due to the great work of the Go team and the compatibility guarantee) a friction less process, which is often even automated.
• In contrary, the updating of Go modules is unfortunately sort of a forgotten and less understood topic. I often observe, that the dependencies are only updated, when this is necessary due to functional requirements. The minimal version selection does not help here and I fear that a root certificates package in x/ will get way less often updated than the same package in the standard library.
• It is already possible to day to ship minor updates for Go and it is done quite often. E.g. for Go 1.15 we currently are at 1.15.7, which makes more than a minor release per month. So I do not really see a problem to ship a new minor release, if there is a certificate, that needs to become distrusted.

In regards to the NSS trust, I did not go into full details here, but it is my observation, that this problem has relaxed over the last couple years. On one hand, all the mentioned Linux distributions have solved this problem one way or the other, so this is a pretty well understood problem with battle proved solutions available. On the other hand does Mozilla provide the PEM of Root Certificates in Mozilla's Root Store with the Websites (TLS/SSL) Trust Bit Enabled (see: CA/Included Certificates). In my understanding, this is already the correct list of certificates with the correct trust bits enabled.

The other part you have identified in my proposal (API to set the default root pool) is less relevant for the use cases I am outlining in the proposal, for the following reasons:

1. On Unix/Linux based systems, Go already today offers the possibility to add additional certificates to the default pool by setting the respective environment variables.
2. There are other proposals covering somewhat related aspects:
   1. crypto/x509: add ability to reload root certificates crypto/x509: add ability to reload root certificates #41888
   2. crypto/x509: Convenient, efficient way to reload a certificate pool after on-disk changes: proposal: crypto/x509: reloading certificates from disk #35887
3. For me, the way the system roots are loaded is not important. The go:linkname trick has been bluntly copied from time/tzdata, so nothing I invented myself.