go_deps: download .mod files through Bazel whenever possible

[jayconrod]

So that developers can make use of --experimental_remote_downloader, which can provide mirroring and other benefits.

A sketch of how this can work:

• Facts contain an entry for each .mod with importpath, version, sum. This set may be stale or incomplete.
• When preparing the synthetic workspace, we list .mod files needed based on require directives and module tags. This set may also be incomplete.
• For each .mod file:
  • Skip if GOPRIVATE or GONOPROXY matches importpath. Each variable contains a list of glob patterns that may match a prefix of the module path. This avoids disclosure of a private module name to an untrusted public proxy.
    -Skip if sum is not known.
  • Use module_ctx.download to attempt to download the .mod file into GOMODCACHE. Typically GOMODCACHE will be go_repository_cache. Download even if the file is already present, since we want to populate the download cache too. Tolerate errors: this is best effort. The URL should be constructed from only the first on the GOPROXY list; don't attempt to handle direct, cache, or fall back to other servers.
• Run go list -m -json all, as we do now.
• Return facts for all .mod files mentioned in the go list -m output so that the facts are not incomplete on the next run.

Follow-up to #2418

[jayconrod]

@fmeum FYI, I started implementing this, and I hit kind of a major misconception I had about go.sum files. I assumed the h1:... sum for a go.mod file was the base64 encoding of its SHA-256 sum, so we'd be able to convert sums to something module_ctx.download could use to verify downloads.

That is unfortunately not the case. Go first computes the go.mod hash, then writes a line to a file with go.mod <hash>, then hashes that file to get the sum for go.sum. So given go.sum, we have no way or deriving the SHA-256 sum for the actual content.

This may still be salvageable. We'll need to download the files without sums, but go list -m should still verify them.

One other modification: go_deps can't write files outside its scratch directory, so it will need to create a directory to be used as a file:// GOPROXY instead of writing directly to GOMODCACHE.

[fmeum]

Could we use facts to also record the mapping from the h1 hash to the corresponding SHA-256 hash?

[jayconrod]

We could, but do you think we'll need it? We don't have a way to convert between the two hashes in Starlark, and only the sha256 hashes are useful to Bazel. Go still knows the h1 hashes, so it will be able to verify files whether downloading them from the proxy or "downloading" them from the scratch directory.

[fmeum]

It's not so much about verification as it is about hitting the repository cache. If ctx.download can be supplied with a SHA-256, it could fetch all the files without network access (assuming up-to-date facts).