Docs: go get of prebuilt aws-go provider fails with "module source tree too large" (500MB Go proxy limit); document GOPRIVATE workaround

[so0k]

Summary

go get of the prebuilt AWS provider for Go fails for recent major versions because the module source tree exceeds the Go module proxy / checksum-DB limit of 500 MB (524288000 bytes):

> go get github.com/cdktn-io/cdktn-provider-aws-go/aws/v25
go: downloading github.com/cdktn-io/cdktn-provider-aws-go/aws/v25 v25.3.0
go: github.com/cdktn-io/cdktn-provider-aws-go/aws/v25: create zip: module source tree too large (max size is 524288000 bytes)

This is a hard limit on proxy.golang.org and sum.golang.org (and enforced by the Go toolchain when it creates the module zip on the direct fallback). Publishing the module works fine — it's consumption that breaks.

We should document the user-side workaround in the cdktn docs, since there is nothing we can fix on the publishing side.

Workaround

Bypass the proxy and checksum verification for the affected module path:

GOPRIVATE=github.com/cdktn-io/cdktn-provider-aws-go go get github.com/cdktn-io/cdktn-provider-aws-go/aws/v25

Note: GOPROXY=direct alone is not sufficient — checksum verification against sum.golang.org still fails with the same size limit. GOPRIVATE (which implies both GONOPROXY and GONOSUMDB) is required.

Observed behavior per major version (with GOPRIVATE set)

Version	Result
aws/v21	✅ works (even without workaround)
aws/v22 (v22.1.0)	✅ works
aws/v23 (v23.8.0)	✅ works
aws/v24 (v24.12.0)	❌ create zip: module source tree too large
aws/v25 (v25.3.0)	❌ create zip: module source tree too large

Open question

Even with GOPRIVATE, v24/v25 still fail at the create zip step: in direct mode the Go toolchain itself creates the module zip and applies the same 500 MB limit to the uncompressed file tree, so sufficiently large versions cannot be consumed via go get/go mod at all.

It's not obvious why v22/v23 pass while v24/v25 don't — a checkout of v22 is ~579 MB on disk, which is over the limit. Likely explanation: the on-disk size includes .git (and possibly other files excluded from the module zip), so the actual zipped source tree for v22/v23 is still under 500 MB, while v24/v25 crossed it. Worth verifying and documenting the size trajectory, since every major version gets closer to being completely unconsumable via the Go toolchain.

Proposed actions

• Add a "Go" caveat section to the prebuilt-providers docs: explain the 500 MB proxy/sumdb limit and the GOPRIVATE workaround, and note that very large versions (currently aws/v24+) cannot be fetched via go get at all.
• Verify the actual module-zip source-tree sizes per major version to confirm why v22/v23 still pass.
• Investigate options for keeping the Go package under the limit (e.g. splitting submodules, trimming generated docs from the Go package) or document an alternative consumption path (vendoring / git submodule) for oversized versions.

References

• Reported in cdk-dev Slack: https://cdk-dev.slack.com/archives/C0A3479CMJ9/p1786338705665899
• Go module proxy size limit: module source tree too large (max size is 524288000 bytes)

[so0k]

Ran a systematic investigation in a local test harness. TL;DR: the exact first-failing release is aws/v24.6.0 (so the table in the issue description needs a correction — v24 didn't fail from the start), the .git-exclusion hypothesis is confirmed, the consuming project's cdktn version is irrelevant, and for v24.6.0+ no client-side workaround exists — GOPRIVATE cannot help, because the 500 MiB cap is enforced by the Go toolchain itself when it builds the module zip in direct mode.

Scenarios tested

1. Consumer-side matrix — does the project's cdktn version matter?
Three minimal Go projects (pinned to cdktn 0.22 / 0.23 / 0.24) each ran go get against provider versions v22.1.0, v23.8.0, v24.5.0, v24.6.0, and v24.12.0, in both proxy mode and GOPRIVATE direct mode — including cross-combinations (e.g. v24.12.0 requested from the cdktn-0.22 project, v22.1.0 from the cdktn-0.24 project).

Result: the same provider version passes or fails identically in every project and every mode. This is structural — go/the proxy build and size-check one immutable zip per module@version, before anything about the importing project is consulted. So this is purely a property of the provider module version, not of cdktn.

2. Boundary bisection.
All 268 tags of cdktn-io/cdktn-provider-aws-go were listed and bisected within v24 using real GOPRIVATE go mod download runs plus the size-check tool below:

v24.0.1  PASS  515,735,376 B
v24.3.0  PASS  520,517,609 B
v24.4.0  PASS  523,263,535 B
v24.5.0  PASS  523,366,927 B   <- last passing release
v24.6.0  FAIL  525,519,743 B   <- FIRST failing release (limit: 524,288,000)
v24.12.0 FAIL  532,890,331 B

No tags exist between v24.5.0 and v24.6.0, so the boundary is unambiguous. Proxy mode fails identically (surfaced as a sum.golang.org 404 wrapping the same create zip: module source tree too large error — i.e. proxy.golang.org can't build the zip either).

How the .git hypothesis was narrowed down

The puzzle was that a v22 checkout is well over 500 MB on disk yet fetches fine. To measure exactly what Go counts, a small tool was built around golang.org/x/mod/zip.CheckDir — the literal function cmd/go calls to validate/size a module before zipping — and run against a shallow clone of each tag. Three numbers per tag: repo incl. .git, .git alone, and the module tree Go actually counts.

Key finding on layout: the Go module root is the aws/ subdirectory (that's where go.mod lives; module path .../aws/vNN), and zip.CheckDir explicitly excludes .git and only walks the module root. So neither .git (~73–87 MB) nor the repo-root LICENSE/README.md count.

tag	repo + .git	module tree (what Go counts)	vs 524,288,000
aws/v21.23.0	526 MB	455,155,786	PASS
aws/v22.1.0	539 MB	465,042,954	PASS
aws/v23.8.0	586 MB	507,806,943	PASS
aws/v24.5.0	604 MB	523,366,927	PASS (last good)
aws/v24.6.0	606 MB	525,519,743	FAIL (first bad)
aws/v25.3.0	648 MB	561,410,520	FAIL

Hypothesis confirmed: every tested version's checkout is over 500 MB from v22 onward (which explains the "v22 is ~579 MB, why does it work?" confusion — checkout size is simply the wrong number to look at), but the counted module tree stayed under the cap until v24.6.0 overshot it by ~1.2 MB.

What tipped it over

README.md diff between the two tags:

-# CDKTN Go bindings for hashicorp/aws provider version 6.49.0
+# CDKTN Go bindings for hashicorp/aws provider version 6.50.0

terraform-provider-aws 6.50.0 added 5 new resources — mostly Bedrock AgentCore (ecsdaemontaskdefinition alone is ~986 KB of generated Go) — accounting for ~72% of the +2.1 MB delta, plus growth in existing Bedrock AgentCore resources and the jsii assembly blob. This is pure upstream growth, not a generator regression: the release automation tracks latest of ~> 6.0 with no size gate, and one routine upstream minor was enough to cross an already-523 MB module over the line. Growth is roughly linear (~1–2 MB per upstream minor; v25.3.0 is already 561 MB), so every future release stays broken without a structural fix, and module content is immutable once tagged — the broken versions can never be repaired retroactively.

Proposed fix

Short term (docs — the original scope of this issue):

• Document GOPRIVATE=github.com/cdktn-io/cdktn-provider-aws-go for v22.x–v24.5.0 (GOPROXY=direct alone is not enough; sumdb verification hits the same limit).
• Document clearly that v24.6.0+ cannot be consumed via go get at all, in any mode — the last usable Go release is v24.5.0 (upstream provider 6.49.0).

Structural (required for v24.6.0+ and all future releases) — options in rough order of preference:

1. Split the Go package into per-service modules (e.g. .../aws-go/ec2/vNN, .../aws-go/s3/vNN), the way cdktf-provider-aws split namespaces at the package level for other reasons. No single service is anywhere near 500 MB, consumers only pull what they import, and it fixes proxy + sumdb + direct in one move. Most work in the generator/release pipeline.
2. Trim the generated tree: the jsii assembly blob is ~40 MB and the per-resource Go is extremely verbose; measure whether stripping embedded docs/comments from generated code buys enough headroom. Buys time (~years at current growth) but doesn't fix the trajectory.
3. Add a size gate to release automation: run golang.org/x/mod/zip.CheckDir against aws/ in CI before tagging, so a release that will be unconsumable is caught (and can be held/split) instead of silently published broken. Cheap, and worth doing regardless of 1/2.
4. Fallback documentation only: vendoring / git submodule consumption path for oversized versions (works, but a poor developer experience and doesn't help transitive consumers).

Reproduction commands and the reusable measurement harness (the zip.CheckDir tool, a measure-any-tag script, and the full consumer matrix script) are available if useful — happy to PR the size gate from item 3.

[so0k]

Follow-up: the structural fix proposed above (option 1, per-service Go modules) is now validated in production, via a parallel set of bindings rather than a change to this repo's pipeline.

cdktn-io/cdktn-aws regroups terraform-provider-aws (pinned 6.62.0) from one jsii submodule per resource into one per service — 257 service groups + a provider module — and publishes Go as 258 sibling modules in one repo, cdktn-io/cdktn-aws-go, tagged <group>/v0.1.1, no root go.mod. npm/PyPI/Maven/NuGet stay a single library (@cdktn/aws).

Measured against the 524,288,000 B cap this issue is about:

• Largest single module (awslexv2models): 60,722,732 B = 11.58 % of the cap (~9× headroom on the worst case; median module 0.18 %). Every module gated in CI by the same zip.CheckDir call go mod download runs — release automation now fails before tagging anything unconsumable (option 3 from the proposal, applied).
• An unsplit monolith of the same bindings floors at 99.89 % of the cap and grows ~1–2 MB per upstream minor — permanently riding the limit even in the best case.
• Verified through the real pipeline this issue broke on: go get github.com/cdktn-io/cdktn-aws-go/awslambda@v0.1.1 resolves through proxy.golang.org + sum.golang.org (30 s cold), and a 36-module consumer resolves everything from the proxy, builds, and synthesizes with provider validation on in ~2 s. No GOPRIVATE, no workarounds.

For this repo's prebuilt cdktn-provider-aws-go: nothing changes retroactively — v24.6.0+ remains unconsumable and module contents are immutable, so the short-term guidance above (last usable Go release v24.5.0, GOPRIVATE for v22.x–v24.5.0) still stands. Go consumers who need current provider versions can migrate to the per-service modules; the import shape changes (awss3.NewAwsS3Bucket(...) etc.), so it's a port, not a drop-in.

Whether the grouped generator family should become a supported mode of @cdktn/provider-generator (making this fixable for the whole prebuilt fleet) is written summarised in this artifact and backed up with the full evidence trail in cdktn-aws/REPORT.md.