Add genception, a Bazel-backed go/packages driver by kasey · Pull Request #17307 · OffchainLabs/prysm

kasey · 2026-08-05T20:39:39Z

Add genception, a Bazel-backed go/packages driver

Part of the gloas-devnet-7 stacked series; based on incremental-state-htr, diff shown against it.

Code generators that resolve Go types through golang.org/x/tools/go/packages cannot run inside the Bazel sandbox. The default driver shells out to go list, and the go toolchain is not available in the sandbox, so there is nothing for it to talk to. That blocks codegen tools that use x/tools/go/packages for semantic parsing of source (rather than trying to deal with source code as raw text).

tools/genception is a GOPACKAGESDRIVER implementation that closes that gap. Instead of invoking the go toolchain, it answers go/packages queries from a package inventory generated inside of a Bazel rules plugin ("aspect"): the per-target *.pkg.json files produced by the rules_go gopackagesdriver aspect, plus an index file listing them. It loads that inventory into an in-memory registry, post-processes it into a complete package graph (absolute paths, stdlib imports, test packages), and serves file=/pattern=/bare-pattern queries against it.

This PR adds the tool and its tests only. Nothing in the tree invokes it yet, so existing builds are unaffected; the Bazel rule that wires it into codegen (tools/methodical.bzl) lands later in the series.

Key changes

Entrypoint

tools/genception/cmd/main.go — reads a packages.DriverRequest from stdin, writes the driver response to stdout. Builds the registry once per invocation (driver.NewJSONDriver), since loading the inventory is IO-heavy.
Deliberately exits 0 even on error. gopls checks the driver's exit code and silently falls back to go list when it is non-zero, which is exactly the failure mode this tool exists to avoid; a hidden fallback is worse than a visibly empty response.

Inventory ingest and package graph (`tools/genception/driver/`)

jsondriver.go — JSONDriver orchestrates the flow: load env, read the inventory index, stream-decode each *.pkg.json into the registry, resolve, then serve queries via Handle.
inventory.go — loadJsonListing reads the index file (a JSON []string of exec-root-relative paths) named by PACKAGE_JSON_INVENTORY.
flatpackage.go — flatPackage is the trimmed JSON form of packages.Package, and owns the Bazel-to-go/packages translations:
- resolveStdlib re-adds stdlib imports by parsing each compiled file's import block, since Bazel is unaware of stdlib and omits those edges from the JSON.
- deriveTestPackage splits _test.go files by their package clause: internal tests fold back into the base package, external tests become a derived <id>_xtest package that imports the package under test.
registry.go — the queryable graph.
- add/update merge packages that share a Go package path. Multiple Bazel labels can map to one package path (e.g. :go_proto and :go_default_library under //proto/prysm/v1alpha1); without the merge, one clobbers the other and source files disappear from the graph. The entry with the superset source list wins wholesale, keeping GoFiles/CompiledGoFiles/Imports from a single consistent target.
- rewritePackage/canonicalizeID normalize package IDs to import paths and strip the @@io_bazel_rules_go//stdlib: label prefix.
- resolveQueryID handles the three query forms from the driver protocol; query/walk return the matched roots plus their transitive dependency closure. Queries that match nothing are logged and dropped rather than emitted as dangling roots.
- indexFiles builds the file-to-package index used by file= queries, after paths are absolute and test packages exist.
resolver.go — rewrites Bazel's symbolic path prefixes (__BAZEL_EXECROOT__, __BAZEL_OUTPUT_BASE__, __BAZEL_WORKSPACE__) into real absolute paths, stat-checking each candidate root.
tagfilt.go — filters the file lists through go/build's MatchFile using the configured build tags, so the response reflects what would actually compile (relevant here given Prysm's develop tag).
response.go — driverResponse mirrors packages.DriverResponse but carries flatPackage; parseGoMinorVersion derives the protocol's GoVersion field from runtime.Version(), returning 0 for unparseable values such as devel builds.
logger.go — logs to the configured file, falling back to stderr. Configured during driver construction rather than init() so that importing the package has no filesystem side effects (stdout is the protocol channel, so log output must not go there).
recorder.go — optional debugging aid; when enabled, dumps the request and response JSON to a timestamped directory.

Tests

env_test.go (required/optional env vars, including PWD's fallback to the process value), inventory_test.go with testdata/json-list.json, registry_test.go (isSuperset, the superset-wins merge in both add orders, query-form resolution, resolved roots), response_test.go (Go version parsing).

Configuration

The driver is configured entirely through environment variables (tools/genception/driver/env.go). Required:

Variable	Purpose
`PACKAGE_JSON_INVENTORY`	Path to the JSON index listing the per-target `*.pkg.json` files
`PACKAGES_BASE`	Bazel output base, used to resolve symbolic path prefixes
`PWD`	Bazel exec root; falls back to the process's `PWD` if unset

Optional: GOTAGS (comma-separated build tags for file filtering), GOPACKAGESDRIVER_LOG_PATH (defaults to genception.log under PWD), and GOPACKAGESDRIVER_RECORDER_PATH (unset disables request/response recording).

Behavior notes

For file= queries the response describes the whole enclosing package, so CompiledGoFiles contains more than the single file named in the query. This is intentional — the caller needs the full package to type-check it.
New Bazel targets: //tools/genception/cmd (public go_binary) and //tools/genception/driver (go_default_library + go_default_test). No existing target depends on them in this PR.

Acknowledgements

I have read CONTRIBUTING.md.
I have included a uniquely named changelog fragment file.
I have added a description with sufficient context for reviewers to understand this PR.
I have tested that my changes work as expected and I added a testing plan to the PR description (if applicable).

_{Stack created with GitHub Stacks CLI • Give Feedback 💬}

nalepae · 2026-08-11T20:35:10Z

+	}
+	bi := 0
+	for i := range a {
+		if a[i] == b[bi] {


Will panic if len(b) == 0.

You're right, I mentally inverted what the check at the top of function did (and did so again when checking out your comment). I think it's not possible in practice, since b is the result of reading the package manifest, which can be assumed to not have an empty file list in practice because of how it is constructed, but doesn't hurt to be safe.

nalepae · 2026-08-11T20:38:37Z

+		// extract test sources and imports into their own flatPackage
+		testFp := pkg.deriveTestPackage()
+		if testFp != nil {
+			r.packages[testFp.ID] = testFp


Items of r.packages are added while iterating on r.packages itself.
Could it cause an issue here?

Hmm yeah, it's pseudorandom whether we visit the test package in the iterator (implementation detail in go's spec, but iterating a sparse bucket slice in practice). If we do visit one of these added packages, things could get weird since we're constructing a package name derived from the package containing the tests. Usually we always name an external test package package foo_test when in the same directory as package foo, but if we 1) had package bar_test in package foo and 2) get unlucky hash dice rolls on the map key bucket on every iteration such that keys keep landing after the previous iteration, I guess you could fail to terminate, or terminate after building a giant map that fills the disk lol. Insanely unlikely scenario but easy enough to just stick these in a separate slice and merge them at the end.

nalepae · 2026-08-11T20:52:09Z

+			}
+
+			if key := stdlibId(imp); key != "" {
+				fp.Imports[imp] = key


Is it guaranteed that fp.Imports is always non-nil?
What if flatPackage decodes from a JSON with no Imports key?

IIRC Imports is always output as an empty dict {} due to how the starlark side of things is coded, but it's easy enough to check+initialize so you don't have to trust the implicit assumption / lore.

nalepae · 2026-08-11T20:57:46Z

+	internalTests, externalTests, nonTests, err := fp.groupTestFiles(fp.GoFiles)
+	if err != nil {
+		log.WithError(err).WithField("package", fp.PkgPath).Warn("unable to group test files; skipping test package derivation")
+		return nil


If we are here, fp.GoFiles is not modified.
==> The package mixes package xxx and package xxx_test files.

If we are here, fp.GoFiles is not modified.

Right, this error handling path represents a situation where go's parser tried to parse a file, only up to package foo, and couldn't parse the package name out for some reason (for instance, an empty file). If such a file is present in a package then the package registry just gives up trying to deal with that package.

Previously I thought about this as a choice between the build failing dramatically or just giving up and letting the test files (potentially in another package) persist, since I don't think it would usually break anything. But looking at this again the simplest choice seems like filtering out the file that couldn't be parsed from all lists. I'll do that instead.

nalepae · 2026-08-11T21:00:07Z

+	if err != nil {
+		log.WithError(err).Error("unable to handle driver request")
+	}
+	_, err = out.Write(resp)


The pd.Handle error is overwritten here.
==> The process exits with 0 code with 0 written bits (but with an error log).

prestonvanloon

LGTM - Manu had some good feedback though

Code generators that resolve types through golang.org/x/tools/go/packages cannot run inside the Bazel sandbox: the go toolchain is unavailable, so the default `go list` driver has nothing to talk to. genception implements the GOPACKAGESDRIVER protocol against a package inventory that Bazel provides, answering file and pattern queries from that inventory instead of shelling out. It exits 0 even on error, because gopls silently falls back to `go list` on a non-zero exit, which is the failure mode this is meant to avoid.

kasey force-pushed the incremental-state-htr branch from ecc9798 to 6b385fa Compare August 5, 2026 21:08

kasey force-pushed the methodical-codegen-driver branch from 1a78c8b to 730d29e Compare August 5, 2026 21:08

kasey force-pushed the incremental-state-htr branch from 6b385fa to 6a0fb66 Compare August 5, 2026 22:00

kasey force-pushed the methodical-codegen-driver branch from 730d29e to 346a8de Compare August 5, 2026 22:00

kasey force-pushed the incremental-state-htr branch from 6a0fb66 to 75b7e27 Compare August 6, 2026 13:27

kasey force-pushed the methodical-codegen-driver branch from beba51a to 56e6c00 Compare August 6, 2026 13:27

kasey force-pushed the incremental-state-htr branch from 75b7e27 to c10d031 Compare August 6, 2026 17:25

kasey force-pushed the methodical-codegen-driver branch from 56e6c00 to a67a389 Compare August 6, 2026 17:25

kasey force-pushed the incremental-state-htr branch from 184d1eb to ec17e1a Compare August 7, 2026 20:00

kasey force-pushed the methodical-codegen-driver branch from a67a389 to e3b9bf1 Compare August 7, 2026 20:00

Base automatically changed from incremental-state-htr to develop August 10, 2026 17:00

kasey force-pushed the methodical-codegen-driver branch 2 times, most recently from 5c101d4 to 0440be1 Compare August 11, 2026 19:34

nalepae reviewed Aug 11, 2026

View reviewed changes

syjn99 self-requested a review August 12, 2026 05:04

prestonvanloon reviewed Aug 12, 2026

View reviewed changes

kasey added 4 commits August 12, 2026 15:31

placate gosec

c44b7ae

fix gosec complaints

07a9d04

edits per Manu's feedback

7aba3e0

kasey force-pushed the methodical-codegen-driver branch from 0440be1 to 7aba3e0 Compare August 12, 2026 20:32

Uh oh!

Conversation

kasey commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Add genception, a Bazel-backed go/packages driver

Key changes

Entrypoint

Inventory ingest and package graph (tools/genception/driver/)

Tests

Configuration

Behavior notes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kasey Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kasey Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kasey Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

prestonvanloon left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kasey commented Aug 5, 2026 •

edited

Loading

Inventory ingest and package graph (`tools/genception/driver/`)

kasey Aug 12, 2026 •

edited

Loading

kasey Aug 12, 2026 •

edited

Loading

kasey Aug 12, 2026 •

edited

Loading