feat(schema): config schema generation pipeline (foundation) - #362
Closed
viniciusdc wants to merge 2 commits into
Closed
feat(schema): config schema generation pipeline (foundation)#362viniciusdc wants to merge 2 commits into
viniciusdc wants to merge 2 commits into
Conversation
… accessor Establishes the architectural foundation for the upcoming JSON Schema + commented YAML reference pipeline (consumed by nebari-docs). The actual generation logic lands in follow-up commits on this branch. What this commit changes: - pkg/provider/provider.go: adds `ConfigType() reflect.Type` to the Provider interface. One-line implementations on all 6 cluster providers (aws, gcp, azure, hetzner, local, existing). - pkg/dnsprovider/provider.go: same addition to DNSProvider, with the one-line implementation on cloudflare. - pkg/nic/config_types.go: adds `(*Client).RegisteredConfigTypes(ctx)` returning a ConfigTypes struct keyed by provider name. Walks the registry; no hard-coded provider list. - pkg/configschema/: skeleton package with Format constants, Options struct, and Generate signature. Body returns "not yet implemented" for now; the invopop/jsonschema + goccy CommentMap wiring lands in the next commit. - cmd/schemagen/: skeleton binary that wires up nic.Client → RegisteredConfigTypes → configschema.Generate. Currently does a dry-run discovery print so reviewers can see the registry-driven enumeration working end-to-end. Run: `go run ./cmd/schemagen`. - Makefile: `make schemas` target wraps the schemagen invocation. Design rationale lives in ADR-0005 (#360) and the architecture review that drove the registry-driven approach. Chuck's prior proposal in #40 arrived at the same architectural conclusion (registry-driven enumeration, new providers picked up automatically); the output format shifted to JSON Schema + commented YAML for the Docusaurus renderer. Refs #40.
Wires up the schema-gen pipeline end-to-end: - pkg/configschema.Generate (FormatJSON) now produces real JSON Schema using invopop/jsonschema with godoc descriptions extracted via Reflector.AddGoComments. Configured to read yaml tags (not json), package-qualify $defs keys to avoid collisions across providers (e.g. aws.Config vs longhorn.Config), and emit additionalProperties:false to match the validator's strict-on-unknown contract. - cmd/schemagen now writes schemas/manifest.json, schemas/nebari-config.json, and schemas/providers/<name>.json. Walks pkg/ to collect package paths for godoc extraction. Supports -version, -providers, -pkg-root flags. - schemas/ committed in-tree. 6 cluster providers (aws, gcp, azure, hetzner, local, existing) + 1 DNS provider (cloudflare) + top-level nebari-config + manifest. Total ~28KB of JSON. - .github/workflows/schemas.yml drift-check on PRs and main pushes, paths-filtered to config/provider/schema-gen files. Authors run `make schemas` locally and commit; CI fails on divergence (kubebuilder CRD-regen pattern, no auto-commit). YAML reference output (FormatYAML) still stubbed; lands in a follow-up commit on this branch.
This was referenced Jul 23, 2026
Draft
Collaborator
Author
|
Superseded by #507. This branch is 41 commits behind main and built on the old #507 re-homes the pipeline onto current main: same design (invopop-based
|
6 tasks
viniciusdc
added a commit
that referenced
this pull request
Aug 15, 2026
Re-home the config schema-generation pipeline onto the current provider
layout (pkg/providers/{cluster,dns}), superseding the stale branch behind
#362 which was 41 commits behind main and built on the removed
pkg/provider/ layout.
- Add ConfigType() reflect.Type to the cluster and DNS provider interfaces,
with one-line implementations on all seven registered providers.
- pkg/configschema wraps invopop/jsonschema: reads yaml tags, package-
qualifies $defs keys so aws.Config and longhorn.Config don't collide,
emits additionalProperties:false, and pulls field descriptions from godoc.
- pkg/nic/config_types.go exposes RegisteredConfigTypes through the registry,
so the generator enumerates providers with no hard-coded list.
- cmd/schemagen writes schemas/manifest.json, nebari-config.json, and
providers/<name>.json. `make schemas` regenerates; a paths-filtered CI
workflow fails on drift.
- schemas/ is committed in-tree for the docs site to consume.
Content fixes surfaced while rendering these schemas in the docs:
- enum tags on certificate.type and the AWS node-group taint effect, so the
allowed values are structured instead of buried in prose.
- drop an internal issue link from the trust_bundle godoc that was leaking
onto the generated, user-facing schema.
Refs #40, #360.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs
nebari-docs.nic configCLI discussion. This PR ships only the internal generation pipeline; user-facing subcommands are out of scope. Design rationale lives there.Status
Draft — JSON Schema pipeline + CI drift gate are live. YAML reference output, godoc cleanup, and
ARCHITECTURE.mdupdates land on this branch before the PR moves out of Draft.What's in this PR
provider.Provideranddnsprovider.DNSProviderinterfaces gainConfigType() reflect.Type. One-line implementations on all 7 registered providers (aws, gcp, azure, hetzner, local, existing, cloudflare).pkg/nic/config_types.go— new(*Client).RegisteredConfigTypes(ctx)accessor +ConfigTypesstruct. Walks the registry; no hard-coded provider list anywhere in the pipeline.pkg/configschema/— wrapsgithub.com/invopop/jsonschema.Generate(ctx, t, FormatJSON, opts)produces JSON Schema with godoc descriptions extracted viaReflector.AddGoComments. Configured to read yaml tags (not json), package-qualify$defskeys to avoid name collisions (e.g.aws.Configvslonghorn.Config), and emitadditionalProperties: false.FormatYAMLis stubbed.cmd/schemagen/— writesschemas/manifest.json,schemas/nebari-config.json,schemas/providers/<name>.json. Walkspkg/to collect package paths for godoc extraction. Flags:-out,-providers,-pkg-root,-version.schemas/committed in-tree (~28KB total). 6 cluster providers + 1 DNS provider + top-level nebari-config + manifest.Makefile—make schemastarget..github/workflows/schemas.yml— drift-check on PRs and main pushes, paths-filtered to config / provider / schema-gen files. Fails on divergence with a clear "runmake schemaslocally" message. No auto-commit (kubebuilder CRD-regen pattern).Planned follow-ups
Best-guess ordering; may shift as the work progresses.
goccy/go-yamlCommentMapintegration forFormatYAML(the human-readable reference document).examples/aws-config.yaml).ARCHITECTURE.md+WALKTHROUGH.mdupdates (directory map + new "Single Provider Registry" subsection).Once those land the PR moves out of Draft.
Docs-site consumption
Docusaurus fetches
schemas/manifest.jsonfromraw.githubusercontent.com/nebari-dev/nebari-infrastructure-core/<ref>/schemas/manifest.json, then each referenced schema. Tagged releases give versioned docs for free;maincan be surfaced as an "experimental" / "unstable" track.Test plan
go build ./...cleanmake schemasregeneratesschemas/with no drift on a clean treego run ./cmd/schemagenenumerates all 7 registered providers via the registrygo run ./cmd/schemagen -providers aws,hetznernarrows correctlyschemas/providers/hetzner.jsonincludes field-level godoc asdescriptionand per-package-qualified$defskeysmake schemas, push, confirm CI fails with the expected message