docs: add "perfect upjet provider" agent skill

Co-authored-by: Breee <11966385+Breee@users.noreply.github.com>

Author: Copilot

.github/skills/upjet-provider/SKILL.md

@@ -0,0 +1,109 @@
+---
+name: upjet-provider
+description: Build a production-grade upjet v2 Crossplane provider from any Terraform provider, following the crossplane-contrib/provider-keycloak reference architecture — repo scaffold, code generation pipeline, external-name/reference configuration, CI, e2e tests, schema-diff and upstream-release automation. Use when creating a new Crossplane provider, porting a Terraform provider to Crossplane, or auditing an existing upjet provider for missing pieces.
+---
+
+# The Perfect Upjet Provider
+
+This skill describes end-to-end how to build a Crossplane provider from a Terraform
+provider using [upjet](https://github.com/crossplane/upjet) v2, matching the
+architecture of [`crossplane-contrib/provider-keycloak`](https://github.com/crossplane-contrib/provider-keycloak)
+— the reference implementation this skill is derived from.
+
+Throughout, substitute:
+
+| Placeholder | Example (keycloak) | Example (litellm) |
+|---|---|---|
+| `<name>` | `keycloak` | `litellm` |
+| `<ORG>` | `crossplane-contrib` | `corewire` |
+| `<TF_SOURCE>` | `keycloak/keycloak` | `BerriAI/litellm` |
+| `<TF_REPO>` | `https://github.com/keycloak/terraform-provider-keycloak` | `https://github.com/BerriAI/terraform-provider-litellm` |
+| `<ROOT_GROUP>` | `keycloak.crossplane.io` | `litellm.crossplane.io` |
+
+## Before you start — inputs you must collect
+
+1. **Terraform provider repo + latest release tag** (`<TF_REPO>`, `v<TERRAFORM_PROVIDER_VERSION>`).
+2. **Registry source address** (`<TF_SOURCE>`, as used in `required_providers`).
+3. **Whether the provider is SDKv2 (`helper/schema`), plugin-framework, or muxed.**
+   This decides `WithTerraformPluginSDKIncludeList` vs `WithTerraformPluginFrameworkIncludeList`.
+4. **The exported provider constructor** (e.g. `provider.KeycloakProvider(nil)`,
+   `litellm.Provider()`) — needed for the no-fork runtime path.
+5. **Docs path inside the TF repo** (`docs/resources`, sometimes `website/docs/r`).
+6. **Auth model** — which provider-config attributes are required, which are secret.
+7. **Release asset naming** — `<name>_<version>_<os>_<arch>.zip`, needed to fetch the
+   schema for the supported `PLATFORMS`.
+
+Do not guess these. Fetch the upstream repo and read `main.go` / `docs/` first.
+
+## Build order
+
+Work in this order; each step is verifiable on its own.
+
+1. **Scaffold the repository** → `references/repository-layout.md`
+2. **Makefile + build submodule** → `references/build-system.md`
+3. **`config/` — schema, metadata, external names, per-group config** → `references/config-patterns.md`
+4. **`internal/clients/<name>.go` — the `terraform.SetupFn`** → `references/clients.md`
+5. **`cmd/provider/main.go` — the controller manager** → `references/provider-main.md`
+6. **`generate/generate.go` — the generation pipeline**, then run `make generate` → `references/generation-pipeline.md`
+7. **Examples + e2e (chainsaw/uptest)** → `references/testing.md`
+8. **CI, release, and upstream-tracking automation** → `references/automation.md`
+9. **Docs (`README`, `CONTRIBUTING`, repo-local `SKILL.md`/`AGENTS.md`)**
+
+A provider is only "complete" when every item in `references/checklist.md` is ticked.
+
+## Non-negotiable rules
+
+- **Terraform CLI version must stay `< 1.6`.** Terraform 1.6+ is BSL-licensed and
+  cannot be used in an Apache-2.0 project. The Makefile must *fail* on a higher
+  version (`check-terraform-version`).
+- **No-fork only.** Embed the Terraform provider as a Go dependency and pass it via
+  `ujconfig.WithTerraformProvider(p)`. Never ship a `terraform` binary in the image
+  and never shell out to it at runtime. The runtime image is `distroless/static`
+  with a single Go binary.
+- **`config/schema.json` and `config/provider-metadata.yaml` are committed.**
+  They are `//go:embed`-ed, so generation and runtime never need network access.
+- **`config/external_name.go` is the single source of truth** for which Terraform
+  resources are exposed. `config/generated.lst` is derived from it by
+  `cmd/generatedlist` and verified in CI (`make generated-lst-check`).
+- **Never hand-edit generated output**: `apis/**/zz_*.go`, `internal/controller/**/zz_*.go`,
+  `package/crds/`, `examples-generated/`. Change `config/` and re-run `make generate`.
+- **CI must run `make generate` and fail on a dirty tree** (`check-diff`). This is
+  what keeps generated code honest.
+- **Every exposed managed resource needs an e2e example** and must be listed in a
+  `cluster/test/cases*.txt` file; enforce it with a coverage check.
+- **Bumping the upstream Terraform provider is never "just a version bump".**
+  Gate it on `make schema-version-diff` (state-schema version changes) and
+  `make crddiff` (breaking CRD changes).
+
+## Quick reference — the commands every provider must support
+
+```bash
+make submodules              # init the crossplane/build submodule (first run)
+make generate                # schema -> docs -> types -> CRDs -> controllers -> lists
+make build                   # build the provider binary + xpkg
+make test                    # unit tests
+make lint                    # golangci-lint
+make local-deploy            # kind + crossplane + locally built provider
+make e2e                     # local-deploy + uptest/chainsaw suite
+make schema-version-diff     # TF state-schema drift vs base branch (CI)
+make schema-diff OLD_PROVIDER_VERSION=x.y.z   # manual two-version schema diff
+make crddiff                 # breaking CRD change detection (CI)
+make generated-lst-check     # generated.lst is in sync with external_name.go
+```
+
+Always run `make lint` (and `make generate`) before committing Go changes.
+
+## Reference files
+
+| File | Read it when |
+|---|---|
+| `references/repository-layout.md` | Creating the repo skeleton; what every file/dir is for |
+| `references/build-system.md` | Writing the Makefile: schema fetch, docs pull, platforms, tool pinning |
+| `references/config-patterns.md` | Choosing external names, references, groups, kinds, sensitive fields |
+| `references/clients.md` | Writing `TerraformSetupBuilder`, credential handling, session/client pooling |
+| `references/provider-main.md` | Controller-manager wiring, flags, feature gates, safe-start, webhooks |
+| `references/generation-pipeline.md` | `generate/generate.go` directives and what each stage produces |
+| `references/testing.md` | Examples, uptest/chainsaw, e2e case lists, coverage gates |
+| `references/automation.md` | CI jobs, release workflows, schema-diff issues, upstream release checks |
+| `references/troubleshooting.md` | Known upjet v2 / crossplane-runtime v2 compile and runtime pitfalls |
+| `references/checklist.md` | Final completeness checklist for a "perfect" provider |

.github/skills/upjet-provider/references/automation.md

@@ -0,0 +1,75 @@
+# CI, release and upstream-tracking automation
+
+## `.github/workflows/ci.yml`
+
+Jobs, in dependency order:
+
+| Job | What it does |
+|---|---|
+| `detect-noop` | `fkirc/skip-duplicate-actions`; optionally computes the e2e scope via `scripts/e2e_dag.py select` |
+| `lint` | `golangci/golangci-lint-action` with the pinned `GOLANGCILINT_VERSION` |
+| `check-diff` | `make generate` then `git diff --exit-code`; plus `make generated-lst-check`, `make e2e-cases-check`, docs freshness |
+| `unit-tests` | `make test` (+ race tests), upload coverage |
+| `build` / `build-publish` | QEMU + buildx multi-arch image and xpkg; publish on `main` / `release-*` |
+| `e2e-tests` | kind + Crossplane + `make uptest`, matrixed over upstream versions where relevant |
+| `schema-version-diff` | `make schema-version-diff` — flags Terraform state-schema version bumps |
+| `crddiff` | `make crddiff` — flags breaking CRD changes |
+
+Cache Go build output using `make go.cachedir` as the cache key input.
+
+`check-diff` is the most valuable job in the whole pipeline: it proves the committed
+generated code matches `config/`.
+
+## Release workflows
+
+- `auto-release.yaml` — scheduled: if there are relevant changes since the last tag
+  and CI is green, bump the semver tag and cut a GitHub Release with generated notes.
+- `tag.yml` — `workflow_dispatch` with `version` + `message` inputs for manual tagging.
+- `quick-release.yml` (optional) — scheduled multi-arch image/xpkg publish, with an
+  optional `crane copy` mirror to a second registry.
+- `backport.yml` (optional) — label-driven backports to `release-*` branches.
+
+## Upstream Terraform provider tracking
+
+Two complementary automations, both **dry-run on pull requests** (and path-filtered
+to their own sources so they don't run on unrelated PRs) and **live on schedule**:
+
+### `schema-diff-issues.yml` → `scripts/schema_diff_issues.py`
+
+Compares `config/schema.json` (everything upstream offers) against
+`config/generated.lst` (what we expose) and files one GitHub issue per
+not-yet-exposed resource, skipping resources already tracked by an open issue.
+This turns "which resources are still missing?" into a live backlog.
+
+### `provider-release-check.yml` → `scripts/check_provider_release.py`
+
+- `--mode issue`: compare the latest upstream release to `TERRAFORM_PROVIDER_VERSION`
+  in the Makefile; file a tracking issue with the full bump checklist.
+- `--mode bump`: perform the mechanical bump in the working tree — update the
+  Makefile, `go get` the new provider module, `go mod tidy`, `make generate` — and
+  emit a PR body summarizing the schema diff (via `version_diff.py`). Committing,
+  branching and PR creation stay in the workflow so git identity is controlled there.
+
+### `scripts/version_diff.py`
+
+Given `generated.lst`, an old `schema.json` and a new one, reports new resources,
+removed resources, `SchemaVersion` bumps (which imply Terraform state migrations)
+and per-attribute additions/removals for generated resources. Exit codes:
+`0` no changes, `1` changes detected, `2` error.
+
+## Renovate
+
+`.github/renovate.json` should:
+
+- keep Go modules and GitHub Actions current (grouped, auto-merge for patch/minor),
+- treat the **Terraform provider dependency specially** — either exclude it from
+  auto-merge or require the `provider-release-check` flow, because bumping it
+  regenerates every CRD and may need a state-schema migration,
+- pick up `# renovate: datasource=github-releases depName=...` comments in the
+  Makefile so `TERRAFORM_PROVIDER_VERSION` is tracked too.
+
+## Repository hygiene
+
+`CODEOWNERS`, a `Dockerfile` based on a digest-pinned `gcr.io/distroless/static`
+running as `USER 65532`, and `package/crossplane.yaml` declaring
+`capabilities: [safe-start]`.

.github/skills/upjet-provider/references/build-system.md

@@ -0,0 +1,106 @@
+# Build system
+
+The Makefile is layered on the `crossplane/build` submodule
+(`git submodule add https://github.com/crossplane/build build`, branch `master`),
+which supplies `common.mk`, `output.mk`, `golang.mk`, `k8s_tools.mk`,
+`imagelight.mk`, `xpkg.mk`, `local.xpkg.mk`, `controlplane.mk`.
+
+## Variables
+
+```makefile
+PROJECT_NAME ?= provider-<name>
+PROJECT_REPO ?= github.com/<ORG>/$(PROJECT_NAME)
+
+export TERRAFORM_VERSION ?= 1.5.7   # MUST stay < 1.6 (BSL)
+TERRAFORM_VERSION_VALID := $(shell [ "$(TERRAFORM_VERSION)" = "`printf "$(TERRAFORM_VERSION)\n1.6" | sort -V | head -n1`" ] && echo 1 || echo 0)
+
+export TERRAFORM_PROVIDER_SOURCE  ?= <TF_SOURCE>
+export TERRAFORM_PROVIDER_REPO    ?= <TF_REPO>
+# renovate: datasource=github-releases depName=<owner>/terraform-provider-<name>
+export TERRAFORM_PROVIDER_VERSION ?= x.y.z
+export TERRAFORM_PROVIDER_DOWNLOAD_NAME ?= terraform-provider-<name>
+export TERRAFORM_PROVIDER_DOWNLOAD_URL_PREFIX ?= ${TERRAFORM_PROVIDER_REPO}/releases/download/v$(TERRAFORM_PROVIDER_VERSION)
+export TERRAFORM_NATIVE_PROVIDER_BINARY ?= terraform-provider-<name>_v$(TERRAFORM_PROVIDER_VERSION)
+export TERRAFORM_DOCS_PATH        ?= docs/resources
+export TERRAFORM_FILE_MIRROR      ?= .terraform.d/plugins
+export TERRAFORM_FILE_MIRROR_REPO ?= ${TERRAFORM_FILE_MIRROR}/registry.terraform.io
+
+PLATFORMS ?= linux_amd64 linux_arm64
+GO_STATIC_PACKAGES = $(GO_PROJECT)/cmd/provider $(GO_PROJECT)/cmd/generator
+GO_LDFLAGS += -X $(GO_PROJECT)/internal/version.Version=$(VERSION)
+GO_SUBDIRS += cmd internal apis config
+TERRAFORM_PROVIDER_SCHEMA := config/schema.json
+```
+
+Pin the toolchain explicitly so CI and local runs agree: `GO_REQUIRED_VERSION`,
+`GOLANGCILINT_VERSION`, `KUBECTL_VERSION`, `KIND_VERSION`, `UP_VERSION`,
+`UPTEST_VERSION`, `CHAINSAW_VERSION`, `CROSSPLANE_VERSION`, `CROSSPLANE_CLI_VERSION`.
+
+## Schema acquisition (`config/schema.json`)
+
+Terraform CLI is used **only at generation time** to dump the provider schema:
+
+1. `download-tf-provider-platforms` → for each entry in `PLATFORMS`, download
+   `${DOWNLOAD_URL_PREFIX}/${DOWNLOAD_NAME}_${VERSION}_${PLATFORM}.zip` into a
+   filesystem mirror at `$(WORK_DIR)/terraform/$(TERRAFORM_FILE_MIRROR_REPO)/...`
+   and unzip it. Use `curl --retry 5 --retry-delay 5 --retry-all-errors`.
+2. Write `main.tf.json` with `required_providers` pinned to
+   `<TF_SOURCE>@$(TERRAFORM_PROVIDER_VERSION)` and `config.tfrc` with a
+   `provider_installation { filesystem_mirror { ... include = ["*/*/*"] } }` block.
+3. `TF_CLI_CONFIG_FILE=... terraform init -no-color` then
+   `terraform providers schema -json=true > config/schema.json`.
+
+Hook it into generation with `generate.init: $(TERRAFORM_PROVIDER_SCHEMA) pull-docs`.
+
+## Docs acquisition (`pull-docs`)
+
+Sparse, shallow, blobless clone of the TF provider repo at the pinned tag, checking
+out only `$(TERRAFORM_DOCS_PATH)`:
+
+```makefile
+git clone -c advice.detachedHead=false --depth 1 --filter=blob:none \
+  --branch "v$(TERRAFORM_PROVIDER_VERSION)" --sparse "$(TERRAFORM_PROVIDER_REPO)" \
+  "$(WORK_DIR)/$(TERRAFORM_PROVIDER_SOURCE)"
+git -C "$(WORK_DIR)/$(TERRAFORM_PROVIDER_SOURCE)" sparse-checkout set "$(TERRAFORM_DOCS_PATH)"
+```
+
+The scraper in `generate/generate.go` reads from `../.work/${TERRAFORM_PROVIDER_SOURCE}/${TERRAFORM_DOCS_PATH}`.
+
+## Generation hooks
+
+```makefile
+generate.init: $(TERRAFORM_PROVIDER_SCHEMA) pull-docs
+generate.done: generated-lst        # + e2e-index, docs-gen if you have them
+```
+
+`generated-lst` / `generated-lst-check` run `go run ./cmd/generatedlist [--check] config/generated.lst`.
+
+## Local development and e2e
+
+```makefile
+controlplane.up:   # kind create cluster + helm install crossplane (pinned chart)
+local-deploy: build controlplane.up local.xpkg.deploy.provider.$(PROJECT_NAME)
+uptest:            # uptest e2e "$(UPTEST_EXAMPLE_LIST)" --setup-script=cluster/test/setup.sh \
+                   #   --default-conditions="Test" --default-timeout=2400s
+e2e: local-deploy uptest
+```
+
+`UPTEST_EXAMPLE_LIST := $(shell grep -v '^\#' cluster/test/cases.txt | paste -sd ',' -)`.
+
+## Schema diffing
+
+- `schema-version-diff` (CI): read `TERRAFORM_PROVIDER_VERSION` and `config/schema.json`
+  from `${GITHUB_BASE_REF}` via `git cat-file -p`, then run
+  `./scripts/version_diff.py config/generated.lst <old> config/schema.json`.
+- `schema-diff OLD_PROVIDER_VERSION=x.y.z` (manual): download the old provider
+  binary for the host platform, dump its schema with terraform, diff against the
+  current one.
+- `crddiff` (CI, optional but recommended): `uptest crddiff revision` against the base
+  branch CRDs to catch breaking CRD changes; allow overriding with
+  `CRDDIFF_ALLOW_BREAKING=true` for intentional major bumps.
+
+## Misc targets
+
+`submodules`, `go.cachedir` (cache key for GitHub Actions), `cobertura` (coverage
+report excluding `zz_*`), `run` (out-of-cluster provider), `test.race` for
+concurrency-sensitive packages.

.github/skills/upjet-provider/references/checklist.md

@@ -0,0 +1,65 @@
+# Completeness checklist
+
+A provider is "perfect" (provider-keycloak-grade) when every box below is ticked.
+
+## Scaffold
+
+- [ ] `build/` submodule wired; `make submodules` works from a clean clone
+- [ ] `Makefile` with pinned `TERRAFORM_VERSION` (< 1.6) and a `check-terraform-version` guard
+- [ ] `TERRAFORM_PROVIDER_*` variables set, with a `# renovate:` comment on the version
+- [ ] `PLATFORMS` covers every platform you publish (`linux_amd64 linux_arm64`)
+- [ ] `Dockerfile` on digest-pinned `gcr.io/distroless/static`, `USER 65532`, no terraform binary
+- [ ] `package/crossplane.yaml` with `capabilities: [safe-start]`
+- [ ] `.golangci.yml`, `CODEOWNERS`, `.github/renovate.json`, `LICENSE`
+
+## Configuration
+
+- [ ] `config/schema.json` and `config/provider-metadata.yaml` committed and embedded
+- [ ] `config/external_name.go` covers every resource you intend to expose
+- [ ] Include lists match the plugin type (SDKv2 vs framework)
+- [ ] `KnownReferencers()` wires the recurring foreign-key fields
+- [ ] One `config/<group>/config.go` per API group, registered in `GetProvider`
+- [ ] Sensitive fields marked `Sensitive` **before** the first release
+- [ ] `config/generated.lst` generated and checked in CI
+
+## Code
+
+- [ ] `internal/clients/<name>.go` with `TerraformSetupBuilder` + unit tests
+- [ ] Required provider attributes validated with explicit errors; credentials never logged
+- [ ] `cmd/provider/main.go` with metrics, feature gates, `PollJitter`, leader election
+- [ ] `cmd/generator/main.go`, `cmd/generatedlist/main.go` (+ `cmd/crdconversion` if multi-version)
+- [ ] `internal/features`, `internal/version`
+- [ ] `apis/v1beta1` ProviderConfig types (crossplane-runtime v2 shapes)
+
+## Generation
+
+- [ ] `generate/generate.go` with all stages: clean → scrape → generator → controller-gen → angryjet → resolver
+- [ ] `generate.init` fetches schema and docs; `generate.done` refreshes derived lists
+- [ ] `make generate` is idempotent — a second run leaves the tree clean
+
+## Testing
+
+- [ ] Unit tests for all hand-written packages; `-race` where concurrency matters
+- [ ] `examples/` manifest for every exposed resource
+- [ ] `cluster/test/setup.sh` + `cases.txt`; `make e2e` green against a real backend
+- [ ] Coverage gate: no example missing, no orphan example
+- [ ] Conversion tests if CRDs are multi-version
+
+## Automation
+
+- [ ] `ci.yml` with detect-noop, lint, check-diff, unit-tests, build, e2e
+- [ ] `schema-version-diff` and `crddiff` jobs
+- [ ] `auto-release` + `tag` workflows
+- [ ] `schema-diff-issues` workflow + `scripts/schema_diff_issues.py`
+- [ ] `provider-release-check` workflow + `scripts/check_provider_release.py`
+- [ ] `scripts/version_diff.py`
+- [ ] Scheduled workflows dry-run on PRs and are path-filtered
+
+## Documentation
+
+- [ ] `README.md`: install, ProviderConfig, a worked example, dev quickstart
+- [ ] `CONTRIBUTING.md`: how to expose a new resource (edit `external_name.go`,
+      add a group config, `make generate`, add an example, add to `cases.txt`)
+- [ ] Repo-local `SKILL.md` / `AGENTS.md` for AI agents, listing the generated
+      paths that must never be hand-edited
+- [ ] Every deviation from this checklist documented with its reason