[INF-8060] Generate remaining resources and add data sources

.github/workflows/check.yml

@@ -41,6 +41,18 @@ jobs:
       - name: Vet
         run: go vet ./...
 
+      # The generated schema/model code under internal/provider/codegen is
+      # committed, so it can silently drift from its inputs (the vendored spec and
+      # codegen/ruletypesgen). Regenerating here proves the committed output is
+      # what the inputs actually produce.
+      - name: Generated code is up to date
+        run: |
+          make generate
+          if ! git diff --exit-code; then
+            echo "::error::make generate produced a diff. Run it locally and commit the result."
+            exit 1
+          fi
+
       - name: Build
         run: |
           mkdir bin

AGENTS.md

@@ -1,9 +1,8 @@
 # Working in this repo (for AI agents)
 
 This is the Ably Terraform provider. It manages Ably resources (apps, keys,
-namespaces, queues, integration rules) via the Ably Control API. The provider's
-schema and model code is being moved onto code generation from the Control API
-spec; `CODEGEN_STRATEGY.md` explains the why, the decisions, and the plan.
+namespaces, queues, integration rules) via the Ably Control API. Its schema and
+model code is generated from the Control API's OpenAPI spec.
 
 ## The test loop (run this on every change)
 
@@ -28,31 +27,17 @@ CI also enforces `gofmt` and `go vet`, so keep `gofmt -l .` clean and
 
 ## Code generation
 
-Schema and model code is generated. Regenerate with:
-
-```sh
-make generate
-```
-
-Generated code lives under `internal/provider/codegen/` and **is committed; do
-not hand-edit it**. Change the inputs and regenerate:
-
-- Simple resources (app, namespace, queue) are generated from the vendored
-  OpenAPI spec `codegen/control-api.yaml` (sourced from the `ably/docs` repo).
-- Integration-rule families are generated from the in-repo `control` rule types
-  via `codegen/ruletypesgen`, with descriptions and metadata sourced from the
-  spec and an overrides table.
-
-Generation produces schema + model only. **CRUD wiring to the control client is
-always hand-written.**
-
-Step-by-step runbooks are in `DEVELOPMENT.md`:
-
-- "Adding a new integration rule"
-- "Porting a resource onto generated code" (reference example:
-  `internal/provider/resource_ably_rule_bodyguard.go`)
-
-and the pipeline details are in `codegen/README.md`.
+Schema and model code for resources and data sources is generated with `make
+generate`, and the output is committed under `internal/provider/codegen/`. **Do
+not hand-edit it**: change the inputs in `codegen/` and regenerate. Generation
+produces schema + model only, so **CRUD wiring to the control client is always
+hand-written**. The pipeline is described in `codegen/README.md`, and
+`DEVELOPMENT.md` has the runbooks for adding a rule or a data source and for
+porting a resource onto generated code.
+
+`internal/provider/spec_coverage_test.go` fails when the Control API spec carries
+rule types or operations the provider hasn't accounted for, so new API surface has
+to be a decision someone writes down rather than something we miss.
 
 ## The account exporter
 

DEVELOPMENT.md

@@ -66,12 +66,41 @@ stays hand-written.
 4. Write the resource shim in `internal/provider/` (see "Porting" below for the
    pattern): a `Schema()` that adopts the generated schema, the CRUD methods
    delegating to the `control` client, and `Metadata`/`ImportState`.
+   For a moderation or before-publish rule, take the shared plumbing in
+   `internal/provider/before_publish_rules.go` instead of writing CRUD by hand:
+   supply the model plus a create-body function and a response-mapping function,
+   and delegate the four CRUD methods to `beforePublishCRUD`. Reference example:
+   `resource_ably_rule_tisane.go`. Do **not** copy a webhook rule for these
+   families; the generic `AblyRule` plumbing bakes in `source` and
+   `request_mode`, which these rules do not have and the API rejects.
 5. Register the resource in `internal/provider/provider.go`.
 6. Add an example under `examples/resources/`, a template under
    `templates/resources/`, and run `tfplugindocs` to generate the doc.
 7. Add an acceptance test and a unit test for any preserve-from-plan / write-only
    handling. Run `make test`.
 
+## Adding a data source
+
+The Control API has no fetch-by-ID endpoint for apps, keys, namespaces or queues,
+only list endpoints, so every entity has a plural data source generated from its
+list response and a singular one that lists and filters locally.
+
+1. Add the read path to the `data_sources` block in `codegen/generator_config.yml`
+   and run `make generate`. That produces
+   `internal/provider/codegen/datasource_<name>/`.
+2. Write the data source in `internal/provider/data_source_ably_<entity>.go`:
+   a model mirroring the generated element attributes, the plural data source
+   adopting the generated schema, and the singular one built with
+   `elementAttributes` (see `data_sources.go`) so both serve the same generated
+   attribute set. Use `findOne` for the lookup: it enforces exactly one of
+   id/name and refuses to guess when a name matches more than one record.
+3. Register both in `DataSources()` in `internal/provider/provider.go`.
+4. Name them after the existing resource, not the spec. The key data sources are
+   `ably_api_key`/`ably_api_keys` because the resource is `ably_api_key`, and the
+   key permissions map is `capabilities` for the same reason.
+5. Add an example under `examples/data-sources/`, a template under
+   `templates/data-sources/`, run `tfplugindocs`, and add an acceptance test.
+
 ## Porting a resource onto generated code
 
 The reference example is `ably_rule_bodyguard`

Makefile

@@ -66,15 +66,17 @@ generate:
 	go run ./codegen/ruletypesgen
 	go run github.com/hashicorp/terraform-plugin-codegen-framework/cmd/tfplugingen-framework@v0.4.1 generate resources --input codegen/spec.json --output internal/provider/codegen
 	go run github.com/hashicorp/terraform-plugin-codegen-framework/cmd/tfplugingen-framework@v0.4.1 generate resources --input codegen/rules_spec.json --output internal/provider/codegen
+	# Data sources (Track A only): the list endpoints, plus /me.
+	go run github.com/hashicorp/terraform-plugin-codegen-framework/cmd/tfplugingen-framework@v0.4.1 generate data-sources --input codegen/spec.json --output internal/provider/codegen
 	gofmt -w internal/provider/codegen
 
-# Refresh the vendored Control API spec from the public ably/docs repo and
+# Refresh the vendored Control API spec from the published copy on ably.com and
 # re-apply any local fixes (codegen/spec-fixes.patch, if present), then
 # regenerate. Never copy the upstream spec over codegen/control-api.yaml by
 # hand: that silently reverts the fixes and the generators skip the affected
 # attributes without erroring. Pass SPEC_SRC=<path> to use a local ably/docs
-# checkout instead of fetching from GitHub.
-SPEC_URL=https://raw.githubusercontent.com/ably/docs/main/static/open-specs/control-v1.yaml
+# checkout instead of fetching over the network.
+SPEC_URL=https://ably.com/docs/open-specs/control-v1.yaml
 refresh-spec:
 ifdef SPEC_SRC
 	cp $(SPEC_SRC) codegen/control-api.yaml

codegen/README.md

@@ -1,22 +1,22 @@
 # Code generation
 
 This directory holds the inputs for generating Terraform schema and model code
-from the Ably Control API's OpenAPI spec, the first step of the strategy in
-[`../CODEGEN_STRATEGY.md`](../CODEGEN_STRATEGY.md).
+from the Ably Control API's OpenAPI spec.
 
 ## What's here
 
-- `control-api.yaml` — a vendored snapshot of the Control API OpenAPI spec. We
-  source it from the `ably/docs` repo (`static/open-specs/control-v1.yaml`),
-  which is the published, description-rich version (~1,160 field descriptions
-  versus ~150 in the `ably/website` rswag output). Generating from it gives the
-  generated schemas correct attribute documentation. We vendor a copy so
-  generation is self-contained and runnable in CI without checking out that
-  repo. Refresh it with `make refresh-spec`, which fetches the latest spec
-  from the public `ably/docs` repo (or copies from a local checkout with
-  `SPEC_SRC=<path>`) and re-applies any local fixes. Never copy the upstream
-  file over this one by hand: that silently reverts the fixes, and the
-  generators skip the affected attributes without erroring.
+- `control-api.yaml` — a vendored snapshot of the Control API OpenAPI spec,
+  taken from the copy the docs site publishes at
+  <https://ably.com/docs/open-specs/control-v1.yaml>. That is the same file as
+  `static/open-specs/control-v1.yaml` in `ably/docs`, as served, so we track what
+  is published rather than what is merged. It is the description-rich version
+  (~1,160 field descriptions versus ~150 in the `ably/website` rswag output),
+  which is what gives the generated schemas usable attribute documentation. We
+  vendor a copy so generation is self-contained and runnable in CI. Refresh it
+  with `make refresh-spec`, which fetches the published spec (or copies from a
+  local checkout with `SPEC_SRC=<path>`) and re-applies any local fixes. Never
+  copy the upstream file over this one by hand: that silently reverts the fixes,
+  and the generators skip the affected attributes without erroring.
 - `spec-fixes.patch` — our local fixes to the vendored spec, re-applied by
   `make refresh-spec` when the file exists. There are none at present (the
   last one, `conflationEnabled` missing `type: boolean` in the namespace
@@ -63,24 +63,41 @@ assert that regeneration produces no diff.
 
 This is deliberately limited right now:
 
-- **Two tracks.** Simple resources (`app`, `namespace`, `queue`) generate from
-  the OpenAPI spec. The integration rules use an OpenAPI `oneOf` + discriminator
-  that `tfplugingen-openapi` cannot handle, so the moderation and before-publish
-  rule families are generated from the in-repo `control` types instead via
-  `ruletypesgen` (see the strategy doc). The webhook/firehose rule families are
-  not generated yet.
+- **Two tracks.** Simple resources (`app`, `namespace`, `queue`) and the data
+  sources generate from the OpenAPI spec. The integration rules use an OpenAPI
+  `oneOf` + discriminator that `tfplugingen-openapi` cannot handle, so the
+  moderation and before-publish rule families are generated from the in-repo
+  `control` types instead via `ruletypesgen`. The webhook/firehose rule families
+  are not generated yet.
 - **Schema + model only.** The tools do not emit CRUD wiring. All wiring to the
   `control` client stays hand-written and is not generated here.
 - **Both tools are tech preview.** `tfplugingen-openapi` last shipped v0.3.0
   (Jan 2024). It works on our spec today; we are not betting anything load
   bearing on a future release.
-- **The generated code is wired into one live resource so far.**
-  `ably_rule_bodyguard` is ported onto it; the rest of the generated packages
-  are committed as the reviewable output of the pipeline. Retrofitting the
-  remaining resources is a separate, deliberate step, partly because some
-  diverge from the spec shape on purpose (e.g. `queue` flattens the API's
-  nested `amqp`/`stomp` objects into flat attributes). See the Phase 1 findings
-  in the strategy doc.
+
+## Don't port app, namespace or queue onto the generated schemas yet
+
+Every rule resource and every data source is on generated schema. The three
+simple resources are not, and the generated versions of them are not
+contract-complete: adopting one as-is is a breaking change. What each would need
+reconciling first (all metadata the spec cannot express):
+
+- **namespace**: no `identified` attribute, and a bare `authenticated` bool
+  defaulting to `false`. The hand-written schema carries the whole INF-7589
+  migration (canonical `identified`, deprecated `authenticated` alias, alias plan
+  modifiers, `ConflictsWith`, deprecation message) that a port must preserve.
+  `batching_interval` also gains a spurious default of 20, where the hand-written
+  schema has null plus an `AtLeast(0)` validator, and `id`/`app_id` lose
+  `RequiresReplace`.
+- **queue**: every `RequiresReplace` is lost (`app_id`, `name`, `ttl`,
+  `max_length`, `region`), and the Control API has no queue-update endpoint, so a
+  ported resource would plan in-place updates it cannot execute. The resource also
+  flattens the API's nested `amqp`/`stomp` objects into flat attributes on
+  purpose, which the generator faithfully un-flattens.
+- **app**: generated `created`/`modified` are Int64 where the hand-written schema
+  has RFC3339 strings, which breaks decoding of every existing state file.
+  `UseStateForUnknown` on `id`/`account_id`/`created` and the
+  `status`/`tls_only`/`apns_use_sandbox_endpoint` defaults are all dropped.
 
 ## Known per-resource quirks (encoded in `generator_config.yml`)
 

codegen/control-api.yaml

@@ -8,7 +8,7 @@ info:
     </br></br>
     Detailed information on using this API can be found in the Ably <a href="https://ably.com/docs/platform/account/control-api">Control API docs</a>.
     </br></br>
-    The <a href="https://swagger.io/specification/">OpenAPI specification</a> file used to generate this REST API is available to view <a href="https://github.com/ably/docs/blob/main/static/open-specs/control-v1.yaml">here</a>.
+    The <a href="https://swagger.io/specification/">OpenAPI specification</a> file used to generate this REST API is available to view <a href="https://ably.com/docs/open-specs/control-v1.yaml">here</a>.
 servers:
 - url: https://control.ably.net/v1
 paths:
@@ -6689,8 +6689,10 @@ TOfReTlUQzgpXRW5h3n2LVXbXQhPGcVitb88Cm2R8cxQwgB1VncM8yvmKhREo2tz
       properties:
         name:
           type: string
-          description: A friendly name for your queue.
-          example: My queue
+          maxLength: 64
+          pattern: "^[A-Za-z0-9_-]+$"
+          description: A friendly name for your queue. May only contain letters, numbers, hyphens and underscores, up to 64 characters.
+          example: my-queue
         ttl:
           type: integer
           description: TTL in minutes.
@@ -6715,15 +6717,15 @@ TOfReTlUQzgpXRW5h3n2LVXbXQhPGcVitb88Cm2R8cxQwgB1VncM8yvmKhREo2tz
         id:
           type: string
           description: The ID of the Ably queue
-          example: 28AB6w:us-east-1-a:My queue
+          example: 28AB6w:us-east-1-a:my-queue
         appId:
           type: string
           description: The Ably application ID.
           example: 28AB6w
         name:
           type: string
           description: The friendly name of the queue.
-          example: My queue
+          example: my-queue
         region:
           type: string
           description: The data center region for the queue.
@@ -6739,7 +6741,7 @@ TOfReTlUQzgpXRW5h3n2LVXbXQhPGcVitb88Cm2R8cxQwgB1VncM8yvmKhREo2tz
             queueName:
               type: string
               description: Name of the Ably queue.
-              example: 28AB6w:My queue
+              example: 28AB6w:my-queue
         stomp:
           type: object
           additionalProperties: false
@@ -6755,7 +6757,7 @@ TOfReTlUQzgpXRW5h3n2LVXbXQhPGcVitb88Cm2R8cxQwgB1VncM8yvmKhREo2tz
             destination:
               type: string
               description: Destination queue.
-              example: /amqp/queue/28AB6w:My queue
+              example: /amqp/queue/28AB6w:my-queue
         state:
           type: string
           description: The current state of the queue.

codegen/generator_config.yml

@@ -1,6 +1,34 @@
 provider:
   name: ably
 
+# Data sources come from the list endpoints, because the Control API has no
+# GET-by-ID for apps, keys, namespaces or queues (only rules have one). So each
+# entity generates a plural data source: the parent path parameter as an input,
+# and a computed set of whole objects. The singular by-id-or-name data sources
+# lift their attributes from these generated sets at runtime, so there is one
+# source of truth per entity; see internal/provider/data_sources.go.
+data_sources:
+  apps:
+    read:
+      path: /accounts/{account_id}/apps
+      method: GET
+  keys:
+    read:
+      path: /apps/{app_id}/keys
+      method: GET
+  namespaces:
+    read:
+      path: /apps/{app_id}/namespaces
+      method: GET
+  queues:
+    read:
+      path: /apps/{app_id}/queues
+      method: GET
+  me:
+    read:
+      path: /me
+      method: GET
+
 resources:
   app:
     create: