docs(connect): migrate connector metadata to regenerated partials#459
Conversation
Apply the metadata-partials workflow from docs-extensions-and-macros (prototype/rpcn-metadata-partials) to the connector reference: - Add 61 regenerated metadata partials under modules/components/partials/metadata/<type>/<name>.adoc, produced from each connector's upstream description by `doc-tools generate rpcn-connector-docs`. - Migrate 55 existing pages to replace their inline `== Metadata` block with an include of the corresponding partial. This makes connector metadata flow from the source description on every generator run instead of being frozen in hand-maintained pages, and fixes the missing oracledb_cdc fields (DOC-2229) as a side effect. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
✅ Deploy Preview for redpanda-connect ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Important Review skippedToo many files! This PR contains 102 files, which is 2 over the limit of 100. To get a review, narrow the scope: Upgrade to a paid plan to raise the limit. ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (102)
You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Feediver1
left a comment
Feediver1
left a comment
There was a problem hiding this comment.
PR Review: migrate connector metadata to regenerated partials (#459)
Files reviewed: 116 files (61 new metadata partials, 55 migrated pages)
Overall assessment: The migration itself is mechanically flawless — every count, banner, include, and path verified. But the PR's declared dependency points at dead PRs: the actual tool is docs-extensions-and-macros#210, still open with an unresolved must-fix, so this PR currently has no shipped generator behind it.
Verification performed (all clean)
- Counts exact: 61 partials + 55 migrated pages = 116 files, matching the merge-base diff (+871/−770) exactly; the PR body's 61 = 55 + 6 orphan arithmetic holds.
- All 55 include lines are byte-uniform (
include::connect:components:partial$metadata/<type>/<name>.adoc[]) and component-qualified — the correct, single-source-safe form (component name verified asconnect), avoiding the unqualified-prefix failure mode that has bitten cross-repo includes before. - Every partial carries the autogeneration banner; all 6 claimed orphans confirmed genuinely orphaned (partial exists, no page includes it).
- DOC-2229 fields verified:
transaction_id,source_ts_ms,commit_ts_msall present in the generatedoracledb_cdcpartial.
Critical issues
- The declared tool dependency is dead. The body says "Depends on #212" and cites #213 — both are CLOSED unmerged in docs-extensions-and-macros. The live tool PR is #210 (open, unreleased). Until #210 merges and ships in a released package version the regen workflow consumes, this PR's core premise — "regenerated on every run" — is false: the partials would be one-time snapshots that silently drift from upstream, and the migration is irreproducible for anyone else. Update the dependency reference and gate merge on the tool being released, not just merged.
- Stale-partial lifecycle (cross-PR): the 55 baked-in includes are exactly the surface exposed by the must-fix flagged on #210 — if a connector's upstream metadata section is ever removed, the current tool leaves the stale partial rendering forever (or, under a delete-based remedy, creates unresolved-include hazards). The empty-file remedy proposed on #210 should land before this pattern becomes load-bearing at 55-page scale.
Jira alignment gap
DOC-2229 asks for two things: the missing metadata fields (✓ addressed) and supported Oracle instance types for the Oracle CDC connector — which appears nowhere in this PR. Either add it, split the ticket, or state explicitly that this PR resolves only the metadata half, so DOC-2229 doesn't get closed prematurely.
Suggestion
The 6 orphan partials (salesforce ×3, redis_pubsub, gateway, command) are precisely the DOC-2229 category — customer-visible metadata missing from published pages — yet they stay unpublished after this PR. Fair that placement needs a human decision, but do it here (six one-line includes) or open a tracked follow-up ticket; a PR-body bullet will be forgotten.
Post-merge checklist
- Bump this repo's package/workflow pin when the tool ships, or scheduled regens keep running the metadata-unaware generator against migrated pages.
- Eyeball the first post-merge cloud-docs sync (the include form is correct for single-sourcing, and the PR's multi-repo build claim covers it, but it's the first at-scale exercise of the pattern).
What works well
- The reproducibility statement (generate → migrate
--write) and the explicit exclusion of unrelateddocs-datachurn are exactly the right hygiene for a bulk generated change. - Honest disclosure of the orphans and the bulk-change review burden in the PR body.
🤖 Generated with Claude Code
Feediver1
left a comment
Feediver1
left a comment
There was a problem hiding this comment.
Note the critical fixes from Claude.
Fix a regression from the metadata-partials migration: on pages where the == Metadata section was the last block, the tool over-consumed and dropped the trailing partial$fields include (and, on some pages, partial$examples) plus the // end::single-source[] closing tag. This removed the entire configuration field reference from 37 pages and left 29 pages with an unclosed single-source tag (breaking cloud-docs single-sourcing). Rebuild each affected page as main's content with only the metadata prose swapped for the metadata partial include, restoring the dropped includes and close tag in their original order. Also wire the 6 orphan metadata partials into their pages where a page exists (salesforce, salesforce_cdc, salesforce_graphql, redis_pubsub, command); gateway has no self-managed page so its partial stays unused.
Apply the metadata field-list normalization (docs-extensions-and-macros #218) to the migrated partials: strip ```text/``` fences around bare field lists and render fields as bullet lists with the field name in inline code. Descriptive bullets ("All headers ...") stay prose, and the === per-operation grouping (nats_kv) and non-field fenced blocks (the csv YAML example) are preserved. 38 of 61 partials updated; the rest were already consistent.
…tials-migration # Conflicts: # modules/components/pages/inputs/oracledb_cdc.adoc
|
Merged What conflicted: #458 landed on Resolution: kept Follow-up to fully migrate oracledb later: the upstream Build after merge: 0 unclosed-tag warnings; oracledb renders the correct |
The auto-docs bot regenerated the partials with the not-yet-released tool, reverting the field-list normalization. Re-apply it (inline-code field names, strip bare-field-list fences; preserve === grouping and YAML examples) and re-drop the orphaned oracledb_cdc partial (the page keeps its verified inline metadata). NOTE: this manual pass is clobbered on every bot regen until docs-extensions-and-macros #218 (normalization) and #217 (block terminator) merge and the workflow's package version is bumped. That release is the durable fix.
…ials The migration replaced hand-authored inline == Metadata sections with includes to generated partials, but 9 connectors document their metadata only in the docs page (their upstream Go description has no == Metadata section), so the generator produced empty partials. The result: the rendered pages lost their metadata entirely. Restore the inline == Metadata sections from main and drop the empty partials for: aws_cloudwatch_logs, aws_dynamodb_cdc, otlp_grpc, otlp_http, postgres_cdc, redpanda_migrator, slack (inputs), cohere_rerank (processor), avro (scanner). These connectors are excluded from the partial migration until their upstream descriptions carry the metadata (same rationale as oracledb_cdc).
2 participants
What
Applies the metadata-partials workflow (tool: redpanda-data/docs-extensions-and-macros#210, merged) to the connector reference:
modules/components/partials/metadata/<type>/<name>.adoc, produced from each connector's upstream description bydoc-tools generate rpcn-connector-docs.== Metadatablock replaced with aninclude::of the matching partial.Fully reproducible:
doc-tools generate rpcn-connector-docsthennode .../migrate-metadata-to-partials.js --write.Why
Connector metadata used to live as prose frozen into hand-maintained pages; the generator never refreshed it. Now it flows from the source description on every run. Also fixes the missing
oracledb_cdcfields flagged in DOC-2229 (transaction_id,source_ts_ms,commit_ts_ms).The migration tool had a bug: on pages where
== Metadatawas the last heading, it over-consumed and deleted the trailingpartial$fieldsinclude (the whole config field reference),partial$exampleswhere present, and the// end::single-source[]close tag — 37 pages lost fields, 29 lost the close tag, 7 lost examples.maincontent with only the metadata prose swapped for the include, restoring the dropped includes/tags in order. Local build now reports 0 unclosed-tag warnings.Orphan partials
Wired the metadata include into the 5 orphan pages that exist:
inputs/salesforce,inputs/salesforce_cdc,inputs/salesforce_graphql,inputs/redis_pubsub,processors/command.gatewayhas no self-managed page, so its partial stays unused.DOC-2229 scope
This resolves only the metadata half of DOC-2229. The supported Oracle instance types ask is not addressed here — leave DOC-2229 open (or split it) so it isn't closed prematurely.
Validation
Full multi-repo Antora build renders metadata correctly on self-managed and cloud (single-sourced) pages; 0 unclosed-tag warnings after the repair.
Notes
docs-datadiff-json churn (unrelated platform-transition data the generator also rewrote).Update: metadata formatting normalized
Applied the field-list normalization from docs-extensions-and-macros#218 to all 61 partials (38 changed): fenced bare-name lists (
text /) are now bullet lists with inline-code field names, consistent across connectors.nats_kv's per-operation===grouping and thecsvYAML example are preserved. Local Antora build: 0 unclosed-tag warnings, no rp-connect-docs xref errors.Note: the mis-scoped
=== Output CSV column order(csv) and=== Credentials(gcp_cloud_storage) content will drop out of the metadata partials once the source fixes ship (redpanda-data/benthos#462, redpanda-data/connect#4620) and the docs regenerate; normalization intentionally preserves===blocks.