Blog post: "Building Pulumi HCL: Pair programming with an Oracle"

[iwahbe]

Proposed changes

An engineering blog post about how we test Pulumi HCL's support. The full HCL blog post was getting unwealdy and I thought this section could stand on it's own. All feedback is appreciated.

Unreleased product version (optional)

Related issues (optional)

[github-actions]

Social Media Review

content/blog/pulumi-hcl-building-against-an-oracle/index.md

X — missing

No copy provided (social.twitter blank in frontmatter). Drafted below.

LinkedIn — missing

No copy provided (social.linkedin blank in frontmatter). Drafted below.

Bluesky — missing

No copy provided (social.bluesky blank in frontmatter). Drafted below.

---

Suggested copy

X (228/255 chars):

Ian Wahbe needed to test Pulumi's new HCL support without deciding by hand what's correct for every case. His fix: make OpenTofu the oracle, then turn 10 Claude sub-agents loose on finding bugs.

Here's the exact prompt he used.

LinkedIn (876/2950 chars):

Building HCL support for Pulumi (in preview now) raised a specific testing problem: how do you know generated code is behaving correctly without a human checking every case by hand?

For OpenTofu compatibility, we found an answer. Take the same .tf file, run tofu apply and pulumi up against the same in-memory providers, and compare what each one actually did. Terraform's own behavior becomes the oracle — no human judgment call required.

That distinction mattered once we started using AI to hunt for bugs. Because the tests are correct by construction, we could point Claude at the test suite without worrying about false positives. We ran 10 sub-agents at once, each one required to prove a genuine failing test before opening a draft PR.

Ian Wahbe shared the exact prompt he used, along with where he thinks AI helps in test-writing and where it can quietly go wrong.

Bluesky (275/300 chars):

Ian Wahbe is building HCL support for Pulumi. Testing it meant knowing what "correct" looks like for every program, without a human checking each case by hand.

His answer: make OpenTofu the oracle, then turn 10 Claude sub-agents loose on it. Here's the exact prompt he used.

To apply these suggestions, comment: @claude please update the social posts in the frontmatter with the suggested copy from the social media review above

To re-run the social media review after updates, comment: /social-review

Updated for commit 34adb3602ef9cb7d4292788024e2cf7cd65f8703 (short: 34adb36) at 2026-07-22 10:53 UTC.

[github-actions]

Pre-merge Review — Last updated 2026-07-22T10:49:20Z

Tip

Summary: This PR adds a new engineering blog post by Ian Wahbe on how Pulumi HCL is tested — using Terraform/OpenTofu itself as an oracle for compatibility (tfcompat) alongside Pulumi's language-agnostic conformance tests — in the mold of the site's other deep engineering posts. The core technical claims (the conformance-test structure, the l2-resource-simple assertions, the tfcompat harness, and the l1_sum example) were checked against the pulumi-labs/pulumi-hcl and pulumi/pulumi source and hold up. The blocker is completeness, not accuracy: the post currently reads as an unfinished draft — a leftover author <!-- TODO --> comment and roughly sixteen literal (link) placeholders render as visible text, and the intro's promise to "reframe each test category around the 3 bits we care about" is never delivered. Verification ran a fact-check against Pulumi source, a code-examples pass over the Go/HCL/PCL snippets, a frontmatter sweep, and prose/style linting.

Review confidence:

Dimension	Level	Notes
mechanics	HIGH	Build clean (content-only), frontmatter valid, real hyperlinks resolve. The (link) placeholders are prose text, not broken hyperlinks — see coherence.
facts	HIGH	Core claims verified against pulumi-labs/pulumi-hcl and pulumi/pulumi source; two soft/self-referential claims are unverifiable, none contradicted.
coherence	MEDIUM	Post reads as an unfinished draft: a leftover <!-- TODO --> comment, ~16 (link) placeholders, and the promised category reframing not delivered.
code correctness	HIGH	Go/HCL/PCL snippets match the referenced source.

Investigation log

• Cross-sibling reads: not run (not in a templated section)
• External claim verification: 13 of 21 claims verified (2 unverifiable, 0 contradicted) · 4 specialists (numerical, cross-reference, capability, framing); 0 cross-specialist corroborations · routed: 0 inline, 13 Pass 1, 2 Pass 2 (verified 2, contradicted 0, unverifiable 0), 6 Pass 3 (verified 1, contradicted 0, unverifiable 5).
• Cited-claim spot-checks: 2 of 2 cited claims fetched and compared
• Frontmatter sweep: ran on body + meta_desc + social.{bluesky, linkedin, twitter}
• Temporal-trigger sweep: ran (recency words present in diff; spot-check in-review)
• Code execution: not run (no static/programs/ change)
• Code-examples checks: ran (3 specialists: structural, existence, body-code-coverage); 0 findings
• Editorial-balance pass: ran (single-subject, N/A)

🚨 Outstanding	⚠️ Low-confidence	💡 Pre-existing	✅ Resolved
4	12	0	0

🔍 Verification trail

26 claims extracted · 13 verified · 2 unverifiable · 0 contradicted

• L3 in content/blog/pulumi-hcl-building-against-an-oracle/index.md "date: 2026-07-22" → ➖ not-a-claim (evidence: This is a frontmatter date field for a blog post, not a falsifiable factual assertion—it simply records the publication date, which coincides with the current date.; source: content/blog/pulumi-hcl-building-against-an-oracle/index.md)
• L23 in content/blog/pulumi-hcl-building-against-an-oracle/index.md "Terraform modules consumed from Pulumi should do exactly the same thing as Terraform modules consumed from Terraform." → ➖ not-a-claim (evidence: This is the blog author's stated design goal for their own project (Pulumi HCL), not a third-party-attributed factual assertion. It describes the intended compatibility semantics of the pulumi-hcl project, consistent with the repo's own fr…; source: repo:pulumi-labs/pulumi-hcl pkg/hcl/run/run.go ("Terraform modules map to Pulumi component resources..."))
• L23 in content/blog/pulumi-hcl-building-against-an-oracle/index.md "Pulumi HCL support is in preview and available to try at github.com/pulumi-labs/pulumi-hcl." → ✅ verified (evidence: Cited URL https://github.com/pulumi-labs/pulumi-hcl returns HTTP 200 and resolves to the pulumi-labs/pulumi-hcl repository, confirming the repo exists at the stated location; the body was truncated so preview-status wording couldn't be dir…; source: https://github.com/pulumi-labs/pulumi-hcl)
• L27 in content/blog/pulumi-hcl-building-against-an-oracle/index.md "A good test corpus has three properties: it is comprehensive, targeted, and correct." → ➖ not-a-claim (evidence: This is a subjective positioning/opinion statement by the blog author about what makes a good test corpus, not a falsifiable factual claim checkable against an external source.; source: content/blog/pulumi-hcl-building-against-an-oracle/index.md)
• L35 in content/blog/pulumi-hcl-building-against-an-oracle/index.md "Anthropic built a C compiler by testing it against building the Linux kernel as a broad conformance test, as described at anthropic.com/engineering/building-c-…" → ✅ verified (evidence: The Anthropic page confirms: "I tasked 16 agents with writing a Rust-based C compiler, from scratch, capable of compiling the Linux kernel... the agent team produced a 100,000-line compiler that can build Linux 6.9 on x86, ARM, and RISC-V,…; source: https://www.anthropic.com/engineering/building-c-compiler)
• L39 in content/blog/pulumi-hcl-building-against-an-oracle/index.md "Correct means that the test asserts the correct behavior. Incidental behavior should not be asserted on, since we don't want to update tests when something cha…" → ➖ not-a-claim (evidence: This is the blog author's own stated testing philosophy/design rationale (why they avoid snapshot testing and prefer asserting on correct behavior), not a falsifiable third-party factual claim requiring external verification.; source: content/blog/pulumi-hcl-building-against-an-oracle/index.md (author's own design rationale))
• L45 in content/blog/pulumi-hcl-building-against-an-oracle/index.md "Each Pulumi conformance test consists of a PCL program, a list of Pulumi providers, and an assertion on the result of applying Pulumi to that program against t…" → ✅ verified (evidence: The claim is a near-verbatim restatement of the blog post's own text: "Each test is a PCL (Pulumi's internal codegen language) program, list of Pulumi providers, and an assertion on the result of applying Pulumi to that program against tho…; source: repo:content/blog/pulumi-hcl-building-against-an-oracle/index.md (line 45); gh search code repo:pulumi/pulumi "pulumi-resource-test")
• L49-54 in content/blog/pulumi-hcl-building-against-an-oracle/index.md "Pulumi's HCL plugin translates the given PCL program defining a resource with input properties into the shown HCL program that declares a simple_resource res…" → ✅ verified (evidence: pulumi-labs/pulumi-hcl source confirms this exact behavior: pkg/codegen/generate.go contains the comment "To make the Pulumi default explicit, we emit create_before_destroy = true", and numerous testdata files (e.g. cmd/pulumi-language-h…; source: gh search code --owner pulumi-labs "create_before_destroy" (pulumi-labs/pulumi-hcl pkg/codegen/generate.go, testdata))
• L56-74 in content/blog/pulumi-hcl-building-against-an-oracle/index.md "Pulumi's HCL plugin translates a PCL program that defines a resource with an input property into a Terraform/OpenTofu HCL program that declares a required prov…" → ✅ verified (evidence: The blog itself shows the exact PCL-to-HCL example: PCL defines resource res of type simple:index:Resource with value = true, and the generated HCL includes `required_providers { simple = { source = "pulumi/simple", version = "2.0.0"…; source: repo:content/blog/pulumi-hcl-building-against-an-oracle/index.md (L56-74); gh search code --owner pulumi-labs "create_before_destroy" (pulumi-labs/pulumi-hcl:pkg/codegen/generate.go))
• L78-109 in content/blog/pulumi-hcl-building-against-an-oracle/index.md "The l2-resource-simple conformance test asserts that the SDKs map contains a key simple-2.0.0, that the snapshot contains exactly 3 resources, that there i…" → ✅ verified (evidence: Source (pkg/testing/pulumi-test-language/tests/l2_resource_simple.go) shows: _, ok := sdks["simple-2.0.0"]; require.True(l, ok, ...), require.Len(l, snap.Resources, 3, "expected 3 resources in snapshot"), `RequireSingleResource(l, snap…; source: gh api repos/pulumi/pulumi/contents/pkg/testing/pulumi-test-language/tests/l2_resource_simple.go)
• L111 in content/blog/pulumi-hcl-building-against-an-oracle/index.md "The l2-resource-simple test will only pass when the language under test can generate a valid program that defines a resource, requiring the language to be corr…" → ✅ verified (evidence: The l2-resource-simple test in pulumi/pulumi (pkg/testing/pulumi-test-language/tests/l2_resource_simple.go) asserts require.Len(l, snap.Resources, 3, "expected 3 resources in snapshot") and checks for the stack, provider, and `simple:ind…; source: gh api repos/pulumi/pulumi/contents/pkg/testing/pulumi-test-language/tests/l2_resource_simple.go)
• L113 in content/blog/pulumi-hcl-building-against-an-oracle/index.md "A Pulumi language is described as conformant if and only if it passes all language conformance tests." → ➖ not-a-claim (evidence: The blog post itself states: "The goal is simple: A Pulumi language is conferment if and only if it passes all language tests." This is the PR author's own stated definition/design goal for their conformance testing system, not a third-par…; source: repo:content/blog/pulumi-hcl-building-against-an-oracle/index.md (line 113))
• L113 in content/blog/pulumi-hcl-building-against-an-oracle/index.md "The number of Pulumi conformance tests grows week by week." → 🤷 unverifiable (evidence: GitHub search confirms Pulumi actively and continually adds conformance tests across releases (e.g. changelog/v3.151.0.md, v3.159.0.md, v3.238.0.md all add new conformance tests), consistent with ongoing growth, but there is no source that…; source: gh search code --owner pulumi "conformance test")
• L117 in content/blog/pulumi-hcl-building-against-an-oracle/index.md "A tfcompat test uses the same .tf file to define both a Pulumi project and an OpenTofu workspace, then runs tofu apply and pulumi up against a set of recor…" → ✅ verified (evidence: The pulumi-hcl tfcompat README states: "tfcompat verifies that running a Terraform .tf program through the real Pulumi engine + pulumi-language-hcl runtime produces the same observable behavior as running the same program through tofu appl…; source: gh api repos/pulumi-labs/pulumi-hcl/contents/tests/testutil/tfcompat/README.md)
• L119-129 in content/blog/pulumi-hcl-building-against-an-oracle/index.md "The TestL1Sum test exercises the sum built-in intrinsic, requiring both the Terraform/OpenTofu path and the Pulumi HCL path to accumulate with arbitrary-prec…" → ✅ verified (evidence: The actual source file tests/tfcompat/l1_sum_test.go in pulumi-labs/pulumi-hcl contains the exact comment: "TestL1Sum exercises the sum built-in: both paths must accumulate with arbitrary-precision numbers so large integers and decimal f…; source: gh api repos/pulumi-labs/pulumi-hcl/contents/tests/tfcompat/l1_sum_test.go)
• L131 in content/blog/pulumi-hcl-building-against-an-oracle/index.md "A tfcompat.Case{} does not need any extra assertions because the tfcompat test framework automatically asserts on the workspace/stack output matching." → ✅ verified (evidence: The tfcompat harness README states: "The harness asserts equality of two things: 1. Stack outputs — terraform.tfstate outputs vs. Pulumi stack outputs. 2. Provider operations — recordings of every CreateContext, ReadContext..." confirming…; source: gh api repos/pulumi-labs/pulumi-hcl/contents/tests/testutil/tfcompat/README.md)
• L133-145 in content/blog/pulumi-hcl-building-against-an-oracle/index.md "The l1_sum tfcompat test file defines three outputs: big_ints computed as sum([9007199254740993, 1]), decimals computed as sum([0.1, 0.2]), and mixe…" → ✅ verified (evidence: The actual file tests/tfcompat/testdata/cases/l1_sum/main.tf in pulumi-labs/pulumi-hcl decodes to exactly: output "big_ints" { value = sum([9007199254740993, 1]) }, output "decimals" { value = sum([0.1, 0.2]) }, and output "mixed" { v…; source: gh api repos/pulumi-labs/pulumi-hcl/contents/tests/tfcompat/testdata/cases/l1_sum/main.tf)
• L147 in content/blog/pulumi-hcl-building-against-an-oracle/index.md "Each tfcompat test locks what it observes against OpenTofu's behavior, making correctness a built-in property of the test framework." → ✅ verified (evidence: The blog post itself states this exact idea at line 117 and 147: "for all valid OpenTofu programs, tofu apply and pulumi up should do the same thing... Each test locks what it observes against OpenTofu's behavior, effectively making *c…; source: content/blog/pulumi-hcl-building-against-an-oracle/index.md (lines 117, 147); gh search code --owner pulumi-labs "tfcompat")
• L151 in content/blog/pulumi-hcl-building-against-an-oracle/index.md "Because tfcompat tests are correct by construction, LLM assistants can be used to find failing tests without needing to filter out false positives or negatives." → ➖ not-a-claim (evidence: This is the blog post author's own design rationale for their tfcompat testing methodology (a first-party description of why their own system's properties make it suitable for LLM-assisted triage), not a third-party factual assertion check…; source: content/blog/pulumi-hcl-building-against-an-oracle/index.md (author's own pipeline description))
• L153 in content/blog/pulumi-hcl-building-against-an-oracle/index.md "The author's verbatim Claude prompt instructs Claude to find bugs by proving each with a genuine failing tfcompat test, starting with 10 sub-agents, avoiding d…" → 🤷 unverifiable (evidence: This claim describes the exact content of a verbatim quoted prompt embedded within the blog post itself (the author's own Claude prompt), not a third-party assertion checkable against an external source. Web search could not locate the spe…; source: WebSearch ran query "Ian Wahbe Pulumi HCL oracle blog 10 sub-agents draft PR failing test"; top results didn't address the claim; intuition: This is a description of a quote within the PR's own content (self-referential), not a third-party factual claim — like…)
• L159 in content/blog/pulumi-hcl-building-against-an-oracle/index.md "LLMs improve velocity but not quality in software engineering." → ✅ verified (framing: strengthened — the claim states an absolute ("not quality") while sources show a more nuanced mixed/degraded-quality effect rather than zero effect; still broa…; evidence: Multiple studies support this framing, e.g. a Cursor adoption study found "a significant, large, but transient increase in project-level development velocity, along with a significant and persistent increase in static analysis warnings and…; source: https://arxiv.org/html/2511.04427v2 ; https://www.sonarsource.com/blog/the-inevitable-rise-of-poor-code-quality-in-ai-accelerated-codebases/; intuition: Stated as a blanket, unqualified generalization ("velocity but not quality") when actual research shows heterogeneous/m…)
• L0 in content/blog/pulumi-hcl-building-against-an-oracle/index.md "" → 🚩 flagged (readthrough: orphaned-structure)
• L45 in content/blog/pulumi-hcl-building-against-an-oracle/index.md "centrally maintained in pulumi/pulumi (link: pulumi-resource-test folder), and then each (link: go) language (link: dotnet) runs (link PCL) them (link: HCL)" → 🚩 flagged (incomplete-link-placeholders)
• L155 in content/blog/pulumi-hcl-building-against-an-oracle/index.md "This is supported by a (link) couple (link) of (link) skills (link), but this strategy does genuinely (link) find (link) high-quality (link) bugs (link)." → 🚩 flagged (incomplete-link-placeholders)
• L159 in content/blog/pulumi-hcl-building-against-an-oracle/index.md "you approach property based testing (link) with a LLM as both generator (link) & reducer (link)" → 🚩 flagged (incomplete-link-placeholders)
• L23 in content/blog/pulumi-hcl-building-against-an-oracle/index.md "Grammar/spelling: 'any more then', 'what they want the their', 'has too be', 'conferment', 'its'/'it's'" → 🚩 flagged (spelling-grammar)

📊 Editorial balance

Single-subject post; balance check N/A.

🚨 Outstanding in this PR

These must be resolved or refuted before merging.

• [L0] content/blog/pulumi-hcl-building-against-an-oracle/index.md — a leftover authoring note ships in the published source (L41):

  <!-- TODO: Reframe each test catagory around the 3 bits we care about -->

  Remove the comment before merging. It signals an intended restructure — mapping each test approach (conformance tests, tfcompat) onto the three corpus properties (comprehensive, targeted, correct) — that the post never carries out: the Comprehensive/Targeted/Correct section sets up the framing, but the two testing sections that follow only reference it in passing. Either do the reframing the note describes, or drop the note. (Also fix the typo in the intended word, "catagory" → "category," if you keep any of it.)

• [L45] content/blog/pulumi-hcl-building-against-an-oracle/index.md — five link placeholders render as literal text in a key explanatory sentence:

  The set of tests is centrally maintained in pulumi/pulumi (link: pulumi-resource-test folder), and then each (link: go) language (link: dotnet) runs (link PCL) them (link: HCL).

  Replace each (link: …) with the real URL (the pulumi-resource-test folder in pulumi/pulumi, and the per-language runners), or rewrite the sentence without them. As written, readers see "(link: go) language (link: dotnet)" in prose.

• [L155] content/blog/pulumi-hcl-building-against-an-oracle/index.md — eight (link) placeholders render as literal text:

  This is supported by a (link) couple (link) of (link) skills (link), but this strategy does genuinely (link) find (link) high-quality (link) bugs (link).

  Fill in the skill/example links (or remove the placeholders). This sentence also reads awkwardly with the markers interleaved between words — consider rewriting once the links are known. Note the doubled word "find find" earlier in the paragraph (L155: "just ask the LLMs to find find failing tests").

• [L159] content/blog/pulumi-hcl-building-against-an-oracle/index.md — three (link) placeholders render as literal text in the conclusion:

  If you can encode correctness into the test harness itself instead of each test case itself, writing tests becomes easier. Like everything else in software engineering, LLMs improve velocity but not quality. In the worst case, they add useless tests locking in current behavior without concern for correctness. In the best case, you approach property based testing (link) with a LLM as both generator (link) & reducer (link) with the test itself guaranteeing correctness.

  Fill in or remove the placeholders. (Separately, "LLMs improve velocity but not quality" is stated more absolutely than the research supports — current studies show a mixed/degraded-quality effect rather than no quality effect. It's the author's editorial take in a conclusion, so this is a judgment call, but consider softening the blanket phrasing.)

⚠️ Low-confidence

Review each and resolve as appropriate — these don't block the PR.

• [L113] content/blog/pulumi-hcl-building-against-an-oracle/index.md — "we are 167 tests deep, and that number grows week by week." The "grows week by week" cadence is consistent with Pulumi's changelog (new conformance tests land across many releases), but the specific "167" count and the weekly cadence aren't independently verifiable and will drift. Not a blocker. Author question: is 167 the current count as of publication, and is "week by week" accurate — or would "continues to grow" read better and age more gracefully?

• [L23] content/blog/pulumi-hcl-building-against-an-oracle/index.md — a cluster of spelling/grammar errors to correct before publishing:

  • L23: "any more then my penmanship" → "than"
  • L23: "what they want the their [Terraform modules]" → "their"
  • L39: "its very easy to miss a regression" → "it's"
  • L111: "The language has too be wired up correctly" → "to be"
  • L113: "A Pulumi language is conferment if and only if" → "conformant" (this is the post's core definition, so the typo is especially worth catching)
  • L113: "but its still a lot of coverage" → "it's"

Style findings

Found by pattern-based linting; Findings may be false positives.

• line 27: [style] difficulty qualifier — Avoid difficulty qualifier 'easy' -- it judges difficulty for the reader (STYLE-GUIDE.md §Inclusive Language).
• line 31: [style] punctuation — Commas and periods go inside quotation marks.
• line 35: [style] weasel word — 'very' is a weasel word!
• line 39: [style] weasel word — 'very' is a weasel word!
• line 39: [style] difficulty qualifier — Avoid difficulty qualifier 'easy' -- it judges difficulty for the reader (STYLE-GUIDE.md §Inclusive Language).
• line 43: [style] heading capitalization — Heading 'Pulumi language: Conformance testing' should use sentence case (capitalize only the first word and proper nouns).
• line 113: [style] difficulty qualifier — Avoid difficulty qualifier 'simple' -- it judges difficulty for the reader (STYLE-GUIDE.md §Inclusive Language).
• line 113: [style] wordiness — 'it was' is too wordy.
• line 117: [style] filler — Don't start a sentence with 'There is'.
• line 117: [style] wordiness — 'witnessed' is too wordy.

📋 Triaged verifier findings

I double-checked these and realized they weren't real findings — click to expand

• [L153] content/blog/pulumi-hcl-building-against-an-oracle/index.md — "The author's verbatim Claude prompt instructs Claude to find bugs by proving each with a genuine failing tfcompat test, starting with 10 sub-agents…" — recorded as 🤷 unverifiable. Mis-sourced: this isn't an external claim — it describes the author's own prompt, which the post quotes verbatim in the blockquote at L153. The description matches the quoted text, so it's self-verifying against the page; a web search for the prompt was the wrong check.

💡 Pre-existing issues in touched files (optional)

No pre-existing issues in touched files.

✅ Resolved since last review

No items resolved since the last review.

📜 Review history

• 2026-07-22T10:49:20Z — Core technical claims verified against pulumi-labs/pulumi-hcl and pulumi/pulumi source; post ships as an unfinished draft — a leftover TODO comment and ~16 (link) placeholders, plus several spelling/grammar fixes, must be resolved before merge. (34adb36)

---

Need a re-review? Want to dispute a finding? Mention @claude and include #update-review.
(For ad-hoc questions or fixes, just @claude — no hashtag.)