Skip to content

terraform-tidy-before-import: extract from terraform-search-import - #66

Open
bbasata wants to merge 15 commits into
mainfrom
mitosis
Open

terraform-tidy-before-import: extract from terraform-search-import#66
bbasata wants to merge 15 commits into
mainfrom
mitosis

Conversation

@bbasata

@bbasata bbasata commented Apr 17, 2026

Copy link
Copy Markdown
Collaborator

@github-actions

github-actions Bot commented Apr 17, 2026

Copy link
Copy Markdown

Tessl Skill Review Results

Skill Status Review Score Change
terraform/code-generation/skills/terraform-search-import ✅ PASSED 88%
terraform/code-generation/skills/terraform-tidy-before-import ✅ PASSED 88%

Detailed Review

terraform/code-generation/skills/terraform-search-import — 88% (PASSED)
  Description: 92%
    specificity: 2/3 - Names the domain (Terraform, cloud resources) and some actions (discover, search, bulk import), but doesn't list multiple concrete specific actions like generating import blocks, writing configuration files, or handling state operations.
    trigger_term_quality: 3/3 - Includes strong natural trigger terms users would say: 'cloud resources', 'Terraform', 'import', 'unmanaged infrastructure', 'IaC', 'auditing cloud resources', 'migrating'. Good coverage of terms a user would naturally use when needing this skill.
    completeness: 3/3 - Clearly answers both 'what' (discover cloud resources using Terraform Search queries and bulk import them) and 'when' (bringing unmanaged infrastructure under Terraform control, auditing cloud resources, or migrating to IaC) with explicit trigger scenarios.
    distinctiveness_conflict_risk: 3/3 - Highly distinctive with a clear niche: Terraform resource discovery and import specifically. The combination of 'Terraform Search queries' and 'bulk import' creates a very specific trigger profile unlikely to conflict with general Terraform or cloud skills.

    Assessment: This is a well-structured description that clearly communicates both purpose and trigger conditions. The 'Use when...' clause with three distinct scenarios is effective. The main weakness is that the capability description could be more specific about the concrete actions performed (e.g., generating import blocks, writing HCL configuration, managing state).

  Content: 83%
    conciseness: 2/3 - The skill is moderately efficient but includes some redundancy—prerequisites repeat the discovery script instructions already shown in the 'Check Provider Support First' section, and the post-generation cleanup section is quite lengthy. Some sections like 'When to Use' explain obvious use cases Claude could infer.
    actionability: 3/3 - The skill provides fully executable HCL code blocks, concrete bash commands, specific filter syntax examples, and a complete end-to-end workflow example. The post-generation cleanup includes before/after code showing exactly what to remove and how to handle sensitive values with write-only placeholders.
    workflow_clarity: 3/3 - The decision tree clearly sequences the process from checking provider support through choosing a workflow. The complete example at the end shows the full sequence with explicit validation steps (terraform validate, terraform plan before apply). The post-generation cleanup has a clear numbered checklist for the fragile configuration cleanup phase.
    progressive_disclosure: 2/3 - The skill references MANUAL-IMPORT.md and external HashiCorp docs, showing some progressive disclosure. However, the content is quite long (~250 lines) with detailed examples that could be split into separate reference files (e.g., query examples, cleanup guide). No bundle files are provided to verify referenced paths like ./scripts/list_resources.sh or references/MANUAL-IMPORT.md exist.

    Assessment: This is a well-structured, highly actionable skill with concrete executable examples and a clear decision tree workflow. Its main weakness is length—several sections are redundant (discovery script appears twice) and detailed examples could be offloaded to reference files. The post-generation cleanup section is particularly valuable with its specific guidance on sensitive values and write-only attributes.

Suggestions:

  • Add more specific concrete actions to the capability portion, e.g., 'generate import blocks, write HCL configuration, reconcile state' to improve specificity.
  • Remove the duplicate discovery script instructions in the Prerequisites section since they already appear in the 'Check Provider Support First' section.
  • Move the detailed query examples (filtered, multi-region, parameterized) into a separate QUERY-EXAMPLES.md reference file to reduce the main skill's token footprint.
terraform/code-generation/skills/terraform-tidy-before-import — 88% (PASSED)
  Description: 92%
    specificity: 3/3 - Lists multiple specific concrete actions: 'Resolves validation errors, sensitive attributes, and computed attributes' and 'De-duplicates literal values.' These are precise, actionable capabilities rather than vague language.
    trigger_term_quality: 2/3 - Includes good domain terms like 'Terraform', 'import', 'validation errors', 'sensitive attributes', 'computed attributes', and 'version control', but misses common user variations like 'terraform import', 'tf import', 'state import', 'terraform plan errors', or '.tf files'.
    completeness: 3/3 - Clearly answers both what ('Resolves validation errors, sensitive attributes, computed attributes; de-duplicates literal values') and when ('before committing generated Terraform code to version control and before importing resources into Terraform'). The 'Use this...' clause provides explicit trigger guidance.
    distinctiveness_conflict_risk: 3/3 - Highly specific niche: Terraform import preparation. The combination of 'Terraform', 'import', 'validation errors', 'sensitive attributes', and 'computed attributes' creates a very distinct trigger profile unlikely to conflict with general Terraform or IaC skills.

    Assessment: This is a strong skill description that clearly defines a specific niche (Terraform import preparation) with concrete actions and explicit 'when to use' guidance. The main weakness is that trigger term coverage could be broader to capture more natural user phrasings like 'terraform import' or 'tf state'. Overall, it is well-structured and would perform reliably in a multi-skill selection scenario.

  Content: 83%
    conciseness: 2/3 - The content is mostly efficient and avoids explaining basic Terraform concepts, but includes some verbose protective language (e.g., repeated 'Never remove these Terraform language arguments or blocks during cleanup' across multiple paragraphs) and the detailed enumeration of built-in arguments that Claude would already know.
    actionability: 3/3 - The skill provides concrete, executable guidance: specific hq command examples, a clear before/after HCL code example showing computed attribute removal, sensitive attribute handling with write-only attributes, and lifecycle blocks. The steps are specific enough to be directly followed.
    workflow_clarity: 3/3 - The workflow is clearly sequenced with numbered steps, includes explicit validation checkpoints (terraform validate at steps 3 and 9), has a feedback loop for error resolution, and includes a clever optimization pattern of running schema analysis as a background task while doing validation. The rename/restore pattern provides safety.
    progressive_disclosure: 2/3 - The skill references 'references/resource-schema-lookup-tables.md' for the schema lookup table details, which is good progressive disclosure. However, the bundle shows no files were provided, so we cannot verify the reference exists. The main content is reasonably structured but the inline before/after example and the parsing section could potentially be split out for better organization.

    Assessment: This is a well-structured, actionable skill for tidying generated Terraform code. Its greatest strengths are the clear multi-step workflow with validation checkpoints, the concrete before/after example, and the practical guidance on handling sensitive and computed attributes. Minor weaknesses include some redundant protective language about preserving built-in arguments and an unverifiable external reference.

Suggestions:

  • Add more natural trigger term variations such as 'terraform import', 'tf import', 'terraform state', '.tf files', or 'terraform plan errors' to improve discoverability.
  • Consolidate the repeated 'never remove built-in arguments' guidance into a single concise rule rather than restating it for both resource and import blocks separately.
  • Ensure the referenced file 'references/resource-schema-lookup-tables.md' is included in the bundle so the progressive disclosure chain is complete.

Checks: frontmatter validity, required fields, body structure, examples, line count.
Review score is informational — not used for pass/fail gating.

bbasata and others added 2 commits April 17, 2026 17:29
Document that terraform-tidy-before-import removes only computed attributes that are not optional, and preserves computed+optional attributes unless they are explicitly set to null.

Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@bbasata bbasata changed the title terraform-tidy-for-import: extract from terraform-search-import terraform-tidy-before-import: extract from terraform-search-import Apr 22, 2026
@bbasata
bbasata marked this pull request as ready for review April 22, 2026 01:01
@bbasata
bbasata requested a review from a team as a code owner April 22, 2026 01:01
Comment on lines +19 to +20
value = null # sensitive
value_wo = null # sensitive

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

curious: is the # sensitive generated by Terraform?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not. It was likely generated by an agent/model that assisted in writing this. Maybe it's better to remove it.

"evals": [
{
"id": 1,
"prompt": "Tidy the generated Terraform in aws_thorough_before_cleanup.tf so it is ready for import and safe to commit. Use the default thoroughness level, keep valid import blocks intact, preserve Terraform language meta-arguments and nested language blocks, and organize the finished configuration into appropriate Terraform files before you wrap up. Keep terraform.tf in any terraform working directories for base provider configuration. Make a best effort: if the final file does not validate or does not meet all criteria, the computer should still present it as output for evaluation.",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How much instruction normally goes into the prompt vs into the skill?
How does it perform if this was just Tidy the generated Terraform config in aws_thorough_before_cleanup.tf?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right on, super helpful feedback. I think this prompt needs to be lighter. Agent fatigue. Time to edit.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't seem to find a schema for this filetype?
I looked at the docs of tessl, which is used in a Github Action of this repo, and I can only find different files: https://docs.tessl.io/evaluate/evaluating-your-codebase#file-formats

How to run this eval?

@bbasata bbasata Apr 28, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image

Yes, spot-on observation! 😃 I took a speculative leap in this pull request.

For the eval, I chose the format described on agentskills.io. I ran it locally (WORKS ON MY MACHINE 😃) with the skill-creator skill from anthropics/skills.

It would be Really Useful to include some output, would it not? 🙃

"expected_output": "A cleaned AWS configuration as close to aws_thorough_after_cleanup.tf as possible. Expect absence of provider-defined computed attributes and top-level timeout blocks while preserving valid import blocks, lifecycle/meta-arguments, and connection timeouts.",
"files": [
"evals/files/aws_thorough_before_cleanup.tf",
"evals/files/aws_thorough_after_cleanup.tf",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As Anthropic denotes these files as input files, do we need to somehow assert that the agent (while using the skill) does not read this file? Else it might be relatively easy to get to this result 🥷

gautambaghel pushed a commit that referenced this pull request May 11, 2026
)

The DESC_EVAL and CONTENT_EVAL jq programs used `\\(.key)` /
`\\(.value...)` inside a single-quoted format string. In jq, `\\` is an
escaped backslash, so the program contains a literal backslash followed
by `(.key)` — not an interpolation — and the PR comment renders as

    \(.key): \(.value.score)/3 - \(.value.reasoning)

for every criterion (e.g. PR #66, PR #64).

Drop the redundant backslash so jq evaluates the interpolation. The
surrounding `join("\\n")` and `"\\n\\n"` are intentionally kept — those
literal `\n` tokens are consumed downstream by `printf '%b'`.

Co-authored-by: sahildmk <sahildmk@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants