diff --git a/agent-skills/AGENTS.md b/agent-skills/AGENTS.md index 841c3c4a4aa..ee2667abbfd 100644 --- a/agent-skills/AGENTS.md +++ b/agent-skills/AGENTS.md @@ -141,7 +141,7 @@ When a task involves Atmos, activate the matching skill for detailed guidance. | AWS ECR: registry login, ECR auth integrations, Docker credential writes | `atmos-aws-ecr` | `agent-skills/skills/atmos-aws-ecr/SKILL.md` | | AWS compliance: Security Hub standards, compliance reports, CIS AWS, PCI DSS, SOC2, HIPAA, NIST | `atmos-aws-compliance` | `agent-skills/skills/atmos-aws-compliance/SKILL.md` | | AWS security: analyze findings, map to components/stacks, structured remediation | `atmos-aws-security` | `agent-skills/skills/atmos-aws-security/SKILL.md` | -| Migrating to Atmos from native Terraform/OpenTofu, Terraform Workspaces, or Terramate: layout, workspace mapping, remote-state bridge, generate_hcl/script decomposition | `atmos-migration` | `agent-skills/skills/atmos-migration/SKILL.md` | +| Migrating to Atmos from Terraform, Terraform Workspaces, Terramate, or a task runner (Make, Just, or Task): layout, workspace mapping, remote-state bridge, task-to-command mapping, generate_hcl/script decomposition | `atmos-migration` | `agent-skills/skills/atmos-migration/SKILL.md` | | Atmos Modernization: replace deprecated patterns with current Atmos naming, CI, Pro, auth, secrets, and dependencies | `atmos-modernization` | `agent-skills/skills/atmos-modernization/SKILL.md` | ## Common Patterns diff --git a/agent-skills/skills/atmos-introspection/SKILL.md b/agent-skills/skills/atmos-introspection/SKILL.md index b0bfd156ae6..b4c3314d0ef 100644 --- a/agent-skills/skills/atmos-introspection/SKILL.md +++ b/agent-skills/skills/atmos-introspection/SKILL.md @@ -312,12 +312,15 @@ atmos terraform plan --help=usage # ONLY the usage line + embedded usage ex # the fastest way to see real invocation examples atmos terraform plan --help=flags # Command-specific flags only, excludes inherited/global flags atmos terraform plan --help=all # Full reference: everything, including inherited/global flags +atmos wrapper --help=hidden # ONLY wrapper's hidden subcommands (internal: true custom + # commands, or hidden built-ins) -- omitted from every other topic ``` Prefer `--help=usage` over the default `--help` when you just need to confirm how a command is invoked -- it skips straight to worked examples instead of the full flag listing. Reach for `--help=all` only when you need the complete inherited-flag surface (e.g. global `--stack`/`--format`/logging flags shared across -commands). +commands). Reach for `--help=hidden` when a command's default action or steps reference a subcommand name +that never shows up in `--help` -- it is almost certainly an `internal: true` custom command, not a typo. ## Introspection Workflow for AI Agents diff --git a/agent-skills/skills/atmos-migration/SKILL.md b/agent-skills/skills/atmos-migration/SKILL.md index ff01d3ecbcb..c8418ea7a65 100644 --- a/agent-skills/skills/atmos-migration/SKILL.md +++ b/agent-skills/skills/atmos-migration/SKILL.md @@ -1,6 +1,6 @@ --- name: atmos-migration -description: "Migrating to Atmos from existing IaC: techniques, tactics, and design patterns for native Terraform, Terraform Workspaces, and Terramate — minimum-disruption paths, file-layout options, workspace mapping, tag/generate_hcl/script decomposition, and the remote-state bridge for progressive migration" +description: "This skill helps you migrate a repository to Atmos. It covers native Terraform, Terraform Workspaces, Terramate, Makefiles, Justfiles, and Taskfiles. It gives minimum-disruption paths, file-layout options, workspace mapping, task-to-command mapping, generate_hcl/script decomposition, and the remote-state bridge for a step-by-step migration." metadata: copyright: Copyright Cloud Posse, LLC 2026 version: "1.0.0" @@ -10,6 +10,9 @@ references: - references/from-terraform-workspaces.md - references/remote-state-bridge.md - references/from-terramate.md + - references/from-makefile.md + - references/from-justfile.md + - references/from-taskfile.md - references/from-component-updater.md --- @@ -17,31 +20,33 @@ references: ## Overview -This skill is the agent's decision guide for migrating an existing Terraform repository to Atmos. -Atmos is designed to **adopt an existing repo without forcing a reorganization** -- the canonical -`components/terraform/` layout is a recommendation, not a requirement. Lead with the minimum -change that delivers value, then escalate only as the user's needs grow. +This skill is a decision guide. Use it to migrate an existing Terraform repository to Atmos. +Atmos can adopt an existing repository without a reorganization. The `components/terraform/` +layout is a recommendation. It is not a requirement. Start with the smallest change that gives +value. Add more only when the user has a real need for it. -For full prose tutorials aimed at end users, link to: +For full tutorials for end users, see: - [Migrating from Native Terraform](https://atmos.tools/migration/native-terraform) - [Migrating from Terraform Workspaces](https://atmos.tools/migration/terraform-workspaces) -- [Migrating from Terragrunt](https://atmos.tools/migration/terragrunt) (not covered by this skill) +- [Migrating from Terragrunt](https://atmos.tools/migration/terragrunt) (this skill does not cover Terragrunt) - Migrating from Terramate -- covered by this skill via [references/from-terramate.md](references/from-terramate.md) (no atmos.tools tutorial yet) +- [Migrating from Makefiles](https://atmos.tools/migration/makefile) +- [Migrating from Justfiles](https://atmos.tools/migration/justfile) +- [Migrating from Taskfile.yml](https://atmos.tools/migration/taskfile) ## Terraform or OpenTofu -Everything in this skill applies identically to **Terraform** and **OpenTofu**. Atmos invokes -whichever binary is configured (`components.terraform.command` in `atmos.yaml`, defaulting to -`terraform`). Migration paths, file layouts, and the remote-state bridge are the same regardless -of which binary the user is running. Use the user's terminology -- if they say "OpenTofu," use -"OpenTofu" in your responses. +This skill applies the same way to Terraform and to OpenTofu. Atmos runs the binary set in +`components.terraform.command` in `atmos.yaml`. The default binary is `terraform`. The migration +steps, file layouts, and the remote-state bridge do not change based on the binary. Use the same +word the user uses. If the user says "OpenTofu," write "OpenTofu" in your response. ## Core Principles -These principles override default agent instincts. Internalize them before proposing changes to a -user's repo. +These principles come before your normal instincts. Read them before you propose a change to the +user's repository. 1. **Migration is opt-in, not all-or-nothing.** Atmos does not require a filesystem reorganization. Point `base_path` at the user's existing layout (e.g., `base_path: "terraform"` @@ -66,11 +71,19 @@ user's repo. and [atmos-templates](../atmos-templates/SKILL.md) skills for the boundary. 6. **Crawl → walk → run.** Get the user to a working `atmos terraform plan` in 20 minutes; defer inheritance, catalogs, and multi-account hierarchies until they have a concrete need. +7. **Task runners are not a blocker.** Atmos custom commands and workflows can replace the + targets, recipes, and tasks that Make, Just, and Task provide. This doesn't have to happen all + at once — a Makefile, Justfile, or Taskfile can stay as a thin wrapper around `atmos` commands + during migration, the same incremental approach described in Principle 6. The end state turns + each leaf target into a custom command; a target chain usually stays a custom command too, + using `dependencies.commands`/`dependencies.workflows` for its prerequisites. Reserve + workflows for fixed, multi-step orchestration across more than one component — not every + dependency chain needs one. ## Decide the Migration Shape First -Before proposing any change, identify which source pattern the user has. Each routes to a -different reference: +Find the user's source pattern before you propose any change. Each pattern points to a different +reference file: | User has... | Use reference | |----------------------------------------------------------------------|--------------------------------------------------| @@ -79,16 +92,56 @@ different reference: | `terraform.workspace`-driven environments with shared state backend | [from-terraform-workspaces.md](references/from-terraform-workspaces.md) | | `.tm.hcl` files, `stack.tm.hcl`, `generate_hcl` blocks (Terramate project) | [from-terramate.md](references/from-terramate.md) | | Need to read outputs from un-migrated TF (legacy or another repo) | [remote-state-bridge.md](references/remote-state-bridge.md) | +| User has a Makefile driving builds/tests/deploys | [from-makefile.md](references/from-makefile.md) | +| User has a Justfile (`just` command runner) | [from-justfile.md](references/from-justfile.md) | +| User has a Taskfile.yml (go-task) | [from-taskfile.md](references/from-taskfile.md) | | `cloudposse/github-action-atmos-component-updater` | [from-component-updater.md](references/from-component-updater.md) | -The remote-state-bridge pattern is what makes **progressive, component-by-component migration** -possible. Without it, a team is forced into a big-bang cutover. Cover it any time the user has -existing Terraform state they need to read from new Atmos components. +The remote-state-bridge pattern makes progressive migration possible. It lets a team migrate one +component at a time. Without it, the team must migrate everything at once. Use this pattern when +the user has existing Terraform state that a new Atmos component must read. + +### Common Problems in Task-Runner Migration + +These behaviors apply to every task runner. Check them before you open a reference file: + +- **The default order can change, and it differs by source tool.** Task runs `deps:` at the same + time by default, so command-level `dependencies.commands`/`dependencies.workflows` -- also + concurrent by default -- is its direct match. Make and Just run dependencies one after another + by default; `make -j` is required for concurrency. Do not describe `dependencies.commands` as + matching Make's/Just's *default* -- it changes the order, and can introduce a race between + prerequisites that were only ever sequential by accident, not by a declared dependency. For an + ordinary Make/Just chain, ordered steps preserve the default; reach for `dependencies.commands` + there only when the source used `-j`, the prerequisites are genuinely independent, or a + prerequisite is shared by more than one caller (it dedups a shared dependency to a single run + regardless of concurrency -- true for every one of these tools). Check the source tool's real + default before you move it. +- **Freshness checks map to `inputs`/`artifacts`, not to plain steps -- and the scope is per + step.** Task's `sources:`/`generates:` fields and non-`.PHONY` Make targets both skip the + *entire* recipe/task when a file has not changed. Atmos's step-level `inputs.sources`/ + `artifacts.paths` fields are the direct match: with no explicit `when:`, declaring them + implicitly means `when: checksum.changed`, and *that one step* is skipped when nothing has + changed since its last successful run -- later steps in the same command still run regardless. + If the source recipe/task runs more than one command and the freshness decision must gate all + of them together, combine them into a single `shell`/`script` step rather than spreading + `inputs`/`artifacts` across several steps. This does not carry over on its own -- add + `inputs`/`artifacts` to the migrated step yourself. The `require`/`assert` step type does not + replace this. It only checks that a file exists, not whether it is fresh. +- **`workflows.base_path` needs to be set explicitly once the user has their own `atmos.yaml`.** + Only fixed, multi-step orchestration across more than one component becomes an Atmos workflow + (Principle 7) -- most target chains stay a custom command with `dependencies.commands` instead. + `atmos workflow ` fails with + `'workflows.base_path' must be configured in 'atmos.yaml'` until you add it (for example, + `workflows.base_path: "stacks/workflows"`). None of this skill's `atmos.yaml` snippets show it + by default -- add it the moment the user's migration reaches its first workflow. + +Each reference file has its own "Common Problems" section with the exact field names and steps +for that tool. This section is only a short summary. ## The Minimum-Viable Migration -When a user says "I want to try Atmos on my existing repo," this is the checklist. Do not deviate -unless the user's setup requires it. +Use this checklist when the user wants to try Atmos on an existing repository. Do not change the +order unless the user's setup requires it. 1. **Install Atmos.** See `atmos.tools/install`. 2. **Create `atmos.yaml`** at the repo root, pointing `base_path` and `components.terraform.base_path` @@ -107,13 +160,13 @@ unless the user's setup requires it. 4. **Run `atmos terraform plan vpc -s dev`** and confirm output matches what `terraform plan -var-file=dev.tfvars` produced before. -A working reference for this shape lives at `examples/native-terraform/` in the Atmos repo. +A working example of this shape is at `examples/native-terraform/` in the Atmos repository. ## File-Layout Options -Pick the layout that matches the user's migration goals. `components/terraform/` is the recommended -Atmos convention, especially for new repos or multi-toolchain projects, but existing layouts can be -preserved when the user wants a lower-disruption migration. +Pick the layout that matches the user's goals. Atmos recommends the `components/terraform/` +layout, especially for a new repository or a multi-tool project. You can keep an existing layout +when the user wants less disruption. | `base_path` | Use when | |------------------------------------------|-------------------------------------------------------------------------| @@ -121,13 +174,13 @@ preserved when the user wants a lower-disruption migration. | `base_path: "terraform"` | TF-only repo with code already in `terraform/`; preserve dir name | | `base_path: "."` + `components.terraform.base_path: "components/terraform"` | Multi-toolchain or new repo; canonical Atmos layout | -For deeper organization patterns (multi-region, multi-account, org hierarchies), defer to the -[atmos-design-patterns](../atmos-design-patterns/SKILL.md) skill. +For more organization patterns, such as multi-region, multi-account, and organization +hierarchies, see the skill [atmos-design-patterns](../atmos-design-patterns/SKILL.md). ## YAML Functions vs Gomplate Datasources -This is a recurring footgun -- agents reach for Gomplate datasources when a YAML function would -be safer and clearer. Prefer the right column: +This is a common mistake: an agent chooses a Gomplate datasource when a YAML function is safer +and clearer. Use the option in the right column: | Goal | Reach for (NOT this) | Use instead | |-------------------------------|---------------------------------------------------|------------------------------------------| @@ -138,61 +191,84 @@ be safer and clearer. Prefer the right column: | Read Terraform output | Templated remote-state datasource | `!terraform.state component output` | | Get current AWS account ID | `gomplate.datasources` AWS plugin | `!aws.account_id` | -YAML functions are type-safe, produce clear errors, work without enabling Gomplate, and don't -require keeping templates valid YAML. Reserve Go templates for control flow (conditionals, loops, -dynamic keys) that YAML functions cannot express. See [atmos-templates](../atmos-templates/SKILL.md) -for when Go templates are the right tool. +A YAML function checks its own types. It gives a clear error message. It works without Gomplate +turned on. It does not require the template text to stay valid YAML. Use a Go template only for +control flow, such as a conditional, a loop, or a dynamic key, that a YAML function cannot +express. See [atmos-templates](../atmos-templates/SKILL.md) for when to use a Go template. ## What Does NOT Need to Change -Lead with this when a user fears a big rewrite. None of the following must change to adopt Atmos: +Tell the user this list first, if they are afraid of a large rewrite. None of these items must +change to adopt Atmos: -- **Terraform code** -- providers, resources, data sources, modules all stay as-is. -- **Module sources** -- `source = "../../modules/foo"` or registry sources keep working. -- **Backend code** -- delete the `backend "s3" {}` block from `.tf` files (Atmos generates - `backend.tf.json`), or leave it and disable backend generation in `atmos.yaml`. Either works. -- **`.tfvars` files** -- consumed via `!include`; convert to YAML later if/when the user wants - deep-merge inheritance. -- **Custom provider configuration** -- providers stay in `.tf` files; pass env vars via stack - `env:` or vars via stack `vars:`. +- **Terraform code.** Providers, resources, data sources, and modules stay the same. +- **Module sources.** A local path, such as `source = "../../modules/foo"`, or a registry + source, keeps working. +- **Backend code.** You can delete the `backend "s3" {}` block from the `.tf` files, because + Atmos creates `backend.tf.json`. Or you can keep the block and turn off backend generation in + `atmos.yaml`. Both methods work. +- **`.tfvars` files.** Atmos reads them through `!include`. Convert them to YAML later, only if + the user wants deep-merge inheritance. +- **Custom provider configuration.** Providers stay in the `.tf` files. Pass environment + variables through stack `env:`. Pass Terraform variables through stack `vars:`. ## When to Escalate to Other Skills -After the minimum migration is working, the user will often ask "how do I do X next?" Route -those questions to the right skill: - -- **Organizing many stacks (orgs, tenants, accounts, regions)** → [atmos-design-patterns](../atmos-design-patterns/SKILL.md) -- **Abstract components, inheritance, catalog patterns** → [atmos-components](../atmos-components/SKILL.md) -- **Deep merging, imports, overrides** → [atmos-stacks](../atmos-stacks/SKILL.md) -- **Vendoring third-party components** → [atmos-vendoring](../atmos-vendoring/SKILL.md) -- **Authentication / provider credentials** → [atmos-auth](../atmos-auth/SKILL.md) -- **Validation policies (OPA, JSON Schema)** → [atmos-validation](../atmos-validation/SKILL.md) -- **CI/CD with affected-detection** → [atmos-ci](../atmos-ci/SKILL.md) -- **Cross-component data sharing via stores** → [atmos-stores](../atmos-stores/SKILL.md) +After the minimum migration works, the user will often ask what to do next. Send each question +to the correct skill: + +- **Organize many stacks**, such as by organization, tenant, account, or region. Use + [atmos-design-patterns](../atmos-design-patterns/SKILL.md). +- **Build abstract components, inheritance, or catalog patterns.** Use + [atmos-components](../atmos-components/SKILL.md). +- **Use deep merging, imports, or overrides.** Use [atmos-stacks](../atmos-stacks/SKILL.md). +- **Vendor third-party components.** Use [atmos-vendoring](../atmos-vendoring/SKILL.md). +- **Set up authentication or provider credentials.** Use [atmos-auth](../atmos-auth/SKILL.md). +- **Add validation policies, such as OPA or JSON Schema.** Use + [atmos-validation](../atmos-validation/SKILL.md). +- **Set up CI/CD with affected-component detection.** Use [atmos-ci](../atmos-ci/SKILL.md). +- **Share data between components through a store.** Use + [atmos-stores](../atmos-stores/SKILL.md). ## Anti-Patterns -Things to push back on if a user (or another agent) proposes them during migration: - -- **"You must move all Terraform into `components/terraform/` before using Atmos."** No -- that is - the recommended layout, not a requirement. Let the user choose between adopting the best-practice - layout now or pointing `base_path` at the existing layout and reorganizing later. -- **"You must rewrite all `.tfvars` as YAML before running Atmos."** No -- native stack YAML is the - best-practice destination for inheritance and composition, but `!include` lets users keep - existing `.tfvars` during a progressive migration. -- **"Delete your workspace state and start over."** No -- bridge it with - `metadata.terraform_workspace` and the remote-state-bridge pattern. -- **"Add Gomplate datasources for everything."** No -- reach for YAML functions first. -- **"Adopt the full multi-account org hierarchy on day one."** No -- start with one stack file. +Push back if a user or another agent proposes one of these methods during migration: + +- **"You must move all Terraform into `components/terraform/` before you use Atmos."** This is + false. That layout is a recommendation, not a requirement. Let the user pick: adopt the + recommended layout now, or point `base_path` at the current layout and reorganize later. +- **"You must rewrite all `.tfvars` files as YAML before you run Atmos."** This is false. Native + stack YAML is the best final format for inheritance and composition. But `!include` lets the + user keep existing `.tfvars` files during a step-by-step migration. +- **"Delete your workspace state and start over."** This is false. Connect the existing state + with `metadata.terraform_workspace` and the remote-state-bridge pattern. +- **"Add a Gomplate datasource for everything."** This is false. Use a YAML function first. +- **"Adopt the full multi-account organization hierarchy on day one."** This is false. Start + with one stack file. +- **"Wrap atmos commands in a Makefile, Justfile, or Taskfile forever."** This is false. A + wrapper is a good bridge while the user builds trust in Atmos. But it is not the final state. + Change each leaf target to a custom command. An ordinary Make or Just dependency chain (for + example, `deploy: build test`) stays a custom command with ordered steps or + `dependencies.commands` -- it does not need a workflow. A Taskfile's `deps:` maps the same way, + onto ordered steps or `dependencies.commands` on the custom command. Reserve workflows for + fixed, multi-step orchestration across more than one component, not for an ordinary target + chain. ## Additional Resources -- [References/from-native-terraform.md](references/from-native-terraform.md) -- scenario-keyed - recipes for vanilla TF migration -- [References/from-terraform-workspaces.md](references/from-terraform-workspaces.md) -- mapping - workspaces to stacks without losing state -- [References/remote-state-bridge.md](references/remote-state-bridge.md) -- the dummy-component - and abstract-component patterns for reading state from un-migrated or external Terraform -- [References/from-terramate.md](references/from-terramate.md) -- construct-by-construct mapping +- [References/from-native-terraform.md](references/from-native-terraform.md): steps for a plain + Terraform migration, matched to each shape. +- [References/from-terraform-workspaces.md](references/from-terraform-workspaces.md): how to map + workspaces to stacks without losing state. +- [References/remote-state-bridge.md](references/remote-state-bridge.md): the dummy-component and + abstract-component patterns. Use them to read state from Terraform that is not yet migrated, or + from an external repository. +- [References/from-terramate.md](references/from-terramate.md): construct-by-construct mapping from Terramate (`stack.tm.hcl`, globals, `generate_hcl`, `script{}`, tags/labels) to Atmos, - including the one remaining known gap (`.tmtriggers`) + including the one remaining known gap (`.tmtriggers`). +- [References/from-makefile.md](references/from-makefile.md): steps for a Makefile, matched to + each shape. +- [References/from-justfile.md](references/from-justfile.md): steps for a Justfile, matched to + each shape. +- [References/from-taskfile.md](references/from-taskfile.md): steps for a Taskfile.yml (go-task) + file, matched to each shape. diff --git a/agent-skills/skills/atmos-migration/references/from-justfile.md b/agent-skills/skills/atmos-migration/references/from-justfile.md new file mode 100644 index 00000000000..8c446f10b0e --- /dev/null +++ b/agent-skills/skills/atmos-migration/references/from-justfile.md @@ -0,0 +1,196 @@ +# Migrating from Justfiles + +This guide shows how to move Just recipes to Atmos. Find the correct shape for the Justfile +below. Then follow the matching steps. For the full tutorial, see +[atmos.tools/migration/justfile](https://atmos.tools/migration/justfile). + +Just recipe bodies do not require tab indentation, unlike Make. Just's named parameters with +default values map closely to Atmos custom command `flags:` and `arguments:`. This is the +closest match of the three task runners this skill covers. If the Justfile also selects a +Terraform environment, also use [from-native-terraform.md](from-native-terraform.md) for the +Terraform-specific steps. + +## Find the Shape of the Justfile + +| Shape | Steps | +|-----------------------------------------------------------------------|-----------------------------------| +| Recipes with named parameters and default values | [Shape A](#shape-a-recipes-with-named-parameters) | +| Recipe dependencies (`build: test`) | [Shape B](#shape-b-recipe-dependencies) | +| `set dotenv-load`, `export VAR := ...`, `set shell := [...]` | [Shape C](#shape-c-environment-and-shell-settings) | + +## Shape A: Recipes with Named Parameters + +**Before:** +```just +# Build the deployable artifact +build: + go build -o bin/handler ./cmd/handler + +# Run static analysis +lint: + golangci-lint run ./... + +[private] +_clean: + rm -rf bin/ +``` + +**Steps:** + +1. Turn the `# comment` above a recipe into the command's `description:` field. Atmos shows this + text in `atmos --help` and `atmos --help`. This replaces `just --list`. +2. Turn a recipe's named parameter with a default value, such as `deploy env='dev':`, into a + command `flags:` entry with a matching `default:` value. Inside a step, read the value as + `{{ .Flags.env }}`. Do not use Just's own `{{env}}` syntax. See + [Common Problems](#--interpolation-looks-like-atmos-templates-but-is-not) below. +3. Set `internal: true` on a command created from a `[private]` recipe. It runs normally + (`atmos ...`, as a `default:` target, or from another command's steps) but is excluded + from `atmos --help` listings and completion suggestions. Only inline the recipe's body into a + caller's step when it is genuinely single-caller logic with no reason to be invoked on its own. + +```yaml +commands: + - name: build + description: Build the deployable artifact + steps: + - type: shell + command: go build -o bin/handler ./cmd/handler + + - name: lint + description: Run static analysis + steps: + - type: shell + command: golangci-lint run ./... +``` + +## Shape B: Recipe Dependencies + +**Before:** +```just +# Run tests (builds first) +test: build + go test ./... + +# Deploy to the given environment (defaults to dev) +deploy env='dev': build test + cd terraform && terraform apply -var-file=envs/{{env}}.tfvars +``` + +**Steps:** use the same method as Make's dependency chains. See +[from-makefile.md Shape B](from-makefile.md#shape-b-target-chains-with-dependencies). Use a +`type: atmos` step with `command: build` to call another custom command -- `type: atmos` preserves +step-level stack context and structured output handling, which a `type: shell` step running +`atmos build` does not. + +```yaml +commands: + - name: test + description: Run tests (builds first) + steps: + - type: atmos + command: build + - type: shell + command: go test ./... + + - name: deploy + description: Deploy to the given environment (defaults to dev) + flags: + - name: env + shorthand: e + default: "dev" + steps: + - type: atmos + command: test + - type: atmos + command: terraform apply infra -s {{ .Flags.env }} +``` + +`infra` is a placeholder Atmos component name, not the `terraform` verb repeated. Move the +recipe's Terraform code to `components/terraform/infra/` (the default +`components.terraform.base_path` is `components/terraform`), then swap `infra` for whatever the +user actually names the component. + +## Shape C: Environment and Shell Settings + +**Before:** +```just +set dotenv-load := true +set shell := ["bash", "-uc"] + +export AWS_REGION := "us-east-1" +``` + +**Steps:** + +- Turn `export VAR := value` into a command or step `env:` map. +- `set dotenv-load` maps to `env: !include .env` on the command, workflow, or step. Atmos parses + the dotenv file natively (including `export VAR=value`, comments, quoting, and `${VAR}` + expansion) and merges the result into `env:`. If the values are secrets rather than plain + config, use Atmos's store or secrets integration instead of a plaintext `.env` file. +- `set shell := [...]` changes the shell for every recipe in the Justfile. Atmos has no matching + command-level setting. Use `type: script` with an explicit `interpreter:` field on the one step + that needs a different interpreter. + +```yaml +commands: + - name: build + description: Build the deployable artifact + env: + <<: !include .env + AWS_REGION: us-east-1 + steps: + - type: shell + command: go build -o bin/handler ./cmd/handler +``` + +## Common Problems + +### `{{ }}` interpolation looks like Atmos templates but is not + +Just's `{{ var }}` syntax and Atmos's `{{ .Flags.var }}` syntax both use Go templates, but they +run in different tools at different times. Do not copy Just interpolation syntax into Atmos +YAML. Change each reference to the matching `{{ .Flags. }}` or +`{{ .Arguments. }}` form. + +### `[private]` recipes map to `internal: true` + +The custom command schema has an `internal: true` field. It excludes the command from `atmos --help` +listings and completion suggestions while leaving it fully runnable -- directly, as a `default:` +target, or from another command's steps. This is the direct equivalent of a `[private]` recipe, +and it covers cases plain step-inlining cannot: a helper called from more than one recipe, or one +a user invokes by name for manual debugging. + +Only inline a `[private]` recipe's logic into a caller's step when it is genuinely single-caller +and has no reason to be invoked on its own -- in that case a separate `internal` command is just +unnecessary indirection. + +If a `[private]` recipe is never called by any public recipe (an orphaned helper, not a +dependency), `internal: true` no longer forces the same discovery you'd get from step-inlining -- +it would just as quietly hide dead code as reachable helper code. Confirm with the user whether +the recipe is still needed at all before migrating it; if it is, ask whether it should become a +`internal` command, a step inside whichever command ends up needing it, or a short script the user +maintains separately. + +### Command echo differs between `just` and Atmos + +By default, Just prints each recipe line before running it (`sh -x`-style), so `just build`'s +visible output includes every command line, not just what those commands print. Atmos `type: +shell` steps run silently by default -- only the command's own stdout/stderr shows. The migrated +command's side effects match the original recipe, but the terminal output will look sparser side +by side. Tell the user this if they compare `just ` output to `atmos ` output +directly; it is a visible difference, not a bug. + +### Confirm `set shell` with the user; `dotenv-load` has a direct replacement + +`set dotenv-load` maps directly to `env: !include .env` -- no confirmation needed unless the +`.env` file holds secrets, in which case ask whether to use Atmos's store or secrets integration +instead. `set shell` has no command-level equivalent; ask the user if a non-default shell matters +to their workflow, then apply `type: script` with `interpreter:` to the specific steps that need +it. + +## What Not To Do + +- Do not assume `{{ }}` means the same thing after you move it into Atmos YAML. +- Do not invent a visibility value beyond the documented `internal: true` boolean (no "public"/"private" enum, no partial visibility). +- Do not drop `set shell` behavior without telling the user; `dotenv-load` maps directly to + `env: !include .env`, so it does not need the same case-by-case confirmation. diff --git a/agent-skills/skills/atmos-migration/references/from-makefile.md b/agent-skills/skills/atmos-migration/references/from-makefile.md new file mode 100644 index 00000000000..d21eaf4c70b --- /dev/null +++ b/agent-skills/skills/atmos-migration/references/from-makefile.md @@ -0,0 +1,292 @@ +# Migrating from Makefiles + +This guide shows how to move Make targets to Atmos. Find the correct shape for the Makefile +below. Then follow the matching steps. For the full tutorial, see +[atmos.tools/migration/makefile](https://atmos.tools/migration/makefile). + +This guide covers the `make` orchestration layer only: targets, target dependencies, variables, +and conditionals. If the Makefile also selects a Terraform environment through `-var-file` or +per-environment directories, also use +[from-native-terraform.md](from-native-terraform.md). That guide covers the Terraform-specific +steps: backend generation, `.tfvars` files, and workspace mapping. + +## Find the Shape of the Makefile + +| Shape | Steps | +|-----------------------------------------------------------------|-------------------------------------| +| Independent leaf targets (`build`, `test`, `lint`, `clean`) | [Shape A](#shape-a-independent-leaf-targets) | +| Target chains with dependencies (`deploy: build test`) | [Shape B](#shape-b-target-chains-with-dependencies) | +| Recursive or parallel Make (`$(MAKE) -C dir`, `make -j`) | [Shape C](#shape-c-recursive-or-parallel-make) | + +Most Makefiles mix all three shapes. Treat each target on its own. Then combine the results. + +## Shape A: Independent Leaf Targets + +**Before:** + +```makefile +.PHONY: build test lint clean help + +help: ## Show available targets + @awk 'BEGIN {FS=":.*##"} /^[a-zA-Z_-]+:.*##/ {printf " %-10s %s\n", $$1, $$2}' $(MAKEFILE_LIST) + +build: ## Compile the deployable artifact + go build -o bin/handler ./cmd/handler + +test: build ## Run unit tests + go test ./... + +lint: ## Run static analysis + golangci-lint run ./... + +clean: ## Remove build artifacts + @rm -rf bin/ +``` + + +**Steps:** + +1. Turn each leaf target into a custom command in `atmos.yaml`. If the Makefile is large, put + the commands in a separate file. See + [Split commands across files](#split-commands-across-files) below. +2. Turn the silent-recipe `@` prefix into `show: { command: false }`. Make's `@` only suppresses + the echoed command line -- the recipe's own stdout/stderr still prints. `output: none` is not + the same thing: it sends the step's stdout *and* stderr to the void, which would hide real + command output (for example, `go test` diagnostics). Reserve `output: none` for a step whose + output genuinely needs to be discarded entirely. +3. Delete the `help` target. Atmos generates the same information from each command's + `description:` field. Run `atmos --help` or `atmos --help` to see it. +4. When one target depends on another leaf target, such as `test: build`, add a step that runs + the dependency's command. Use a `type: atmos` step with `command: build` to call the other + custom command -- `type: atmos` preserves step-level stack context and gives you structured + output handling (captured stdout/stderr, exit code) that a `type: shell` step invoking + `atmos build` does not. + +```yaml +commands: + - name: build + description: Compile the deployable artifact + steps: + - type: shell + command: go build -o bin/handler ./cmd/handler + + - name: test + description: Run unit tests + steps: + - type: atmos + command: build + - type: shell + command: go test ./... + + - name: lint + description: Run static analysis + steps: + - type: shell + command: golangci-lint run ./... + + - name: clean + description: Remove build artifacts + steps: + - type: shell + command: rm -rf bin/ + show: + command: false +``` + +## Shape B: Target Chains with Dependencies + +**Before:** + +```makefile +ENV ?= dev + +deploy: build test ## Plan and apply the given ENV (default: dev) + cd terraform && terraform apply -var-file=envs/$(ENV).tfvars +``` + + +**Steps:** + +1. Turn `ENV ?= dev` into a command `flags:` entry with `default: "dev"`. +2. GNU Make's own default is to build a target's prerequisites one at a time, in the order + listed, not concurrently (`-j` is required for that) -- so ordered steps, not + `dependencies.commands`, are the default-preserving match for a plain target list. Reach for + command-level `dependencies.commands: [build, test]` instead only when: a prerequisite is + shared by more than one target (it dedups a shared dependency to a single run, the same + guarantee `make` already gives for free, independent of concurrency), the prerequisites are + genuinely independent, or the source target actually used `-j`. Here, `build` is shared -- + Shape A's `test` already depends on it via its own `atmos build` step -- so + `dependencies.commands` is the right call for `deploy`. Convert Shape A's `test` from its + `atmos build` step to `dependencies.commands: [build]` at the same time, so the scheduler + still orders `build` before `test` even though both now run through the same + concurrent-by-default mechanism, instead of listing `build` and `test` as a flat, unordered + sibling list on `deploy`. +3. Move the Terraform-specific line, `terraform apply -var-file=envs/$(ENV).tfvars`, to + [from-native-terraform.md Shape B](from-native-terraform.md#shape-b-single-dir-with--var-file-from-a-makefile). + That guide shows how the Terraform side maps to stacks. Here, the line becomes a single + `type: atmos` step, because `terraform apply` is a native Atmos verb. +4. Turn `ifeq ($(ENV),prod)` conditionals into a Go template conditional inside a custom command: + `{{ if eq .Flags.env "prod" }}...{{ end }}`. This is the same pattern used for `--verbose` and + other boolean flags. Inside a workflow, use `when: !cel 'stack == "prod"'` on the step + instead. + +```yaml +commands: + - name: test + description: Run unit tests + dependencies: + commands: [build] + steps: + - type: shell + command: go test ./... + + - name: deploy + description: Plan and apply the given environment (default dev) + flags: + - name: env + shorthand: e + default: "dev" + dependencies: + commands: [build, test] + steps: + - type: atmos + command: terraform apply infra -s {{ .Flags.env }} +``` + +`build` still runs exactly once for the whole `atmos deploy` invocation -- `test`'s own edge on +`build` orders it correctly ahead of `test`, and `deploy`'s own steps wait for both to finish. + +`infra` is a placeholder Atmos component name, not the `terraform` verb repeated. Move the +target's Terraform code to `components/terraform/infra/` (the default +`components.terraform.base_path` is `components/terraform`), then swap `infra` for whatever the +user actually names the component. + +## Shape C: Recursive or Parallel Make + +**Before:** + +```makefile +SERVICES := vpc eks rds + +build-all: + for dir in $(SERVICES); do $(MAKE) -C services/$$dir build; done + +build-parallel: + $(MAKE) -j4 build-all +``` + + +**Steps:** + +1. `build-all`'s recipe is a shell `for` loop, not multiple Make targets -- `make -j` only + parallelizes independent targets within a single `make` invocation, it does not parallelize + commands inside one recipe's shell script. So `$(MAKE) -j4 build-all` still runs `vpc`, `eks`, + and `rds` one at a time, in order, exactly like plain `build-all` would. `-j4` here does + nothing. +2. Turn `$(MAKE) -C dir target` recursion over a fixed set of directories into a `matrix` step. + Define a `service` axis, and call the per-service command once for each value. Using + `max_concurrency` on the matrix step is a deliberate upgrade over the sequential source + behavior, not a literal translation of `-j4` -- call it out to whoever is reviewing the + migration, since it changes execution behavior (all services build concurrently instead of + one at a time). +3. The per-directory `build` target here is a different recipe than Shape A's top-level `build` + command -- it lives in each `services//` directory, not at the repo root. Give it its own + custom command, `build-service`, with a `service` flag/argument that maps into the + `services/` path. Do not reuse Shape A's `build` command as-is; it has no way to receive + a per-service value. + +```yaml +commands: + - name: build-service + description: Build a single service + flags: + - name: service + required: true + steps: + - type: shell + command: go build -o bin/{{ .Flags.service }} ./services/{{ .Flags.service }} + + - name: build-all + description: Build every service + steps: + - name: build-services + type: matrix + matrix: + service: [vpc, eks, rds] + max_concurrency: 4 + steps: + - type: atmos + command: build-service --service {{ .matrix.service }} +``` + +## Common Problems + +### Tabs, `.PHONY`, and file-timestamp caching + +Atmos steps have no tab requirement. Do not confuse `.PHONY` with a caching feature. If a +Makefile target is not `.PHONY` and uses file timestamps to skip work when inputs have not +changed, turn it into step `inputs.sources`/`artifacts.paths` -- with no explicit `when:`, that +implicitly means `when: checksum.changed`, and the step is skipped when nothing has changed since +its last successful run. It does not carry over automatically; tell the user to add +`inputs`/`artifacts` to the migrated step themselves. Content hashing (the default) is a +deliberate upgrade over Make's own mtime comparison -- a fresh `git clone`/CI checkout resets +every file's mtime, which makes Make think everything changed even when it didn't; use +`when: timestamp.changed` instead for Make's exact mtime semantics. Task's `sources:`/`generates:` +feature maps to the same fields. See +[from-taskfile.md](from-taskfile.md#sourcesgenerates-becomes-inputsartifacts) for more detail. +The scope is different, too: Make's freshness check gates the target's entire recipe, but +`inputs`/`artifacts` are declared per step -- skipping one step does not stop later steps in the +same command from running. If a target's recipe has more than one command line and the freshness +decision must gate all of them together, combine them into a single `shell`/`script` step rather +than spreading `inputs`/`artifacts` across several steps. + +### Silent recipes and command echo + +`@command` suppresses the echo of one command line -- it does not touch the recipe's own output. +It maps to `show: { command: false }` on that one step, not `output: none`. `output: none` +discards the step's stdout and stderr entirely, which is a much bigger change in behavior than +Make's `@` prefix ever was; reserve it for steps whose output genuinely needs to be thrown away. + +### Split commands across files + +When a Makefile uses `include foo.mk` to split its content across files, split the Atmos config +the same way. Put the extra commands in a file such as `atmos.d/commands.yaml` or +`.atmos.d/commands.yaml`. Atmos auto-discovers `atmos.d/`/`.atmos.d/` in the config directory +(and, as a lower-priority fallback, at the git/worktree root) -- no `import:` entry is needed for +this specific location. Use `import:` only when splitting across a directory Atmos does not +auto-discover. See [Imports](https://atmos.tools/cli/configuration/imports). + +### Complex `$(eval)` and `$(call)` macros + +Do not try to convert deeply macro-driven Makefiles into flags and arguments one by one. Put +genuinely dynamic logic in a `shell` or `script` step, or in a script that the step calls. Atmos +custom commands replace task orchestration. They do not replace a general-purpose macro +language. + +## What Not To Do + +- Do not confuse `.PHONY` with a caching feature -- it is not one. Do not drop file-timestamp + caching without adding the matching `inputs`/`artifacts` fields to the migrated step; it is a + direct match, not a gap, but it does not carry over on its own. Remember the scope difference + too: it gates one step, not the whole recipe -- combine multiple command lines into a single + step if the freshness decision must cover all of them. +- Do not describe `dependencies.commands` as matching Make's *default* prerequisite order -- + `make` builds prerequisites one at a time, in the order listed, unless `-j` is given. + `dependencies.commands` runs concurrently by default, which changes that order. Ordered steps + are the default-preserving match for a plain `target: dep1 dep2`; reach for + `dependencies.commands` only when a prerequisite is shared by more than one target (it dedups a + shared dependency to a single run regardless of concurrency, the way `make` already does), the + prerequisites are genuinely independent, or the source used `-j`. +- Do not turn every private or helper target into its own discoverable command by default. If the + helper is called from only one recipe, put its logic in a step inside the command or workflow + that needs it. If it needs to be called from more than one recipe, or invoked directly for + debugging, make it a custom command with `internal: true` instead -- it stays runnable but is + excluded from `atmos --help` listings and completion suggestions. +- Do not treat "wrap `atmos` commands in the Makefile" as the final state. It is a valid bridge + during early migration. Leaf targets should become custom commands. A target chain usually stays + a custom command too, using `dependencies.commands` for its prerequisites -- do not prescribe a + workflow for every dependency chain. Reserve workflows for fixed, multi-step orchestration + across more than one component. +- Do not invent `when:` conditions that check flag values on workflow steps. The `when:` field + checks CEL context values, such as `stack`, `ci`, and `local`. Flag-based conditionals belong + in the custom command's own Go templates. diff --git a/agent-skills/skills/atmos-migration/references/from-native-terraform.md b/agent-skills/skills/atmos-migration/references/from-native-terraform.md index f583a5d35fc..55bc209e4fa 100644 --- a/agent-skills/skills/atmos-migration/references/from-native-terraform.md +++ b/agent-skills/skills/atmos-migration/references/from-native-terraform.md @@ -43,32 +43,32 @@ terraform/ **Recipe:** 1. `atmos.yaml` at repo root, **no file moves**: - ```yaml - base_path: "./" - components: - terraform: - base_path: "terraform" # Point at the existing dir - apply_auto_approve: false - deploy_run_init: true - auto_generate_backend_file: false - stacks: - base_path: "stacks" - included_paths: ["**/*"] - excluded_paths: ["**/_defaults.yaml"] - ``` + ```yaml + base_path: "./" + components: + terraform: + base_path: "terraform" # Point at the existing dir + apply_auto_approve: false + deploy_run_init: true + auto_generate_backend_file: false + stacks: + base_path: "stacks" + included_paths: ["**/*"] + excluded_paths: ["**/_defaults.yaml"] + ``` 2. Create `stacks/dev.yaml`: - ```yaml - import: - - _defaults - components: - terraform: - vpc: - vars: !include ../terraform/vpc/envs/dev.tfvars - database: - vars: !include ../terraform/database/envs/dev.tfvars - ``` + ```yaml + import: + - _defaults + components: + terraform: + vpc: + vars: !include ../terraform/vpc/envs/dev.tfvars + database: + vars: !include ../terraform/database/envs/dev.tfvars + ``` 3. Run `atmos terraform plan vpc -s dev`. Compare to the previous - `cd terraform/vpc && terraform plan -var-file=envs/dev.tfvars` output. + `cd terraform/vpc && terraform plan -var-file=envs/dev.tfvars` output. The user keeps their `.tfvars` files and TF code unchanged. Later, they can convert per-env `.tfvars` to native YAML to get deep-merge inheritance across environments. @@ -90,24 +90,46 @@ With a Makefile like `terraform plan -var-file=envs/$(ENV).tfvars`. **Recipe:** 1. `atmos.yaml`: - ```yaml - base_path: "./" - components: - terraform: - base_path: "." # The whole repo is one component - stacks: - base_path: "stacks" - ``` -2. Treat the single TF dir as one component (e.g., `infra`): - ```yaml - # stacks/dev.yaml - components: - terraform: - infra: - vars: !include ../terraform/envs/dev.tfvars - ``` -3. The Makefile can stay as a thin wrapper around `atmos terraform plan infra -s dev` during - transition, then be deleted. + ```yaml + base_path: "./" + components: + terraform: + base_path: "." # The whole repo is one component + stacks: + base_path: "stacks" + included_paths: + - "**/*" + ``` +2. **The component name must match the physical directory name, unless you set + `metadata.component`** -- Atmos resolves a component to + `/`, so with `base_path: "."` a stack + component named `terraform` resolves to the real directory with no extra configuration: + ```yaml + # stacks/dev.yaml + components: + terraform: + terraform: + vars: !include ../terraform/envs/dev.tfvars + ``` + If the user wants a friendlier stack component name, such as `infra`, without moving or + renaming the directory, set `metadata.component` to the physical directory name instead: + ```yaml + # stacks/dev.yaml + components: + terraform: + infra: + metadata: + component: terraform # points at the existing `terraform/` directory + vars: !include ../terraform/envs/dev.tfvars + ``` + `metadata.component` is a pointer to the physical directory, resolved the same way the + component name normally is (relative to `components.terraform.base_path`, or absolute). It is + also how you point multiple stack instances at one shared component (see + [remote-state-bridge.md](remote-state-bridge.md)) -- the no-move rename above is the same + mechanism applied to a single-component repo. Only rename the directory on disk (e.g. + `terraform/` to `infra/`) if the user prefers a physical rename over an alias. +3. The Makefile can stay as a thin wrapper around `atmos terraform plan terraform -s dev` during + transition, then be deleted. ## Shape C: Multiple Root Modules with Shared Modules diff --git a/agent-skills/skills/atmos-migration/references/from-taskfile.md b/agent-skills/skills/atmos-migration/references/from-taskfile.md new file mode 100644 index 00000000000..563362ca683 --- /dev/null +++ b/agent-skills/skills/atmos-migration/references/from-taskfile.md @@ -0,0 +1,212 @@ +# Migrating from Taskfile.yml (go-task) + +This guide shows how to move Task's tasks to Atmos. Find the correct shape for the Taskfile +below. Then follow the matching steps. For the full tutorial, see +[atmos.tools/migration/taskfile](https://atmos.tools/migration/taskfile). + +Task and Atmos both use declarative YAML. This makes the migration mostly mechanical. It is the +simplest of the three task-runner migrations this skill covers. If the Taskfile also selects a +Terraform environment, also use [from-native-terraform.md](from-native-terraform.md) for the +Terraform-specific steps. + +## Find the Shape of the Taskfile + +| Shape | Steps | +|------------------------------------------------------------|--------------------------------------| +| Simple task (`desc`/`cmds`) | [Shape A](#shape-a-simple-tasks) | +| Task with `deps:` (parallel by default) | [Shape B](#shape-b-task-dependencies-parallel-by-default) | +| `vars:`/`env:` and `sources:`/`generates:` | [Shape C](#shape-c-variables-and-up-to-date-checks) | +| `includes:` (multi-file composition) | see [Common Problems](#includes-multi-file-composition) | + +`deps:` maps to command-level `dependencies.commands` (concurrent by default, deduped -- the +direct match). `sources:`/`generates:` maps to step-level `inputs.sources`/`artifacts.paths` +(implicit `when: checksum.changed` -- the direct match). Neither is a gap; both need the user to +add the matching field during migration, since neither carries over automatically. + +## Shape A: Simple Tasks + +**Before:** +```yaml +version: '3' + +tasks: + build: + desc: Compile the deployable artifact + cmds: + - go build -o bin/handler ./cmd/handler + + lint: + desc: Run static analysis + cmds: + - golangci-lint run ./... +``` + +**Steps:** + +1. Turn `desc:` into the command's `description:` field. +2. Turn each entry in `cmds:` into a `type: shell` step. If the line is a native Atmos verb, such + as `terraform plan` or `terraform apply`, use a `type: atmos` step instead. `type: atmos` is + reserved for native Atmos verbs only. If the line calls another custom command (for example + `atmos build`), keep it as a `type: shell` step with `command: atmos build` -- do not use + `type: atmos` for that. +3. Set `internal: true` on a command created from an `internal: true` task. It runs normally + (`atmos ...`, as a `default:` target, or from another command's steps) but is excluded + from `atmos --help` listings and completion suggestions. Only inline the task's body into a + caller's step when it is genuinely single-caller logic with no reason to be invoked on its own. + +```yaml +commands: + - name: build + description: Compile the deployable artifact + steps: + - type: shell + command: go build -o bin/handler ./cmd/handler + + - name: lint + description: Run static analysis + steps: + - type: shell + command: golangci-lint run ./... +``` + +## Shape B: Task Dependencies (Parallel by Default) + +**Before:** +```yaml +tasks: + test: + desc: Run unit tests + deps: [build] + cmds: + - go test ./... + + deploy: + desc: Plan and apply the given environment + deps: [test, lint] + cmds: + - terraform -chdir=terraform apply -var-file=envs/dev.tfvars +``` + +Task runs `deps:` at the same time by default. Atmos custom-command and workflow steps run one +after another by default -- so a `deps:` entry is not a step and never becomes one. It maps to +the command-level `dependencies.commands` field, which resolves through the same DAG scheduler as +`parallel`/`matrix` `needs:` and runs concurrently by default -- matching Task's `deps:` behavior +directly, not working around it with a hand-built `parallel` step: + +```yaml +commands: + - name: deploy + description: Plan and apply the given environment + dependencies: + commands: [test, lint] + steps: + - type: atmos + command: terraform apply infra -s dev +``` + +`infra` is a placeholder Atmos component name, not the `terraform` verb repeated. Move the +task's Terraform code to `components/terraform/infra/` (the default +`components.terraform.base_path` is `components/terraform`), then swap `infra` for whatever the +user actually names the component. + +`dependencies.commands` also matches a behavior Task itself has that a hand-rolled `parallel` +step does not: if two commands both depend on the same one -- for example both `test` and `lint` +depending on `build` -- Atmos runs `build` exactly once and dedups it, the same as Task's own +`deps:` graph. A `parallel` step calling `atmos build` from two different places would run it +twice. If one dependency itself depends on another (`lint` depends on `build`, and `deploy` +depends on `test` and `lint`), declare that directly on `lint`'s own `dependencies.commands` -- +the scheduler resolves the whole transitive graph itself, still deduping `build` to a single run. + +Reach for a `parallel` step instead of `dependencies.commands` only for concurrency inside a +single command's own steps, not between named commands -- for example, running several shell +commands side by side that were never their own Task tasks to begin with. + +## Shape C: Variables and Up-to-Date Checks + +**Before:** +```yaml +vars: + ENV: '{{.ENV | default "dev"}}' + +tasks: + build: + desc: Compile the deployable artifact + cmds: + - go build -o bin/handler ./cmd/handler + sources: + - cmd/**/*.go + generates: + - bin/handler +``` + +**Steps:** + +- Turn `vars: ENV: '{{.ENV | default "dev"}}'` into a command `flags:` entry with + `default: "dev"`. Task's Sprig `default` filter becomes the plain `default:` field. +- Turn `env:` into an `env:` map. The two are almost identical. + +### `sources:`/`generates:` becomes `inputs`/`artifacts` + +Task skips a task's `cmds:` when its `sources:` files match its `generates:` outputs, checked by +default with a content hash (Task also supports `method: timestamp` for an mtime-based check). +The step-level `inputs.sources` and `artifacts.paths` fields are the direct match, with the same +checksum-by-default/timestamp-as-an-option choice: + +```yaml +commands: + - name: build + description: Compile the deployable artifact + steps: + - type: shell + command: go build -o bin/handler ./cmd/handler + inputs: + sources: ["cmd/**/*.go"] + artifacts: + paths: ["bin/handler"] +``` + +With no explicit `when:`, declaring `inputs`/`artifacts` on a step is enough -- it implicitly +means `when: checksum.changed`, and the step is skipped when the hash of the matched source files +matches the hash recorded after the last successful run. This does not carry over on its own -- +add `inputs`/`artifacts` to the migrated step yourself, matching the Taskfile's own +`sources:`/`generates:` lists. The `require`/`assert` step type is a different, older step type -- +it only checks that a file, tool, or directory exists, not whether it is fresh, so it does not +replace `inputs`/`artifacts`. + +## Common Problems + +### `includes:` (multi-file composition) + +Task's `includes:` field combines several Taskfiles into one. Atmos has two matching methods. +Pick the one that fits the content being split: + +- To split command definitions across files, put them in files such as `atmos.d/commands.yaml` + or `.atmos.d/commands.yaml`. Atmos auto-discovers `atmos.d/`/`.atmos.d/` in the config + directory (and, as a lower-priority fallback, at the git/worktree root) -- no `import:` entry + is needed for this specific location. Use `import:` only when splitting across a directory + Atmos does not auto-discover. See [Imports](https://atmos.tools/cli/configuration/imports). +- To split multi-step chains, use separate workflow files. Atmos workflows already live one file + per purpose, under `workflows.base_path`. Unlike `atmos.d`/`.atmos.d`, there is no default for + `workflows.base_path` -- add it explicitly (for example `workflows.base_path: + "stacks/workflows"`) the first time the user's migration reaches a workflow, or `atmos + workflow ` fails with `'workflows.base_path' must be configured in 'atmos.yaml'`. + +### `sources`/`generates` maps to a different field than `steps` + +See [Shape C](#sourcesgenerates-becomes-inputsartifacts) above. It does not carry over +automatically -- the user must add `inputs`/`artifacts` to the migrated step themselves. State +that directly. Do not gloss over it, and do not claim it "just works" without the field. + +## What Not To Do + +- Do not drop `sources:`/`generates:` without adding the matching `inputs`/`artifacts` fields to + the migrated step. It is a direct match, not a gap, but it does not carry over on its own. +- Do not turn `deps:` into plain sequential steps, or into a hand-built `parallel` step, without + first considering command-level `dependencies.commands` -- it is the direct match: concurrent + by default, and it dedups a dependency shared by more than one command the same way Task's own + `deps:` graph does. +- Do not describe `require`/`assert` as a freshness or caching check. It only checks that + something exists. +- Do not turn every `internal: true` task into its own discoverable command by default. If it is + called from only one task, inline it into that caller's step. If it needs to be called from + more than one task, or invoked directly for debugging, make it an `internal: true` custom command. diff --git a/cmd/cmd_utils_test.go b/cmd/cmd_utils_test.go index 8b22523f7e1..94705f89854 100644 --- a/cmd/cmd_utils_test.go +++ b/cmd/cmd_utils_test.go @@ -2419,6 +2419,153 @@ func TestExecuteCustomCommandShellStepPropagatesCILogGroupSentinel(t *testing.T) assert.Equal(t, "1", string(got)) } +func TestCreateCustomCommandHidden(t *testing.T) { + atmosConfig := &schema.AtmosConfiguration{} + + t.Run("hidden command is excluded from help listings", func(t *testing.T) { + parentCmd := &cobra.Command{Use: "atmos"} + hiddenCmd, err := createCustomCommand(atmosConfig, &schema.Command{ + Name: "helper", + Description: "hidden helper", + Internal: true, + }, parentCmd) + require.NoError(t, err) + assert.True(t, hiddenCmd.Hidden) + assert.False(t, hiddenCmd.IsAvailableCommand(), "a hidden command must not report as available for help listings") + }) + + t.Run("unset hidden keeps the command visible", func(t *testing.T) { + parentCmd := &cobra.Command{Use: "atmos"} + visibleCmd, err := createCustomCommand(atmosConfig, &schema.Command{ + Name: "visible", + Description: "visible command", + }, parentCmd) + require.NoError(t, err) + assert.False(t, visibleCmd.Hidden) + assert.True(t, visibleCmd.IsAvailableCommand()) + }) +} + +func TestHiddenCommandStillExecutesDirectly(t *testing.T) { + _ = NewTestKit(t) + ensureIOInitialized(t) + + workDir := t.TempDir() + sentinelFile := filepath.Join(workDir, "sentinel.txt") + + atmosConfig := schema.AtmosConfiguration{BasePath: workDir} + parentCmd := &cobra.Command{Use: "atmos"} + commands := []schema.Command{{ + Name: "hidden-helper", + Description: "a hidden helper command", + Internal: true, + WorkingDirectory: workDir, + // TaskTypeShell runs through Atmos's own in-process mvdan/sh interpreter (see + // pkg/runner/step/shell.go), not the host shell, so printf and redirection here + // are cross-platform including on Windows -- not a platform-specific binary call. + Steps: []schema.Task{{ + Name: "write-sentinel", + Type: schema.TaskTypeShell, + Command: `printf %s "ran" > sentinel.txt`, + }}, + }} + + require.NoError(t, processCustomCommands(atmosConfig, commands, parentCmd)) + hiddenCmd := findSubcommand(parentCmd, "hidden-helper") + require.NotNil(t, hiddenCmd) + require.True(t, hiddenCmd.Hidden) + require.False(t, hiddenCmd.IsAvailableCommand()) + + // Hidden must not gate execution: invoking it directly still runs its steps. + hiddenCmd.PreRun(hiddenCmd, nil) + hiddenCmd.Run(hiddenCmd, nil) + + got, err := os.ReadFile(sentinelFile) + require.NoError(t, err) + assert.Equal(t, "ran", string(got)) +} + +func TestNestedCommandVisibilityIsIndependentOfParent(t *testing.T) { + atmosConfig := schema.AtmosConfiguration{} + parentCmd := &cobra.Command{Use: "atmos"} + + commands := []schema.Command{ + { + Name: "hidden-parent", + Description: "hidden parent with a visible child", + Internal: true, + Commands: []schema.Command{ + {Name: "visible-child", Description: "visible child of a hidden parent"}, + }, + }, + { + Name: "visible-parent", + Description: "visible parent with a hidden child", + Commands: []schema.Command{ + {Name: "hidden-child", Description: "hidden child of a visible parent", Internal: true}, + }, + }, + } + + require.NoError(t, processCustomCommands(atmosConfig, commands, parentCmd)) + + hiddenParent := findSubcommand(parentCmd, "hidden-parent") + require.NotNil(t, hiddenParent) + assert.True(t, hiddenParent.Hidden) + visibleChild := findSubcommand(hiddenParent, "visible-child") + require.NotNil(t, visibleChild) + assert.False(t, visibleChild.Hidden, "a visible child must stay visible even though its parent is hidden") + + visibleParent := findSubcommand(parentCmd, "visible-parent") + require.NotNil(t, visibleParent) + assert.False(t, visibleParent.Hidden) + hiddenChild := findSubcommand(visibleParent, "hidden-child") + require.NotNil(t, hiddenChild) + assert.True(t, hiddenChild.Hidden, "a hidden child must stay hidden even though its parent is visible") +} + +func TestDefaultDispatchToHiddenChildStillExecutes(t *testing.T) { + _ = NewTestKit(t) + ensureIOInitialized(t) + + workDir := t.TempDir() + sentinelFile := filepath.Join(workDir, "sentinel.txt") + + atmosConfig := schema.AtmosConfiguration{BasePath: workDir} + parentCmd := &cobra.Command{Use: "atmos"} + commands := []schema.Command{{ + Name: "wrapper", + Description: "visible wrapper whose default action is a hidden helper", + Default: "helper", + Commands: []schema.Command{ + { + Name: "helper", + Description: "hidden implementation", + Internal: true, + WorkingDirectory: workDir, + // TaskTypeShell runs through Atmos's own in-process mvdan/sh interpreter (see + // pkg/runner/step/shell.go), not the host shell, so printf and redirection here + // are cross-platform including on Windows -- not a platform-specific binary call. + Steps: []schema.Task{{ + Name: "write-sentinel", + Type: schema.TaskTypeShell, + Command: `printf %s "ran" > sentinel.txt`, + }}, + }, + }, + }} + + require.NoError(t, processCustomCommands(atmosConfig, commands, parentCmd)) + wrapperCmd := findSubcommand(parentCmd, "wrapper") + require.NotNil(t, wrapperCmd) + + wrapperCmd.Run(wrapperCmd, nil) + + got, err := os.ReadFile(sentinelFile) + require.NoError(t, err) + assert.Equal(t, "ran", string(got)) +} + // TestExecuteCustomCommandStepWhenFlagsFact covers the end-to-end wiring of a custom command's // --flag value into a step's `when:` CEL expression via the `flags` fact -- including the // `hasRunnableStep` pre-check loop, which runs before the main step loop and must see the same diff --git a/cmd/help_template.go b/cmd/help_template.go index 0206ad5fa1e..cf511134770 100644 --- a/cmd/help_template.go +++ b/cmd/help_template.go @@ -400,6 +400,53 @@ func printAvailableCommands(ctx *helpRenderContext, cmd *cobra.Command) { }) } +// hiddenSubcommands returns cmd's direct subcommands that are marked Hidden, in Commands() order. +// This includes both `hidden: true` custom commands and any hidden built-in commands. +func hiddenSubcommands(cmd *cobra.Command) []*cobra.Command { + var hidden []*cobra.Command + for _, c := range cmd.Commands() { + if c.Hidden { + hidden = append(hidden, c) + } + } + return hidden +} + +// printHiddenCommands prints the list of hidden subcommands for the `--help=hidden` topic. +// Unlike printAvailableCommands, this shows ONLY commands excluded from ordinary help output +// (via `hidden: true` on a custom command, or a hidden built-in) -- the inverse listing, for +// discovering what a bare `--help` deliberately omits. +func printHiddenCommands(ctx *helpRenderContext, cmd *cobra.Command) { + defer perf.Track(nil, "cmd.printHiddenCommands")() + + hidden := hiddenSubcommands(cmd) + if len(hidden) == 0 { + fmt.Fprintln(ctx.writer, ctx.styles.muted.Render(fmt.Sprintf("%s has no hidden subcommands.", cmd.CommandPath()))) + return + } + + parentExperimental := isExperimentalCommand(cmd) + + var mdRenderer *markdown.Renderer + if ctx.atmosConfig != nil { + mdRenderer, _ = markdown.NewTerminalMarkdownRenderer(*ctx.atmosConfig) + } + + maxCmdWidth := 0 + for _, c := range hidden { + if width := calculateCommandWidth(c, parentExperimental); width > maxCmdWidth { + maxCmdWidth = width + } + } + + fmt.Fprintln(ctx.writer, ctx.styles.heading.Render("HIDDEN COMMANDS")) + fmt.Fprintln(ctx.writer) + for _, c := range hidden { + formatCommandLine(ctx, c, maxCmdWidth, mdRenderer, parentExperimental) + } + fmt.Fprintln(ctx.writer) +} + // getConfigAliases returns all available config alias commands. func getConfigAliases(cmd *cobra.Command) []*cobra.Command { var aliases []*cobra.Command diff --git a/cmd/help_topics.go b/cmd/help_topics.go index eaea403b9a7..bfb7be98d5b 100644 --- a/cmd/help_topics.go +++ b/cmd/help_topics.go @@ -9,12 +9,14 @@ const ( helpTopicUsage helpTopic = "usage" helpTopicFlags helpTopic = "flags" helpTopicAll helpTopic = "all" + helpTopicHidden helpTopic = "hidden" ) var supportedHelpTopics = []helpTopic{ helpTopicUsage, helpTopicFlags, helpTopicAll, + helpTopicHidden, } type helpTopicRequest struct { diff --git a/cmd/help_topics_render.go b/cmd/help_topics_render.go index 59c03d303f2..b9a18e97044 100644 --- a/cmd/help_topics_render.go +++ b/cmd/help_topics_render.go @@ -72,6 +72,8 @@ func printHelpForTopic(ctx *helpRenderContext, cmd *cobra.Command, topic helpTop printCompatibilityFlags(ctx.writer, cmd, ctx.styles) case helpTopicAll: printFullHelp(ctx, cmd) + case helpTopicHidden: + printHiddenCommands(ctx, cmd) default: printDefaultHelp(ctx, cmd) } @@ -107,13 +109,17 @@ func printDefaultHelp(ctx *helpRenderContext, cmd *cobra.Command) { printLocalFlagsOnly(ctx.writer, cmd, ctx.atmosConfig, ctx.styles) printCompatibilityFlags(ctx.writer, cmd, ctx.styles) printFooter(ctx.writer, cmd, ctx.styles) - printHelpTopicHint(ctx.writer, ctx.styles) + printHelpTopicHint(ctx.writer, ctx.styles, cmd) } -func printHelpTopicHint(w io.Writer, styles *helpStyles) { +func printHelpTopicHint(w io.Writer, styles *helpStyles, cmd *cobra.Command) { defer perf.Track(nil, "cmd.printHelpTopicHint")() usageMsg := "Use `--help=usage` for examples or `--help=all` for all flags and full help." + if len(hiddenSubcommands(cmd)) > 0 { + usageMsg = "Use `--help=usage` for examples, `--help=all` for all flags and full help, " + + "or `--help=hidden` to list hidden subcommands." + } usageMsg = renderMarkdownDescription(usageMsg) fmt.Fprintf(w, "\n%s\n", styles.muted.Render(usageMsg)) } diff --git a/cmd/help_topics_test.go b/cmd/help_topics_test.go index b11921294a0..5509ee4d708 100644 --- a/cmd/help_topics_test.go +++ b/cmd/help_topics_test.go @@ -56,6 +56,16 @@ func TestNormalizeHelpTopicArgs(t *testing.T) { expectedSet: true, expectedChange: true, }, + { + name: "hidden topic", + args: []string{"terraform", "plan", "--help=hidden"}, + expectedArgs: []string{"terraform", "plan", "--help"}, + expectedTopic: helpTopicHidden, + expectedRaw: "hidden", + expectedValid: true, + expectedSet: true, + expectedChange: true, + }, { name: "uppercase topic", args: []string{"terraform", "plan", "--help=USAGE"}, @@ -153,6 +163,42 @@ func TestTopicHelpRendering_AllIncludesGlobalFlags(t *testing.T) { assert.Contains(t, output, "--global") } +func testHelpCommandWithHiddenChild(t *testing.T) *cobra.Command { + t.Helper() + + parent := &cobra.Command{Use: "parent", Short: "parent command"} + visible := &cobra.Command{Use: "visible", Short: "a visible child", Run: func(cmd *cobra.Command, args []string) {}} + hidden := &cobra.Command{Use: "secret", Short: "a hidden child", Hidden: true, Run: func(cmd *cobra.Command, args []string) {}} + parent.AddCommand(visible, hidden) + + return parent +} + +func TestTopicHelpRendering_HiddenShowsOnlyHiddenSubcommands(t *testing.T) { + output := renderTopicHelpForTest(t, helpTopicRequest{topic: helpTopicHidden, explicit: true, valid: true}, testHelpCommandWithHiddenChild(t)) + + assert.Contains(t, output, "HIDDEN COMMANDS") + assert.Contains(t, output, "secret") + assert.Contains(t, output, "a hidden child") + assert.NotContains(t, output, "visible") + assert.NotContains(t, output, "a visible child") +} + +func TestTopicHelpRendering_HiddenWithNoHiddenSubcommandsShowsMessage(t *testing.T) { + output := renderTopicHelpForTest(t, helpTopicRequest{topic: helpTopicHidden, explicit: true, valid: true}, testHelpCommand(t)) + + assert.Contains(t, output, "has no hidden subcommands") + assert.NotContains(t, output, "HIDDEN COMMANDS") +} + +func TestTopicHelpRendering_DefaultHintMentionsHiddenWhenPresent(t *testing.T) { + output := renderTopicHelpForTest(t, helpTopicRequest{valid: true}, testHelpCommandWithHiddenChild(t)) + + // Match "help=hidden" rather than "--help=hidden": word-wrapping in the rendered hint + // can split the leading "--" onto the previous line at narrower terminal widths. + assert.Contains(t, output, "help=hidden") +} + func TestTopicHelpRendering_RootDefaultFiltersPersistentGlobals(t *testing.T) { root := &cobra.Command{ Use: "atmos", diff --git a/docs/prd/topic-specific-cli-help.md b/docs/prd/topic-specific-cli-help.md index b3965977806..eda9125e87c 100644 --- a/docs/prd/topic-specific-cli-help.md +++ b/docs/prd/topic-specific-cli-help.md @@ -2,7 +2,7 @@ ## Status: Implemented -**Last Updated**: 2026-07-06 +**Last Updated**: 2026-08-06 ## Overview @@ -14,6 +14,7 @@ The v1 topic set is intentionally small: - `--help=usage` shows only usage and embedded usage examples. - `--help=flags` shows command-specific flags, excluding inherited/global flags. - `--help=all` shows the full help page, including inherited/global flags. +- `--help=hidden` shows only the command's hidden subcommands (custom commands with `internal: true`, or a hidden built-in) -- the commands every other topic, including `--help=all`, omits. ## Motivation @@ -78,13 +79,25 @@ atmos terraform plan --help=all Shows the complete help page, including inherited/global flags. +Hidden subcommands: + +```shell +atmos wrapper --help=hidden +``` + +Shows only the command's directly hidden subcommands (custom commands with `internal: true`, plus any +hidden built-ins), formatted the same as an ordinary command listing. Shows a one-line "has no hidden +subcommands" message instead of an empty section when there are none. Default help includes a hint for +`--help=hidden` only on commands that actually have a hidden subcommand, so the hint doesn't clutter +help output for the common case. + Unknown topic: ```shell atmos terraform plan --help=advanced ``` -Exits non-zero and lists valid topics: `usage`, `flags`, `all`. +Exits non-zero and lists valid topics: `usage`, `flags`, `all`, `hidden`. ## Implementation Notes @@ -97,6 +110,7 @@ Exits non-zero and lists valid topics: `usage`, `flags`, `all`. - `usage`: usage and examples. - `flags`: command-specific flags and compatibility flags. - `all`: current full help. + - `hidden`: only the command's hidden direct subcommands, or a "no hidden subcommands" message. - For the root command, filter root persistent flags from default and `--help=flags` output because they are global flags even though Cobra exposes them as local root flags. ## Acceptance Criteria @@ -105,5 +119,7 @@ Exits non-zero and lists valid topics: `usage`, `flags`, `all`. - `atmos --help=usage` excludes flags and includes usage examples when present. - `atmos --help=flags` excludes inherited/global flags. - `atmos --help=all` includes inherited/global flags. +- `atmos --help=hidden` shows only hidden direct subcommands, and never a command that's + already visible in the default listing. - `atmos --help=` exits non-zero and lists valid topics. - Existing `-h`, bare `--help`, and pager behavior continue to work. diff --git a/internal/exec/validate_stacks.go b/internal/exec/validate_stacks.go index 303cb8f0468..4bd41e276ac 100644 --- a/internal/exec/validate_stacks.go +++ b/internal/exec/validate_stacks.go @@ -401,20 +401,22 @@ func createComponentStackMap( }, } - // Find Atmos stack name - if atmosConfig.Stacks.NameTemplate != "" { - stackName, err = ProcessTmpl(atmosConfig, "validate-stacks-name-template", atmosConfig.Stacks.NameTemplate, configAndStacksInfo.ComponentSection, atmosConfig.Templates.Settings.IgnoreMissingTemplateValues) - if err != nil { - return nil, err - } - } else { - context := cfg.GetContextFromVars(varsSection) - configAndStacksInfo.Context = context - stackName, err = cfg.GetContextPrefix(stackManifest, context, GetStackNamePattern(atmosConfig), stackManifest) - if err != nil { - return nil, err - } + // Find the Atmos stack name. Precedence matches the rest of the CLI + // (resolveStackName, used by `describe stacks`/`list stacks`, and + // processStackContextPrefix, used by `terraform plan` and friends): + // manifest `name:` > name_template > name_pattern > filename. Neither + // name_template nor name_pattern is required -- filename-based zero-config + // stack naming (#1934) applies here too, so `validate stacks` no longer + // hard-fails on a repo that every other command resolves stacks for fine. + if comp, ok := configAndStacksInfo.ComponentSection[cfg.ComponentSectionName].(string); !ok || comp == "" { + configAndStacksInfo.ComponentSection[cfg.ComponentSectionName] = componentName } + var resolvedContext schema.Context + stackName, resolvedContext, err = resolveStackName(atmosConfig, stackManifest, getStackManifestName(stackSection), configAndStacksInfo, varsSection) + if err != nil { + return nil, err + } + configAndStacksInfo.Context = resolvedContext _, ok = terraformComponentStackMap[componentName] if !ok { diff --git a/internal/exec/validate_stacks_test.go b/internal/exec/validate_stacks_test.go index 61ac6eabf76..7b9f23ffc60 100644 --- a/internal/exec/validate_stacks_test.go +++ b/internal/exec/validate_stacks_test.go @@ -71,6 +71,74 @@ func TestValidateStacksWithMergeContext(t *testing.T) { }) } +// nativeTerraformExampleDir returns the absolute path to examples/native-terraform +// using runtime.Caller(0) so the path is source-file-relative (CWD-independent). +func nativeTerraformExampleDir(t *testing.T) string { + t.Helper() + _, callerFile, _, ok := runtime.Caller(0) + require.True(t, ok, "runtime.Caller(0) must succeed") + dir := filepath.Join(filepath.Dir(callerFile), "..", "..", "examples", "native-terraform") + absDir, err := filepath.Abs(dir) + require.NoError(t, err, "cannot resolve fixture path") + return absDir +} + +// TestValidateStacksWithoutNameTemplateOrPattern is a regression test: `atmos +// validate stacks` used to hard-fail with ErrMissingStackNameTemplateAndPattern +// whenever `stacks.name_template`/`stacks.name_pattern` was unset, even though +// `terraform plan`, `list stacks`, and `describe component` all resolve stack +// names fine via filename/`name:`-based zero-config fallback (#1934) in that +// case. The createComponentStackMap function's stack-name derivation now +// mirrors that same precedence (resolveStackName) instead of requiring +// name_template/name_pattern itself. The examples/native-terraform directory +// is the real repro: its atmos.yaml sets neither field on purpose (see its +// own comment) and is the example this repo's atmos-migration skill points +// users at. +func TestValidateStacksWithoutNameTemplateOrPattern(t *testing.T) { + absPath := nativeTerraformExampleDir(t) + + atmosConfig := &schema.AtmosConfiguration{ + BasePath: absPath, + StacksBaseAbsolutePath: filepath.Join(absPath, "stacks"), + Stacks: schema.Stacks{ + BasePath: "stacks", + IncludedPaths: []string{"**/*"}, + ExcludedPaths: []string{"**/_defaults.yaml"}, + }, + Logs: schema.Logs{ + Level: u.LogLevelDebug, + }, + Components: schema.Components{ + Terraform: schema.Terraform{ + BasePath: "components/terraform", + }, + }, + Settings: schema.AtmosSettings{ + ListMergeStrategy: "replace", + }, + } + atmosConfig.TerraformDirAbsolutePath = filepath.Join(absPath, "components", "terraform") + atmosConfig.HelmfileDirAbsolutePath = filepath.Join(absPath, "components", "helmfile") + atmosConfig.PackerDirAbsolutePath = filepath.Join(absPath, "components", "packer") + + // Populate StackConfigFilesAbsolutePaths the same way cfg.InitCliConfig does for a + // real CLI run -- without it, FindStacksMap has no files to process and + // createComponentStackMap's loop body (where the bug lives) never runs, letting + // the test pass vacuously regardless of the fix. + includeStackAbsPaths, err := u.JoinPaths(atmosConfig.StacksBaseAbsolutePath, atmosConfig.Stacks.IncludedPaths) + require.NoError(t, err) + stackConfigFilesAbsolutePaths, _, err := cfg.FindAllStackConfigsInPaths(atmosConfig, includeStackAbsPaths, atmosConfig.Stacks.ExcludedPaths) + require.NoError(t, err) + require.NotEmpty(t, stackConfigFilesAbsolutePaths, "fixture must have discoverable stack manifests for this test to exercise the bug") + atmosConfig.StackConfigFilesAbsolutePaths = stackConfigFilesAbsolutePaths + + require.Empty(t, atmosConfig.Stacks.NameTemplate, "fixture must exercise the no-name_template case") + require.Empty(t, atmosConfig.Stacks.NamePattern, "fixture must exercise the no-name_pattern case") + + err = ValidateStacks(atmosConfig) + assert.NoError(t, err, "validate stacks must succeed via filename/name:-based stack naming, same as terraform plan/list stacks/describe component") +} + func TestMergeContextInProcessYAMLConfigFile(t *testing.T) { // Test that ProcessYAMLConfigFileWithContext properly tracks import chain absPath := validateStacksTestDataDir(t) diff --git a/lychee.toml b/lychee.toml index 5c83bb8bbc8..31bc836c3da 100644 --- a/lychee.toml +++ b/lychee.toml @@ -196,7 +196,8 @@ exclude = [ # verified to return 200 outside CI via curl. "playwright\\.dev", # Reproducible Builds project docs intermittently refuse connections from CI - # runners; the page returns 200 outside CI, verified via curl. + # runners; the page returns 200 outside CI, verified via curl. Supersedes + # the narrower reproducible-builds.org/docs/source-date-epoch/ exclude. "reproducible-builds\\.org", # Microsoft's Language Server Protocol spec site intermittently resets # connections from CI runners; the page returns 200 outside CI, verified via curl. diff --git a/pkg/config/command_merge_core_test.go b/pkg/config/command_merge_core_test.go index 4b420d26c11..05debc5c33d 100644 --- a/pkg/config/command_merge_core_test.go +++ b/pkg/config/command_merge_core_test.go @@ -256,6 +256,76 @@ func TestMergeCommandArraysPathLeafOverridesNestedAndPreservesSiblings(t *testin assert.Equal(t, "Base examples", examples["description"]) } +// TestMergeMainCommandArrayDoesNotInheritUnrelatedSubcommands is a regression +// test for a directory's own atmos.yaml `commands:` entry silently inheriting +// a same-named command's subcommand tree (and other subcommand-referencing +// fields, such as `default:`) from an unrelated discovered source -- for +// example a different, outer project's git-root `.atmos.d` command that +// happens to share a name. The mergeMainCommandArray function must treat a +// leaf command (no `commands:` key) in the directory's own atmos.yaml as +// fully self-contained, not as a partial override of the discovered command. +func TestMergeMainCommandArrayDoesNotInheritUnrelatedSubcommands(t *testing.T) { + discovered := []interface{}{ + map[string]interface{}{ + "name": "build", + "description": "Build and documentation commands for Atmos development", + "default": "binary", + commandsKey: []interface{}{ + map[string]interface{}{"name": "binary", "description": "Build the Atmos binary"}, + map[string]interface{}{"name": "deps", "description": "Download Go module dependencies"}, + }, + }, + } + mainCommands := []interface{}{ + map[string]interface{}{ + "name": "build", + "description": "My own build command", + }, + } + + merged := mergeMainCommandArray(discovered, mainCommands) + require.Len(t, merged, 1) + + build := requireCommandMap(t, merged, "build") + assert.Equal(t, "My own build command", build["description"]) + _, hasCommands := build[commandsKey] + assert.False(t, hasCommands, "leaf command in the directory's own atmos.yaml must not inherit an unrelated source's subcommand tree") + _, hasDefault := build["default"] + assert.False(t, hasDefault, "leaf command must not inherit a default: pointing at a subcommand that no longer exists") +} + +// TestMergeMainCommandArrayStillComposesWhenSecondOptsIn verifies +// mergeMainCommandArray's strict mode only replaces a colliding command +// outright when the directory's own atmos.yaml command omits `commands:` +// entirely. When it explicitly defines `commands:` (opting into extending a +// shared command tree, e.g. via `.atmos.d` fragments within the same +// project), the normal deep-merge behavior still applies. +func TestMergeMainCommandArrayStillComposesWhenSecondOptsIn(t *testing.T) { + discovered := []interface{}{ + map[string]interface{}{ + "name": "casts", + commandsKey: []interface{}{ + map[string]interface{}{"name": "setup", "description": "Base setup"}, + }, + }, + } + mainCommands := []interface{}{ + map[string]interface{}{ + "name": "casts", + commandsKey: []interface{}{ + map[string]interface{}{"name": "generate", "description": "Local generate"}, + }, + }, + } + + merged := mergeMainCommandArray(discovered, mainCommands) + require.Len(t, merged, 1) + + casts := requireCommandMap(t, merged, "casts") + findCommandMap(t, casts[commandsKey], "setup") + findCommandMap(t, casts[commandsKey], "generate") +} + // TestNormalizeCommandArraySkipsNilEntries verifies that normalizeCommandArray // drops nil command entries (normalizeCommandDefinition returns cmd unchanged // when it's not a map, and nil normalizes to nil, which must be skipped). @@ -290,18 +360,18 @@ func TestNormalizeCommandDefinitionWithoutName(t *testing.T) { func TestMergeCommandDefinitionsNonMapOperands(t *testing.T) { t.Run("first is not a map", func(t *testing.T) { second := map[string]interface{}{"name": "second"} - result := mergeCommandDefinitions("not-a-map", second) + result := mergeCommandDefinitions("not-a-map", second, false) assert.Equal(t, second, result) }) t.Run("second is not a map", func(t *testing.T) { first := map[string]interface{}{"name": "first"} - result := mergeCommandDefinitions(first, "not-a-map") + result := mergeCommandDefinitions(first, "not-a-map", false) assert.Equal(t, "not-a-map", result) }) t.Run("both are not maps", func(t *testing.T) { - result := mergeCommandDefinitions(42, "second-value") + result := mergeCommandDefinitions(42, "second-value", false) assert.Equal(t, "second-value", result) }) } diff --git a/pkg/config/load.go b/pkg/config/load.go index 9142f79f652..427b9b19254 100644 --- a/pkg/config/load.go +++ b/pkg/config/load.go @@ -1458,9 +1458,13 @@ func processConfigImportsAndReapply(path string, tempViper *viper.Viper, content finalCommands = mergeCommandArrays(finalCommands, importedCommands) } - // Add main, with main overriding all others on duplicates + // Add main, with main overriding all others on duplicates. Strict: the + // directory's own atmos.yaml is authoritative for any command name it + // defines directly, so it must not inherit a same-named command's + // subcommands from a discovered default (e.g. an unrelated ancestor + // project's git-root .atmos.d) just because it doesn't repeat `commands:`. if mainCommands != nil { - finalCommands = mergeCommandArrays(finalCommands, mainCommands) + finalCommands = mergeMainCommandArray(finalCommands, mainCommands) } tempViper.Set(commandsKey, finalCommands) @@ -2101,7 +2105,29 @@ func overlayProfileSettings(v *viper.Viper, settings map[string]any, prefix stri // When duplicates exist based on name, the second parameter takes precedence (override behavior). // This ensures local commands can override imported/remote commands. func mergeCommandArrays(first, second interface{}) []interface{} { - return mergeNormalizedCommandArrays(normalizeCommandArray(first), normalizeCommandArray(second)) + return mergeNormalizedCommandArrays(normalizeCommandArray(first), normalizeCommandArray(second), false) +} + +// mergeMainCommandArray merges commands gathered from `.atmos.d`/imports (first) +// against a directory's own inline atmos.yaml `commands:` block (second). Unlike +// mergeCommandArrays, when second's definition of a command omits a nested +// `commands:` key, the merged result has none either -- it does not silently +// inherit a same-named command's subcommand tree from first. That matters when +// first was discovered from a git-root `.atmos.d` belonging to a different, +// unrelated outer project (e.g. this directory has no `.git` of its own and sits +// inside someone else's monorepo): the directory's own atmos.yaml is authoritative +// for any command name it defines directly, so a coincidental name collision with +// an ancestor's `.atmos.d` command must not graft that command's subcommands on. +// +// Composing multiple `.atmos.d`/import fragments of the SAME project still goes +// through the lenient mergeCommandArrays (see mergeConfigFile and +// processConfigImportsAndReapply's default/imported-commands step): a fragment +// that omits `commands:` there is expected to preserve another fragment's +// subcommands, which is how "Split commands across files" (see the +// atmos-migration skill) is meant to work. Only the boundary between discovered +// defaults and the directory's own complete atmos.yaml uses strict mode. +func mergeMainCommandArray(first, second interface{}) []interface{} { + return mergeNormalizedCommandArrays(normalizeCommandArray(first), normalizeCommandArray(second), true) } func normalizeCommandArray(commands interface{}) []interface{} { @@ -2116,7 +2142,7 @@ func normalizeCommandArray(commands interface{}) []interface{} { if normalized == nil { continue } - result = mergeNormalizedCommandArrays(result, []interface{}{normalized}) + result = mergeNormalizedCommandArrays(result, []interface{}{normalized}, false) } return result @@ -2184,7 +2210,14 @@ func normalizeCommandDefinition(cmd interface{}) interface{} { return current } -func mergeNormalizedCommandArrays(first, second []interface{}) []interface{} { +// mergeNormalizedCommandArrays merges two already-normalized command arrays by +// name, later entries overriding earlier ones. The strict flag is forwarded to +// mergeCommandDefinitions for every name collision -- see mergeMainCommandArray +// for what it changes and why. Every current caller passes an already-deduped +// `first`/`second` (normalizeCommandArray or a prior merge step already +// resolved same-source collisions), so strict only takes effect at the +// first-vs-second boundary, which is the boundary it is meant for. +func mergeNormalizedCommandArrays(first, second []interface{}, strict bool) []interface{} { // Build a map of commands by name, with later entries overriding earlier ones. commandMap := make(map[string]interface{}) var orderedNames []string @@ -2205,7 +2238,7 @@ func mergeNormalizedCommandArrays(first, second []interface{}) []interface{} { // Store or merge the command. Nested command groups are merged // recursively so imports can extend a shared command tree. if existing, exists := commandMap[name]; exists { - commandMap[name] = mergeCommandDefinitions(existing, cmd) + commandMap[name] = mergeCommandDefinitions(existing, cmd, strict) } else { commandMap[name] = cmd } @@ -2229,7 +2262,18 @@ func mergeNormalizedCommandArrays(first, second []interface{}) []interface{} { return result } -func mergeCommandDefinitions(first, second interface{}) interface{} { +// mergeCommandDefinitions deep-merges two command definitions that share a +// name, second's fields taking precedence. When strict is true and second +// does not itself define a nested `commands:` key, second is treated as a +// complete, standalone leaf command: it replaces first outright instead of +// being field-merged with it. A field-by-field merge would otherwise leave +// dangling references into first's dropped `commands:` tree (for example a +// `default:` naming a subcommand that no longer exists once `commands:` is +// gone), so strict mode does not partially merge here at all -- see +// mergeMainCommandArray for why this distinction exists. When second does +// define `commands:`, or strict is false, the normal field-by-field merge +// runs and nested `commands:` arrays are deep-merged if both sides have one. +func mergeCommandDefinitions(first, second interface{}, strict bool) interface{} { firstMap, ok := first.(map[string]interface{}) if !ok { return second @@ -2239,6 +2283,10 @@ func mergeCommandDefinitions(first, second interface{}) interface{} { return second } + if _, secondHasCommands := secondMap[commandsKey]; strict && !secondHasCommands { + return second + } + // Start with the first map's size. Avoid len(first)+len(second) here // because CodeQL correctly treats unchecked allocation arithmetic as a // potential overflow, and the map will grow if secondMap adds new keys. @@ -2246,6 +2294,7 @@ func mergeCommandDefinitions(first, second interface{}) interface{} { for key, value := range firstMap { merged[key] = value } + for key, value := range secondMap { if key == commandsKey { if existing, ok := merged[key]; ok { diff --git a/pkg/datafetcher/schema/atmos/config/1.0.json b/pkg/datafetcher/schema/atmos/config/1.0.json index ecf175f2ac3..7b18eb26957 100644 --- a/pkg/datafetcher/schema/atmos/config/1.0.json +++ b/pkg/datafetcher/schema/atmos/config/1.0.json @@ -3438,7 +3438,7 @@ "$ref": "#/$defs/yamlFunction" } ], - "description": "Internal hides this command from help/list output while leaving it fully invocable,\nmirroring go-task's `internal: true` (maps to Cobra's Command.Hidden)." + "description": "Internal excludes the command from `atmos --help` / `atmos \u003cgroup\u003e --help` subcommand\nlistings, shell-completion suggestions, and the AI `atmos_list_commands` tool, while\nleaving it fully runnable: `atmos \u003cname\u003e ...` still executes it directly, and\n`atmos \u003cname\u003e --help` still renders its own help when invoked explicitly. Use for\nhelper commands meant to be called by other commands or run manually for debugging,\nanalogous to Just's `[private]` recipes or Task's `internal: true` tasks (maps to Cobra's\nCommand.Hidden)." } }, "type": "object", diff --git a/pkg/io/line_prefix_writer.go b/pkg/io/line_prefix_writer.go index dd422aa8e94..e0510d28b58 100644 --- a/pkg/io/line_prefix_writer.go +++ b/pkg/io/line_prefix_writer.go @@ -27,6 +27,12 @@ type LinePrefixWriter struct { prefix string w stdio.Writer buffer []byte + // pending holds the already-prefixed, already-normalized suffix of a line + // that a prior write to w left unwritten (a short write or a write error + // after n > 0 bytes). It is retried byte-for-byte before any new line, so + // bytes w already accepted are never re-sent and the line's prefix is + // never re-applied. + pending []byte } // NewLinePrefixWriter creates a writer that prefixes every rendered line with @@ -66,11 +72,10 @@ func (w *LinePrefixWriter) Write(p []byte) (int, error) { w.buffer = append(w.buffer, p...) - // Hold writeMu across every line this call flushes, not per individual writeLine -- one - // upstream Write can resolve into several complete lines (e.g. a "\r"-separated progress - // update followed later by its completion), and releasing the shared lock between them let a - // concurrently-writing sibling node's entire output interleave in the gap, splitting what - // should read as one contiguous burst from this node. + // Hold writeMu for the whole flush so that every line produced by this + // single Write call reaches the shared writer as one contiguous block. + // Locking per-line let a concurrent node's writer interleave a line in + // between two lines emitted from the same Write call. w.writeMu.Lock() defer w.writeMu.Unlock() if err := w.flushCompleteLinesLocked(); err != nil { @@ -79,21 +84,20 @@ func (w *LinePrefixWriter) Write(p []byte) (int, error) { return len(p), nil } -// Flush writes any trailing partial line. +// Flush writes any trailing partial line, plus any complete lines still buffered. func (w *LinePrefixWriter) Flush() error { defer perf.Track(nil, "io.LinePrefixWriter.Flush")() w.mu.Lock() defer w.mu.Unlock() - if len(w.buffer) == 0 { + if len(w.buffer) == 0 && len(w.pending) == 0 { return nil } - // See Write: writeMu is held across the whole flush (complete lines plus the trailing - // partial one) so this node's output can't be split by a concurrently-writing sibling. w.writeMu.Lock() defer w.writeMu.Unlock() + if err := w.flushCompleteLinesLocked(); err != nil { return err } @@ -101,15 +105,20 @@ func (w *LinePrefixWriter) Flush() error { return nil } line := append([]byte(nil), w.buffer...) - if err := w.writeLine(line); err != nil { - return err - } w.buffer = w.buffer[:0] - return nil + return w.writeLineLocked(line) } -// flushCompleteLinesLocked writes buffered complete lines while w.mu and w.writeMu are held. +// flushCompleteLinesLocked writes any pending suffix left over from a prior +// short or failed write, then any buffered complete lines, while w.mu and +// w.writeMu are held. It leaves any trailing partial line buffered for a +// later Write or Flush. func (w *LinePrefixWriter) flushCompleteLinesLocked() error { + if len(w.pending) > 0 { + if err := w.writePendingLocked(); err != nil { + return err + } + } for { idx := lineEndIndex(w.buffer) if idx < 0 { @@ -120,31 +129,50 @@ func (w *LinePrefixWriter) flushCompleteLinesLocked() error { end++ } line := append([]byte(nil), w.buffer[:end]...) - if err := w.writeLine(line); err != nil { + w.buffer = w.buffer[end:] + if err := w.writeLineLocked(line); err != nil { return err } - w.buffer = w.buffer[end:] } } -// writeLine writes one already-delimited line with the configured prefix. Callers (Write, Flush) -// must already hold writeMu -- this method does not lock it itself, so multiple lines from one -// flush can be written as a single atomic burst relative to other writers sharing that lock. -func (w *LinePrefixWriter) writeLine(line []byte) error { +// writeLineLocked writes one already-delimited raw line with the configured +// prefix. If the underlying write is short or fails after n > 0 bytes, the +// unwritten encoded suffix is kept in w.pending (not w.buffer) so a later +// call retries exactly those bytes, without re-applying the prefix or +// resending bytes w already accepted. Callers must hold w.writeMu. +func (w *LinePrefixWriter) writeLineLocked(line []byte) error { if w.w == nil { return nil } - line = bytes.ReplaceAll(line, crlfBytes, lfBytes) - line = bytes.ReplaceAll(line, crBytes, lfBytes) + normalized := bytes.ReplaceAll(line, crlfBytes, lfBytes) + normalized = bytes.ReplaceAll(normalized, crBytes, lfBytes) if w.prefix == "" { - _, err := w.w.Write(line) - return err + w.pending = normalized + } else { + w.pending = append([]byte(w.prefix), normalized...) } + return w.writePendingLocked() +} - _, err := stdio.WriteString(w.w, w.prefix+string(line)) - return err +// writePendingLocked writes w.pending to the underlying writer, retaining +// only the unwritten suffix if the write is short or fails. A nil-error +// short write (n < len(w.pending) with err == nil) is treated as +// stdio.ErrShortWrite so callers still see and retry it. The caller must +// already hold w.writeMu. +func (w *LinePrefixWriter) writePendingLocked() error { + n, err := w.w.Write(w.pending) + if err == nil && n < len(w.pending) { + err = stdio.ErrShortWrite + } + if err != nil { + w.pending = append([]byte(nil), w.pending[n:]...) + return err + } + w.pending = nil + return nil } // lineEndIndex returns the first complete line-ending byte position or -1 when absent. diff --git a/pkg/io/line_prefix_writer_test.go b/pkg/io/line_prefix_writer_test.go index fa245157787..2bbcb40a66a 100644 --- a/pkg/io/line_prefix_writer_test.go +++ b/pkg/io/line_prefix_writer_test.go @@ -4,6 +4,7 @@ import ( "bytes" "errors" "fmt" + stdio "io" "strings" "sync" "testing" @@ -226,6 +227,31 @@ func TestLinePrefixWriterKeepsBufferedOutputAfterWriteErrors(t *testing.T) { require.Equal(t, "[node] partial", target.out.String()) } +func TestLinePrefixWriterRetriesNilErrorShortWriteWithoutDuplicating(t *testing.T) { + target := &linePrefixPartialWriter{acceptOnFirst: 6} + writer := NewLinePrefixWriter("node", target, nil) + + _, err := writer.Write([]byte("line\n")) + require.ErrorIs(t, err, stdio.ErrShortWrite) + require.Equal(t, "[node]", target.out.String()) + + require.NoError(t, writer.Flush()) + require.Equal(t, "[node] line\n", target.out.String()) +} + +func TestLinePrefixWriterRetriesPartialFailedWriteWithoutDuplicating(t *testing.T) { + expectedErr := errors.New("write failed") + target := &linePrefixPartialWriter{acceptOnFirst: 6, errOnFirst: expectedErr} + writer := NewLinePrefixWriter("node", target, nil) + + _, err := writer.Write([]byte("line\n")) + require.ErrorIs(t, err, expectedErr) + require.Equal(t, "[node]", target.out.String()) + + require.NoError(t, writer.Flush()) + require.Equal(t, "[node] line\n", target.out.String()) +} + type linePrefixErrorWriter struct { err error } @@ -247,3 +273,27 @@ func (w *linePrefixFailOnceWriter) Write(p []byte) (int, error) { } return w.out.Write(p) } + +// linePrefixPartialWriter accepts only acceptOnFirst bytes of the first Write +// call (optionally paired with errOnFirst), then accepts everything on later +// calls. It simulates a real io.Writer's short-write and partial-failure +// behavior, where n > 0 bytes reach the underlying sink before an error. +type linePrefixPartialWriter struct { + acceptOnFirst int + errOnFirst error + failed bool + out bytes.Buffer +} + +func (w *linePrefixPartialWriter) Write(p []byte) (int, error) { + if !w.failed { + w.failed = true + n := w.acceptOnFirst + if n > len(p) { + n = len(p) + } + w.out.Write(p[:n]) + return n, w.errOnFirst + } + return w.out.Write(p) +} diff --git a/pkg/schema/command.go b/pkg/schema/command.go index aad18e53929..eae8dcd5a1d 100644 --- a/pkg/schema/command.go +++ b/pkg/schema/command.go @@ -73,8 +73,13 @@ type Command struct { // top-level `command_aliases:` map (CommandAliases), which redirects to a possibly-unrelated // command via a subprocess re-exec. Aliases []string `yaml:"aliases,omitempty" json:"aliases,omitempty" mapstructure:"aliases"` - // Internal hides this command from help/list output while leaving it fully invocable, - // mirroring go-task's `internal: true` (maps to Cobra's Command.Hidden). + // Internal excludes the command from `atmos --help` / `atmos --help` subcommand + // listings, shell-completion suggestions, and the AI `atmos_list_commands` tool, while + // leaving it fully runnable: `atmos ...` still executes it directly, and + // `atmos --help` still renders its own help when invoked explicitly. Use for + // helper commands meant to be called by other commands or run manually for debugging, + // analogous to Just's `[private]` recipes or Task's `internal: true` tasks (maps to Cobra's + // Command.Hidden). Internal bool `yaml:"internal,omitempty" json:"internal,omitempty" mapstructure:"internal"` } diff --git a/website/docs/cli/configuration/commands/command/commands.mdx b/website/docs/cli/configuration/commands/command/commands.mdx index 1eb3b5d848d..dad77dd07c0 100644 --- a/website/docs/cli/configuration/commands/command/commands.mdx +++ b/website/docs/cli/configuration/commands/command/commands.mdx @@ -29,6 +29,36 @@ Run a nested subcommand by chaining the names: atmos list stacks ``` +## Loading Environment Variables from a Dotenv File + +A command's `env:` field accepts [`!include`](/functions/yaml/include) to load values directly +from a `.env` file, instead of listing each variable inline: + +```yaml +commands: + - name: deploy + description: Deploy the application + env: !include .env + steps: + - atmos terraform apply app -s {{ .Arguments.stack }} +``` + +Use a YAML merge key to combine dotenv values with inline overrides: + +```yaml +commands: + - name: deploy + description: Deploy the application + env: + <<: !include .env + AWS_REGION: us-east-2 + steps: + - atmos terraform apply app -s {{ .Arguments.stack }} +``` + +See [Environment Variables](/cli/configuration/env#including-dotenv-files) for the full dotenv +`!include` behavior, including merging multiple files and precedence rules. + ## Path-Based Names When intermediate command levels only exist to spell out the command path, you can put the full path in `name` instead of writing nested `commands` arrays: diff --git a/website/docs/cli/configuration/commands/command/index.mdx b/website/docs/cli/configuration/commands/command/index.mdx index f07bbc168bc..0ebd2a8a0c7 100644 --- a/website/docs/cli/configuration/commands/command/index.mdx +++ b/website/docs/cli/configuration/commands/command/index.mdx @@ -48,7 +48,7 @@ atmos deploy vpc -s plat-ue2-prod
`name`
-
Required. The command name as typed after `atmos` (for example, `deploy`). For nested commands, this is the subcommand name. Names appear in `atmos help`.
+
Required. The command name as typed after `atmos` (for example, `deploy`). For nested commands, this is the subcommand name. Names appear in `atmos help`, unless the command is marked `internal`.
`description`
Required. Help text shown in `atmos help` and `atmos --help`. Supports multi-line YAML for usage examples.
@@ -90,7 +90,7 @@ atmos deploy vpc -s plat-ue2-prod
Alternative names this command is also invocable under, registered natively and in-process (the same underlying command object, just with extra names) — distinct from the top-level [`aliases`](/cli/configuration/aliases) config, which redirects to a possibly-unrelated command via a subprocess. See [Aliases](#aliases).
`internal`
-
Set to `true` to hide this command from `atmos help` and `atmos list` output while leaving it fully invocable — useful for a command meant to be called only as a [dependency](/cli/configuration/commands/dependencies) of another command, not run directly. Defaults to `false`.
+
Set to `true` to exclude the command from `atmos --help` listings and completion suggestions while keeping it directly runnable (`atmos ...`) and its own `--help` output intact — useful for a command meant to be called only as a [dependency](/cli/configuration/commands/dependencies) of another command, not run directly. Defaults to `false`. List a parent's internal subcommands with `atmos --help=hidden`.
## Aliases diff --git a/website/docs/migration/justfile.mdx b/website/docs/migration/justfile.mdx new file mode 100644 index 00000000000..d4ab3086e85 --- /dev/null +++ b/website/docs/migration/justfile.mdx @@ -0,0 +1,259 @@ +--- +title: Migrating from Justfiles +sidebar_position: 5 +sidebar_label: From Justfiles +description: How to migrate from Justfiles to Atmos +id: justfile +--- +import FAQItem from '@site/src/components/FAQItem' +import File from '@site/src/components/File' +import Intro from '@site/src/components/Intro' +import TaskList from '@site/src/components/TaskList' +import Terminal from '@site/src/components/Terminal' +import Tabs from '@theme/Tabs' +import TabItem from '@theme/TabItem' + + +A Justfile gives your team an ad-hoc way to run build, test, and deploy tasks, through named +recipes. Atmos gives you the same capability as a native, documented feature: custom commands +and workflows. Just's named parameters with default values map more directly onto Atmos flags +and arguments than Make's or Task's syntax does, so this migration is direct. Your Terraform +code and scripts do not need to change. Atmos replaces only the Justfile's recipes. + + +:::tip Using an AI Coding Assistant? +Install the `atmos-migration` skill so Claude Code, Cursor, GitHub Copilot, and other AI coding +assistants can apply this guide directly to your repository: + +```shell +atmos ai skill install atmos-migration +``` + +See [AI Agent Skills](/ai/skills) for details. +::: + +## Why Some Teams Choose Just Over Make + +Just recipes do not need tab indentation, the way Make recipes do. A recipe's parameters are +named and typed, instead of set through implicit `$(VAR)` substitution. Neither point changes the +migration to Atmos. A custom command does not care how the source tool formatted its recipes. It +only cares what the recipe does. + +## Key Concepts at a Glance + +| Justfile concept | Atmos equivalent | +|----------------------------------------------------------|-----------------------------------------------------------------| +| `recipe param='default':` | Command `flags:`/`arguments:` with a `default:` | +| `recipe: dep1 dep2` (recipe dependency) | Steps in order, or a `parallel` step with `needs:` | +| `# comment` above a recipe | Command `description:` (replaces `just --list`) | +| `export VAR := value` | Command or step `env:` map | +| `set dotenv-load` | `env: !include .env` on the command, workflow, or step | +| `set shell := [...]` | Per-step `type: script` with `interpreter:` | +| `[private]` recipe | Command `internal: true` | +| `{{ }}` interpolation | `{{ .Flags. }}` / `{{ .Arguments. }}` (a different tool) | + +## Before and After + + + + ```just + set dotenv-load := true + + export AWS_REGION := "us-east-1" + + # Build the deployable artifact + build: + go build -o bin/handler ./cmd/handler + + # Run tests (builds first) + test: build + go test ./... + + # Deploy to the given environment (defaults to dev) + deploy env='dev': build test + cd terraform && terraform apply -var-file=envs/{{env}}.tfvars + + [private] + _clean: + rm -rf bin/ + ``` + + + ```yaml + # AWS_REGION (and anything else in .env) applies to every Atmos operation, not just one + # command's own steps -- this matches the source Justfile's `set dotenv-load`/`export`, + # which apply to every recipe. A command-level `env:` map only reaches that command's own + # steps, so `test` invoking `atmos build` (a separate process) and `deploy` invoking + # `atmos test` would not inherit it. + env: + <<: !include .env + AWS_REGION: us-east-1 + + commands: + - name: build + description: Build the deployable artifact + steps: + - type: shell + command: go build -o bin/handler ./cmd/handler + + - name: test + description: Run tests (builds first) + steps: + - type: atmos + command: build + - type: shell + command: go test ./... + + - name: deploy + description: Deploy to the given environment (defaults to dev) + flags: + - name: env + shorthand: e + default: "dev" + steps: + - type: atmos + command: test + - type: atmos + command: terraform apply infra -s {{ .Flags.env }} + ``` + + + +`infra` is a placeholder Atmos component name, not the `terraform` verb repeated. Move the old +`terraform/` directory's `.tf` files to `components/terraform/infra/` (the default +`components.terraform.base_path` is `components/terraform`), then swap `infra` for whatever you +actually name the component. Alternatively, keep the existing `terraform/` directory where it is: +set `components.terraform.base_path: "."` and add `metadata.component: terraform` on the `infra` +stack component -- `metadata.component` points the stack component at the physical directory, so +no files need to move. + +`-s {{ .Flags.env }}` only selects *which stack* runs, the way `terraform.apply` picks a stack by +name (`dev`, `staging`, `prod`); it does not, by itself, load that environment's Terraform +variables the way the source `-var-file=envs/{{env}}.tfvars` did. Bring the per-environment +`.tfvars` files in through each stack file instead, one per environment (`stacks/dev.yaml`, +`stacks/staging.yaml`, `stacks/prod.yaml`), each pointing at its own file: + + +```yaml +components: + terraform: + infra: + vars: !include ../components/terraform/infra/envs/dev.tfvars +``` + + +See [Migrating from Native Terraform](/migration/native-terraform) for the full `.tfvars`/stack +mapping. + +## Named Parameters Become Flags and Arguments + +Just's `recipe param='default':` syntax maps directly to Atmos `flags:` (or `arguments:` for a +positional value), each with a matching `default:`. Read the value inside a step as +`{{ .Flags.env }}`. Do not use Just's own `{{env}}` syntax. It is a different template engine +that runs at a different time. + + +```yaml +commands: + - name: deploy + description: Deploy to the given environment + flags: + - name: env + shorthand: e + default: "dev" + steps: + - type: atmos + command: terraform apply infra -s {{ .Flags.env }} +``` + + + +```bash +atmos deploy --env staging +``` + + +## Recipe Dependencies Become Steps + +A recipe dependency, such as `deploy: build test`, gets the same treatment as a Makefile target +chain. See [Migrating from Makefiles](/migration/makefile) for the general method, including when +to use a `parallel` step instead of plain steps in order. + +## `[private]` Recipes and Environment Settings + +Set `internal: true` on the custom command. The command still runs (`atmos ...`, or as a +`default:` target, or from another command's steps), but it's excluded from `atmos --help` +listings and completion suggestions — matching a `[private]` recipe's behavior in `just --list`. +Reserve inlining the logic into a caller's step for a helper that's genuinely single-caller and +has no reason to be invoked on its own. + +Atmos loads `.env` files natively with `env: !include .env`. Atmos parses the dotenv format +(including `export VAR=value`, comments, quoting, and `${VAR}` expansion) and merges the result +into the command, workflow, or step `env:` map: + + +```yaml +commands: + - name: build + description: Build the deployable artifact + env: !include .env + steps: + - type: shell + command: go build -o bin/handler ./cmd/handler +``` + + +If the `.env` file holds secrets rather than plain configuration, prefer Atmos's store or secrets +integration instead of a plaintext `.env` file. + +## What You Gain + +- **A larger set of built-in step types.** A Justfile recipe runs shell commands only. Atmos + adds step types for orchestration (`parallel`, `matrix`, `wait`), user prompts (`confirm`, + `choose`, `input`), and output (`table`, `markdown`, `toast`), with more than 30 types in + total. See [Step Types](/workflows/steps/type) for the full list. +- **Interactive commands.** A step such as `confirm` or `choose` can pause a command and ask the + user a question. A Justfile recipe cannot do this without a custom shell script. +- **Automatic tool installation.** A command can list the tools it needs under + `dependencies.tools`. Atmos installs the correct version before the command runs. See + [Toolchain Configuration](/cli/configuration/toolchain). A Justfile has no built-in match for + this. +- **One interface across every command.** `just --list` shows recipe names and parameters. + Atmos adds `atmos --help` with the same flag and argument format for every command, + plus typed flags, shorthands, and defaults enforced by Atmos itself. + +## Migration Checklist + + +- [ ] List every Justfile recipe. Mark each one as independent, or part of a chain +- [ ] Turn recipe parameters with defaults into command `flags:`/`arguments:` +- [ ] Turn recipe dependencies into steps in order, or into a workflow +- [ ] Turn `export VAR := value` into `env:` maps +- [ ] Replace `set dotenv-load` with `env: !include .env` (or a store/secrets integration for secret values) +- [ ] Mark `[private]` recipes as `internal: true` custom commands +- [ ] Stop relying on `just --list`. `atmos --help` replaces it + + +## Common Questions + + +Yes. Use `env: !include .env` on the command, workflow, or step. Atmos parses the dotenv file +natively and merges its values into `env:`. Reserve Atmos's store or secrets integration for +values that are actually secret, since a plain `.env` file is not encrypted. + + + +Set `internal: true` on the custom command. It stays fully runnable, including as a `default:` +target or from another command's steps, but disappears from `atmos --help` listings and +completion suggestions. + + + +No. Just's `{{ var }}` and Atmos's `{{ .Flags.var }}` both use Go templates, but they are +different engines. Change each reference to `{{ .Flags. }}` or `{{ .Arguments. }}`. + + + +See [Migrating from Makefiles](/migration/makefile) or +[Migrating from Taskfile.yml](/migration/taskfile). + diff --git a/website/docs/migration/makefile.mdx b/website/docs/migration/makefile.mdx new file mode 100644 index 00000000000..f06f2f31e38 --- /dev/null +++ b/website/docs/migration/makefile.mdx @@ -0,0 +1,339 @@ +--- +title: Migrating from Makefiles +sidebar_position: 4 +sidebar_label: From Makefiles +description: How to migrate from Makefiles to Atmos +id: makefile +--- +import FAQItem from '@site/src/components/FAQItem' +import File from '@site/src/components/File' +import Intro from '@site/src/components/Intro' +import TaskList from '@site/src/components/TaskList' +import Terminal from '@site/src/components/Terminal' +import Tabs from '@theme/Tabs' +import TabItem from '@theme/TabItem' + + +A Makefile gives your team an ad-hoc way to run build, test, and deploy tasks. Atmos gives you +the same capability as a native, documented feature: custom commands and workflows. Your +Terraform code, scripts, and other tools do not need to change, and neither does your Makefile. +Atmos is the front door either way: if you want, a custom command can simply call `make ` +as its one step, making Atmos a thin wrapper around the Makefile you already have, not the other +way around. Moving a target's own recipe logic into native `steps:` is worthwhile, but optional. + + +:::tip Using an AI Coding Assistant? +Install the `atmos-migration` skill so Claude Code, Cursor, GitHub Copilot, and other AI coding +assistants can apply this guide directly to your repository: + +```shell +atmos ai skill install atmos-migration +``` + +See [AI Agent Skills](/ai/skills) for details. +::: + +## Key Concepts at a Glance + +| Makefile concept | Atmos equivalent | +|--------------------------------------------------|------------------------------------------------------------| +| `.PHONY` target, no dependencies | [Custom command](/cli/configuration/commands) | +| `VAR ?= default`, `$(VAR)` | Command `flags:` with a `default:` | +| `target: dep1 dep2` (dependency order) | Steps in order (`make`'s own default is sequential); command [`dependencies.commands`](/cli/configuration/commands/dependencies#named-command-and-workflow-dependencies) only when a prerequisite is shared by more than one target, or the source used `-j` | +| Recipe shell lines (`@`-silenced, tab-indented) | Steps (`type: shell`, `output: none`) | +| `ifeq (...)` conditional | Go template `{{ if }}` in a command, or `when:` in a workflow | +| `$(MAKE) -j`, `$(MAKE) -C dir` | `parallel` / `matrix` steps | +| Self-documenting `help` target (`awk`-parsed) | Free: `atmos --help` and each command's `description:` | + +## Before and After + + + + ```makefile + .PHONY: build test lint clean deploy + + ENV ?= dev + + build: + go build -o bin/handler ./cmd/handler + + test: build + go test ./... + + lint: + golangci-lint run ./... + + clean: + @rm -rf bin/ + + deploy: build test + cd terraform && terraform apply -var-file=envs/$(ENV).tfvars + ``` + + + ```yaml + commands: + - name: build + description: Compile the deployable artifact + steps: + - type: shell + command: go build -o bin/handler ./cmd/handler + + - name: test + description: Run unit tests + dependencies: + commands: [build] + steps: + - type: shell + command: go test ./... + + - name: lint + description: Run static analysis + steps: + - type: shell + command: golangci-lint run ./... + + - name: clean + description: Remove build artifacts + steps: + - type: shell + command: rm -rf bin/ + output: none + + - name: deploy + description: Plan and apply the given environment (default dev) + flags: + - name: env + shorthand: e + default: "dev" + dependencies: + commands: [build, test] + steps: + - type: atmos + command: terraform apply infra -s {{ .Flags.env }} + ``` + + + +`infra` is a placeholder Atmos component name, not the `terraform` verb repeated. Move the old +`terraform/` directory's `.tf` files to `components/terraform/infra/` (the default +`components.terraform.base_path` is `components/terraform`), then swap `infra` for whatever you +actually name the component. + +## Independent Targets Become Custom Commands + +Turn each `.PHONY` target that has no dependency into its own +[custom command](/cli/configuration/commands). In the example above, these are `build`, `test`, +`lint`, and `clean`. Give each command a `description:`. Then `atmos --help` shows the same +information as a hand-written, `awk`-parsed `help` target, so you can delete that target. Turn +`VAR ?= default` into a `flags:` entry with a matching `default:`. + + +```yaml +commands: + - name: build + description: Compile the deployable artifact + steps: + - type: shell + command: go build -o bin/handler ./cmd/handler +``` + + + +```bash +atmos build +``` + + +## Target Chains Become Ordered Custom-Command Steps + +A `type: atmos` step runs the `atmos` binary itself, so it can call any atmos command, a native +verb such as `terraform apply` or another custom command such as `build`. To reuse a command +from another command or workflow step, prefer `type: atmos` with `command: build` over +`type: shell` with `command: atmos build`: `type: atmos` auto-applies `-s ` from the +step's `stack:` field and renders through Atmos's structured step output. + +For a chain such as `deploy: build test`, steps calling each prerequisite in order is the +default-preserving match — GNU Make's own default is to build a target's prerequisites one at a +time, in the order listed, not concurrently (`-j` is required for that). Do not reach for +`dependencies.commands` here just because it exists; it runs concurrently by default, which +changes the order unless you've confirmed the prerequisites are genuinely independent. + +`dependencies.commands` earns its place for a different reason: `make` only builds a shared +prerequisite once per invocation, even when more than one target depends on it — plain steps +lose that property, since each caller's `type: atmos`/`type: shell` step runs it again. +[`dependencies.commands`](/cli/configuration/commands/dependencies#named-command-and-workflow-dependencies) +dedups a dependency shared by more than one command to a single run, the same guarantee `make` +gives you for free, independent of whether you also want the concurrency. If order still matters +between the shared dependents (`test` must still finish before `deploy`'s own steps run, and +`build` before `test`), declare that edge directly on the later command's own +`dependencies.commands`, rather than listing every prerequisite as a flat, sibling list on the +caller — the scheduler then waits for it, concurrency aside: + +```yaml +commands: + - name: build + steps: + - type: shell + command: go build ./... + + - name: test + dependencies: + commands: [build] + steps: + - type: shell + command: go test ./... + + - name: deploy + dependencies: + commands: [build, test] + steps: + - type: atmos + command: terraform apply infra -s dev +``` + +`build` still runs exactly once for the whole `atmos deploy` invocation — `test`'s own edge on +`build` orders it correctly ahead of `test`, and `deploy`'s own steps wait for both to finish. + +If the Makefile's `deploy` target also selects a Terraform environment, for example through +`-var-file=envs/$(ENV).tfvars`, that part is a Terraform migration step, not a Make migration +step. See [Migrating from Native Terraform](/migration/native-terraform) for how the Terraform +side maps to stacks and `.tfvars` files. + +## Parallel and Recursive Make Become `parallel` and `matrix` + +`$(MAKE) -j` runs independent targets at the same time. It maps to a +[`parallel`](/workflows/steps/type/parallel) step. `$(MAKE) -C dir target` runs the same target +across a fixed set of directories. It maps to a [`matrix`](/workflows/steps/type/matrix) step: + + +```yaml +commands: + - name: build-all + description: Build every service + steps: + - name: build-services + type: matrix + matrix: + service: [vpc, eks, rds] + max_concurrency: 4 + steps: + - type: shell + command: atmos build --service {{ .matrix.service }} +``` + + +## File-Timestamp Targets Become `inputs`/`artifacts` + +Make skips a target's recipe when every prerequisite (the target's own `inputs`) is older than +the target file it already produced (the target's own `artifacts`) — comparing mtimes. The +step-level [`inputs.sources`](/workflows/steps/inputs) and +[`artifacts.paths`](/workflows/steps/artifacts) fields are the direct match: + +```yaml +commands: + - name: build + description: Compile the deployable artifact + steps: + - type: shell + command: go build -o bin/handler ./cmd/handler + inputs: + sources: ["cmd/**/*.go"] + artifacts: + paths: ["bin/handler"] +``` + +With no explicit `when:`, declaring `inputs`/`artifacts` on a step is enough — it implicitly means +`when: checksum.changed`, and the step is skipped when the hash of the matched source files +matches the hash recorded after the last successful run. That default is a deliberate upgrade +over Make's own mtime comparison: a fresh `git clone`/CI checkout resets every file's mtime, +which makes Make (and the mtime-based `timestamp.changed` fact) think everything changed even +when it didn't. Use `when: timestamp.changed` instead if you want Make's exact mtime semantics. +See [inputs](/workflows/steps/inputs) for the full field reference. It does not carry over on its +own — add `inputs`/`artifacts` to the migrated step yourself. + +The scope is different, too: Make's freshness check gates the target's *entire* recipe, but +`inputs`/`artifacts` are declared per step — skipping one step does not stop later steps in the +same command from running. If a target's recipe has more than one command line and the freshness +decision must gate all of them together, combine them into a single `shell`/`script` step rather +than spreading `inputs`/`artifacts` across several steps. + +## What Stays the Same + +- **Your Terraform code, scripts, and CLIs.** Atmos wraps them. It does not replace them. +- **Environment variables you already export.** Move them into a command or step `env:` map. +- **The Makefile itself, for as long as you want.** A custom command can call `make ` + directly as its one step — Atmos becomes a consistent facade over the tools you already have, + not a mandatory rewrite of them. Move a target's own recipe logic into native `steps:` only when + you want something `make` doesn't give you: richer step types, freshness checks, or interactive + prompts. + +## What You Gain + +- **One command for discovery.** Run `atmos --help` to list every command (recommended over the + interactive `atmos help`, which pages the same listing). Run `atmos --help` to see its + flags and arguments. A Makefile has no built-in match for this. It needs a hand-written, + `awk`-parsed `help` target instead. +- **One consistent interface.** Every custom command uses the same flag and argument syntax. + A Makefile has no fixed convention. Each target can parse its own variables in its own way. +- **More than 30 built-in step types.** Atmos ships step types for shell commands, orchestration + (`parallel`, `matrix`, `wait`), user prompts (`confirm`, `choose`, `input`), and output + (`table`, `markdown`, `toast`). See [Step Types](/workflows/steps/type) for the full list. A + plain Makefile recipe has none of this. You would write it yourself, in shell. +- **Interactive commands.** A step such as `confirm` or `choose` can pause a command and ask the + user a question. A Makefile recipe cannot do this without a custom shell script. +- **Automatic tool installation.** A command can list the tools it needs under + `dependencies.tools`. Atmos installs the correct version before the command runs. See + [Toolchain Configuration](/cli/configuration/toolchain). A Makefile has no built-in match for + this. The user must install each tool by hand, or write a separate setup script. + +## Migration Checklist + + +- [ ] List every Makefile target. Mark each one as independent, or part of a chain +- [ ] Turn independent targets into custom commands in `atmos.yaml` +- [ ] Turn `VAR ?= default` variables into command `flags:` with defaults +- [ ] Turn dependency chains into steps in order (`make`'s own default) -- reach for `dependencies.commands` only when a prerequisite is shared by more than one target, or the source used `-j` +- [ ] Replace `$(MAKE) -j`/`-C` with `parallel`/`matrix` steps +- [ ] Delete any hand-written `help` target. `atmos --help` replaces it +- [ ] Keep the Makefile for as long as you want -- call `make ` from a step, or migrate the logic into `steps:`, whichever fits +- [ ] Turn file-timestamp targets into step `inputs.sources`/`artifacts.paths` + + +## Common Questions + + +No, and not every target has to move into Atmos either. A custom command can call `make ` +as its one step (`command: make build`), which gives you `atmos build`'s consistent flags, +`atmos --help` discovery, and tool-dependency management as a facade over the Makefile you already +have — the point of migrating is a consistent interface over every tool, not necessarily rewriting +every tool. Move a target's logic into native `steps:` when you want something `make` doesn't +have: richer step types, freshness checks, or interactive prompts. Neither is required, and there +is no fixed deadline for either. + + + +Turn them into step `inputs.sources`/`artifacts.paths` -- with no explicit `when:`, that +implicitly skips the step when nothing has changed since its last successful run. It does not +carry over automatically; you add it to the migrated step yourself. See +[File-Timestamp Targets Become inputs/artifacts](#file-timestamp-targets-become-inputsartifacts). + + + +Yes, through either step type: `type: atmos` with `command: `, or +`type: shell` with `command: atmos `. Prefer `type: atmos` — it auto-applies +`-s ` and gets Atmos's structured step output, which `type: shell` does not. This preserves +`make`'s own sequential default. Reach for `dependencies.commands` instead only when the same +prerequisite is shared by more than one target — it dedups a shared dependency to a single run, +the way `make` already does — or when the source target actually used `-j`. + + + +See [Migrating from Justfiles](/migration/justfile) or +[Migrating from Taskfile.yml](/migration/taskfile). + + + +See [Migrating from Native Terraform](/migration/native-terraform) for that part of the +migration. This guide covers only the `make` orchestration layer. + diff --git a/website/docs/migration/native-terraform.mdx b/website/docs/migration/native-terraform.mdx index 8034f05e710..ec995d7ace1 100644 --- a/website/docs/migration/native-terraform.mdx +++ b/website/docs/migration/native-terraform.mdx @@ -19,6 +19,17 @@ import TabItem from '@theme/TabItem' You're already 90% there. Your Terraform code doesn't need to change. Atmos gives you a documented, conventional way to manage your infrastructure—whether you're using Makefiles, shell scripts, or just raw Terraform commands. +:::tip Using an AI Coding Assistant? +Install the `atmos-migration` skill so Claude Code, Cursor, GitHub Copilot, and other AI coding +assistants can apply this guide directly to your repository: + +```shell +atmos ai skill install atmos-migration +``` + +See [AI Agent Skills](/ai/skills) for details. +::: + ## Why This Guide? Most teams don't use Terraform in isolation. You're probably already using: diff --git a/website/docs/migration/taskfile.mdx b/website/docs/migration/taskfile.mdx new file mode 100644 index 00000000000..d3316737fa0 --- /dev/null +++ b/website/docs/migration/taskfile.mdx @@ -0,0 +1,300 @@ +--- +title: Migrating from Taskfile.yml +sidebar_position: 6 +sidebar_label: From Taskfile (go-task) +description: How to migrate from Taskfile.yml (go-task) to Atmos +id: taskfile +--- +import FAQItem from '@site/src/components/FAQItem' +import File from '@site/src/components/File' +import Intro from '@site/src/components/Intro' +import TaskList from '@site/src/components/TaskList' +import Terminal from '@site/src/components/Terminal' +import Tabs from '@theme/Tabs' +import TabItem from '@theme/TabItem' + + +A Taskfile gives your team an ad-hoc way to run build, test, and deploy tasks, defined in YAML. +Atmos gives you the same capability as a native, documented feature: custom commands and +workflows. Task and Atmos both use declarative YAML, so most of the mapping from one field to +another is direct. Two Task features map to a dedicated field rather than plain steps, covered +below. Your Terraform code and scripts do not need to change. + + +:::tip Using an AI Coding Assistant? +Install the `atmos-migration` skill so Claude Code, Cursor, GitHub Copilot, and other AI coding +assistants can apply this guide directly to your repository: + +```shell +atmos ai skill install atmos-migration +``` + +See [AI Agent Skills](/ai/skills) for details. +::: + +## Key Concepts at a Glance + +| Taskfile.yml concept | Atmos equivalent | +|--------------------------------------------------|------------------------------------------------------------------| +| `desc:` | Command `description:` | +| `cmds:` (list of shell commands) | `steps:` (`type: shell`, or `type: atmos` for any atmos command) | +| `deps:` (runs at the same time by default) | Command [`dependencies.commands`](/cli/configuration/commands/dependencies#named-command-and-workflow-dependencies), concurrent by default | +| `vars:` / `env:` | Command `flags:` (with `default:`) / `env:` map | +| `sources:` / `generates:` (freshness check) | Step [`inputs.sources`](/workflows/steps/inputs) / [`artifacts.paths`](/workflows/steps/artifacts) | +| `includes:` (multi-file composition) | Auto-discovered `atmos.d/*.yaml`, or separate workflow files | +| `internal: true` task | Command `internal: true` | + +## `deps:` Becomes `dependencies.commands` + +Task runs `deps:` at the same time by default. Atmos custom-command and workflow steps, in +contrast, run one after another by default — so a `deps:` entry is not a step and never becomes +one. It maps to the command-level +[`dependencies.commands`](/cli/configuration/commands/dependencies#named-command-and-workflow-dependencies) +field, which resolves through the same DAG scheduler as +[`parallel`](/workflows/steps/type/parallel)/[`matrix`](/workflows/steps/type/matrix) `needs:` and +runs concurrently by default — matching Task's `deps:` behavior directly, not working around it: + + + + ```yaml + tasks: + deploy: + desc: Plan and apply the given environment + deps: [test, lint] + cmds: + - terraform -chdir=terraform apply -var-file=envs/dev.tfvars + ``` + + + ```yaml + commands: + - name: deploy + description: Plan and apply the given environment + dependencies: + commands: [test, lint] + steps: + - type: atmos + command: terraform apply infra -s dev + ``` + + + +`infra` is a placeholder Atmos component name, not the `terraform` verb repeated. Move the old +`terraform/` directory's `.tf` files to `components/terraform/infra/` (the default +`components.terraform.base_path` is `components/terraform`), then swap `infra` for whatever you +actually name the component. + +`dependencies.commands` also matches a behavior Task itself has that a hand-rolled `parallel` +step does not: if two commands both depend on the same one — for example both `test` and `lint` +depending on `build` — Atmos runs `build` exactly once and dedups it, the same as Task's own +`deps:` graph. A `parallel` step calling `atmos build` from two different places would run it +twice. If one dependency itself depends on another (`lint` depends on `build`, and `deploy` +depends on `test` and `lint`), declare that directly on `lint`'s own `dependencies.commands` — the +scheduler resolves the whole transitive graph itself, still deduping `build` to a single run. + +Reach for a `parallel` step instead of `dependencies.commands` only for concurrency inside a +single command's own steps, not between named commands — for example, running several shell +commands side by side that were never their own Task tasks to begin with. + +## `sources:`/`generates:` Becomes `inputs`/`artifacts` + +Task skips a task's `cmds:` when its `sources:` files match its `generates:` outputs, checked by +default with a content hash (Task also supports `method: timestamp` for an mtime-based check). +The step-level [`inputs.sources`](/workflows/steps/inputs) and +[`artifacts.paths`](/workflows/steps/artifacts) fields are the direct match, with the same +checksum-by-default/timestamp-as-an-option choice: + +```yaml +commands: + - name: build + description: Compile the deployable artifact + steps: + - type: shell + command: go build -o bin/handler ./cmd/handler + inputs: + sources: ["cmd/**/*.go"] + artifacts: + paths: ["bin/handler"] +``` + +With no explicit `when:`, declaring `inputs`/`artifacts` on a step is enough — it implicitly means +`when: checksum.changed`, and the step is skipped when the hash of the matched source files +matches the hash recorded after the last successful run. Run the same command twice in a row and +the second run skips `build` entirely; edit a matched source file and the next run executes it +again. See [inputs](/workflows/steps/inputs) for the full field reference, including the +`timestamp.changed` fact for an mtime-based check matching Task's `method: timestamp` mode. +[`require`/`assert`](/workflows/steps/type/require) is a different, older step type — it only +checks that a file, tool, or directory exists, not whether it is fresh, so it does not replace +`inputs`/`artifacts`. + +The scope is different, too: Task's `sources:`/`generates:` gates the task's *entire* `cmds:` +list, but `inputs`/`artifacts` are declared per step — skipping one step does not stop later +steps in the same command from running. If a task has more than one `cmds:` entry and the +freshness decision must gate all of them together, combine them into a single `shell`/`script` +step rather than spreading `inputs`/`artifacts` across several migrated steps. + +## `internal: true` Tasks + +Set `internal: true` on the custom command. The command still runs (`atmos ...`, or as a +`default:` target, or from another command's steps), but it's excluded from `atmos --help` +listings and completion suggestions — matching an `internal: true` task's behavior in +`task --list`. Reserve inlining the logic into a caller's step for a helper that's genuinely +single-caller and has no reason to be invoked on its own. + +## Before and After + + + + ```yaml + version: '3' + + vars: + ENV: '{{.ENV | default "dev"}}' + + tasks: + build: + desc: Compile the deployable artifact + cmds: + - go build -o bin/handler ./cmd/handler + sources: + - cmd/**/*.go + generates: + - bin/handler + + test: + desc: Run unit tests + deps: [build] + cmds: + - go test ./... + + lint: + desc: Run static analysis + cmds: + - golangci-lint run ./... + ``` + + + ```yaml + commands: + - name: build + description: Compile the deployable artifact + steps: + - type: shell + command: go build -o bin/handler ./cmd/handler + inputs: + sources: ["cmd/**/*.go"] + artifacts: + paths: ["bin/handler"] + + - name: test + description: Run unit tests + dependencies: + commands: [build] + steps: + - type: shell + command: go test ./... + + - name: lint + description: Run static analysis + steps: + - type: shell + command: golangci-lint run ./... + ``` + + + + +```yaml +commands: + - name: deploy + description: Plan and apply the given environment (default dev) + flags: + - name: env + default: "dev" +``` + + + +```bash +atmos deploy --env staging +``` + + +## `includes:` Splits into Commands and Workflows + +Task's `includes:` field joins several Taskfiles into one. Atmos has two matching methods. +Choose the one that fits what you are splitting: + +- To split command definitions across files, put them in files such as `atmos.d/commands.yaml` or + `.atmos.d/commands.yaml`. Atmos auto-discovers `atmos.d/`/`.atmos.d/` in the config directory + (and, as a lower-priority fallback, at the git/worktree root) -- no `import:` entry is needed + for this specific location. Use `import:` only when splitting across a directory Atmos does not + auto-discover. See [Imports](/cli/configuration/imports). +- To split multi-step chains, use separate workflow files. Atmos workflows already live one file + per purpose, under `workflows.base_path`. Unlike `atmos.d`/`.atmos.d`, there is no default for + `workflows.base_path` -- add it explicitly (for example `workflows.base_path: + "stacks/workflows"`) the first time your migration reaches a workflow, or `atmos workflow + ` fails with `'workflows.base_path' must be configured in 'atmos.yaml'`. + +## What You Gain + +- **A larger set of built-in step types.** A Taskfile task runs shell commands only. Atmos adds + step types for orchestration (`parallel`, `matrix`, `wait`), user prompts (`confirm`, `choose`, + `input`), and output (`table`, `markdown`, `toast`), with more than 30 types in total. See + [Step Types](/workflows/steps/type) for the full list. +- **Interactive commands.** A step such as `confirm` or `choose` can pause a command and ask the + user a question. A Taskfile task cannot do this without a custom shell script. +- **Automatic tool installation.** A command can list the tools it needs under + `dependencies.tools`. Atmos installs the correct version before the command runs. See + [Toolchain Configuration](/cli/configuration/toolchain). Task has no built-in match for this. +- **One interface across every command.** `task --list` shows task names and descriptions. + Atmos adds `atmos --help` with the same flag and argument format for every command, + in addition to the CLI-wide `atmos --help`. + +## Migration Checklist + + +- [ ] List every task. Mark each one as independent, or part of a `deps:` chain +- [ ] Turn `desc:`/`cmds:` into command `description:`/`steps:` +- [ ] Turn any `deps:` chain into `dependencies.commands`, to keep Task's concurrent-by-default behavior +- [ ] Turn `vars:`/`env:` into command `flags:` (with defaults) and `env:` maps +- [ ] Turn `sources:`/`generates:` into step `inputs.sources`/`artifacts.paths` -- combine multiple `cmds:` entries into one step if the freshness decision must gate all of them together +- [ ] Split `includes:` into auto-discovered `atmos.d/*.yaml` files, separate workflow files, or both +- [ ] Mark `internal: true` tasks as `internal: true` custom commands + + +## Common Questions + + +Yes, once you migrate it. Turn `sources:`/`generates:` into step `inputs.sources`/`artifacts.paths` +— with no explicit `when:`, that implicitly skips the step when nothing has changed since its last +successful run. Unlike Task's own check, the scope is per step, not per task — if the task has +more than one `cmds:` entry, combine them into a single step so the freshness decision still gates +all of them together. It does not carry over automatically; you add it to the migrated step +yourself. See +[sources:/generates: Becomes inputs/artifacts](#sourcesgenerates-becomes-inputsartifacts). + + + +Command-level `dependencies.commands`. It resolves through the same DAG scheduler as `parallel`/ +`matrix` `needs:` and runs concurrently by default, matching Task's `deps:` behavior directly — +including deduping a dependency shared by more than one command to a single run. Reach for a +`parallel` step only for concurrency inside one command's own steps, not between named commands. + + + +Yes, through either step type: `type: atmos` with `command: `, or +`type: shell` with `command: atmos `. Prefer `type: atmos` — it auto-applies +`-s ` and gets Atmos's structured step output, which `type: shell` does not. + + + +Set `internal: true` on the custom command. It stays fully runnable, including as a `default:` +target or from another command's steps, but disappears from `atmos --help` listings and +completion suggestions. + + + +See [Migrating from Makefiles](/migration/makefile) or +[Migrating from Justfiles](/migration/justfile). + diff --git a/website/docs/migration/terraform-workspaces.mdx b/website/docs/migration/terraform-workspaces.mdx index ef4ea4ea216..98a9524e4f1 100644 --- a/website/docs/migration/terraform-workspaces.mdx +++ b/website/docs/migration/terraform-workspaces.mdx @@ -15,6 +15,17 @@ import TaskList from '@site/src/components/TaskList' Terraform workspaces solve a simple problem: deploy the same code to multiple environments using one state backend. Atmos can adopt that workspace-backed state while moving environment configuration into explicit stack files. +:::tip Using an AI Coding Assistant? +Install the `atmos-migration` skill so Claude Code, Cursor, GitHub Copilot, and other AI coding +assistants can apply this guide directly to your repository: + +```shell +atmos ai skill install atmos-migration +``` + +See [AI Agent Skills](/ai/skills) for details. +::: + ## How Workspaces Fit With Atmos Workspaces are a valid way to organize Terraform state. When migrating to Atmos, the important shift is making configuration explicit in stacks while preserving the state layout that already works for your team. diff --git a/website/sidebars.js b/website/sidebars.js index 6f76be339a9..c04f7208e7d 100644 --- a/website/sidebars.js +++ b/website/sidebars.js @@ -75,7 +75,10 @@ module.exports = { 'migration/native-terraform', 'migration/terragrunt', 'migration/terraform-workspaces', - 'migration/terramate' + 'migration/terramate', + 'migration/makefile', + 'migration/justfile', + 'migration/taskfile' ] }, ]