From 2d3d3c09395f73922eff9a5f8b2df4c71d1d6f71 Mon Sep 17 00:00:00 2001 From: Erik Osterman Date: Thu, 6 Aug 2026 15:25:21 -0500 Subject: [PATCH 01/17] feat(migration): add Makefile, Justfile, and Taskfile migration guides Extend the atmos-migration skill and docs to cover moving task-runner orchestration (Make, Just, Task) to Atmos custom commands and workflows, alongside the existing native-Terraform/Terraform-Workspaces coverage. Co-Authored-By: Claude Sonnet 5 --- agent-skills/AGENTS.md | 2 +- agent-skills/skills/atmos-migration/SKILL.md | 258 +++++++++++------- .../references/from-justfile.md | 162 +++++++++++ .../references/from-makefile.md | 210 ++++++++++++++ .../references/from-taskfile.md | 173 ++++++++++++ website/docs/migration/justfile.mdx | 195 +++++++++++++ website/docs/migration/makefile.mdx | 238 ++++++++++++++++ website/docs/migration/taskfile.mdx | 231 ++++++++++++++++ website/sidebars.js | 5 +- 9 files changed, 1371 insertions(+), 103 deletions(-) create mode 100644 agent-skills/skills/atmos-migration/references/from-justfile.md create mode 100644 agent-skills/skills/atmos-migration/references/from-makefile.md create mode 100644 agent-skills/skills/atmos-migration/references/from-taskfile.md create mode 100644 website/docs/migration/justfile.mdx create mode 100644 website/docs/migration/makefile.mdx create mode 100644 website/docs/migration/taskfile.mdx diff --git a/agent-skills/AGENTS.md b/agent-skills/AGENTS.md index b8cc58390ac..848079b056c 100644 --- a/agent-skills/AGENTS.md +++ b/agent-skills/AGENTS.md @@ -129,7 +129,7 @@ When a task involves Atmos, activate the matching skill for detailed guidance. | AWS ECR: registry login, ECR auth integrations, Docker credential writes | `atmos-aws-ecr` | `agent-skills/skills/atmos-aws-ecr/SKILL.md` | | AWS compliance: Security Hub standards, compliance reports, CIS AWS, PCI DSS, SOC2, HIPAA, NIST | `atmos-aws-compliance` | `agent-skills/skills/atmos-aws-compliance/SKILL.md` | | AWS security: analyze findings, map to components/stacks, structured remediation | `atmos-aws-security` | `agent-skills/skills/atmos-aws-security/SKILL.md` | -| Migrating to Atmos from native Terraform/OpenTofu or Terraform Workspaces: layout, workspace mapping, remote-state bridge | `atmos-migration` | `agent-skills/skills/atmos-migration/SKILL.md` | +| Migrating to Atmos from Terraform, Terraform Workspaces, or a task runner (Make, Just, or Task): layout, workspace mapping, remote-state bridge, task-to-command mapping | `atmos-migration` | `agent-skills/skills/atmos-migration/SKILL.md` | | Atmos Modernization: replace deprecated patterns with current Atmos naming, CI, Pro, auth, secrets, and dependencies | `atmos-modernization` | `agent-skills/skills/atmos-modernization/SKILL.md` | ## Common Patterns diff --git a/agent-skills/skills/atmos-migration/SKILL.md b/agent-skills/skills/atmos-migration/SKILL.md index ae5a18ecfcb..0ba1fa79d32 100644 --- a/agent-skills/skills/atmos-migration/SKILL.md +++ b/agent-skills/skills/atmos-migration/SKILL.md @@ -1,6 +1,6 @@ --- name: atmos-migration -description: "Migrating to Atmos from existing IaC: techniques, tactics, and design patterns for native Terraform and Terraform Workspaces — minimum-disruption paths, file-layout options, workspace mapping, and the remote-state bridge for progressive migration" +description: "This skill helps you migrate a repository to Atmos. It covers native Terraform, Terraform Workspaces, Makefiles, Justfiles, and Taskfiles. It gives minimum-disruption paths, file-layout options, workspace mapping, task-to-command mapping, and the remote-state bridge for a step-by-step migration." metadata: copyright: Copyright Cloud Posse, LLC 2026 version: "1.0.0" @@ -8,64 +8,79 @@ references: - references/from-native-terraform.md - references/from-terraform-workspaces.md - references/remote-state-bridge.md + - references/from-makefile.md + - references/from-justfile.md + - references/from-taskfile.md --- # Migrating to Atmos ## Overview -This skill is the agent's decision guide for migrating an existing Terraform repository to Atmos. -Atmos is designed to **adopt an existing repo without forcing a reorganization** -- the canonical -`components/terraform/` layout is a recommendation, not a requirement. Lead with the minimum -change that delivers value, then escalate only as the user's needs grow. +This skill is a decision guide. Use it to migrate an existing Terraform repository to Atmos. +Atmos can adopt an existing repository without a reorganization. The `components/terraform/` +layout is a recommendation. It is not a requirement. Start with the smallest change that gives +value. Add more only when the user has a real need for it. -For full prose tutorials aimed at end users, link to: +For full tutorials for end users, see: - [Migrating from Native Terraform](https://atmos.tools/migration/native-terraform) - [Migrating from Terraform Workspaces](https://atmos.tools/migration/terraform-workspaces) -- [Migrating from Terragrunt](https://atmos.tools/migration/terragrunt) (not covered by this skill) +- [Migrating from Terragrunt](https://atmos.tools/migration/terragrunt) (this skill does not cover Terragrunt) +- [Migrating from Makefiles](https://atmos.tools/migration/makefile) +- [Migrating from Justfiles](https://atmos.tools/migration/justfile) +- [Migrating from Taskfile.yml](https://atmos.tools/migration/taskfile) ## Terraform or OpenTofu -Everything in this skill applies identically to **Terraform** and **OpenTofu**. Atmos invokes -whichever binary is configured (`components.terraform.command` in `atmos.yaml`, defaulting to -`terraform`). Migration paths, file layouts, and the remote-state bridge are the same regardless -of which binary the user is running. Use the user's terminology -- if they say "OpenTofu," use -"OpenTofu" in your responses. +This skill applies the same way to Terraform and to OpenTofu. Atmos runs the binary set in +`components.terraform.command` in `atmos.yaml`. The default binary is `terraform`. The migration +steps, file layouts, and the remote-state bridge do not change based on the binary. Use the same +word the user uses. If the user says "OpenTofu," write "OpenTofu" in your response. ## Core Principles -These principles override default agent instincts. Internalize them before proposing changes to a -user's repo. - -1. **Migration is opt-in, not all-or-nothing.** Atmos does not require a filesystem - reorganization. Point `base_path` at the user's existing layout (e.g., `base_path: "terraform"` - or `base_path: "."`) when preserving layout lowers adoption risk. The `components/terraform/` - convention is still the best-practice layout for new or fully migrated repos because Atmos - supports multiple toolchains (Terraform, Helmfile, Packer, Ansible); it is not a prerequisite - for adopting Atmos in Terraform-only repos. -2. **Existing `.tfvars` files may be kept during migration.** Use `!include` to pull them into - stacks when the user wants minimal disruption. Converting values into native stack YAML remains - the best-practice end state when the user wants deep-merge inheritance and richer stack - composition, but it can happen progressively. -3. **No Terraform code changes are required.** Don't rewrite providers, backends, or modules - during migration. Atmos generates `backend.tf.json` and `*.auto.tfvars.json` at runtime. -4. **Workspaces are not the enemy.** If the user has `terraform.workspace`-driven environments, - Atmos can map onto their existing state via `metadata.terraform_workspace` and - `workspace_key_prefix`. They do not have to abandon their workspace state to adopt Atmos. -5. **Prefer YAML functions over Gomplate datasources.** When both can express the same thing - (`!include` vs `gomplate.datasources` for files, `!exec` vs templated shell, `!env` vs - `gomplate getenv`, `!store` vs custom datasource URLs), reach for the YAML function first. - YAML functions are type-safe, can't break YAML parsing, produce clear errors, and don't - require enabling Gomplate. See the [atmos-yaml-functions](../atmos-yaml-functions/SKILL.md) - and [atmos-templates](../atmos-templates/SKILL.md) skills for the boundary. -6. **Crawl → walk → run.** Get the user to a working `atmos terraform plan` in 20 minutes; defer - inheritance, catalogs, and multi-account hierarchies until they have a concrete need. +These principles come before your normal instincts. Read them before you propose a change to the +user's repository. + +1. **Migration is optional. It is not all-or-nothing.** Atmos does not require a new file layout. + Point `base_path` at the user's existing layout, for example `base_path: "terraform"` or + `base_path: "."`. This lowers the risk of adoption. The `components/terraform/` layout is + still the best practice for a new repository or a fully migrated repository, because Atmos + supports more than one tool, such as Terraform, Helmfile, Packer, and Ansible. It is not a + requirement for a Terraform-only repository. +2. **You can keep existing `.tfvars` files during migration.** Use `!include` to add them to a + stack. This gives the smallest disruption. Native stack YAML is still the best final state, if + the user wants deep-merge inheritance and richer stack composition. The user can convert to + YAML step by step. +3. **Terraform code changes are not necessary.** Do not rewrite providers, backends, or modules + during migration. Atmos creates the files `backend.tf.json` and `*.auto.tfvars.json` when it + runs. +4. **Terraform workspaces are not a problem.** If the user has environments controlled by + `terraform.workspace`, Atmos can map onto the existing state. Use + `metadata.terraform_workspace` and `workspace_key_prefix` to do this. The user does not need to + delete their workspace state to adopt Atmos. +5. **Use a YAML function before you use a Gomplate datasource.** Some tasks have both options: + `!include` instead of `gomplate.datasources` for files, `!exec` instead of a templated shell + command, `!env` instead of `gomplate getenv`, and `!store` instead of a custom datasource URL. + Choose the YAML function first. A YAML function checks its own types. It cannot break YAML + parsing. It gives a clear error message. It does not require you to turn on Gomplate. See the + skills [atmos-yaml-functions](../atmos-yaml-functions/SKILL.md) and + [atmos-templates](../atmos-templates/SKILL.md) for more detail on this choice. +6. **Start small. Add complexity later.** Get a working `atmos terraform plan` command for the + user in 20 minutes. Wait to add inheritance, catalogs, and multi-account hierarchies until the + user has a real need for them. +7. **Task runners are not a problem.** Atmos custom commands and workflows can replace the + targets, recipes, and tasks that Make, Just, and Task provide. This does not need to happen at + once. A Makefile, Justfile, or Taskfile can stay as a thin wrapper around `atmos` commands + during the migration. This is the same method used for a Terraform-wrapping Makefile, + described in Principle 6. The final state has each leaf target as a custom command and each + target chain as a workflow. Reach this state step by step. ## Decide the Migration Shape First -Before proposing any change, identify which source pattern the user has. Each routes to a -different reference: +Find the user's source pattern before you propose any change. Each pattern points to a different +reference file: | User has... | Use reference | |----------------------------------------------------------------------|--------------------------------------------------| @@ -73,21 +88,42 @@ different reference: | Multiple TF root modules in scattered dirs | [from-native-terraform.md](references/from-native-terraform.md) | | `terraform.workspace`-driven environments with shared state backend | [from-terraform-workspaces.md](references/from-terraform-workspaces.md) | | Need to read outputs from un-migrated TF (legacy or another repo) | [remote-state-bridge.md](references/remote-state-bridge.md) | +| User has a Makefile driving builds/tests/deploys | [from-makefile.md](references/from-makefile.md) | +| User has a Justfile (`just` command runner) | [from-justfile.md](references/from-justfile.md) | +| User has a Taskfile.yml (go-task) | [from-taskfile.md](references/from-taskfile.md) | -The remote-state-bridge pattern is what makes **progressive, component-by-component migration** -possible. Without it, a team is forced into a big-bang cutover. Cover it any time the user has -existing Terraform state they need to read from new Atmos components. +The remote-state-bridge pattern makes progressive migration possible. It lets a team migrate one +component at a time. Without it, the team must migrate everything at once. Use this pattern when +the user has existing Terraform state that a new Atmos component must read. + +### Two Common Problems in Task-Runner Migration + +Two behaviors apply to every task runner. Check them before you open a reference file: + +- **The default order can change.** Task runs `deps:` at the same time by default. Make and Just + run dependencies one after another, unless the user adds a flag such as `make -j`. Atmos steps + always run one after another, unless you put them inside a `parallel` or `matrix` step. Check + the source tool's real default. Do not assume the step order stays the same when you move it to + Atmos. +- **Atmos has no file-freshness cache.** Task's `sources:`/`generates:` fields and non-`.PHONY` + Make targets both skip work when a file has not changed. Atmos steps always run. The + `require`/`assert` step type does not replace this. It only checks that a file exists. It does + not check if the file is new. Tell the user this directly. + +Each reference file has its own "Common Problems" section with the exact field names and steps +for that tool. This section is only a short summary. ## The Minimum-Viable Migration -When a user says "I want to try Atmos on my existing repo," this is the checklist. Do not deviate -unless the user's setup requires it. +Use this checklist when the user wants to try Atmos on an existing repository. Do not change the +order unless the user's setup requires it. 1. **Install Atmos.** See `atmos.tools/install`. -2. **Create `atmos.yaml`** at the repo root, pointing `base_path` and `components.terraform.base_path` - at the user's existing layout. Do not ask them to move files. -3. **Create one stack file** for one environment. Use `!include` of an existing `.tfvars` file so - nothing has to be rewritten: +2. **Create `atmos.yaml`** at the root of the repository. Set `base_path` and + `components.terraform.base_path` to match the user's current layout. Do not ask the user to + move files. +3. **Create one stack file** for one environment. Use `!include` to add an existing `.tfvars` + file. This avoids a rewrite: ```yaml # stacks/dev.yaml import: @@ -97,16 +133,17 @@ unless the user's setup requires it. vpc: vars: !include ../path/to/existing/dev.tfvars ``` -4. **Run `atmos terraform plan vpc -s dev`** and confirm output matches what `terraform plan - -var-file=dev.tfvars` produced before. +4. **Run `atmos terraform plan vpc -s dev`**. Compare the output to the output of + `terraform plan -var-file=dev.tfvars` from before the migration. Confirm the two outputs + match. -A working reference for this shape lives at `examples/native-terraform/` in the Atmos repo. +A working example of this shape is at `examples/native-terraform/` in the Atmos repository. ## File-Layout Options -Pick the layout that matches the user's migration goals. `components/terraform/` is the recommended -Atmos convention, especially for new repos or multi-toolchain projects, but existing layouts can be -preserved when the user wants a lower-disruption migration. +Pick the layout that matches the user's goals. Atmos recommends the `components/terraform/` +layout, especially for a new repository or a multi-tool project. You can keep an existing layout +when the user wants less disruption. | `base_path` | Use when | |------------------------------------------|-------------------------------------------------------------------------| @@ -114,13 +151,13 @@ preserved when the user wants a lower-disruption migration. | `base_path: "terraform"` | TF-only repo with code already in `terraform/`; preserve dir name | | `base_path: "."` + `components.terraform.base_path: "components/terraform"` | Multi-toolchain or new repo; canonical Atmos layout | -For deeper organization patterns (multi-region, multi-account, org hierarchies), defer to the -[atmos-design-patterns](../atmos-design-patterns/SKILL.md) skill. +For more organization patterns, such as multi-region, multi-account, and organization +hierarchies, see the skill [atmos-design-patterns](../atmos-design-patterns/SKILL.md). ## YAML Functions vs Gomplate Datasources -This is a recurring footgun -- agents reach for Gomplate datasources when a YAML function would -be safer and clearer. Prefer the right column: +This is a common mistake: an agent chooses a Gomplate datasource when a YAML function is safer +and clearer. Use the option in the right column: | Goal | Reach for (NOT this) | Use instead | |-------------------------------|---------------------------------------------------|------------------------------------------| @@ -131,58 +168,77 @@ be safer and clearer. Prefer the right column: | Read Terraform output | Templated remote-state datasource | `!terraform.state component output` | | Get current AWS account ID | `gomplate.datasources` AWS plugin | `!aws.account_id` | -YAML functions are type-safe, produce clear errors, work without enabling Gomplate, and don't -require keeping templates valid YAML. Reserve Go templates for control flow (conditionals, loops, -dynamic keys) that YAML functions cannot express. See [atmos-templates](../atmos-templates/SKILL.md) -for when Go templates are the right tool. +A YAML function checks its own types. It gives a clear error message. It works without Gomplate +turned on. It does not require the template text to stay valid YAML. Use a Go template only for +control flow, such as a conditional, a loop, or a dynamic key, that a YAML function cannot +express. See [atmos-templates](../atmos-templates/SKILL.md) for when to use a Go template. ## What Does NOT Need to Change -Lead with this when a user fears a big rewrite. None of the following must change to adopt Atmos: +Tell the user this list first, if they are afraid of a large rewrite. None of these items must +change to adopt Atmos: -- **Terraform code** -- providers, resources, data sources, modules all stay as-is. -- **Module sources** -- `source = "../../modules/foo"` or registry sources keep working. -- **Backend code** -- delete the `backend "s3" {}` block from `.tf` files (Atmos generates - `backend.tf.json`), or leave it and disable backend generation in `atmos.yaml`. Either works. -- **`.tfvars` files** -- consumed via `!include`; convert to YAML later if/when the user wants - deep-merge inheritance. -- **Custom provider configuration** -- providers stay in `.tf` files; pass env vars via stack - `env:` or vars via stack `vars:`. +- **Terraform code.** Providers, resources, data sources, and modules stay the same. +- **Module sources.** A local path, such as `source = "../../modules/foo"`, or a registry + source, keeps working. +- **Backend code.** You can delete the `backend "s3" {}` block from the `.tf` files, because + Atmos creates `backend.tf.json`. Or you can keep the block and turn off backend generation in + `atmos.yaml`. Both methods work. +- **`.tfvars` files.** Atmos reads them through `!include`. Convert them to YAML later, only if + the user wants deep-merge inheritance. +- **Custom provider configuration.** Providers stay in the `.tf` files. Pass environment + variables through stack `env:`. Pass Terraform variables through stack `vars:`. ## When to Escalate to Other Skills -After the minimum migration is working, the user will often ask "how do I do X next?" Route -those questions to the right skill: - -- **Organizing many stacks (orgs, tenants, accounts, regions)** → [atmos-design-patterns](../atmos-design-patterns/SKILL.md) -- **Abstract components, inheritance, catalog patterns** → [atmos-components](../atmos-components/SKILL.md) -- **Deep merging, imports, overrides** → [atmos-stacks](../atmos-stacks/SKILL.md) -- **Vendoring third-party components** → [atmos-vendoring](../atmos-vendoring/SKILL.md) -- **Authentication / provider credentials** → [atmos-auth](../atmos-auth/SKILL.md) -- **Validation policies (OPA, JSON Schema)** → [atmos-validation](../atmos-validation/SKILL.md) -- **CI/CD with affected-detection** → [atmos-ci](../atmos-ci/SKILL.md) -- **Cross-component data sharing via stores** → [atmos-stores](../atmos-stores/SKILL.md) +After the minimum migration works, the user will often ask what to do next. Send each question +to the correct skill: + +- **Organize many stacks**, such as by organization, tenant, account, or region. Use + [atmos-design-patterns](../atmos-design-patterns/SKILL.md). +- **Build abstract components, inheritance, or catalog patterns.** Use + [atmos-components](../atmos-components/SKILL.md). +- **Use deep merging, imports, or overrides.** Use [atmos-stacks](../atmos-stacks/SKILL.md). +- **Vendor third-party components.** Use [atmos-vendoring](../atmos-vendoring/SKILL.md). +- **Set up authentication or provider credentials.** Use [atmos-auth](../atmos-auth/SKILL.md). +- **Add validation policies, such as OPA or JSON Schema.** Use + [atmos-validation](../atmos-validation/SKILL.md). +- **Set up CI/CD with affected-component detection.** Use [atmos-ci](../atmos-ci/SKILL.md). +- **Share data between components through a store.** Use + [atmos-stores](../atmos-stores/SKILL.md). ## Anti-Patterns -Things to push back on if a user (or another agent) proposes them during migration: - -- **"You must move all Terraform into `components/terraform/` before using Atmos."** No -- that is - the recommended layout, not a requirement. Let the user choose between adopting the best-practice - layout now or pointing `base_path` at the existing layout and reorganizing later. -- **"You must rewrite all `.tfvars` as YAML before running Atmos."** No -- native stack YAML is the - best-practice destination for inheritance and composition, but `!include` lets users keep - existing `.tfvars` during a progressive migration. -- **"Delete your workspace state and start over."** No -- bridge it with - `metadata.terraform_workspace` and the remote-state-bridge pattern. -- **"Add Gomplate datasources for everything."** No -- reach for YAML functions first. -- **"Adopt the full multi-account org hierarchy on day one."** No -- start with one stack file. +Push back if a user or another agent proposes one of these methods during migration: + +- **"You must move all Terraform into `components/terraform/` before you use Atmos."** This is + false. That layout is a recommendation, not a requirement. Let the user pick: adopt the + recommended layout now, or point `base_path` at the current layout and reorganize later. +- **"You must rewrite all `.tfvars` files as YAML before you run Atmos."** This is false. Native + stack YAML is the best final format for inheritance and composition. But `!include` lets the + user keep existing `.tfvars` files during a step-by-step migration. +- **"Delete your workspace state and start over."** This is false. Connect the existing state + with `metadata.terraform_workspace` and the remote-state-bridge pattern. +- **"Add a Gomplate datasource for everything."** This is false. Use a YAML function first. +- **"Adopt the full multi-account organization hierarchy on day one."** This is false. Start + with one stack file. +- **"Wrap atmos commands in a Makefile, Justfile, or Taskfile forever."** This is false. A + wrapper is a good bridge while the user builds trust in Atmos. But it is not the final state. + Change each leaf target to a custom command. Change each dependency chain to a workflow, once + the team is ready. ## Additional Resources -- [References/from-native-terraform.md](references/from-native-terraform.md) -- scenario-keyed - recipes for vanilla TF migration -- [References/from-terraform-workspaces.md](references/from-terraform-workspaces.md) -- mapping - workspaces to stacks without losing state -- [References/remote-state-bridge.md](references/remote-state-bridge.md) -- the dummy-component - and abstract-component patterns for reading state from un-migrated or external Terraform +- [References/from-native-terraform.md](references/from-native-terraform.md): steps for a plain + Terraform migration, matched to each shape. +- [References/from-terraform-workspaces.md](references/from-terraform-workspaces.md): how to map + workspaces to stacks without losing state. +- [References/remote-state-bridge.md](references/remote-state-bridge.md): the dummy-component and + abstract-component patterns. Use them to read state from Terraform that is not yet migrated, or + from an external repository. +- [References/from-makefile.md](references/from-makefile.md): steps for a Makefile, matched to + each shape. +- [References/from-justfile.md](references/from-justfile.md): steps for a Justfile, matched to + each shape. +- [References/from-taskfile.md](references/from-taskfile.md): steps for a Taskfile.yml (go-task) + file, matched to each shape. diff --git a/agent-skills/skills/atmos-migration/references/from-justfile.md b/agent-skills/skills/atmos-migration/references/from-justfile.md new file mode 100644 index 00000000000..8dfadbf92b7 --- /dev/null +++ b/agent-skills/skills/atmos-migration/references/from-justfile.md @@ -0,0 +1,162 @@ +# Migrating from Justfiles + +This guide shows how to move Just recipes to Atmos. Find the correct shape for the Justfile +below. Then follow the matching steps. For the full tutorial, see +[atmos.tools/migration/justfile](https://atmos.tools/migration/justfile). + +Just recipe bodies do not require tab indentation, unlike Make. Just's named parameters with +default values map closely to Atmos custom command `flags:` and `arguments:`. This is the +closest match of the three task runners this skill covers. If the Justfile also selects a +Terraform environment, also use [from-native-terraform.md](from-native-terraform.md) for the +Terraform-specific steps. + +## Find the Shape of the Justfile + +| Shape | Steps | +|-----------------------------------------------------------------------|-----------------------------------| +| Recipes with named parameters and default values | [Shape A](#shape-a-recipes-with-named-parameters) | +| Recipe dependencies (`build: test`) | [Shape B](#shape-b-recipe-dependencies) | +| `set dotenv-load`, `export VAR := ...`, `set shell := [...]` | [Shape C](#shape-c-environment-and-shell-settings) | + +## Shape A: Recipes with Named Parameters + +**Before:** +```just +# Build the deployable artifact +build: + go build -o bin/handler ./cmd/handler + +# Run static analysis +lint: + golangci-lint run ./... + +[private] +_clean: + rm -rf bin/ +``` + +**Steps:** + +1. Turn the `# comment` above a recipe into the command's `description:` field. Atmos shows this + text in `atmos help` and `atmos --help`. This replaces `just --list`. +2. Turn a recipe's named parameter with a default value, such as `deploy env='dev':`, into a + command `flags:` entry with a matching `default:` value. Inside a step, read the value as + `{{ .Flags.env }}`. Do not use Just's own `{{env}}` syntax. See + [Common Problems](#-interpolation-looks-like-atmos-templates-but-is-not) below. +3. Do not create a separate command for a `[private]` recipe. There is no `hidden` or `private` + field on Atmos custom commands. Put the recipe's body in a step of the command that calls it. + +```yaml +commands: + - name: build + description: Build the deployable artifact + steps: + - type: shell + command: go build -o bin/handler ./cmd/handler + + - name: lint + description: Run static analysis + steps: + - type: shell + command: golangci-lint run ./... +``` + +## Shape B: Recipe Dependencies + +**Before:** +```just +# Run tests (builds first) +test: build + go test ./... + +# Deploy to the given environment (defaults to dev) +deploy env='dev': build test + cd terraform && terraform apply -var-file=envs/{{env}}.tfvars +``` + +**Steps:** use the same method as Make's dependency chains. See +[from-makefile.md Shape B](from-makefile.md#shape-b-target-chains-with-dependencies). Do not use +a `type: atmos` step to call another custom command. That step type is only for native Atmos +verbs. Call the dependency with a `type: shell` step and `command: atmos build`. + +```yaml +commands: + - name: test + description: Run tests (builds first) + steps: + - type: shell + command: atmos build + - type: shell + command: go test ./... + + - name: deploy + description: Deploy to the given environment (defaults to dev) + flags: + - name: env + shorthand: e + default: "dev" + steps: + - type: shell + command: atmos test + - type: atmos + command: terraform apply infra -s {{ .Flags.env }} +``` + +## Shape C: Environment and Shell Settings + +**Before:** +```just +set dotenv-load := true +set shell := ["bash", "-uc"] + +export AWS_REGION := "us-east-1" +``` + +**Steps:** + +- Turn `export VAR := value` into a command or step `env:` map. +- Atmos has no confirmed built-in feature that matches `set dotenv-load`. Do not claim it does. + Add a step that loads the `.env` file directly, or, if the values are secrets, use Atmos's + store or secrets integration instead of a plain `.env` file. Ask the user which option fits + their case. +- `set shell := [...]` changes the shell for every recipe in the Justfile. Atmos has no matching + command-level setting. Use `type: script` with an explicit `interpreter:` field on the one step + that needs a different interpreter. + +```yaml +commands: + - name: build + description: Build the deployable artifact + env: + AWS_REGION: us-east-1 + steps: + - type: shell + command: go build -o bin/handler ./cmd/handler +``` + +## Common Problems + +### `{{ }}` interpolation looks like Atmos templates but is not + +Just's `{{ var }}` syntax and Atmos's `{{ .Flags.var }}` syntax both use Go templates, but they +run in different tools at different times. Do not copy Just interpolation syntax into Atmos +YAML. Change each reference to the matching `{{ .Flags. }}` or +`{{ .Arguments. }}` form. + +### `[private]` recipes have no equivalent field + +There is no `hidden` or `private` field in the custom command schema. Put a private helper +recipe's logic in a step inside the command or workflow that calls it. Do not create a separate +command just to hide it from `--list` or `--help`. + +### Confirm `set dotenv-load` and `set shell` with the user + +Do not drop either setting without comment. Ask the user if the behavior matters to their +workflow, such as loading secrets or using non-default shell syntax. Then pick the correct +replacement for their case. + +## What Not To Do + +- Do not assume `{{ }}` means the same thing after you move it into Atmos YAML. +- Do not invent a `hidden` or `private` command field. It does not exist. +- Do not drop `dotenv-load` or `set shell` behavior without telling the user. diff --git a/agent-skills/skills/atmos-migration/references/from-makefile.md b/agent-skills/skills/atmos-migration/references/from-makefile.md new file mode 100644 index 00000000000..2cdc20007fb --- /dev/null +++ b/agent-skills/skills/atmos-migration/references/from-makefile.md @@ -0,0 +1,210 @@ +# Migrating from Makefiles + +This guide shows how to move Make targets to Atmos. Find the correct shape for the Makefile +below. Then follow the matching steps. For the full tutorial, see +[atmos.tools/migration/makefile](https://atmos.tools/migration/makefile). + +This guide covers the `make` orchestration layer only: targets, target dependencies, variables, +and conditionals. If the Makefile also selects a Terraform environment through `-var-file` or +per-environment directories, also use +[from-native-terraform.md](from-native-terraform.md). That guide covers the Terraform-specific +steps: backend generation, `.tfvars` files, and workspace mapping. + +## Find the Shape of the Makefile + +| Shape | Steps | +|-----------------------------------------------------------------|-------------------------------------| +| Independent leaf targets (`build`, `test`, `lint`, `clean`) | [Shape A](#shape-a-independent-leaf-targets) | +| Target chains with dependencies (`deploy: build test`) | [Shape B](#shape-b-target-chains-with-dependencies) | +| Recursive or parallel Make (`$(MAKE) -C dir`, `make -j`) | [Shape C](#shape-c-recursive-or-parallel-make) | + +Most Makefiles mix all three shapes. Treat each target on its own. Then combine the results. + +## Shape A: Independent Leaf Targets + +**Before:** +```makefile +.PHONY: build test lint clean help + +help: ## Show available targets + @awk 'BEGIN {FS=":.*##"} /^[a-zA-Z_-]+:.*##/ {printf " %-10s %s\n", $$1, $$2}' $(MAKEFILE_LIST) + +build: ## Compile the deployable artifact + go build -o bin/handler ./cmd/handler + +test: build ## Run unit tests + go test ./... + +lint: ## Run static analysis + golangci-lint run ./... + +clean: ## Remove build artifacts + @rm -rf bin/ +``` + +**Steps:** + +1. Turn each leaf target into a custom command in `atmos.yaml`. If the Makefile is large, put + the commands in a separate file. See + [Split commands across files](#split-commands-across-files) below. +2. Turn the silent-recipe `@` prefix into the step field `output: none`. +3. Delete the `help` target. Atmos generates the same information from each command's + `description:` field. Run `atmos help` or `atmos --help` to see it. +4. When one target depends on another leaf target, such as `test: build`, add a step that runs + the dependency's command. Do not use a `type: atmos` step to call a custom command. That step + type is only for native Atmos verbs, such as `terraform plan`. Use a `type: shell` step with + `command: atmos build` instead. + +```yaml +commands: + - name: build + description: Compile the deployable artifact + steps: + - type: shell + command: go build -o bin/handler ./cmd/handler + + - name: test + description: Run unit tests + steps: + - type: shell + command: atmos build + - type: shell + command: go test ./... + + - name: lint + description: Run static analysis + steps: + - type: shell + command: golangci-lint run ./... + + - name: clean + description: Remove build artifacts + steps: + - type: shell + command: rm -rf bin/ + output: none +``` + +## Shape B: Target Chains with Dependencies + +**Before:** +```makefile +ENV ?= dev + +deploy: build test ## Plan and apply the given ENV (default: dev) + cd terraform && terraform apply -var-file=envs/$(ENV).tfvars +``` + +**Steps:** + +1. Turn `ENV ?= dev` into a command `flags:` entry with `default: "dev"`. +2. Turn the target order (`deploy: build test`) into steps that run in the same order. In this + example, the steps call the Shape A commands, one after the other. +3. Check if the prerequisites are truly independent. In this example, `build` must finish before + `test` runs, but nothing else depends on their order relative to each other. When two + prerequisites do not depend on each other, use a `parallel` step with `needs:` instead of + listing them one after the other. See [Shape C](#shape-c-recursive-or-parallel-make) for the + general `parallel`/`matrix` pattern. +4. Move the Terraform-specific line, `terraform apply -var-file=envs/$(ENV).tfvars`, to + [from-native-terraform.md Shape B](from-native-terraform.md#shape-b-single-dir-with--var-file-from-a-makefile). + That guide shows how the Terraform side maps to stacks. Here, the line becomes a single + `type: atmos` step, because `terraform apply` is a native Atmos verb. +5. Turn `ifeq ($(ENV),prod)` conditionals into a Go template conditional inside a custom command: + `{{ if eq .Flags.env "prod" }}...{{ end }}`. This is the same pattern used for `--verbose` and + other boolean flags. Inside a workflow, use `when: !cel 'stack == "prod"'` on the step + instead. + +```yaml +commands: + - name: deploy + description: Plan and apply the given environment (default dev) + flags: + - name: env + shorthand: e + default: "dev" + steps: + - type: shell + command: atmos build + - type: shell + command: atmos test + - type: atmos + command: terraform apply infra -s {{ .Flags.env }} +``` + +## Shape C: Recursive or Parallel Make + +**Before:** +```makefile +SERVICES := vpc eks rds + +build-all: + for dir in $(SERVICES); do $(MAKE) -C services/$$dir build; done + +build-parallel: + $(MAKE) -j4 build-all +``` + +**Steps:** + +1. Turn `$(MAKE) -j` into a `parallel` step. Use `max_concurrency` to set the fan-out width. +2. Turn `$(MAKE) -C dir target` recursion over a fixed set of directories into a `matrix` step. + Define a `service` axis, and call the per-service command once for each value. + +```yaml +commands: + - name: build-all + description: Build every service + steps: + - name: build-services + type: matrix + matrix: + service: [vpc, eks, rds] + max_concurrency: 4 + steps: + - type: shell + command: atmos build --service {{ .matrix.service }} +``` + +## Common Problems + +### Tabs, `.PHONY`, and file-timestamp caching + +Atmos steps have no tab requirement. Do not confuse `.PHONY` with a caching feature. If a +Makefile target is not `.PHONY` and uses file timestamps to skip work when inputs have not +changed, that caching behavior has no equivalent in Atmos. Atmos steps always run. Tell the user +this directly. Do not imply that the behavior carries over. Task's `sources:`/`generates:` +feature has the same problem. See +[from-taskfile.md](from-taskfile.md#the-sourcesgenerates-gap) for more detail. + +### Silent recipes and command echo + +`@command` suppresses the echo of one command line. It maps to the step field `output: none` on +that one step. It does not mean you should add `output: none` to every step. + +### Split commands across files + +When a Makefile uses `include foo.mk` to split its content across files, split the Atmos config +the same way. Put the extra commands in a file such as `atmos.d/commands.yaml`. Then add that +file to the root config with `import:` in `atmos.yaml`. See +[Imports](https://atmos.tools/cli/configuration/imports). + +### Complex `$(eval)` and `$(call)` macros + +Do not try to convert deeply macro-driven Makefiles into flags and arguments one by one. Put +genuinely dynamic logic in a `shell` or `script` step, or in a script that the step calls. Atmos +custom commands replace task orchestration. They do not replace a general-purpose macro +language. + +## What Not To Do + +- Do not build file-timestamp or `.PHONY` caching as an Atmos feature. It does not exist. State + this directly instead of dropping the behavior without comment. +- Do not turn every private or helper recipe into its own discoverable command. There is no + `hidden` or `private` field on custom commands. Put the helper logic in a step inside the + command or workflow that needs it. +- Do not treat "wrap `atmos` commands in the Makefile" as the final state. It is a valid bridge + during early migration, as shown in Shape B, step 4. Leaf targets should become custom + commands. Target chains should become workflows. +- Do not invent `when:` conditions that check flag values on workflow steps. The `when:` field + checks CEL context values, such as `stack`, `ci`, and `local`. Flag-based conditionals belong + in the custom command's own Go templates. diff --git a/agent-skills/skills/atmos-migration/references/from-taskfile.md b/agent-skills/skills/atmos-migration/references/from-taskfile.md new file mode 100644 index 00000000000..7552ea95400 --- /dev/null +++ b/agent-skills/skills/atmos-migration/references/from-taskfile.md @@ -0,0 +1,173 @@ +# Migrating from Taskfile.yml (go-task) + +This guide shows how to move Task's tasks to Atmos. Find the correct shape for the Taskfile +below. Then follow the matching steps. For the full tutorial, see +[atmos.tools/migration/taskfile](https://atmos.tools/migration/taskfile). + +Task and Atmos both use declarative YAML. This makes the migration mostly mechanical. It is the +simplest of the three task-runner migrations this skill covers. If the Taskfile also selects a +Terraform environment, also use [from-native-terraform.md](from-native-terraform.md) for the +Terraform-specific steps. + +## Find the Shape of the Taskfile + +| Shape | Steps | +|------------------------------------------------------------|--------------------------------------| +| Simple task (`desc`/`cmds`) | [Shape A](#shape-a-simple-tasks) | +| Task with `deps:` (parallel by default) | [Shape B](#shape-b-task-dependencies-parallel-by-default) | +| `vars:`/`env:` and `sources:`/`generates:` | [Shape C](#shape-c-variables-and-up-to-date-checks) | +| `includes:` (multi-file composition) | see [Common Problems](#includes-multi-file-composition) | + +## Shape A: Simple Tasks + +**Before:** +```yaml +version: '3' + +tasks: + build: + desc: Compile the deployable artifact + cmds: + - go build -o bin/handler ./cmd/handler + + lint: + desc: Run static analysis + cmds: + - golangci-lint run ./... +``` + +**Steps:** + +1. Turn `desc:` into the command's `description:` field. +2. Turn each entry in `cmds:` into a `type: shell` step. If the line is a `terraform` or `atmos` + call, use a `type: atmos` step instead, because `type: atmos` is reserved for native Atmos + verbs. + +```yaml +commands: + - name: build + description: Compile the deployable artifact + steps: + - type: shell + command: go build -o bin/handler ./cmd/handler + + - name: lint + description: Run static analysis + steps: + - type: shell + command: golangci-lint run ./... +``` + +## Shape B: Task Dependencies (Parallel by Default) + +**Before:** +```yaml +tasks: + test: + desc: Run unit tests + deps: [build] + cmds: + - go test ./... + + deploy: + desc: Plan and apply the given environment + deps: [test, lint] + cmds: + - terraform -chdir=terraform apply -var-file=envs/dev.tfvars +``` + +This is the most important problem in this migration. Task runs `deps:` at the same time by +default. Atmos custom-command and workflow steps run one after another by default. This is the +opposite default. If you turn `deps: [test, lint]` into two plain steps that run one after +another, the command becomes slower. It also changes what happens when one task fails. To keep +Task's default behavior, put the dependency tasks inside a `parallel` step: + +```yaml +commands: + - name: deploy + description: Plan and apply the given environment + steps: + - name: checks + type: parallel + fail: + mode: wait_all + steps: + - name: test + type: shell + command: atmos test + - name: lint + type: shell + command: atmos lint + - type: atmos + command: terraform apply infra -s dev +``` + +If the Taskfile's `deps:` list needs its own internal order, add `needs:` to the steps inside the +`parallel` block. Do not assume the tasks should run one after another just because that is +Atmos's default for steps outside a `parallel` or `matrix` block. + +## Shape C: Variables and Up-to-Date Checks + +**Before:** +```yaml +vars: + ENV: '{{.ENV | default "dev"}}' + +tasks: + build: + desc: Compile the deployable artifact + cmds: + - go build -o bin/handler ./cmd/handler + sources: + - cmd/**/*.go + generates: + - bin/handler +``` + +**Steps:** + +- Turn `vars: ENV: '{{.ENV | default "dev"}}'` into a command `flags:` entry with + `default: "dev"`. Task's Sprig `default` filter becomes the plain `default:` field. +- Turn `env:` into an `env:` map. The two are almost identical. + +### The `sources`/`generates` gap + +Task skips a task's `cmds:` when its `sources:` files match its `generates:` outputs. It checks +this with a file hash. Atmos steps always run. There is no built-in check for whether a file is +up to date. The `require`/`assert` step type does not fix this. It only checks that a file, tool, +or directory exists. It does not compare hashes or timestamps. + +If the user depends on `sources:`/`generates:` to skip a slow step, such as code generation, tell +them plainly that this behavior does not carry over. Then offer two honest choices: + +1. Accept that the step always runs. This is correct for most fast build steps. +2. Add a hash or timestamp check inside the shell step itself. This is a script the user + maintains. It is not a built-in Atmos feature. + +## Common Problems + +### `includes:` (multi-file composition) + +Task's `includes:` field combines several Taskfiles into one. Atmos has two matching methods. +Pick the one that fits the content being split: + +- To split command definitions across files, put them in files such as + `atmos.d/commands.yaml`. Add each file to the root config with `import:` in `atmos.yaml`. See + [Imports](https://atmos.tools/cli/configuration/imports). +- To split multi-step chains, use separate workflow files. Atmos workflows already live one file + per purpose, under `workflows.base_path` (`stacks/workflows/` by default). No extra step is + needed. + +### `sources`/`generates` has no built-in match + +See [Shape C](#the-sourcesgenerates-gap) above. This is the largest real gap in this migration. +State it directly. Do not gloss over it. + +## What Not To Do + +- Do not drop `sources:`/`generates:` caching without comment. State the change directly. Let + the user decide how, or whether, to replace it. +- Do not turn `deps:` into plain sequential steps without warning the user about the change in + default concurrency. +- Do not describe `require`/`assert` as a freshness or caching check. It only checks that + something exists. diff --git a/website/docs/migration/justfile.mdx b/website/docs/migration/justfile.mdx new file mode 100644 index 00000000000..ceac1538df9 --- /dev/null +++ b/website/docs/migration/justfile.mdx @@ -0,0 +1,195 @@ +--- +title: Migrating from Justfiles +sidebar_position: 5 +sidebar_label: From Justfiles +description: How to migrate from Justfiles to Atmos +id: justfile +--- +import FAQItem from '@site/src/components/FAQItem' +import File from '@site/src/components/File' +import Intro from '@site/src/components/Intro' +import TaskList from '@site/src/components/TaskList' +import Terminal from '@site/src/components/Terminal' +import Tabs from '@theme/Tabs' +import TabItem from '@theme/TabItem' + + +A Justfile gives your team an ad-hoc way to run build, test, and deploy tasks, through named +recipes. Atmos gives you the same capability as a native, documented feature: custom commands +and workflows. Just's named parameters with default values map more directly onto Atmos flags +and arguments than Make's or Task's syntax does, so this migration is direct. Your Terraform +code and scripts do not need to change. Atmos replaces only the Justfile's recipes. + + +## Why Some Teams Choose Just Over Make + +Just recipes do not need tab indentation, the way Make recipes do. A recipe's parameters are +named and typed, instead of set through implicit `$(VAR)` substitution. Neither point changes the +migration to Atmos. A custom command does not care how the source tool formatted its recipes. It +only cares what the recipe does. + +## Key Concepts at a Glance + +| Justfile concept | Atmos equivalent | +|----------------------------------------------------------|-----------------------------------------------------------------| +| `recipe param='default':` | Command `flags:`/`arguments:` with a `default:` | +| `recipe: dep1 dep2` (recipe dependency) | Steps in order, or a `parallel` step with `needs:` | +| `# comment` above a recipe | Command `description:` (replaces `just --list`) | +| `export VAR := value` | Command or step `env:` map | +| `set dotenv-load` | No built-in match. Add a step, or use a store | +| `set shell := [...]` | Per-step `type: script` with `interpreter:` | +| `[private]` recipe | No match. Put the logic in a step instead | +| `{{ }}` interpolation | `{{ .Flags. }}` / `{{ .Arguments. }}` (a different tool) | + +## Before and After + + + + ```just + set dotenv-load := true + + export AWS_REGION := "us-east-1" + + # Build the deployable artifact + build: + go build -o bin/handler ./cmd/handler + + # Run tests (builds first) + test: build + go test ./... + + # Deploy to the given environment (defaults to dev) + deploy env='dev': build test + cd terraform && terraform apply -var-file=envs/{{env}}.tfvars + + [private] + _clean: + rm -rf bin/ + ``` + + + ```yaml + commands: + - name: build + description: Build the deployable artifact + env: + AWS_REGION: us-east-1 + steps: + - type: shell + command: go build -o bin/handler ./cmd/handler + + - name: test + description: Run tests (builds first) + steps: + - type: shell + command: atmos build + - type: shell + command: go test ./... + + - name: deploy + description: Deploy to the given environment (defaults to dev) + flags: + - name: env + shorthand: e + default: "dev" + steps: + - type: shell + command: atmos test + - type: atmos + command: terraform apply infra -s {{ .Flags.env }} + ``` + + + +## Named Parameters Become Flags and Arguments + +Just's `recipe param='default':` syntax maps directly to Atmos `flags:` (or `arguments:` for a +positional value), each with a matching `default:`. Read the value inside a step as +`{{ .Flags.env }}`. Do not use Just's own `{{env}}` syntax. It is a different template engine +that runs at a different time. + + +```yaml +commands: + - name: deploy + description: Deploy to the given environment + flags: + - name: env + shorthand: e + default: "dev" + steps: + - type: atmos + command: terraform apply infra -s {{ .Flags.env }} +``` + + + +```bash +atmos deploy --env staging +``` + + +## Recipe Dependencies Become Steps + +A recipe dependency, such as `deploy: build test`, gets the same treatment as a Makefile target +chain. See [Migrating from Makefiles](/migration/makefile) for the general method, including when +to use a `parallel` step instead of plain steps in order. + +## `[private]` Recipes and Environment Settings + +There is no `hidden` or `private` field on Atmos custom commands. Put a `[private]` recipe's +logic in a step of the command that calls it. Do not create a separate command for it. + +`set dotenv-load` has no confirmed native `.env`-loading feature in Atmos. Do not assume it does. +Either add a step that loads the `.env` file directly, or, if the values are secrets, use Atmos's +store or secrets integration instead. + +## What You Gain + +- **A larger set of built-in step types.** A Justfile recipe runs shell commands only. Atmos + adds step types for orchestration (`parallel`, `matrix`, `wait`), user prompts (`confirm`, + `choose`, `input`), and output (`table`, `markdown`, `toast`), with more than 30 types in + total. See [Step Types](/workflows/steps/type) for the full list. +- **Interactive commands.** A step such as `confirm` or `choose` can pause a command and ask the + user a question. A Justfile recipe cannot do this without a custom shell script. +- **Automatic tool installation.** A command can list the tools it needs under + `dependencies.tools`. Atmos installs the correct version before the command runs. See + [Toolchain Configuration](/cli/configuration/toolchain). A Justfile has no built-in match for + this. +- **One interface across every command.** `just --list` shows recipe names and parameters. + Atmos adds `atmos --help` with the same flag and argument format for every command, + plus typed flags, shorthands, and defaults enforced by Atmos itself. + +## Migration Checklist + + +- [ ] List every Justfile recipe. Mark each one as independent, or part of a chain +- [ ] Turn recipe parameters with defaults into command `flags:`/`arguments:` +- [ ] Turn recipe dependencies into steps in order, or into a workflow +- [ ] Turn `export VAR := value` into `env:` maps +- [ ] Decide how to replace `set dotenv-load`: an explicit step, or a store +- [ ] Put `[private]` recipes into the steps of the commands that call them +- [ ] Stop relying on `just --list`. `atmos help` replaces it + + +## Common Questions + + +Not on its own. Atmos has no confirmed native match for this feature. Add a step that loads the +`.env` file, or use Atmos's store or secrets integration instead of a plain `.env` file. + + + +There is no `hidden` or `private` field on a custom command. Put the helper logic in a step +inside the command or workflow that uses it, instead of exposing it as its own command. + + + +No. Just's `{{ var }}` and Atmos's `{{ .Flags.var }}` both use Go templates, but they are +different engines. Change each reference to `{{ .Flags. }}` or `{{ .Arguments. }}`. + + + +See [Migrating from Makefiles](/migration/makefile) or +[Migrating from Taskfile.yml](/migration/taskfile). + diff --git a/website/docs/migration/makefile.mdx b/website/docs/migration/makefile.mdx new file mode 100644 index 00000000000..420fc702730 --- /dev/null +++ b/website/docs/migration/makefile.mdx @@ -0,0 +1,238 @@ +--- +title: Migrating from Makefiles +sidebar_position: 4 +sidebar_label: From Makefiles +description: How to migrate from Makefiles to Atmos +id: makefile +--- +import FAQItem from '@site/src/components/FAQItem' +import File from '@site/src/components/File' +import Intro from '@site/src/components/Intro' +import TaskList from '@site/src/components/TaskList' +import Terminal from '@site/src/components/Terminal' +import Tabs from '@theme/Tabs' +import TabItem from '@theme/TabItem' + + +A Makefile gives your team an ad-hoc way to run build, test, and deploy tasks. Atmos gives you +the same capability as a native, documented feature: custom commands and workflows. Your +Terraform code, scripts, and other tools do not need to change. Atmos replaces only the +Makefile's targets and recipes. You can keep the Makefile as a thin wrapper around `atmos` +commands while you migrate. + + +## Key Concepts at a Glance + +| Makefile concept | Atmos equivalent | +|--------------------------------------------------|------------------------------------------------------------| +| `.PHONY` target, no dependencies | [Custom command](/cli/configuration/commands) | +| `VAR ?= default`, `$(VAR)` | Command `flags:` with a `default:` | +| `target: dep1 dep2` (dependency order) | Steps in order, or a `parallel` step with `needs:` | +| Recipe shell lines (`@`-silenced, tab-indented) | Steps (`type: shell`, `output: none`) | +| `ifeq (...)` conditional | Go template `{{ if }}` in a command, or `when:` in a workflow | +| `$(MAKE) -j`, `$(MAKE) -C dir` | `parallel` / `matrix` steps | +| Self-documenting `help` target (`awk`-parsed) | Free: `atmos help` and each command's `description:` | + +## Before and After + + + + ```makefile + .PHONY: build test lint clean deploy + + ENV ?= dev + + build: + go build -o bin/handler ./cmd/handler + + test: build + go test ./... + + lint: + golangci-lint run ./... + + clean: + @rm -rf bin/ + + deploy: build test + cd terraform && terraform apply -var-file=envs/$(ENV).tfvars + ``` + + + ```yaml + commands: + - name: build + description: Compile the deployable artifact + steps: + - type: shell + command: go build -o bin/handler ./cmd/handler + + - name: test + description: Run unit tests + steps: + - type: shell + command: atmos build + - type: shell + command: go test ./... + + - name: lint + description: Run static analysis + steps: + - type: shell + command: golangci-lint run ./... + + - name: clean + description: Remove build artifacts + steps: + - type: shell + command: rm -rf bin/ + output: none + + - name: deploy + description: Plan and apply the given environment (default dev) + flags: + - name: env + shorthand: e + default: "dev" + steps: + - type: shell + command: atmos build + - type: shell + command: atmos test + - type: atmos + command: terraform apply infra -s {{ .Flags.env }} + ``` + + + +## Independent Targets Become Custom Commands + +Turn each `.PHONY` target that has no dependency into its own +[custom command](/cli/configuration/commands). In the example above, these are `build`, `test`, +`lint`, and `clean`. Give each command a `description:`. Then `atmos help` shows the same +information as a hand-written, `awk`-parsed `help` target, so you can delete that target. Turn +`VAR ?= default` into a `flags:` entry with a matching `default:`. + + +```yaml +commands: + - name: build + description: Compile the deployable artifact + steps: + - type: shell + command: go build -o bin/handler ./cmd/handler +``` + + + +```bash +atmos build +``` + + +## Target Chains Become Workflows + +A step cannot call another custom command through `type: atmos`. That step type is only for +native Atmos verbs, such as `terraform plan` or `terraform apply`. To reuse a command from +another command or workflow step, call it directly with `type: shell` and +`command: atmos build`. + +If the Makefile's `deploy` target also selects a Terraform environment, for example through +`-var-file=envs/$(ENV).tfvars`, that part is a Terraform migration step, not a Make migration +step. See [Migrating from Native Terraform](/migration/native-terraform) for how the Terraform +side maps to stacks and `.tfvars` files. + +## Parallel and Recursive Make Become `parallel` and `matrix` + +`$(MAKE) -j` runs independent targets at the same time. It maps to a +[`parallel`](/workflows/steps/type/parallel) step. `$(MAKE) -C dir target` runs the same target +across a fixed set of directories. It maps to a [`matrix`](/workflows/steps/type/matrix) step: + + +```yaml +commands: + - name: build-all + description: Build every service + steps: + - name: build-services + type: matrix + matrix: + service: [vpc, eks, rds] + max_concurrency: 4 + steps: + - type: shell + command: atmos build --service {{ .matrix.service }} +``` + + +## What Stays the Same + +- **Your Terraform code, scripts, and CLIs.** Atmos wraps them. It does not replace them. +- **Environment variables you already export.** Move them into a command or step `env:` map. +- **The Makefile itself, for a while.** Keep it as a thin wrapper around `atmos` commands until + your team is ready. Then remove it, once each leaf target and chain has become a command or a + workflow. + +One gap to state directly: a target that skips work based on file timestamps has no match in +Atmos. Atmos steps always run. If this caching matters to your build, plan for it. Do not assume +it carries over on its own. + +## What You Gain + +- **One command for discovery.** Run `atmos help` to list every command. Run + `atmos --help` to see its flags and arguments. A Makefile has no built-in match for + this. It needs a hand-written, `awk`-parsed `help` target instead. +- **One consistent interface.** Every custom command uses the same flag and argument syntax. + A Makefile has no fixed convention. Each target can parse its own variables in its own way. +- **More than 30 built-in step types.** Atmos ships step types for shell commands, orchestration + (`parallel`, `matrix`, `wait`), user prompts (`confirm`, `choose`, `input`), and output + (`table`, `markdown`, `toast`). See [Step Types](/workflows/steps/type) for the full list. A + plain Makefile recipe has none of this. You would write it yourself, in shell. +- **Interactive commands.** A step such as `confirm` or `choose` can pause a command and ask the + user a question. A Makefile recipe cannot do this without a custom shell script. +- **Automatic tool installation.** A command can list the tools it needs under + `dependencies.tools`. Atmos installs the correct version before the command runs. See + [Toolchain Configuration](/cli/configuration/toolchain). A Makefile has no built-in match for + this. The user must install each tool by hand, or write a separate setup script. + +## Migration Checklist + + +- [ ] List every Makefile target. Mark each one as independent, or part of a chain +- [ ] Turn independent targets into custom commands in `atmos.yaml` +- [ ] Turn `VAR ?= default` variables into command `flags:` with defaults +- [ ] Turn dependency chains into steps in order, or into a workflow +- [ ] Replace `$(MAKE) -j`/`-C` with `parallel`/`matrix` steps +- [ ] Delete any hand-written `help` target. `atmos help` replaces it +- [ ] Keep the Makefile as a thin wrapper during the transition, if you want one +- [ ] Confirm no target depends on file-timestamp caching before you remove it + + +## Common Questions + + +No. It can stay as a thin wrapper around `atmos` commands during a step-by-step migration. Each +leaf target should become a custom command over time. Each dependency chain should become a +workflow. There is no fixed deadline for this. + + + +There is no direct match in Atmos. Atmos steps always run. If a target relies on file timestamps +to skip slow work, plan for that change directly. Do not assume the behavior stays the same. + + + +Yes, but not through `type: atmos`. That step type is only for native Atmos verbs, such as +`terraform plan`. To call another custom command, use a `type: shell` step with +`command: atmos `. + + + +See [Migrating from Justfiles](/migration/justfile) or +[Migrating from Taskfile.yml](/migration/taskfile). + + + +See [Migrating from Native Terraform](/migration/native-terraform) for that part of the +migration. This guide covers only the `make` orchestration layer. + diff --git a/website/docs/migration/taskfile.mdx b/website/docs/migration/taskfile.mdx new file mode 100644 index 00000000000..b266e805614 --- /dev/null +++ b/website/docs/migration/taskfile.mdx @@ -0,0 +1,231 @@ +--- +title: Migrating from Taskfile.yml +sidebar_position: 6 +sidebar_label: From Taskfile (go-task) +description: How to migrate from Taskfile.yml (go-task) to Atmos +id: taskfile +--- +import FAQItem from '@site/src/components/FAQItem' +import File from '@site/src/components/File' +import Intro from '@site/src/components/Intro' +import TaskList from '@site/src/components/TaskList' +import Terminal from '@site/src/components/Terminal' +import Tabs from '@theme/Tabs' +import TabItem from '@theme/TabItem' + + +A Taskfile gives your team an ad-hoc way to run build, test, and deploy tasks, defined in YAML. +Atmos gives you the same capability as a native, documented feature: custom commands and +workflows. Task and Atmos both use declarative YAML, so most of the mapping from one field to +another is direct. Two real differences in behavior still need attention first, covered below. +Your Terraform code and scripts do not need to change. + + +## Key Concepts at a Glance + +| Taskfile.yml concept | Atmos equivalent | +|--------------------------------------------------|------------------------------------------------------------------| +| `desc:` | Command `description:` | +| `cmds:` (list of shell commands) | `steps:` (`type: shell`, or `type: atmos` for native verbs) | +| `deps:` (runs at the same time by default) | A `parallel` step. Plain steps in order change the default | +| `vars:` / `env:` | Command `flags:` (with `default:`) / `env:` map | +| `sources:` / `generates:` (freshness check) | No direct match. Steps always run | +| `includes:` (multi-file composition) | `atmos.d/*.yaml` + root `import:`, or separate workflow files | + +## The Parallel-by-Default Difference + +This is the most important behavior change in this migration. Task runs `deps:` at the same time +by default. This is the opposite of Atmos, where custom-command and workflow steps run one after +another by default. If you turn `deps: [test, lint]` into two plain steps in order, the migrated +command becomes slower. The failure behavior also changes. To keep Task's default, put the +dependency tasks inside a [`parallel`](/workflows/steps/type/parallel) step: + + + + ```yaml + tasks: + deploy: + desc: Plan and apply the given environment + deps: [test, lint] + cmds: + - terraform -chdir=terraform apply -var-file=envs/dev.tfvars + ``` + + + ```yaml + commands: + - name: deploy + description: Plan and apply the given environment + steps: + - name: checks + type: parallel + fail: + mode: wait_all + steps: + - name: test + type: shell + command: atmos test + - name: lint + type: shell + command: atmos lint + - type: atmos + command: terraform apply infra -s dev + ``` + + + +If a Taskfile's `deps:` list needs an order between its own entries, add `needs:` to the steps +inside the `parallel` block. Do not default to sequential steps just because that is Atmos's own +default outside a `parallel` or `matrix` block. + +## The `sources`/`generates` Gap + +Task skips a task's `cmds:` when its `sources:` files match its `generates:` outputs, through a +file-hash check. Atmos steps always run. There is no built-in match for this. +[`require`/`assert`](/workflows/steps/type/require) does not replace it. That step type only +checks that a file, tool, or directory exists. It does not check freshness. + +If you rely on `sources:`/`generates:` for caching, for example to skip a slow codegen step, +plan for this directly. It is a real gap, not something Atmos preserves on its own. Two honest +choices: + +1. Accept that the step always runs. This is correct for most fast build steps. +2. Add a hash or timestamp check inside the shell step itself. This is a script you write and + maintain, not a built-in Atmos feature. + +## Before and After + + + + ```yaml + version: '3' + + vars: + ENV: '{{.ENV | default "dev"}}' + + tasks: + build: + desc: Compile the deployable artifact + cmds: + - go build -o bin/handler ./cmd/handler + sources: + - cmd/**/*.go + generates: + - bin/handler + + test: + desc: Run unit tests + deps: [build] + cmds: + - go test ./... + + lint: + desc: Run static analysis + cmds: + - golangci-lint run ./... + ``` + + + ```yaml + commands: + - name: build + description: Compile the deployable artifact + steps: + - type: shell + command: go build -o bin/handler ./cmd/handler + + - name: test + description: Run unit tests + steps: + - type: shell + command: atmos build + - type: shell + command: go test ./... + + - name: lint + description: Run static analysis + steps: + - type: shell + command: golangci-lint run ./... + ``` + + + + +```yaml +commands: + - name: deploy + description: Plan and apply the given environment (default dev) + flags: + - name: env + default: "dev" +``` + + + +```bash +atmos deploy --env staging +``` + + +## `includes:` Splits into Commands and Workflows + +Task's `includes:` field joins several Taskfiles into one. Atmos has two matching methods. +Choose the one that fits what you are splitting: + +- To split command definitions across files, put them in files such as `atmos.d/commands.yaml`. + Add each file to the root config with `import:` in `atmos.yaml`. See + [Imports](/cli/configuration/imports). +- To split multi-step chains, use separate workflow files. Atmos workflows already live one file + per purpose, under `workflows.base_path` (`stacks/workflows/` by default). No extra step is + needed. + +## What You Gain + +- **A larger set of built-in step types.** A Taskfile task runs shell commands only. Atmos adds + step types for orchestration (`parallel`, `matrix`, `wait`), user prompts (`confirm`, `choose`, + `input`), and output (`table`, `markdown`, `toast`), with more than 30 types in total. See + [Step Types](/workflows/steps/type) for the full list. +- **Interactive commands.** A step such as `confirm` or `choose` can pause a command and ask the + user a question. A Taskfile task cannot do this without a custom shell script. +- **Automatic tool installation.** A command can list the tools it needs under + `dependencies.tools`. Atmos installs the correct version before the command runs. See + [Toolchain Configuration](/cli/configuration/toolchain). Task has no built-in match for this. +- **One interface across every command.** `task --list` shows task names and descriptions. + Atmos adds `atmos --help` with the same flag and argument format for every command, + in addition to the CLI-wide `atmos help`. + +## Migration Checklist + + +- [ ] List every task. Mark each one as independent, or part of a `deps:` chain +- [ ] Turn `desc:`/`cmds:` into command `description:`/`steps:` +- [ ] Put any `deps:` chain in a `parallel` step, to keep Task's default behavior +- [ ] Turn `vars:`/`env:` into command `flags:` (with defaults) and `env:` maps +- [ ] Decide how to handle `sources:`/`generates:` freshness checks. Do not assume parity +- [ ] Split `includes:` into `atmos.d/*.yaml` imports, separate workflow files, or both + + +## Common Questions + + +No. Atmos steps always run. There is no built-in file-hash freshness check. +`require`/`assert` only checks that something exists, not whether it is fresh. If this caching +matters to you, plan for it directly. See [The sources/generates Gap](#the-sourcesgenerates-gap). + + + +A `parallel` step. Task runs `deps:` at the same time by default. Atmos steps run one after +another by default. Put the dependency tasks in a `parallel` step to keep the original behavior, +instead of running them one after another without meaning to. + + + +Yes, but not through `type: atmos`. That step type is only for native Atmos verbs, such as +`terraform plan`. Use a `type: shell` step with `command: atmos ` instead. + + + +See [Migrating from Makefiles](/migration/makefile) or +[Migrating from Justfiles](/migration/justfile). + diff --git a/website/sidebars.js b/website/sidebars.js index 882a1b8936c..67413b417d2 100644 --- a/website/sidebars.js +++ b/website/sidebars.js @@ -74,7 +74,10 @@ module.exports = { items: [ 'migration/native-terraform', 'migration/terragrunt', - 'migration/terraform-workspaces' + 'migration/terraform-workspaces', + 'migration/makefile', + 'migration/taskfile', + 'migration/justfile' ] }, ] From 71420cb8d2193c565589c2bd69a0d9b10cdbc357 Mon Sep 17 00:00:00 2001 From: Erik Osterman Date: Thu, 6 Aug 2026 17:00:30 -0500 Subject: [PATCH 02/17] fix(config): stop directory's own commands: inheriting unrelated .atmos.d subcommands A directory's own inline atmos.yaml commands: entry named the same as a command discovered from git-root .atmos.d (e.g. an unrelated outer project's dev tooling) silently inherited that command's subcommand tree and other subcommand-referencing fields such as default:. Treat a leaf command with no commands: key as fully authoritative instead of merging it field-by-field against the discovered default. Co-Authored-By: Claude Sonnet 5 --- pkg/config/command_merge_core_test.go | 76 +++++++++++++++++++++++++-- pkg/config/load.go | 63 +++++++++++++++++++--- 2 files changed, 129 insertions(+), 10 deletions(-) diff --git a/pkg/config/command_merge_core_test.go b/pkg/config/command_merge_core_test.go index 4b420d26c11..05debc5c33d 100644 --- a/pkg/config/command_merge_core_test.go +++ b/pkg/config/command_merge_core_test.go @@ -256,6 +256,76 @@ func TestMergeCommandArraysPathLeafOverridesNestedAndPreservesSiblings(t *testin assert.Equal(t, "Base examples", examples["description"]) } +// TestMergeMainCommandArrayDoesNotInheritUnrelatedSubcommands is a regression +// test for a directory's own atmos.yaml `commands:` entry silently inheriting +// a same-named command's subcommand tree (and other subcommand-referencing +// fields, such as `default:`) from an unrelated discovered source -- for +// example a different, outer project's git-root `.atmos.d` command that +// happens to share a name. The mergeMainCommandArray function must treat a +// leaf command (no `commands:` key) in the directory's own atmos.yaml as +// fully self-contained, not as a partial override of the discovered command. +func TestMergeMainCommandArrayDoesNotInheritUnrelatedSubcommands(t *testing.T) { + discovered := []interface{}{ + map[string]interface{}{ + "name": "build", + "description": "Build and documentation commands for Atmos development", + "default": "binary", + commandsKey: []interface{}{ + map[string]interface{}{"name": "binary", "description": "Build the Atmos binary"}, + map[string]interface{}{"name": "deps", "description": "Download Go module dependencies"}, + }, + }, + } + mainCommands := []interface{}{ + map[string]interface{}{ + "name": "build", + "description": "My own build command", + }, + } + + merged := mergeMainCommandArray(discovered, mainCommands) + require.Len(t, merged, 1) + + build := requireCommandMap(t, merged, "build") + assert.Equal(t, "My own build command", build["description"]) + _, hasCommands := build[commandsKey] + assert.False(t, hasCommands, "leaf command in the directory's own atmos.yaml must not inherit an unrelated source's subcommand tree") + _, hasDefault := build["default"] + assert.False(t, hasDefault, "leaf command must not inherit a default: pointing at a subcommand that no longer exists") +} + +// TestMergeMainCommandArrayStillComposesWhenSecondOptsIn verifies +// mergeMainCommandArray's strict mode only replaces a colliding command +// outright when the directory's own atmos.yaml command omits `commands:` +// entirely. When it explicitly defines `commands:` (opting into extending a +// shared command tree, e.g. via `.atmos.d` fragments within the same +// project), the normal deep-merge behavior still applies. +func TestMergeMainCommandArrayStillComposesWhenSecondOptsIn(t *testing.T) { + discovered := []interface{}{ + map[string]interface{}{ + "name": "casts", + commandsKey: []interface{}{ + map[string]interface{}{"name": "setup", "description": "Base setup"}, + }, + }, + } + mainCommands := []interface{}{ + map[string]interface{}{ + "name": "casts", + commandsKey: []interface{}{ + map[string]interface{}{"name": "generate", "description": "Local generate"}, + }, + }, + } + + merged := mergeMainCommandArray(discovered, mainCommands) + require.Len(t, merged, 1) + + casts := requireCommandMap(t, merged, "casts") + findCommandMap(t, casts[commandsKey], "setup") + findCommandMap(t, casts[commandsKey], "generate") +} + // TestNormalizeCommandArraySkipsNilEntries verifies that normalizeCommandArray // drops nil command entries (normalizeCommandDefinition returns cmd unchanged // when it's not a map, and nil normalizes to nil, which must be skipped). @@ -290,18 +360,18 @@ func TestNormalizeCommandDefinitionWithoutName(t *testing.T) { func TestMergeCommandDefinitionsNonMapOperands(t *testing.T) { t.Run("first is not a map", func(t *testing.T) { second := map[string]interface{}{"name": "second"} - result := mergeCommandDefinitions("not-a-map", second) + result := mergeCommandDefinitions("not-a-map", second, false) assert.Equal(t, second, result) }) t.Run("second is not a map", func(t *testing.T) { first := map[string]interface{}{"name": "first"} - result := mergeCommandDefinitions(first, "not-a-map") + result := mergeCommandDefinitions(first, "not-a-map", false) assert.Equal(t, "not-a-map", result) }) t.Run("both are not maps", func(t *testing.T) { - result := mergeCommandDefinitions(42, "second-value") + result := mergeCommandDefinitions(42, "second-value", false) assert.Equal(t, "second-value", result) }) } diff --git a/pkg/config/load.go b/pkg/config/load.go index e7fe0ee97bf..80e8fe54402 100644 --- a/pkg/config/load.go +++ b/pkg/config/load.go @@ -1315,9 +1315,13 @@ func processConfigImportsAndReapply(path string, tempViper *viper.Viper, content finalCommands = mergeCommandArrays(finalCommands, importedCommands) } - // Add main, with main overriding all others on duplicates + // Add main, with main overriding all others on duplicates. Strict: the + // directory's own atmos.yaml is authoritative for any command name it + // defines directly, so it must not inherit a same-named command's + // subcommands from a discovered default (e.g. an unrelated ancestor + // project's git-root .atmos.d) just because it doesn't repeat `commands:`. if mainCommands != nil { - finalCommands = mergeCommandArrays(finalCommands, mainCommands) + finalCommands = mergeMainCommandArray(finalCommands, mainCommands) } tempViper.Set(commandsKey, finalCommands) @@ -1927,7 +1931,29 @@ func overlayProfileSettings(v *viper.Viper, settings map[string]any, prefix stri // When duplicates exist based on name, the second parameter takes precedence (override behavior). // This ensures local commands can override imported/remote commands. func mergeCommandArrays(first, second interface{}) []interface{} { - return mergeNormalizedCommandArrays(normalizeCommandArray(first), normalizeCommandArray(second)) + return mergeNormalizedCommandArrays(normalizeCommandArray(first), normalizeCommandArray(second), false) +} + +// mergeMainCommandArray merges commands gathered from `.atmos.d`/imports (first) +// against a directory's own inline atmos.yaml `commands:` block (second). Unlike +// mergeCommandArrays, when second's definition of a command omits a nested +// `commands:` key, the merged result has none either -- it does not silently +// inherit a same-named command's subcommand tree from first. That matters when +// first was discovered from a git-root `.atmos.d` belonging to a different, +// unrelated outer project (e.g. this directory has no `.git` of its own and sits +// inside someone else's monorepo): the directory's own atmos.yaml is authoritative +// for any command name it defines directly, so a coincidental name collision with +// an ancestor's `.atmos.d` command must not graft that command's subcommands on. +// +// Composing multiple `.atmos.d`/import fragments of the SAME project still goes +// through the lenient mergeCommandArrays (see mergeConfigFile and +// processConfigImportsAndReapply's default/imported-commands step): a fragment +// that omits `commands:` there is expected to preserve another fragment's +// subcommands, which is how "Split commands across files" (see the +// atmos-migration skill) is meant to work. Only the boundary between discovered +// defaults and the directory's own complete atmos.yaml uses strict mode. +func mergeMainCommandArray(first, second interface{}) []interface{} { + return mergeNormalizedCommandArrays(normalizeCommandArray(first), normalizeCommandArray(second), true) } func normalizeCommandArray(commands interface{}) []interface{} { @@ -1942,7 +1968,7 @@ func normalizeCommandArray(commands interface{}) []interface{} { if normalized == nil { continue } - result = mergeNormalizedCommandArrays(result, []interface{}{normalized}) + result = mergeNormalizedCommandArrays(result, []interface{}{normalized}, false) } return result @@ -2010,7 +2036,14 @@ func normalizeCommandDefinition(cmd interface{}) interface{} { return current } -func mergeNormalizedCommandArrays(first, second []interface{}) []interface{} { +// mergeNormalizedCommandArrays merges two already-normalized command arrays by +// name, later entries overriding earlier ones. The strict flag is forwarded to +// mergeCommandDefinitions for every name collision -- see mergeMainCommandArray +// for what it changes and why. Every current caller passes an already-deduped +// `first`/`second` (normalizeCommandArray or a prior merge step already +// resolved same-source collisions), so strict only takes effect at the +// first-vs-second boundary, which is the boundary it is meant for. +func mergeNormalizedCommandArrays(first, second []interface{}, strict bool) []interface{} { // Build a map of commands by name, with later entries overriding earlier ones. commandMap := make(map[string]interface{}) var orderedNames []string @@ -2031,7 +2064,7 @@ func mergeNormalizedCommandArrays(first, second []interface{}) []interface{} { // Store or merge the command. Nested command groups are merged // recursively so imports can extend a shared command tree. if existing, exists := commandMap[name]; exists { - commandMap[name] = mergeCommandDefinitions(existing, cmd) + commandMap[name] = mergeCommandDefinitions(existing, cmd, strict) } else { commandMap[name] = cmd } @@ -2055,7 +2088,18 @@ func mergeNormalizedCommandArrays(first, second []interface{}) []interface{} { return result } -func mergeCommandDefinitions(first, second interface{}) interface{} { +// mergeCommandDefinitions deep-merges two command definitions that share a +// name, second's fields taking precedence. When strict is true and second +// does not itself define a nested `commands:` key, second is treated as a +// complete, standalone leaf command: it replaces first outright instead of +// being field-merged with it. A field-by-field merge would otherwise leave +// dangling references into first's dropped `commands:` tree (for example a +// `default:` naming a subcommand that no longer exists once `commands:` is +// gone), so strict mode does not partially merge here at all -- see +// mergeMainCommandArray for why this distinction exists. When second does +// define `commands:`, or strict is false, the normal field-by-field merge +// runs and nested `commands:` arrays are deep-merged if both sides have one. +func mergeCommandDefinitions(first, second interface{}, strict bool) interface{} { firstMap, ok := first.(map[string]interface{}) if !ok { return second @@ -2065,6 +2109,10 @@ func mergeCommandDefinitions(first, second interface{}) interface{} { return second } + if _, secondHasCommands := secondMap[commandsKey]; strict && !secondHasCommands { + return second + } + // Start with the first map's size. Avoid len(first)+len(second) here // because CodeQL correctly treats unchecked allocation arithmetic as a // potential overflow, and the map will grow if secondMap adds new keys. @@ -2072,6 +2120,7 @@ func mergeCommandDefinitions(first, second interface{}) interface{} { for key, value := range firstMap { merged[key] = value } + for key, value := range secondMap { if key == commandsKey { if existing, ok := merged[key]; ok { From 4435caf19d18aa6d9145d58c83ce8878998dda8a Mon Sep 17 00:00:00 2001 From: Erik Osterman Date: Thu, 6 Aug 2026 17:01:02 -0500 Subject: [PATCH 03/17] fix(validate): stacks validation no longer requires name_template/name_pattern createComponentStackMap derived a component's logical stack name via a stricter, older code path that predated zero-config filename-based stack naming (#1934), so atmos validate stacks hard-failed on any repo that terraform plan, list stacks, and describe component already resolved stacks for fine, including this repo's own examples/native-terraform. Reuse resolveStackName's precedence (manifest name > name_template > name_pattern > filename) instead. Co-Authored-By: Claude Sonnet 5 --- internal/exec/validate_stacks.go | 28 ++++++----- internal/exec/validate_stacks_test.go | 68 +++++++++++++++++++++++++++ 2 files changed, 83 insertions(+), 13 deletions(-) diff --git a/internal/exec/validate_stacks.go b/internal/exec/validate_stacks.go index f7197c44888..d4cb36001a4 100644 --- a/internal/exec/validate_stacks.go +++ b/internal/exec/validate_stacks.go @@ -401,20 +401,22 @@ func createComponentStackMap( }, } - // Find Atmos stack name - if atmosConfig.Stacks.NameTemplate != "" { - stackName, err = ProcessTmpl(atmosConfig, "validate-stacks-name-template", atmosConfig.Stacks.NameTemplate, configAndStacksInfo.ComponentSection, atmosConfig.Templates.Settings.IgnoreMissingTemplateValues) - if err != nil { - return nil, err - } - } else { - context := cfg.GetContextFromVars(varsSection) - configAndStacksInfo.Context = context - stackName, err = cfg.GetContextPrefix(stackManifest, context, GetStackNamePattern(atmosConfig), stackManifest) - if err != nil { - return nil, err - } + // Find the Atmos stack name. Precedence matches the rest of the CLI + // (resolveStackName, used by `describe stacks`/`list stacks`, and + // processStackContextPrefix, used by `terraform plan` and friends): + // manifest `name:` > name_template > name_pattern > filename. Neither + // name_template nor name_pattern is required -- filename-based zero-config + // stack naming (#1934) applies here too, so `validate stacks` no longer + // hard-fails on a repo that every other command resolves stacks for fine. + if comp, ok := configAndStacksInfo.ComponentSection[cfg.ComponentSectionName].(string); !ok || comp == "" { + configAndStacksInfo.ComponentSection[cfg.ComponentSectionName] = componentName } + var resolvedContext schema.Context + stackName, resolvedContext, err = resolveStackName(atmosConfig, stackManifest, getStackManifestName(stackSection), configAndStacksInfo, varsSection) + if err != nil { + return nil, err + } + configAndStacksInfo.Context = resolvedContext _, ok = terraformComponentStackMap[componentName] if !ok { diff --git a/internal/exec/validate_stacks_test.go b/internal/exec/validate_stacks_test.go index eaf4fe69f77..fa33e9ff5e8 100644 --- a/internal/exec/validate_stacks_test.go +++ b/internal/exec/validate_stacks_test.go @@ -71,6 +71,74 @@ func TestValidateStacksWithMergeContext(t *testing.T) { }) } +// nativeTerraformExampleDir returns the absolute path to examples/native-terraform +// using runtime.Caller(0) so the path is source-file-relative (CWD-independent). +func nativeTerraformExampleDir(t *testing.T) string { + t.Helper() + _, callerFile, _, ok := runtime.Caller(0) + require.True(t, ok, "runtime.Caller(0) must succeed") + dir := filepath.Join(filepath.Dir(callerFile), "..", "..", "examples", "native-terraform") + absDir, err := filepath.Abs(dir) + require.NoError(t, err, "cannot resolve fixture path") + return absDir +} + +// TestValidateStacksWithoutNameTemplateOrPattern is a regression test: `atmos +// validate stacks` used to hard-fail with ErrMissingStackNameTemplateAndPattern +// whenever `stacks.name_template`/`stacks.name_pattern` was unset, even though +// `terraform plan`, `list stacks`, and `describe component` all resolve stack +// names fine via filename/`name:`-based zero-config fallback (#1934) in that +// case. The createComponentStackMap function's stack-name derivation now +// mirrors that same precedence (resolveStackName) instead of requiring +// name_template/name_pattern itself. The examples/native-terraform directory +// is the real repro: its atmos.yaml sets neither field on purpose (see its +// own comment) and is the example this repo's atmos-migration skill points +// users at. +func TestValidateStacksWithoutNameTemplateOrPattern(t *testing.T) { + absPath := nativeTerraformExampleDir(t) + + atmosConfig := &schema.AtmosConfiguration{ + BasePath: absPath, + StacksBaseAbsolutePath: filepath.Join(absPath, "stacks"), + Stacks: schema.Stacks{ + BasePath: "stacks", + IncludedPaths: []string{"**/*"}, + ExcludedPaths: []string{"**/_defaults.yaml"}, + }, + Logs: schema.Logs{ + Level: u.LogLevelDebug, + }, + Components: schema.Components{ + Terraform: schema.Terraform{ + BasePath: "components/terraform", + }, + }, + Settings: schema.AtmosSettings{ + ListMergeStrategy: "replace", + }, + } + atmosConfig.TerraformDirAbsolutePath = filepath.Join(absPath, "components", "terraform") + atmosConfig.HelmfileDirAbsolutePath = filepath.Join(absPath, "components", "helmfile") + atmosConfig.PackerDirAbsolutePath = filepath.Join(absPath, "components", "packer") + + // Populate StackConfigFilesAbsolutePaths the same way cfg.InitCliConfig does for a + // real CLI run -- without it, FindStacksMap has no files to process and + // createComponentStackMap's loop body (where the bug lives) never runs, letting + // the test pass vacuously regardless of the fix. + includeStackAbsPaths, err := u.JoinPaths(atmosConfig.StacksBaseAbsolutePath, atmosConfig.Stacks.IncludedPaths) + require.NoError(t, err) + stackConfigFilesAbsolutePaths, _, err := cfg.FindAllStackConfigsInPaths(atmosConfig, includeStackAbsPaths, atmosConfig.Stacks.ExcludedPaths) + require.NoError(t, err) + require.NotEmpty(t, stackConfigFilesAbsolutePaths, "fixture must have discoverable stack manifests for this test to exercise the bug") + atmosConfig.StackConfigFilesAbsolutePaths = stackConfigFilesAbsolutePaths + + require.Empty(t, atmosConfig.Stacks.NameTemplate, "fixture must exercise the no-name_template case") + require.Empty(t, atmosConfig.Stacks.NamePattern, "fixture must exercise the no-name_pattern case") + + err = ValidateStacks(atmosConfig) + assert.NoError(t, err, "validate stacks must succeed via filename/name:-based stack naming, same as terraform plan/list stacks/describe component") +} + func TestMergeContextInProcessYAMLConfigFile(t *testing.T) { // Test that ProcessYAMLConfigFileWithContext properly tracks import chain absPath := validateStacksTestDataDir(t) From fd4f988b613d711f06f70c0786ae69ecf89d06cf Mon Sep 17 00:00:00 2001 From: Erik Osterman Date: Thu, 6 Aug 2026 17:01:59 -0500 Subject: [PATCH 04/17] fix(migration): correct field-tested gaps in task-runner migration guides Field-testing the new Makefile/Justfile/Taskfile migration references against real fixtures and the real atmos binary surfaced several gaps: from-native-terraform.md's Shape B recipe used a component name that never resolved (component names must match the physical directory); workflows.base_path has no default and needs to be called out; an orphaned [private] Justfile recipe and Just's command-echo behavior weren't addressed; from-taskfile.md overstated the need for `import:` when atmos.d/.atmos.d is auto-discovered; and the migration docs sidebar order contradicted the pages' own sidebar_position values. Co-Authored-By: Claude Sonnet 5 --- agent-skills/skills/atmos-migration/SKILL.md | 9 +++++++-- .../atmos-migration/references/from-justfile.md | 16 ++++++++++++++++ .../references/from-native-terraform.md | 17 ++++++++++++++--- .../atmos-migration/references/from-taskfile.md | 14 +++++++++----- website/sidebars.js | 4 ++-- 5 files changed, 48 insertions(+), 12 deletions(-) diff --git a/agent-skills/skills/atmos-migration/SKILL.md b/agent-skills/skills/atmos-migration/SKILL.md index e63b865422f..6e7b0734664 100644 --- a/agent-skills/skills/atmos-migration/SKILL.md +++ b/agent-skills/skills/atmos-migration/SKILL.md @@ -94,9 +94,9 @@ The remote-state-bridge pattern makes progressive migration possible. It lets a component at a time. Without it, the team must migrate everything at once. Use this pattern when the user has existing Terraform state that a new Atmos component must read. -### Two Common Problems in Task-Runner Migration +### Common Problems in Task-Runner Migration -Two behaviors apply to every task runner. Check them before you open a reference file: +These behaviors apply to every task runner. Check them before you open a reference file: - **The default order can change.** Task runs `deps:` at the same time by default. Make and Just run dependencies one after another, unless the user adds a flag such as `make -j`. Atmos steps @@ -107,6 +107,11 @@ Two behaviors apply to every task runner. Check them before you open a reference Make targets both skip work when a file has not changed. Atmos steps always run. The `require`/`assert` step type does not replace this. It only checks that a file exists. It does not check if the file is new. Tell the user this directly. +- **`workflows.base_path` needs to be set explicitly once the user has their own `atmos.yaml`.** + A target chain becomes an Atmos workflow (Principle 7), but `atmos workflow ` fails with + `'workflows.base_path' must be configured in 'atmos.yaml'` until you add it (for example, + `workflows.base_path: "stacks/workflows"`). None of this skill's `atmos.yaml` snippets show it + by default -- add it the moment the user's migration reaches its first workflow. Each reference file has its own "Common Problems" section with the exact field names and steps for that tool. This section is only a short summary. diff --git a/agent-skills/skills/atmos-migration/references/from-justfile.md b/agent-skills/skills/atmos-migration/references/from-justfile.md index 1444ef9350c..a1e4b92595c 100644 --- a/agent-skills/skills/atmos-migration/references/from-justfile.md +++ b/agent-skills/skills/atmos-migration/references/from-justfile.md @@ -149,6 +149,22 @@ There is no `hidden` or `private` field in the custom command schema. Put a priv recipe's logic in a step inside the command or workflow that calls it. Do not create a separate command just to hide it from `--list` or `--help`. +If a `[private]` recipe is never called by any public recipe (an orphaned helper, not a +dependency), there is no public recipe to fold its steps into. Do not create a public `atmos` +command just to preserve it -- that changes its visibility, which is the opposite of what +`[private]` meant. Confirm with the user whether the recipe is still needed at all; if it is, +ask where its logic should live (its own step inside whichever command ends up needing it, or a +short script the user maintains separately) rather than migrating it by default. + +### Command echo differs between `just` and Atmos + +By default, Just prints each recipe line before running it (`sh -x`-style), so `just build`'s +visible output includes every command line, not just what those commands print. Atmos `type: +shell` steps run silently by default -- only the command's own stdout/stderr shows. The migrated +command's side effects match the original recipe, but the terminal output will look sparser side +by side. Tell the user this if they compare `just ` output to `atmos ` output +directly; it is a visible difference, not a bug. + ### Confirm `set dotenv-load` and `set shell` with the user Do not drop either setting without comment. Ask the user if the behavior matters to their diff --git a/agent-skills/skills/atmos-migration/references/from-native-terraform.md b/agent-skills/skills/atmos-migration/references/from-native-terraform.md index f583a5d35fc..783d6cbda54 100644 --- a/agent-skills/skills/atmos-migration/references/from-native-terraform.md +++ b/agent-skills/skills/atmos-migration/references/from-native-terraform.md @@ -97,16 +97,27 @@ With a Makefile like `terraform plan -var-file=envs/$(ENV).tfvars`. base_path: "." # The whole repo is one component stacks: base_path: "stacks" + included_paths: + - "**/*" ``` -2. Treat the single TF dir as one component (e.g., `infra`): +2. **The component name must match the physical directory name** -- Atmos resolves a component + to `/`, so with `base_path: "."` the + component name has to be `terraform` (the real directory), not an invented name like `infra`. + Renaming the component in Atmos config does not rename the directory on disk: ```yaml # stacks/dev.yaml components: terraform: - infra: + terraform: vars: !include ../terraform/envs/dev.tfvars ``` -3. The Makefile can stay as a thin wrapper around `atmos terraform plan infra -s dev` during + If the user wants a friendlier component name without moving files, rename the directory + itself (e.g. `terraform/` to `infra/`) rather than trying to alias it in `atmos.yaml` -- there + is no `metadata.component` override needed here since this is a single-component repo, and + `metadata.component` is for pointing multiple stack instances at one shared component (see + [remote-state-bridge.md](remote-state-bridge.md)), not for renaming a component's own + directory. +3. The Makefile can stay as a thin wrapper around `atmos terraform plan terraform -s dev` during transition, then be deleted. ## Shape C: Multiple Root Modules with Shared Modules diff --git a/agent-skills/skills/atmos-migration/references/from-taskfile.md b/agent-skills/skills/atmos-migration/references/from-taskfile.md index 0dc01487317..57787a6d183 100644 --- a/agent-skills/skills/atmos-migration/references/from-taskfile.md +++ b/agent-skills/skills/atmos-migration/references/from-taskfile.md @@ -151,12 +151,16 @@ them plainly that this behavior does not carry over. Then offer two honest choic Task's `includes:` field combines several Taskfiles into one. Atmos has two matching methods. Pick the one that fits the content being split: -- To split command definitions across files, put them in files such as - `atmos.d/commands.yaml`. Add each file to the root config with `import:` in `atmos.yaml`. See - [Imports](https://atmos.tools/cli/configuration/imports). +- To split command definitions across files, put them in files such as `atmos.d/commands.yaml` + or `.atmos.d/commands.yaml`. Atmos auto-discovers `atmos.d/`/`.atmos.d/` in the config + directory (and, as a lower-priority fallback, at the git/worktree root) -- no `import:` entry + is needed for this specific location. Use `import:` only when splitting across a directory + Atmos does not auto-discover. See [Imports](https://atmos.tools/cli/configuration/imports). - To split multi-step chains, use separate workflow files. Atmos workflows already live one file - per purpose, under `workflows.base_path` (`stacks/workflows/` by default). No extra step is - needed. + per purpose, under `workflows.base_path`. Unlike `atmos.d`/`.atmos.d`, there is no default for + `workflows.base_path` -- add it explicitly (for example `workflows.base_path: + "stacks/workflows"`) the first time the user's migration reaches a workflow, or `atmos + workflow ` fails with `'workflows.base_path' must be configured in 'atmos.yaml'`. ### `sources`/`generates` has no built-in match diff --git a/website/sidebars.js b/website/sidebars.js index bb5bc7996d1..7af4b0aad11 100644 --- a/website/sidebars.js +++ b/website/sidebars.js @@ -76,8 +76,8 @@ module.exports = { 'migration/terragrunt', 'migration/terraform-workspaces', 'migration/makefile', - 'migration/taskfile', - 'migration/justfile' + 'migration/justfile', + 'migration/taskfile' ] }, ] From 230ff819e9d622baf5dc01e6432c0b221df330f8 Mon Sep 17 00:00:00 2001 From: Erik Osterman Date: Thu, 6 Aug 2026 18:36:53 -0500 Subject: [PATCH 05/17] fix(migration): address CodeRabbit review on PR #2896 - Use the terraform component name (matching the physical terraform/ directory) instead of the never-resolving infra in every single-directory Makefile/Justfile/Taskfile example, consistent with the from-native-terraform.md Shape B fix. - Reserve type: atmos for native Atmos verbs only in from-taskfile.md's Shape A guidance; calling another custom command still needs type: shell. - Fix from-justfile.md's Common Problems link fragment (verified against the actual github-slugger algorithm). - Stop telling readers import: is required for auto-discovered atmos.d/.atmos.d files in from-makefile.md and website/docs/migration/taskfile.mdx. - Document that workflows.base_path has no default in website/docs/migration/taskfile.mdx, matching the equivalent fix already applied to the agent-skill references. - Normalize from-native-terraform.md's odd-space (3/5/7) list-continuation and nested-YAML indentation to even, matching the EditorConfig multiple-of-2 rule applied to the other reference files earlier in this branch. Co-Authored-By: Claude Sonnet 5 --- .../references/from-justfile.md | 4 +- .../references/from-makefile.md | 10 +- .../references/from-native-terraform.md | 102 +++++++++--------- .../references/from-taskfile.md | 10 +- website/docs/migration/justfile.mdx | 4 +- website/docs/migration/makefile.mdx | 2 +- website/docs/migration/taskfile.mdx | 20 ++-- 7 files changed, 80 insertions(+), 72 deletions(-) diff --git a/agent-skills/skills/atmos-migration/references/from-justfile.md b/agent-skills/skills/atmos-migration/references/from-justfile.md index a1e4b92595c..89b2c7af09e 100644 --- a/agent-skills/skills/atmos-migration/references/from-justfile.md +++ b/agent-skills/skills/atmos-migration/references/from-justfile.md @@ -42,7 +42,7 @@ _clean: 2. Turn a recipe's named parameter with a default value, such as `deploy env='dev':`, into a command `flags:` entry with a matching `default:` value. Inside a step, read the value as `{{ .Flags.env }}`. Do not use Just's own `{{env}}` syntax. See - [Common Problems](#-interpolation-looks-like-atmos-templates-but-is-not) below. + [Common Problems](#--interpolation-looks-like-atmos-templates-but-is-not) below. 3. Do not create a separate command for a `[private]` recipe. There is no `hidden` or `private` field on Atmos custom commands. Put the recipe's body in a step of the command that calls it. @@ -99,7 +99,7 @@ commands: - type: shell command: atmos test - type: atmos - command: terraform apply infra -s {{ .Flags.env }} + command: terraform apply terraform -s {{ .Flags.env }} ``` ## Shape C: Environment and Shell Settings diff --git a/agent-skills/skills/atmos-migration/references/from-makefile.md b/agent-skills/skills/atmos-migration/references/from-makefile.md index 234ae528270..2229ecb0c01 100644 --- a/agent-skills/skills/atmos-migration/references/from-makefile.md +++ b/agent-skills/skills/atmos-migration/references/from-makefile.md @@ -132,7 +132,7 @@ commands: - type: shell command: atmos test - type: atmos - command: terraform apply infra -s {{ .Flags.env }} + command: terraform apply terraform -s {{ .Flags.env }} ``` ## Shape C: Recursive or Parallel Make @@ -190,9 +190,11 @@ that one step. It does not mean you should add `output: none` to every step. ### Split commands across files When a Makefile uses `include foo.mk` to split its content across files, split the Atmos config -the same way. Put the extra commands in a file such as `atmos.d/commands.yaml`. Then add that -file to the root config with `import:` in `atmos.yaml`. See -[Imports](https://atmos.tools/cli/configuration/imports). +the same way. Put the extra commands in a file such as `atmos.d/commands.yaml` or +`.atmos.d/commands.yaml`. Atmos auto-discovers `atmos.d/`/`.atmos.d/` in the config directory +(and, as a lower-priority fallback, at the git/worktree root) -- no `import:` entry is needed for +this specific location. Use `import:` only when splitting across a directory Atmos does not +auto-discover. See [Imports](https://atmos.tools/cli/configuration/imports). ### Complex `$(eval)` and `$(call)` macros diff --git a/agent-skills/skills/atmos-migration/references/from-native-terraform.md b/agent-skills/skills/atmos-migration/references/from-native-terraform.md index 783d6cbda54..aa79b438dd0 100644 --- a/agent-skills/skills/atmos-migration/references/from-native-terraform.md +++ b/agent-skills/skills/atmos-migration/references/from-native-terraform.md @@ -43,32 +43,32 @@ terraform/ **Recipe:** 1. `atmos.yaml` at repo root, **no file moves**: - ```yaml - base_path: "./" - components: - terraform: - base_path: "terraform" # Point at the existing dir - apply_auto_approve: false - deploy_run_init: true - auto_generate_backend_file: false - stacks: - base_path: "stacks" - included_paths: ["**/*"] - excluded_paths: ["**/_defaults.yaml"] - ``` + ```yaml + base_path: "./" + components: + terraform: + base_path: "terraform" # Point at the existing dir + apply_auto_approve: false + deploy_run_init: true + auto_generate_backend_file: false + stacks: + base_path: "stacks" + included_paths: ["**/*"] + excluded_paths: ["**/_defaults.yaml"] + ``` 2. Create `stacks/dev.yaml`: - ```yaml - import: - - _defaults - components: - terraform: - vpc: - vars: !include ../terraform/vpc/envs/dev.tfvars - database: - vars: !include ../terraform/database/envs/dev.tfvars - ``` + ```yaml + import: + - _defaults + components: + terraform: + vpc: + vars: !include ../terraform/vpc/envs/dev.tfvars + database: + vars: !include ../terraform/database/envs/dev.tfvars + ``` 3. Run `atmos terraform plan vpc -s dev`. Compare to the previous - `cd terraform/vpc && terraform plan -var-file=envs/dev.tfvars` output. + `cd terraform/vpc && terraform plan -var-file=envs/dev.tfvars` output. The user keeps their `.tfvars` files and TF code unchanged. Later, they can convert per-env `.tfvars` to native YAML to get deep-merge inheritance across environments. @@ -90,35 +90,35 @@ With a Makefile like `terraform plan -var-file=envs/$(ENV).tfvars`. **Recipe:** 1. `atmos.yaml`: - ```yaml - base_path: "./" - components: - terraform: - base_path: "." # The whole repo is one component - stacks: - base_path: "stacks" - included_paths: - - "**/*" - ``` + ```yaml + base_path: "./" + components: + terraform: + base_path: "." # The whole repo is one component + stacks: + base_path: "stacks" + included_paths: + - "**/*" + ``` 2. **The component name must match the physical directory name** -- Atmos resolves a component - to `/`, so with `base_path: "."` the - component name has to be `terraform` (the real directory), not an invented name like `infra`. - Renaming the component in Atmos config does not rename the directory on disk: - ```yaml - # stacks/dev.yaml - components: - terraform: - terraform: - vars: !include ../terraform/envs/dev.tfvars - ``` - If the user wants a friendlier component name without moving files, rename the directory - itself (e.g. `terraform/` to `infra/`) rather than trying to alias it in `atmos.yaml` -- there - is no `metadata.component` override needed here since this is a single-component repo, and - `metadata.component` is for pointing multiple stack instances at one shared component (see - [remote-state-bridge.md](remote-state-bridge.md)), not for renaming a component's own - directory. + to `/`, so with `base_path: "."` the + component name has to be `terraform` (the real directory), not an invented name like `infra`. + Renaming the component in Atmos config does not rename the directory on disk: + ```yaml + # stacks/dev.yaml + components: + terraform: + terraform: + vars: !include ../terraform/envs/dev.tfvars + ``` + If the user wants a friendlier component name without moving files, rename the directory + itself (e.g. `terraform/` to `infra/`) rather than trying to alias it in `atmos.yaml` -- there + is no `metadata.component` override needed here since this is a single-component repo, and + `metadata.component` is for pointing multiple stack instances at one shared component (see + [remote-state-bridge.md](remote-state-bridge.md)), not for renaming a component's own + directory. 3. The Makefile can stay as a thin wrapper around `atmos terraform plan terraform -s dev` during - transition, then be deleted. + transition, then be deleted. ## Shape C: Multiple Root Modules with Shared Modules diff --git a/agent-skills/skills/atmos-migration/references/from-taskfile.md b/agent-skills/skills/atmos-migration/references/from-taskfile.md index 57787a6d183..e62d3674202 100644 --- a/agent-skills/skills/atmos-migration/references/from-taskfile.md +++ b/agent-skills/skills/atmos-migration/references/from-taskfile.md @@ -39,9 +39,11 @@ tasks: **Steps:** 1. Turn `desc:` into the command's `description:` field. -2. Turn each entry in `cmds:` into a `type: shell` step. If the line is a `terraform` or `atmos` - call, use a `type: atmos` step instead, because `type: atmos` is reserved for native Atmos - verbs. +2. Turn each entry in `cmds:` into a `type: shell` step. If the line is a native Atmos verb, such + as `terraform plan` or `terraform apply`, use a `type: atmos` step instead. `type: atmos` is + reserved for native Atmos verbs only. If the line calls another custom command (for example + `atmos build`), keep it as a `type: shell` step with `command: atmos build` -- do not use + `type: atmos` for that. ```yaml commands: @@ -99,7 +101,7 @@ commands: type: shell command: atmos lint - type: atmos - command: terraform apply infra -s dev + command: terraform apply terraform -s dev ``` If the Taskfile's `deps:` list needs its own internal order, add `needs:` to the steps inside the diff --git a/website/docs/migration/justfile.mdx b/website/docs/migration/justfile.mdx index ceac1538df9..e8cd6e2c336 100644 --- a/website/docs/migration/justfile.mdx +++ b/website/docs/migration/justfile.mdx @@ -96,7 +96,7 @@ only cares what the recipe does. - type: shell command: atmos test - type: atmos - command: terraform apply infra -s {{ .Flags.env }} + command: terraform apply terraform -s {{ .Flags.env }} ``` @@ -119,7 +119,7 @@ commands: default: "dev" steps: - type: atmos - command: terraform apply infra -s {{ .Flags.env }} + command: terraform apply terraform -s {{ .Flags.env }} ``` diff --git a/website/docs/migration/makefile.mdx b/website/docs/migration/makefile.mdx index 420fc702730..412bfa2de46 100644 --- a/website/docs/migration/makefile.mdx +++ b/website/docs/migration/makefile.mdx @@ -100,7 +100,7 @@ commands while you migrate. - type: shell command: atmos test - type: atmos - command: terraform apply infra -s {{ .Flags.env }} + command: terraform apply terraform -s {{ .Flags.env }} ``` diff --git a/website/docs/migration/taskfile.mdx b/website/docs/migration/taskfile.mdx index b266e805614..735c442fa3f 100644 --- a/website/docs/migration/taskfile.mdx +++ b/website/docs/migration/taskfile.mdx @@ -30,7 +30,7 @@ Your Terraform code and scripts do not need to change. | `deps:` (runs at the same time by default) | A `parallel` step. Plain steps in order change the default | | `vars:` / `env:` | Command `flags:` (with `default:`) / `env:` map | | `sources:` / `generates:` (freshness check) | No direct match. Steps always run | -| `includes:` (multi-file composition) | `atmos.d/*.yaml` + root `import:`, or separate workflow files | +| `includes:` (multi-file composition) | Auto-discovered `atmos.d/*.yaml`, or separate workflow files | ## The Parallel-by-Default Difference @@ -69,7 +69,7 @@ dependency tasks inside a [`parallel`](/workflows/steps/type/parallel) step: type: shell command: atmos lint - type: atmos - command: terraform apply infra -s dev + command: terraform apply terraform -s dev ``` @@ -173,12 +173,16 @@ atmos deploy --env staging Task's `includes:` field joins several Taskfiles into one. Atmos has two matching methods. Choose the one that fits what you are splitting: -- To split command definitions across files, put them in files such as `atmos.d/commands.yaml`. - Add each file to the root config with `import:` in `atmos.yaml`. See - [Imports](/cli/configuration/imports). +- To split command definitions across files, put them in files such as `atmos.d/commands.yaml` or + `.atmos.d/commands.yaml`. Atmos auto-discovers `atmos.d/`/`.atmos.d/` in the config directory + (and, as a lower-priority fallback, at the git/worktree root) -- no `import:` entry is needed + for this specific location. Use `import:` only when splitting across a directory Atmos does not + auto-discover. See [Imports](/cli/configuration/imports). - To split multi-step chains, use separate workflow files. Atmos workflows already live one file - per purpose, under `workflows.base_path` (`stacks/workflows/` by default). No extra step is - needed. + per purpose, under `workflows.base_path`. Unlike `atmos.d`/`.atmos.d`, there is no default for + `workflows.base_path` -- add it explicitly (for example `workflows.base_path: + "stacks/workflows"`) the first time your migration reaches a workflow, or `atmos workflow + ` fails with `'workflows.base_path' must be configured in 'atmos.yaml'`. ## What You Gain @@ -203,7 +207,7 @@ Choose the one that fits what you are splitting: - [ ] Put any `deps:` chain in a `parallel` step, to keep Task's default behavior - [ ] Turn `vars:`/`env:` into command `flags:` (with defaults) and `env:` maps - [ ] Decide how to handle `sources:`/`generates:` freshness checks. Do not assume parity -- [ ] Split `includes:` into `atmos.d/*.yaml` imports, separate workflow files, or both +- [ ] Split `includes:` into auto-discovered `atmos.d/*.yaml` files, separate workflow files, or both ## Common Questions From 4a67d3d294818d901a6a1a48a891f1e31edae9dc Mon Sep 17 00:00:00 2001 From: Erik Osterman Date: Fri, 7 Aug 2026 07:40:16 -0500 Subject: [PATCH 06/17] feat(commands): add hidden custom commands and --help=hidden topic Custom commands can now set `hidden: true` to stay runnable (directly, as a `default:` target, or from another command's steps) while dropping out of `--help` listings, completions, and the AI `atmos_list_commands` tool. This closes the gap the Just/Task/Make migration guides used to call "no match", where a `[private]`/`internal: true` recipe or task needed to be reusable across callers or invoked directly for debugging rather than folded into a single caller's step. Add a matching `--help=hidden` topic to reveal a command's hidden subcommands on demand; the default-help hint only mentions it when a command actually has one, to avoid cluttering the common case. Refresh the affected migration guides (website + agent-skills mirrors) to point at `hidden: true` instead of the old "no match" guidance, and add previously-missing coverage for Task's `internal: true` flag. Co-Authored-By: Claude Sonnet 5 --- .../skills/atmos-introspection/SKILL.md | 5 +- .../references/from-justfile.md | 54 ++++--- .../references/from-makefile.md | 8 +- .../references/from-taskfile.md | 7 + cmd/cmd_utils.go | 9 +- cmd/cmd_utils_test.go | 141 ++++++++++++++++++ cmd/help_template.go | 47 ++++++ cmd/help_topics.go | 2 + cmd/help_topics_render.go | 10 +- cmd/help_topics_test.go | 46 ++++++ docs/prd/topic-specific-cli-help.md | 20 ++- pkg/datafetcher/schema/atmos/config/1.0.json | 14 ++ pkg/schema/command.go | 9 +- .../commands/command/commands.mdx | 30 ++++ .../configuration/commands/command/index.mdx | 3 + website/docs/migration/justfile.mdx | 57 +++++-- website/docs/migration/makefile.mdx | 26 +++- website/docs/migration/native-terraform.mdx | 11 ++ website/docs/migration/taskfile.mdx | 34 ++++- .../docs/migration/terraform-workspaces.mdx | 11 ++ 20 files changed, 487 insertions(+), 57 deletions(-) diff --git a/agent-skills/skills/atmos-introspection/SKILL.md b/agent-skills/skills/atmos-introspection/SKILL.md index ff0084803f3..cae7d322b91 100644 --- a/agent-skills/skills/atmos-introspection/SKILL.md +++ b/agent-skills/skills/atmos-introspection/SKILL.md @@ -311,12 +311,15 @@ atmos terraform plan --help=usage # ONLY the usage line + embedded usage ex # the fastest way to see real invocation examples atmos terraform plan --help=flags # Command-specific flags only, excludes inherited/global flags atmos terraform plan --help=all # Full reference: everything, including inherited/global flags +atmos wrapper --help=hidden # ONLY wrapper's hidden subcommands (hidden: true custom + # commands, or hidden built-ins) -- omitted from every other topic ``` Prefer `--help=usage` over the default `--help` when you just need to confirm how a command is invoked -- it skips straight to worked examples instead of the full flag listing. Reach for `--help=all` only when you need the complete inherited-flag surface (e.g. global `--stack`/`--format`/logging flags shared across -commands). +commands). Reach for `--help=hidden` when a command's default action or steps reference a subcommand name +that never shows up in `--help` -- it is almost certainly a `hidden: true` custom command, not a typo. ## Introspection Workflow for AI Agents diff --git a/agent-skills/skills/atmos-migration/references/from-justfile.md b/agent-skills/skills/atmos-migration/references/from-justfile.md index 89b2c7af09e..5b9a9e2f912 100644 --- a/agent-skills/skills/atmos-migration/references/from-justfile.md +++ b/agent-skills/skills/atmos-migration/references/from-justfile.md @@ -43,8 +43,10 @@ _clean: command `flags:` entry with a matching `default:` value. Inside a step, read the value as `{{ .Flags.env }}`. Do not use Just's own `{{env}}` syntax. See [Common Problems](#--interpolation-looks-like-atmos-templates-but-is-not) below. -3. Do not create a separate command for a `[private]` recipe. There is no `hidden` or `private` - field on Atmos custom commands. Put the recipe's body in a step of the command that calls it. +3. Set `hidden: true` on a command created from a `[private]` recipe. It runs normally + (`atmos ...`, as a `default:` target, or from another command's steps) but is excluded + from `atmos --help` listings and completion suggestions. Only inline the recipe's body into a + caller's step when it is genuinely single-caller logic with no reason to be invoked on its own. ```yaml commands: @@ -115,10 +117,10 @@ export AWS_REGION := "us-east-1" **Steps:** - Turn `export VAR := value` into a command or step `env:` map. -- Atmos has no confirmed built-in feature that matches `set dotenv-load`. Do not claim it does. - Add a step that loads the `.env` file directly, or, if the values are secrets, use Atmos's - store or secrets integration instead of a plain `.env` file. Ask the user which option fits - their case. +- `set dotenv-load` maps to `env: !include .env` on the command, workflow, or step. Atmos parses + the dotenv file natively (including `export VAR=value`, comments, quoting, and `${VAR}` + expansion) and merges the result into `env:`. If the values are secrets rather than plain + config, use Atmos's store or secrets integration instead of a plaintext `.env` file. - `set shell := [...]` changes the shell for every recipe in the Justfile. Atmos has no matching command-level setting. Use `type: script` with an explicit `interpreter:` field on the one step that needs a different interpreter. @@ -128,6 +130,7 @@ commands: - name: build description: Build the deployable artifact env: + <<: !include .env AWS_REGION: us-east-1 steps: - type: shell @@ -143,18 +146,24 @@ run in different tools at different times. Do not copy Just interpolation syntax YAML. Change each reference to the matching `{{ .Flags. }}` or `{{ .Arguments. }}` form. -### `[private]` recipes have no equivalent field +### `[private]` recipes map to `hidden: true` -There is no `hidden` or `private` field in the custom command schema. Put a private helper -recipe's logic in a step inside the command or workflow that calls it. Do not create a separate -command just to hide it from `--list` or `--help`. +The custom command schema has a `hidden: true` field. It excludes the command from `atmos --help` +listings and completion suggestions while leaving it fully runnable -- directly, as a `default:` +target, or from another command's steps. This is the direct equivalent of a `[private]` recipe, +and it covers cases plain step-inlining cannot: a helper called from more than one recipe, or one +a user invokes by name for manual debugging. + +Only inline a `[private]` recipe's logic into a caller's step when it is genuinely single-caller +and has no reason to be invoked on its own -- in that case a separate `hidden` command is just +unnecessary indirection. If a `[private]` recipe is never called by any public recipe (an orphaned helper, not a -dependency), there is no public recipe to fold its steps into. Do not create a public `atmos` -command just to preserve it -- that changes its visibility, which is the opposite of what -`[private]` meant. Confirm with the user whether the recipe is still needed at all; if it is, -ask where its logic should live (its own step inside whichever command ends up needing it, or a -short script the user maintains separately) rather than migrating it by default. +dependency), `hidden: true` no longer forces the same discovery you'd get from step-inlining -- +it would just as quietly hide dead code as reachable helper code. Confirm with the user whether +the recipe is still needed at all before migrating it; if it is, ask whether it should become a +`hidden` command, a step inside whichever command ends up needing it, or a short script the user +maintains separately. ### Command echo differs between `just` and Atmos @@ -165,14 +174,17 @@ command's side effects match the original recipe, but the terminal output will l by side. Tell the user this if they compare `just ` output to `atmos ` output directly; it is a visible difference, not a bug. -### Confirm `set dotenv-load` and `set shell` with the user +### Confirm `set shell` with the user; `dotenv-load` has a direct replacement -Do not drop either setting without comment. Ask the user if the behavior matters to their -workflow, such as loading secrets or using non-default shell syntax. Then pick the correct -replacement for their case. +`set dotenv-load` maps directly to `env: !include .env` -- no confirmation needed unless the +`.env` file holds secrets, in which case ask whether to use Atmos's store or secrets integration +instead. `set shell` has no command-level equivalent; ask the user if a non-default shell matters +to their workflow, then apply `type: script` with `interpreter:` to the specific steps that need +it. ## What Not To Do - Do not assume `{{ }}` means the same thing after you move it into Atmos YAML. -- Do not invent a `hidden` or `private` command field. It does not exist. -- Do not drop `dotenv-load` or `set shell` behavior without telling the user. +- Do not invent a visibility value beyond the documented `hidden: true` boolean (no "public"/"private" enum, no partial visibility). +- Do not drop `set shell` behavior without telling the user; `dotenv-load` maps directly to + `env: !include .env`, so it does not need the same case-by-case confirmation. diff --git a/agent-skills/skills/atmos-migration/references/from-makefile.md b/agent-skills/skills/atmos-migration/references/from-makefile.md index 2229ecb0c01..021c356c2c1 100644 --- a/agent-skills/skills/atmos-migration/references/from-makefile.md +++ b/agent-skills/skills/atmos-migration/references/from-makefile.md @@ -207,9 +207,11 @@ language. - Do not build file-timestamp or `.PHONY` caching as an Atmos feature. It does not exist. State this directly instead of dropping the behavior without comment. -- Do not turn every private or helper recipe into its own discoverable command. There is no - `hidden` or `private` field on custom commands. Put the helper logic in a step inside the - command or workflow that needs it. +- Do not turn every private or helper target into its own discoverable command by default. If the + helper is called from only one recipe, put its logic in a step inside the command or workflow + that needs it. If it needs to be called from more than one recipe, or invoked directly for + debugging, make it a custom command with `hidden: true` instead -- it stays runnable but is + excluded from `atmos --help` listings and completion suggestions. - Do not treat "wrap `atmos` commands in the Makefile" as the final state. It is a valid bridge during early migration, as shown in Shape B, step 4. Leaf targets should become custom commands. Target chains should become workflows. diff --git a/agent-skills/skills/atmos-migration/references/from-taskfile.md b/agent-skills/skills/atmos-migration/references/from-taskfile.md index e62d3674202..e49d4fa91a7 100644 --- a/agent-skills/skills/atmos-migration/references/from-taskfile.md +++ b/agent-skills/skills/atmos-migration/references/from-taskfile.md @@ -44,6 +44,10 @@ tasks: reserved for native Atmos verbs only. If the line calls another custom command (for example `atmos build`), keep it as a `type: shell` step with `command: atmos build` -- do not use `type: atmos` for that. +3. Set `hidden: true` on a command created from an `internal: true` task. It runs normally + (`atmos ...`, as a `default:` target, or from another command's steps) but is excluded + from `atmos --help` listings and completion suggestions. Only inline the task's body into a + caller's step when it is genuinely single-caller logic with no reason to be invoked on its own. ```yaml commands: @@ -177,3 +181,6 @@ State it directly. Do not gloss over it. default concurrency. - Do not describe `require`/`assert` as a freshness or caching check. It only checks that something exists. +- Do not turn every `internal: true` task into its own discoverable command by default. If it is + called from only one task, inline it into that caller's step. If it needs to be called from + more than one task, or invoked directly for debugging, make it a `hidden: true` custom command. diff --git a/cmd/cmd_utils.go b/cmd/cmd_utils.go index fd3e1d3f611..92b2dee63e1 100644 --- a/cmd/cmd_utils.go +++ b/cmd/cmd_utils.go @@ -480,10 +480,11 @@ func createCustomCommand( parentCommand *cobra.Command, ) (*cobra.Command, error) { customCommand := &cobra.Command{ - Use: commandConfig.Name, - Short: commandConfig.Description, - Long: commandConfig.Description, - Args: customCommandArgsValidator(commandConfig), + Use: commandConfig.Name, + Short: commandConfig.Description, + Long: commandConfig.Description, + Hidden: commandConfig.Hidden, + Args: customCommandArgsValidator(commandConfig), Annotations: map[string]string{ annotationCustomCommand: annotationValueTrue, }, diff --git a/cmd/cmd_utils_test.go b/cmd/cmd_utils_test.go index 18650e1d916..6c9734fd225 100644 --- a/cmd/cmd_utils_test.go +++ b/cmd/cmd_utils_test.go @@ -2398,6 +2398,147 @@ func TestExecuteCustomCommandShellStepPropagatesCILogGroupSentinel(t *testing.T) assert.Equal(t, "1", string(got)) } +func TestCreateCustomCommandHidden(t *testing.T) { + atmosConfig := &schema.AtmosConfiguration{} + + t.Run("hidden command is excluded from help listings", func(t *testing.T) { + parentCmd := &cobra.Command{Use: "atmos"} + hiddenCmd, err := createCustomCommand(atmosConfig, &schema.Command{ + Name: "helper", + Description: "hidden helper", + Hidden: true, + }, parentCmd) + require.NoError(t, err) + assert.True(t, hiddenCmd.Hidden) + assert.False(t, hiddenCmd.IsAvailableCommand(), "a hidden command must not report as available for help listings") + }) + + t.Run("unset hidden keeps the command visible", func(t *testing.T) { + parentCmd := &cobra.Command{Use: "atmos"} + visibleCmd, err := createCustomCommand(atmosConfig, &schema.Command{ + Name: "visible", + Description: "visible command", + }, parentCmd) + require.NoError(t, err) + assert.False(t, visibleCmd.Hidden) + assert.True(t, visibleCmd.IsAvailableCommand()) + }) +} + +func TestHiddenCommandStillExecutesDirectly(t *testing.T) { + _ = NewTestKit(t) + ensureIOInitialized(t) + + workDir := t.TempDir() + sentinelFile := filepath.Join(workDir, "sentinel.txt") + + atmosConfig := schema.AtmosConfiguration{BasePath: workDir} + parentCmd := &cobra.Command{Use: "atmos"} + commands := []schema.Command{{ + Name: "hidden-helper", + Description: "a hidden helper command", + Hidden: true, + WorkingDirectory: workDir, + Steps: []schema.Task{{ + Name: "write-sentinel", + Type: schema.TaskTypeShell, + Command: `printf %s "ran" > sentinel.txt`, + }}, + }} + + require.NoError(t, processCustomCommands(atmosConfig, commands, parentCmd)) + hiddenCmd := findSubcommand(parentCmd, "hidden-helper") + require.NotNil(t, hiddenCmd) + require.True(t, hiddenCmd.Hidden) + require.False(t, hiddenCmd.IsAvailableCommand()) + + // Hidden must not gate execution: invoking it directly still runs its steps. + hiddenCmd.PreRun(hiddenCmd, nil) + hiddenCmd.Run(hiddenCmd, nil) + + got, err := os.ReadFile(sentinelFile) + require.NoError(t, err) + assert.Equal(t, "ran", string(got)) +} + +func TestNestedCommandVisibilityIsIndependentOfParent(t *testing.T) { + atmosConfig := schema.AtmosConfiguration{} + parentCmd := &cobra.Command{Use: "atmos"} + + commands := []schema.Command{ + { + Name: "hidden-parent", + Description: "hidden parent with a visible child", + Hidden: true, + Commands: []schema.Command{ + {Name: "visible-child", Description: "visible child of a hidden parent"}, + }, + }, + { + Name: "visible-parent", + Description: "visible parent with a hidden child", + Commands: []schema.Command{ + {Name: "hidden-child", Description: "hidden child of a visible parent", Hidden: true}, + }, + }, + } + + require.NoError(t, processCustomCommands(atmosConfig, commands, parentCmd)) + + hiddenParent := findSubcommand(parentCmd, "hidden-parent") + require.NotNil(t, hiddenParent) + assert.True(t, hiddenParent.Hidden) + visibleChild := findSubcommand(hiddenParent, "visible-child") + require.NotNil(t, visibleChild) + assert.False(t, visibleChild.Hidden, "a visible child must stay visible even though its parent is hidden") + + visibleParent := findSubcommand(parentCmd, "visible-parent") + require.NotNil(t, visibleParent) + assert.False(t, visibleParent.Hidden) + hiddenChild := findSubcommand(visibleParent, "hidden-child") + require.NotNil(t, hiddenChild) + assert.True(t, hiddenChild.Hidden, "a hidden child must stay hidden even though its parent is visible") +} + +func TestDefaultDispatchToHiddenChildStillExecutes(t *testing.T) { + _ = NewTestKit(t) + ensureIOInitialized(t) + + workDir := t.TempDir() + sentinelFile := filepath.Join(workDir, "sentinel.txt") + + atmosConfig := schema.AtmosConfiguration{BasePath: workDir} + parentCmd := &cobra.Command{Use: "atmos"} + commands := []schema.Command{{ + Name: "wrapper", + Description: "visible wrapper whose default action is a hidden helper", + Default: "helper", + Commands: []schema.Command{ + { + Name: "helper", + Description: "hidden implementation", + Hidden: true, + WorkingDirectory: workDir, + Steps: []schema.Task{{ + Name: "write-sentinel", + Type: schema.TaskTypeShell, + Command: `printf %s "ran" > sentinel.txt`, + }}, + }, + }, + }} + + require.NoError(t, processCustomCommands(atmosConfig, commands, parentCmd)) + wrapperCmd := findSubcommand(parentCmd, "wrapper") + require.NotNil(t, wrapperCmd) + + wrapperCmd.Run(wrapperCmd, nil) + + got, err := os.ReadFile(sentinelFile) + require.NoError(t, err) + assert.Equal(t, "ran", string(got)) +} + func TestFindTypedValue(t *testing.T) { tests := []struct { name string diff --git a/cmd/help_template.go b/cmd/help_template.go index f81adb6992e..cfde6d4a99a 100644 --- a/cmd/help_template.go +++ b/cmd/help_template.go @@ -398,6 +398,53 @@ func printAvailableCommands(ctx *helpRenderContext, cmd *cobra.Command) { }) } +// hiddenSubcommands returns cmd's direct subcommands that are marked Hidden, in Commands() order. +// This includes both `hidden: true` custom commands and any hidden built-in commands. +func hiddenSubcommands(cmd *cobra.Command) []*cobra.Command { + var hidden []*cobra.Command + for _, c := range cmd.Commands() { + if c.Hidden { + hidden = append(hidden, c) + } + } + return hidden +} + +// printHiddenCommands prints the list of hidden subcommands for the `--help=hidden` topic. +// Unlike printAvailableCommands, this shows ONLY commands excluded from ordinary help output +// (via `hidden: true` on a custom command, or a hidden built-in) -- the inverse listing, for +// discovering what a bare `--help` deliberately omits. +func printHiddenCommands(ctx *helpRenderContext, cmd *cobra.Command) { + defer perf.Track(nil, "cmd.printHiddenCommands")() + + hidden := hiddenSubcommands(cmd) + if len(hidden) == 0 { + fmt.Fprintln(ctx.writer, ctx.styles.muted.Render(fmt.Sprintf("%s has no hidden subcommands.", cmd.CommandPath()))) + return + } + + parentExperimental := isExperimentalCommand(cmd) + + var mdRenderer *markdown.Renderer + if ctx.atmosConfig != nil { + mdRenderer, _ = markdown.NewTerminalMarkdownRenderer(*ctx.atmosConfig) + } + + maxCmdWidth := 0 + for _, c := range hidden { + if width := calculateCommandWidth(c, parentExperimental); width > maxCmdWidth { + maxCmdWidth = width + } + } + + fmt.Fprintln(ctx.writer, ctx.styles.heading.Render("HIDDEN COMMANDS")) + fmt.Fprintln(ctx.writer) + for _, c := range hidden { + formatCommandLine(ctx, c, maxCmdWidth, mdRenderer, parentExperimental) + } + fmt.Fprintln(ctx.writer) +} + // getConfigAliases returns all available config alias commands. func getConfigAliases(cmd *cobra.Command) []*cobra.Command { var aliases []*cobra.Command diff --git a/cmd/help_topics.go b/cmd/help_topics.go index eaea403b9a7..bfb7be98d5b 100644 --- a/cmd/help_topics.go +++ b/cmd/help_topics.go @@ -9,12 +9,14 @@ const ( helpTopicUsage helpTopic = "usage" helpTopicFlags helpTopic = "flags" helpTopicAll helpTopic = "all" + helpTopicHidden helpTopic = "hidden" ) var supportedHelpTopics = []helpTopic{ helpTopicUsage, helpTopicFlags, helpTopicAll, + helpTopicHidden, } type helpTopicRequest struct { diff --git a/cmd/help_topics_render.go b/cmd/help_topics_render.go index 59c03d303f2..b9a18e97044 100644 --- a/cmd/help_topics_render.go +++ b/cmd/help_topics_render.go @@ -72,6 +72,8 @@ func printHelpForTopic(ctx *helpRenderContext, cmd *cobra.Command, topic helpTop printCompatibilityFlags(ctx.writer, cmd, ctx.styles) case helpTopicAll: printFullHelp(ctx, cmd) + case helpTopicHidden: + printHiddenCommands(ctx, cmd) default: printDefaultHelp(ctx, cmd) } @@ -107,13 +109,17 @@ func printDefaultHelp(ctx *helpRenderContext, cmd *cobra.Command) { printLocalFlagsOnly(ctx.writer, cmd, ctx.atmosConfig, ctx.styles) printCompatibilityFlags(ctx.writer, cmd, ctx.styles) printFooter(ctx.writer, cmd, ctx.styles) - printHelpTopicHint(ctx.writer, ctx.styles) + printHelpTopicHint(ctx.writer, ctx.styles, cmd) } -func printHelpTopicHint(w io.Writer, styles *helpStyles) { +func printHelpTopicHint(w io.Writer, styles *helpStyles, cmd *cobra.Command) { defer perf.Track(nil, "cmd.printHelpTopicHint")() usageMsg := "Use `--help=usage` for examples or `--help=all` for all flags and full help." + if len(hiddenSubcommands(cmd)) > 0 { + usageMsg = "Use `--help=usage` for examples, `--help=all` for all flags and full help, " + + "or `--help=hidden` to list hidden subcommands." + } usageMsg = renderMarkdownDescription(usageMsg) fmt.Fprintf(w, "\n%s\n", styles.muted.Render(usageMsg)) } diff --git a/cmd/help_topics_test.go b/cmd/help_topics_test.go index b11921294a0..5509ee4d708 100644 --- a/cmd/help_topics_test.go +++ b/cmd/help_topics_test.go @@ -56,6 +56,16 @@ func TestNormalizeHelpTopicArgs(t *testing.T) { expectedSet: true, expectedChange: true, }, + { + name: "hidden topic", + args: []string{"terraform", "plan", "--help=hidden"}, + expectedArgs: []string{"terraform", "plan", "--help"}, + expectedTopic: helpTopicHidden, + expectedRaw: "hidden", + expectedValid: true, + expectedSet: true, + expectedChange: true, + }, { name: "uppercase topic", args: []string{"terraform", "plan", "--help=USAGE"}, @@ -153,6 +163,42 @@ func TestTopicHelpRendering_AllIncludesGlobalFlags(t *testing.T) { assert.Contains(t, output, "--global") } +func testHelpCommandWithHiddenChild(t *testing.T) *cobra.Command { + t.Helper() + + parent := &cobra.Command{Use: "parent", Short: "parent command"} + visible := &cobra.Command{Use: "visible", Short: "a visible child", Run: func(cmd *cobra.Command, args []string) {}} + hidden := &cobra.Command{Use: "secret", Short: "a hidden child", Hidden: true, Run: func(cmd *cobra.Command, args []string) {}} + parent.AddCommand(visible, hidden) + + return parent +} + +func TestTopicHelpRendering_HiddenShowsOnlyHiddenSubcommands(t *testing.T) { + output := renderTopicHelpForTest(t, helpTopicRequest{topic: helpTopicHidden, explicit: true, valid: true}, testHelpCommandWithHiddenChild(t)) + + assert.Contains(t, output, "HIDDEN COMMANDS") + assert.Contains(t, output, "secret") + assert.Contains(t, output, "a hidden child") + assert.NotContains(t, output, "visible") + assert.NotContains(t, output, "a visible child") +} + +func TestTopicHelpRendering_HiddenWithNoHiddenSubcommandsShowsMessage(t *testing.T) { + output := renderTopicHelpForTest(t, helpTopicRequest{topic: helpTopicHidden, explicit: true, valid: true}, testHelpCommand(t)) + + assert.Contains(t, output, "has no hidden subcommands") + assert.NotContains(t, output, "HIDDEN COMMANDS") +} + +func TestTopicHelpRendering_DefaultHintMentionsHiddenWhenPresent(t *testing.T) { + output := renderTopicHelpForTest(t, helpTopicRequest{valid: true}, testHelpCommandWithHiddenChild(t)) + + // Match "help=hidden" rather than "--help=hidden": word-wrapping in the rendered hint + // can split the leading "--" onto the previous line at narrower terminal widths. + assert.Contains(t, output, "help=hidden") +} + func TestTopicHelpRendering_RootDefaultFiltersPersistentGlobals(t *testing.T) { root := &cobra.Command{ Use: "atmos", diff --git a/docs/prd/topic-specific-cli-help.md b/docs/prd/topic-specific-cli-help.md index b3965977806..428ea1a07a4 100644 --- a/docs/prd/topic-specific-cli-help.md +++ b/docs/prd/topic-specific-cli-help.md @@ -2,7 +2,7 @@ ## Status: Implemented -**Last Updated**: 2026-07-06 +**Last Updated**: 2026-08-06 ## Overview @@ -14,6 +14,7 @@ The v1 topic set is intentionally small: - `--help=usage` shows only usage and embedded usage examples. - `--help=flags` shows command-specific flags, excluding inherited/global flags. - `--help=all` shows the full help page, including inherited/global flags. +- `--help=hidden` shows only the command's hidden subcommands (custom commands with `hidden: true`, or a hidden built-in) -- the commands every other topic, including `--help=all`, omits. ## Motivation @@ -78,13 +79,25 @@ atmos terraform plan --help=all Shows the complete help page, including inherited/global flags. +Hidden subcommands: + +```shell +atmos wrapper --help=hidden +``` + +Shows only the command's directly hidden subcommands (custom commands with `hidden: true`, plus any +hidden built-ins), formatted the same as an ordinary command listing. Shows a one-line "has no hidden +subcommands" message instead of an empty section when there are none. Default help includes a hint for +`--help=hidden` only on commands that actually have a hidden subcommand, so the hint doesn't clutter +help output for the common case. + Unknown topic: ```shell atmos terraform plan --help=advanced ``` -Exits non-zero and lists valid topics: `usage`, `flags`, `all`. +Exits non-zero and lists valid topics: `usage`, `flags`, `all`, `hidden`. ## Implementation Notes @@ -97,6 +110,7 @@ Exits non-zero and lists valid topics: `usage`, `flags`, `all`. - `usage`: usage and examples. - `flags`: command-specific flags and compatibility flags. - `all`: current full help. + - `hidden`: only the command's hidden direct subcommands, or a "no hidden subcommands" message. - For the root command, filter root persistent flags from default and `--help=flags` output because they are global flags even though Cobra exposes them as local root flags. ## Acceptance Criteria @@ -105,5 +119,7 @@ Exits non-zero and lists valid topics: `usage`, `flags`, `all`. - `atmos --help=usage` excludes flags and includes usage examples when present. - `atmos --help=flags` excludes inherited/global flags. - `atmos --help=all` includes inherited/global flags. +- `atmos --help=hidden` shows only hidden direct subcommands, and never a command that's + already visible in the default listing. - `atmos --help=` exits non-zero and lists valid topics. - Existing `-h`, bare `--help`, and pager behavior continue to work. diff --git a/pkg/datafetcher/schema/atmos/config/1.0.json b/pkg/datafetcher/schema/atmos/config/1.0.json index d45854e87ad..dd7f8f9e435 100644 --- a/pkg/datafetcher/schema/atmos/config/1.0.json +++ b/pkg/datafetcher/schema/atmos/config/1.0.json @@ -3259,6 +3259,20 @@ } ] }, + "hidden": { + "anyOf": [ + { + "type": "boolean" + }, + { + "type": "null" + }, + { + "$ref": "#/$defs/yamlFunction" + } + ], + "description": "Hidden excludes the command from `atmos --help` / `atmos \u003cgroup\u003e --help` subcommand\nlistings, shell-completion suggestions, and the AI `atmos_list_commands` tool, while\nleaving it fully runnable: `atmos \u003cname\u003e ...` still executes it directly, and\n`atmos \u003cname\u003e --help` still renders its own help when invoked explicitly. Use for\nhelper commands meant to be called by other commands or run manually for debugging,\nanalogous to Just's `[private]` recipes or Task's `internal: true` tasks." + }, "identity": { "anyOf": [ { diff --git a/pkg/schema/command.go b/pkg/schema/command.go index 3eeda5f4a89..58b0f968120 100644 --- a/pkg/schema/command.go +++ b/pkg/schema/command.go @@ -30,7 +30,14 @@ type Command struct { Steps Tasks `yaml:"steps" json:"steps" mapstructure:"steps"` Commands []Command `yaml:"commands" json:"commands" mapstructure:"commands"` Verbose bool `yaml:"verbose" json:"verbose" mapstructure:"verbose"` - Identity string `yaml:"identity,omitempty" json:"identity,omitempty" mapstructure:"identity"` + // Hidden excludes the command from `atmos --help` / `atmos --help` subcommand + // listings, shell-completion suggestions, and the AI `atmos_list_commands` tool, while + // leaving it fully runnable: `atmos ...` still executes it directly, and + // `atmos --help` still renders its own help when invoked explicitly. Use for + // helper commands meant to be called by other commands or run manually for debugging, + // analogous to Just's `[private]` recipes or Task's `internal: true` tasks. + Hidden bool `yaml:"hidden,omitempty" json:"hidden,omitempty" mapstructure:"hidden"` + Identity string `yaml:"identity,omitempty" json:"identity,omitempty" mapstructure:"identity"` } // CommandArgument defines a positional argument for a custom command. diff --git a/website/docs/cli/configuration/commands/command/commands.mdx b/website/docs/cli/configuration/commands/command/commands.mdx index 1eb3b5d848d..dad77dd07c0 100644 --- a/website/docs/cli/configuration/commands/command/commands.mdx +++ b/website/docs/cli/configuration/commands/command/commands.mdx @@ -29,6 +29,36 @@ Run a nested subcommand by chaining the names: atmos list stacks ``` +## Loading Environment Variables from a Dotenv File + +A command's `env:` field accepts [`!include`](/functions/yaml/include) to load values directly +from a `.env` file, instead of listing each variable inline: + +```yaml +commands: + - name: deploy + description: Deploy the application + env: !include .env + steps: + - atmos terraform apply app -s {{ .Arguments.stack }} +``` + +Use a YAML merge key to combine dotenv values with inline overrides: + +```yaml +commands: + - name: deploy + description: Deploy the application + env: + <<: !include .env + AWS_REGION: us-east-2 + steps: + - atmos terraform apply app -s {{ .Arguments.stack }} +``` + +See [Environment Variables](/cli/configuration/env#including-dotenv-files) for the full dotenv +`!include` behavior, including merging multiple files and precedence rules. + ## Path-Based Names When intermediate command levels only exist to spell out the command path, you can put the full path in `name` instead of writing nested `commands` arrays: diff --git a/website/docs/cli/configuration/commands/command/index.mdx b/website/docs/cli/configuration/commands/command/index.mdx index aa1470c99b1..6666302959b 100644 --- a/website/docs/cli/configuration/commands/command/index.mdx +++ b/website/docs/cli/configuration/commands/command/index.mdx @@ -85,4 +85,7 @@ atmos deploy vpc -s plat-ue2-prod
`verbose`
Set to `true` to print verbose execution output for the command. Defaults to `false`.
+ +
`hidden`
+
Set to `true` to exclude the command from `atmos --help` listings and completion suggestions while keeping it directly runnable (`atmos ...`) and its own `--help` output intact. Defaults to `false`. List a parent's hidden subcommands with `atmos --help=hidden`.
diff --git a/website/docs/migration/justfile.mdx b/website/docs/migration/justfile.mdx index e8cd6e2c336..1b465817df2 100644 --- a/website/docs/migration/justfile.mdx +++ b/website/docs/migration/justfile.mdx @@ -21,6 +21,17 @@ and arguments than Make's or Task's syntax does, so this migration is direct. Yo code and scripts do not need to change. Atmos replaces only the Justfile's recipes. +:::tip Using an AI Coding Assistant? +Install the `atmos-migration` skill so Claude Code, Cursor, GitHub Copilot, and other AI coding +assistants can apply this guide directly to your repository: + +```shell +atmos ai skill install atmos-migration +``` + +See [AI Agent Skills](/ai/agent-skills) for details. +::: + ## Why Some Teams Choose Just Over Make Just recipes do not need tab indentation, the way Make recipes do. A recipe's parameters are @@ -36,9 +47,9 @@ only cares what the recipe does. | `recipe: dep1 dep2` (recipe dependency) | Steps in order, or a `parallel` step with `needs:` | | `# comment` above a recipe | Command `description:` (replaces `just --list`) | | `export VAR := value` | Command or step `env:` map | -| `set dotenv-load` | No built-in match. Add a step, or use a store | +| `set dotenv-load` | `env: !include .env` on the command, workflow, or step | | `set shell := [...]` | Per-step `type: script` with `interpreter:` | -| `[private]` recipe | No match. Put the logic in a step instead | +| `[private]` recipe | Command `hidden: true` | | `{{ }}` interpolation | `{{ .Flags. }}` / `{{ .Arguments. }}` (a different tool) | ## Before and After @@ -137,12 +148,30 @@ to use a `parallel` step instead of plain steps in order. ## `[private]` Recipes and Environment Settings -There is no `hidden` or `private` field on Atmos custom commands. Put a `[private]` recipe's -logic in a step of the command that calls it. Do not create a separate command for it. +Set `hidden: true` on the custom command. The command still runs (`atmos ...`, or as a +`default:` target, or from another command's steps), but it's excluded from `atmos --help` +listings and completion suggestions — matching a `[private]` recipe's behavior in `just --list`. +Reserve inlining the logic into a caller's step for a helper that's genuinely single-caller and +has no reason to be invoked on its own. + +Atmos loads `.env` files natively with `env: !include .env`. Atmos parses the dotenv format +(including `export VAR=value`, comments, quoting, and `${VAR}` expansion) and merges the result +into the command, workflow, or step `env:` map: + + +```yaml +commands: + - name: build + description: Build the deployable artifact + env: !include .env + steps: + - type: shell + command: go build -o bin/handler ./cmd/handler +``` + -`set dotenv-load` has no confirmed native `.env`-loading feature in Atmos. Do not assume it does. -Either add a step that loads the `.env` file directly, or, if the values are secrets, use Atmos's -store or secrets integration instead. +If the `.env` file holds secrets rather than plain configuration, prefer Atmos's store or secrets +integration instead of a plaintext `.env` file. ## What You Gain @@ -167,21 +196,23 @@ store or secrets integration instead. - [ ] Turn recipe parameters with defaults into command `flags:`/`arguments:` - [ ] Turn recipe dependencies into steps in order, or into a workflow - [ ] Turn `export VAR := value` into `env:` maps -- [ ] Decide how to replace `set dotenv-load`: an explicit step, or a store -- [ ] Put `[private]` recipes into the steps of the commands that call them +- [ ] Replace `set dotenv-load` with `env: !include .env` (or a store/secrets integration for secret values) +- [ ] Mark `[private]` recipes as `hidden: true` custom commands - [ ] Stop relying on `just --list`. `atmos help` replaces it ## Common Questions -Not on its own. Atmos has no confirmed native match for this feature. Add a step that loads the -`.env` file, or use Atmos's store or secrets integration instead of a plain `.env` file. +Yes. Use `env: !include .env` on the command, workflow, or step. Atmos parses the dotenv file +natively and merges its values into `env:`. Reserve Atmos's store or secrets integration for +values that are actually secret, since a plain `.env` file is not encrypted. -There is no `hidden` or `private` field on a custom command. Put the helper logic in a step -inside the command or workflow that uses it, instead of exposing it as its own command. +Set `hidden: true` on the custom command. It stays fully runnable, including as a `default:` +target or from another command's steps, but disappears from `atmos --help` listings and +completion suggestions. diff --git a/website/docs/migration/makefile.mdx b/website/docs/migration/makefile.mdx index 412bfa2de46..ee30fb38dd1 100644 --- a/website/docs/migration/makefile.mdx +++ b/website/docs/migration/makefile.mdx @@ -21,6 +21,17 @@ Makefile's targets and recipes. You can keep the Makefile as a thin wrapper arou commands while you migrate. +:::tip Using an AI Coding Assistant? +Install the `atmos-migration` skill so Claude Code, Cursor, GitHub Copilot, and other AI coding +assistants can apply this guide directly to your repository: + +```shell +atmos ai skill install atmos-migration +``` + +See [AI Agent Skills](/ai/agent-skills) for details. +::: + ## Key Concepts at a Glance | Makefile concept | Atmos equivalent | @@ -132,10 +143,11 @@ atmos build ## Target Chains Become Workflows -A step cannot call another custom command through `type: atmos`. That step type is only for -native Atmos verbs, such as `terraform plan` or `terraform apply`. To reuse a command from -another command or workflow step, call it directly with `type: shell` and -`command: atmos build`. +A `type: atmos` step runs the `atmos` binary itself, so it can call any atmos command, a native +verb such as `terraform apply` or another custom command such as `build`. To reuse a command +from another command or workflow step, prefer `type: atmos` with `command: build` over +`type: shell` with `command: atmos build`: `type: atmos` auto-applies `-s ` from the +step's `stack:` field and renders through Atmos's structured step output. If the Makefile's `deploy` target also selects a Terraform environment, for example through `-var-file=envs/$(ENV).tfvars`, that part is a Terraform migration step, not a Make migration @@ -222,9 +234,9 @@ to skip slow work, plan for that change directly. Do not assume the behavior sta -Yes, but not through `type: atmos`. That step type is only for native Atmos verbs, such as -`terraform plan`. To call another custom command, use a `type: shell` step with -`command: atmos `. +Yes, through either step type: `type: atmos` with `command: `, or +`type: shell` with `command: atmos `. Prefer `type: atmos` — it auto-applies +`-s ` and gets Atmos's structured step output, which `type: shell` does not. diff --git a/website/docs/migration/native-terraform.mdx b/website/docs/migration/native-terraform.mdx index 8034f05e710..12a494c1128 100644 --- a/website/docs/migration/native-terraform.mdx +++ b/website/docs/migration/native-terraform.mdx @@ -19,6 +19,17 @@ import TabItem from '@theme/TabItem' You're already 90% there. Your Terraform code doesn't need to change. Atmos gives you a documented, conventional way to manage your infrastructure—whether you're using Makefiles, shell scripts, or just raw Terraform commands. +:::tip Using an AI Coding Assistant? +Install the `atmos-migration` skill so Claude Code, Cursor, GitHub Copilot, and other AI coding +assistants can apply this guide directly to your repository: + +```shell +atmos ai skill install atmos-migration +``` + +See [AI Agent Skills](/ai/agent-skills) for details. +::: + ## Why This Guide? Most teams don't use Terraform in isolation. You're probably already using: diff --git a/website/docs/migration/taskfile.mdx b/website/docs/migration/taskfile.mdx index 735c442fa3f..91842599d68 100644 --- a/website/docs/migration/taskfile.mdx +++ b/website/docs/migration/taskfile.mdx @@ -21,16 +21,28 @@ another is direct. Two real differences in behavior still need attention first, Your Terraform code and scripts do not need to change. +:::tip Using an AI Coding Assistant? +Install the `atmos-migration` skill so Claude Code, Cursor, GitHub Copilot, and other AI coding +assistants can apply this guide directly to your repository: + +```shell +atmos ai skill install atmos-migration +``` + +See [AI Agent Skills](/ai/agent-skills) for details. +::: + ## Key Concepts at a Glance | Taskfile.yml concept | Atmos equivalent | |--------------------------------------------------|------------------------------------------------------------------| | `desc:` | Command `description:` | -| `cmds:` (list of shell commands) | `steps:` (`type: shell`, or `type: atmos` for native verbs) | +| `cmds:` (list of shell commands) | `steps:` (`type: shell`, or `type: atmos` for any atmos command) | | `deps:` (runs at the same time by default) | A `parallel` step. Plain steps in order change the default | | `vars:` / `env:` | Command `flags:` (with `default:`) / `env:` map | | `sources:` / `generates:` (freshness check) | No direct match. Steps always run | | `includes:` (multi-file composition) | Auto-discovered `atmos.d/*.yaml`, or separate workflow files | +| `internal: true` task | Command `hidden: true` | ## The Parallel-by-Default Difference @@ -93,6 +105,14 @@ choices: 2. Add a hash or timestamp check inside the shell step itself. This is a script you write and maintain, not a built-in Atmos feature. +## `internal: true` Tasks + +Set `hidden: true` on the custom command. The command still runs (`atmos ...`, or as a +`default:` target, or from another command's steps), but it's excluded from `atmos --help` +listings and completion suggestions — matching an `internal: true` task's behavior in +`task --list`. Reserve inlining the logic into a caller's step for a helper that's genuinely +single-caller and has no reason to be invoked on its own. + ## Before and After @@ -208,6 +228,7 @@ Choose the one that fits what you are splitting: - [ ] Turn `vars:`/`env:` into command `flags:` (with defaults) and `env:` maps - [ ] Decide how to handle `sources:`/`generates:` freshness checks. Do not assume parity - [ ] Split `includes:` into auto-discovered `atmos.d/*.yaml` files, separate workflow files, or both +- [ ] Mark `internal: true` tasks as `hidden: true` custom commands ## Common Questions @@ -225,8 +246,15 @@ instead of running them one after another without meaning to. -Yes, but not through `type: atmos`. That step type is only for native Atmos verbs, such as -`terraform plan`. Use a `type: shell` step with `command: atmos ` instead. +Yes, through either step type: `type: atmos` with `command: `, or +`type: shell` with `command: atmos `. Prefer `type: atmos` — it auto-applies +`-s ` and gets Atmos's structured step output, which `type: shell` does not. + + + +Set `hidden: true` on the custom command. It stays fully runnable, including as a `default:` +target or from another command's steps, but disappears from `atmos --help` listings and +completion suggestions. diff --git a/website/docs/migration/terraform-workspaces.mdx b/website/docs/migration/terraform-workspaces.mdx index ef4ea4ea216..4e7cab6da55 100644 --- a/website/docs/migration/terraform-workspaces.mdx +++ b/website/docs/migration/terraform-workspaces.mdx @@ -15,6 +15,17 @@ import TaskList from '@site/src/components/TaskList' Terraform workspaces solve a simple problem: deploy the same code to multiple environments using one state backend. Atmos can adopt that workspace-backed state while moving environment configuration into explicit stack files. +:::tip Using an AI Coding Assistant? +Install the `atmos-migration` skill so Claude Code, Cursor, GitHub Copilot, and other AI coding +assistants can apply this guide directly to your repository: + +```shell +atmos ai skill install atmos-migration +``` + +See [AI Agent Skills](/ai/agent-skills) for details. +::: + ## How Workspaces Fit With Atmos Workspaces are a valid way to organize Terraform state. When migrating to Atmos, the important shift is making configuration explicit in stacks while preserving the state layout that already works for your team. From 9cdf342834119f925d9d0d6312dc37f08d5acc86 Mon Sep 17 00:00:00 2001 From: Erik Osterman Date: Fri, 7 Aug 2026 07:43:53 -0500 Subject: [PATCH 07/17] fix(links): exclude reproducible-builds.org from link check MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Check Markdown Links workflow failed on the SOURCE_DATE_EPOCH citation in docs/prd/archive-step.md with "Connection refused". The domain refuses connections from every network tested (CI, curl, and WebFetch), not just this path or CI specifically — an upstream outage, not a broken/moved link — so exclude it the same way other known-flaky external docs are already handled in this file. Co-Authored-By: Claude Sonnet 5 --- lychee.toml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/lychee.toml b/lychee.toml index 3f81ffe0012..4f4f8f3650b 100644 --- a/lychee.toml +++ b/lychee.toml @@ -189,4 +189,10 @@ exclude = [ # This PDF intermittently times out from CI runners across multiple docs # referencing it; verified to return 200 outside CI via curl. "cis\\.upenn\\.edu/~bcpierce/papers/diff3-short\\.pdf", + # reproducible-builds.org (the canonical SOURCE_DATE_EPOCH spec, cited from + # docs/prd/archive-step.md) is the correct, unchanged URL, but the whole + # domain currently refuses connections from every network tested (CI, curl, + # and WebFetch all got connection-refused) -- an upstream outage, not a + # broken/moved link. Revisit this exclude if the outage is long resolved. + "reproducible-builds\\.org", ] From d1fc916b8a5dd37c702be075e6c5ff156c8d19ce Mon Sep 17 00:00:00 2001 From: Erik Osterman Date: Fri, 7 Aug 2026 08:12:56 -0500 Subject: [PATCH 08/17] fix(migration): address CodeRabbit review on PR #2896 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - lychee.toml: narrow the reproducible-builds.org exclude to the exact SOURCE_DATE_EPOCH path instead of the whole domain, so other links on that domain stay covered by the link check. - justfile.mdx/makefile.mdx/taskfile.mdx (+ agent-skills mirrors): the "after" Terraform-apply examples ran `terraform apply terraform`, confusing the atmos verb with a component literally named "terraform" that didn't match the shown legacy `terraform/` directory layout. Rename the placeholder component to `infra` and add a one-line note on where it maps to under `components.terraform.base_path`. - cmd_utils_test.go: document why the hidden-command tests' printf/ redirection is cross-platform (Atmos's TaskTypeShell runs through the in-process mvdan/sh interpreter, not the host shell) rather than replacing it — flagged as a platform-specific-binary risk, but it isn't one. Co-Authored-By: Claude Sonnet 5 --- .../atmos-migration/references/from-justfile.md | 7 ++++++- .../atmos-migration/references/from-makefile.md | 7 ++++++- .../atmos-migration/references/from-taskfile.md | 7 ++++++- cmd/cmd_utils_test.go | 6 ++++++ lychee.toml | 13 +++++++------ website/docs/migration/justfile.mdx | 9 +++++++-- website/docs/migration/makefile.mdx | 7 ++++++- website/docs/migration/taskfile.mdx | 7 ++++++- 8 files changed, 50 insertions(+), 13 deletions(-) diff --git a/agent-skills/skills/atmos-migration/references/from-justfile.md b/agent-skills/skills/atmos-migration/references/from-justfile.md index 5b9a9e2f912..178ad3e0127 100644 --- a/agent-skills/skills/atmos-migration/references/from-justfile.md +++ b/agent-skills/skills/atmos-migration/references/from-justfile.md @@ -101,9 +101,14 @@ commands: - type: shell command: atmos test - type: atmos - command: terraform apply terraform -s {{ .Flags.env }} + command: terraform apply infra -s {{ .Flags.env }} ``` +`infra` is a placeholder Atmos component name, not the `terraform` verb repeated. Move the +recipe's Terraform code to `components/terraform/infra/` (the default +`components.terraform.base_path` is `components/terraform`), then swap `infra` for whatever the +user actually names the component. + ## Shape C: Environment and Shell Settings **Before:** diff --git a/agent-skills/skills/atmos-migration/references/from-makefile.md b/agent-skills/skills/atmos-migration/references/from-makefile.md index 021c356c2c1..7139916ce42 100644 --- a/agent-skills/skills/atmos-migration/references/from-makefile.md +++ b/agent-skills/skills/atmos-migration/references/from-makefile.md @@ -132,9 +132,14 @@ commands: - type: shell command: atmos test - type: atmos - command: terraform apply terraform -s {{ .Flags.env }} + command: terraform apply infra -s {{ .Flags.env }} ``` +`infra` is a placeholder Atmos component name, not the `terraform` verb repeated. Move the +target's Terraform code to `components/terraform/infra/` (the default +`components.terraform.base_path` is `components/terraform`), then swap `infra` for whatever the +user actually names the component. + ## Shape C: Recursive or Parallel Make **Before:** diff --git a/agent-skills/skills/atmos-migration/references/from-taskfile.md b/agent-skills/skills/atmos-migration/references/from-taskfile.md index e49d4fa91a7..13e9cf53114 100644 --- a/agent-skills/skills/atmos-migration/references/from-taskfile.md +++ b/agent-skills/skills/atmos-migration/references/from-taskfile.md @@ -105,9 +105,14 @@ commands: type: shell command: atmos lint - type: atmos - command: terraform apply terraform -s dev + command: terraform apply infra -s dev ``` +`infra` is a placeholder Atmos component name, not the `terraform` verb repeated. Move the +task's Terraform code to `components/terraform/infra/` (the default +`components.terraform.base_path` is `components/terraform`), then swap `infra` for whatever the +user actually names the component. + If the Taskfile's `deps:` list needs its own internal order, add `needs:` to the steps inside the `parallel` block. Do not assume the tasks should run one after another just because that is Atmos's default for steps outside a `parallel` or `matrix` block. diff --git a/cmd/cmd_utils_test.go b/cmd/cmd_utils_test.go index 6c9734fd225..5f0f61ed67a 100644 --- a/cmd/cmd_utils_test.go +++ b/cmd/cmd_utils_test.go @@ -2439,6 +2439,9 @@ func TestHiddenCommandStillExecutesDirectly(t *testing.T) { Description: "a hidden helper command", Hidden: true, WorkingDirectory: workDir, + // TaskTypeShell runs through Atmos's own in-process mvdan/sh interpreter (see + // pkg/runner/step/shell.go), not the host shell, so printf and redirection here + // are cross-platform including on Windows -- not a platform-specific binary call. Steps: []schema.Task{{ Name: "write-sentinel", Type: schema.TaskTypeShell, @@ -2519,6 +2522,9 @@ func TestDefaultDispatchToHiddenChildStillExecutes(t *testing.T) { Description: "hidden implementation", Hidden: true, WorkingDirectory: workDir, + // TaskTypeShell runs through Atmos's own in-process mvdan/sh interpreter (see + // pkg/runner/step/shell.go), not the host shell, so printf and redirection here + // are cross-platform including on Windows -- not a platform-specific binary call. Steps: []schema.Task{{ Name: "write-sentinel", Type: schema.TaskTypeShell, diff --git a/lychee.toml b/lychee.toml index 4f4f8f3650b..b1e2c1d9c8c 100644 --- a/lychee.toml +++ b/lychee.toml @@ -189,10 +189,11 @@ exclude = [ # This PDF intermittently times out from CI runners across multiple docs # referencing it; verified to return 200 outside CI via curl. "cis\\.upenn\\.edu/~bcpierce/papers/diff3-short\\.pdf", - # reproducible-builds.org (the canonical SOURCE_DATE_EPOCH spec, cited from - # docs/prd/archive-step.md) is the correct, unchanged URL, but the whole - # domain currently refuses connections from every network tested (CI, curl, - # and WebFetch all got connection-refused) -- an upstream outage, not a - # broken/moved link. Revisit this exclude if the outage is long resolved. - "reproducible-builds\\.org", + # The SOURCE_DATE_EPOCH spec page, cited from docs/prd/archive-step.md, is + # the correct, unchanged URL, but currently refuses connections from every + # network tested (CI, curl, and WebFetch all got connection-refused) -- an + # upstream outage, not a broken/moved link. Scoped to this one path so other + # reproducible-builds.org links stay covered by the link check. Revisit this + # exclude once the outage is long resolved. + "reproducible-builds\\.org/docs/source-date-epoch/", ] diff --git a/website/docs/migration/justfile.mdx b/website/docs/migration/justfile.mdx index 1b465817df2..3b59fc49c1f 100644 --- a/website/docs/migration/justfile.mdx +++ b/website/docs/migration/justfile.mdx @@ -107,11 +107,16 @@ only cares what the recipe does. - type: shell command: atmos test - type: atmos - command: terraform apply terraform -s {{ .Flags.env }} + command: terraform apply infra -s {{ .Flags.env }} ``` +`infra` is a placeholder Atmos component name, not the `terraform` verb repeated. Move the old +`terraform/` directory's `.tf` files to `components/terraform/infra/` (the default +`components.terraform.base_path` is `components/terraform`), then swap `infra` for whatever you +actually name the component. + ## Named Parameters Become Flags and Arguments Just's `recipe param='default':` syntax maps directly to Atmos `flags:` (or `arguments:` for a @@ -130,7 +135,7 @@ commands: default: "dev" steps: - type: atmos - command: terraform apply terraform -s {{ .Flags.env }} + command: terraform apply infra -s {{ .Flags.env }} ``` diff --git a/website/docs/migration/makefile.mdx b/website/docs/migration/makefile.mdx index ee30fb38dd1..ae6c89897fe 100644 --- a/website/docs/migration/makefile.mdx +++ b/website/docs/migration/makefile.mdx @@ -111,11 +111,16 @@ See [AI Agent Skills](/ai/agent-skills) for details. - type: shell command: atmos test - type: atmos - command: terraform apply terraform -s {{ .Flags.env }} + command: terraform apply infra -s {{ .Flags.env }} ``` +`infra` is a placeholder Atmos component name, not the `terraform` verb repeated. Move the old +`terraform/` directory's `.tf` files to `components/terraform/infra/` (the default +`components.terraform.base_path` is `components/terraform`), then swap `infra` for whatever you +actually name the component. + ## Independent Targets Become Custom Commands Turn each `.PHONY` target that has no dependency into its own diff --git a/website/docs/migration/taskfile.mdx b/website/docs/migration/taskfile.mdx index 91842599d68..fe56632c947 100644 --- a/website/docs/migration/taskfile.mdx +++ b/website/docs/migration/taskfile.mdx @@ -81,11 +81,16 @@ dependency tasks inside a [`parallel`](/workflows/steps/type/parallel) step: type: shell command: atmos lint - type: atmos - command: terraform apply terraform -s dev + command: terraform apply infra -s dev ``` +`infra` is a placeholder Atmos component name, not the `terraform` verb repeated. Move the old +`terraform/` directory's `.tf` files to `components/terraform/infra/` (the default +`components.terraform.base_path` is `components/terraform`), then swap `infra` for whatever you +actually name the component. + If a Taskfile's `deps:` list needs an order between its own entries, add `needs:` to the steps inside the `parallel` block. Do not default to sequential steps just because that is Atmos's own default outside a `parallel` or `matrix` block. From 04f54d5778a000b023b689c22bac558ffc9e88f5 Mon Sep 17 00:00:00 2001 From: Erik Osterman Date: Fri, 7 Aug 2026 09:18:45 -0500 Subject: [PATCH 09/17] fix(io): make LinePrefixWriter's cross-node line batches atomic TestExecuteTerraformConcurrentHooksUseNodeWriters was failing in CI (reproduced locally under `go test -race -count=200`, ~30% failure rate): concurrent nodes' hook output was interleaving mid-record instead of staying grouped per node. writeLine() acquired the shared writeMu once per line, but a single Write() call can flush multiple buffered lines at once (e.g. a \r-terminated segment held back by a prior Write, completed by the next). Between the two per-line lock acquisitions for one node's burst, another node's own burst could interleave into the shared writer. Fixed by collecting a burst's complete lines up front and writing them under one writeMu acquisition (writeLinesLocked), so a whole burst lands as one contiguous block. Preserves the existing partial-write-error retry behavior: a failed line and everything after it, plus any trailing partial content, are restored to the buffer for the next Write/Flush to retry. Verified with `go test ./pkg/scheduler/adapters/... -race -count=500` (0 failures, was reproducibly failing before) and the full pkg/io suite, race detector, 5x. Co-Authored-By: Claude Sonnet 5 --- pkg/io/line_prefix_writer.go | 86 ++++++++++++++++++++++++------------ 1 file changed, 57 insertions(+), 29 deletions(-) diff --git a/pkg/io/line_prefix_writer.go b/pkg/io/line_prefix_writer.go index fab94879562..2e6ab56dbca 100644 --- a/pkg/io/line_prefix_writer.go +++ b/pkg/io/line_prefix_writer.go @@ -71,7 +71,7 @@ func (w *LinePrefixWriter) Write(p []byte) (int, error) { return len(p), nil } -// Flush writes any trailing partial line. +// Flush writes any trailing partial line, plus any complete lines still buffered. func (w *LinePrefixWriter) Flush() error { defer perf.Track(nil, "io.LinePrefixWriter.Flush")() @@ -81,57 +81,85 @@ func (w *LinePrefixWriter) Flush() error { if len(w.buffer) == 0 { return nil } - if err := w.flushCompleteLinesLocked(); err != nil { - return err + lines, rest := splitBufferedLines(w.buffer) + if len(rest) > 0 { + lines = append(lines, rest) + rest = nil } - if len(w.buffer) == 0 { + return w.writeLinesLocked(lines, rest) +} + +// flushCompleteLinesLocked writes buffered complete lines while w.mu is held, +// leaving any trailing partial line buffered for a later Write or Flush. +func (w *LinePrefixWriter) flushCompleteLinesLocked() error { + lines, rest := splitBufferedLines(w.buffer) + if len(lines) == 0 { return nil } - line := append([]byte(nil), w.buffer...) - if err := w.writeLine(line); err != nil { - return err - } - w.buffer = w.buffer[:0] - return nil + return w.writeLinesLocked(lines, rest) } -// flushCompleteLinesLocked writes buffered complete lines while w.mu is held. -func (w *LinePrefixWriter) flushCompleteLinesLocked() error { +// splitBufferedLines splits buf into its complete, delimited lines and any +// trailing partial content, without mutating buf. +func splitBufferedLines(buf []byte) (lines [][]byte, rest []byte) { + rest = buf for { - idx := lineEndIndex(w.buffer) + idx := lineEndIndex(rest) if idx < 0 { - return nil + return lines, rest } end := idx + 1 - if w.buffer[idx] == carriageReturnByte && end < len(w.buffer) && w.buffer[end] == lineFeedByte { + if rest[idx] == carriageReturnByte && end < len(rest) && rest[end] == lineFeedByte { end++ } - line := append([]byte(nil), w.buffer[:end]...) - if err := w.writeLine(line); err != nil { - return err - } - w.buffer = w.buffer[end:] + lines = append(lines, append([]byte(nil), rest[:end]...)) + rest = rest[end:] } } -// writeLine writes one already-delimited line with the configured prefix. -func (w *LinePrefixWriter) writeLine(line []byte) error { +// writeLinesLocked writes lines, in order, under a single writeMu acquisition +// so the whole batch lands on the shared underlying writer as one contiguous +// block instead of being interleaved line-by-line with a concurrent writer's +// own batch -- e.g. a \r-terminated segment held back by a prior Write plus +// the line that completes it in the next Write. On error, the line that +// failed and everything after it, plus rest, are restored to w.buffer so a +// later Write or Flush can retry them; nothing already written is repeated. +// The caller must already hold w.mu. +func (w *LinePrefixWriter) writeLinesLocked(lines [][]byte, rest []byte) error { if w.w == nil { + w.buffer = rest return nil } w.writeMu.Lock() defer w.writeMu.Unlock() - line = bytes.ReplaceAll(line, crlfBytes, lfBytes) - line = bytes.ReplaceAll(line, crBytes, lfBytes) + for i, line := range lines { + normalized := bytes.ReplaceAll(line, crlfBytes, lfBytes) + normalized = bytes.ReplaceAll(normalized, crBytes, lfBytes) - if w.prefix == "" { - _, err := w.w.Write(line) - return err + var err error + if w.prefix == "" { + _, err = w.w.Write(normalized) + } else { + _, err = stdio.WriteString(w.w, w.prefix+string(normalized)) + } + if err != nil { + w.buffer = append(joinLines(lines[i:]), rest...) + return err + } } + w.buffer = rest + return nil +} - _, err := stdio.WriteString(w.w, w.prefix+string(line)) - return err +// joinLines concatenates lines back into a single buffer, for restoring +// unwritten output after a partial-batch write failure. +func joinLines(lines [][]byte) []byte { + var out []byte + for _, line := range lines { + out = append(out, line...) + } + return out } // lineEndIndex returns the first complete line-ending byte position or -1 when absent. From d33cbf1319cf3408a0abf0343fa4384046fbb331 Mon Sep 17 00:00:00 2001 From: Erik Osterman Date: Fri, 7 Aug 2026 09:35:58 -0500 Subject: [PATCH 10/17] fix(security): remediate 7 npm Dependabot alerts in website/ Bump pnpm.overrides for transitively-pulled packages to their patched versions, all within the semver-major bump the dependabot.yml ignore policy blocks: - js-yaml 3.15.0 -> 3.15.1, 4.3.0 -> 4.3.1 (GHSA-5p4m-2wfm-xmqj, GHSA for the 4.x line): quadratic CPU consumption in !!omap resolution, fixes #269, #268. - mermaid 11.16.0 -> 11.16.1: fixes #267 (radar diagram DoS), #266 (config API prototype pollution), #265 (CSS injection), #264 (Architecture diagram prototype pollution), #263 (XY Chart infinite-loop DoS). No open CodeQL alerts. Verified with `atmos lint --changed` (0 issues) and `npm run build` in website/ (succeeds, same pre-existing unrelated broken-anchor warning as before this change). Co-Authored-By: Claude Sonnet 5 --- website/package.json | 6 +++--- website/pnpm-lock.yaml | 40 ++++++++++++++++++++-------------------- 2 files changed, 23 insertions(+), 23 deletions(-) diff --git a/website/package.json b/website/package.json index 4dc9ffa1cf7..8b2007e4cee 100644 --- a/website/package.json +++ b/website/package.json @@ -110,13 +110,13 @@ "follow-redirects@^1": "^1.16.0", "http-proxy-middleware@^2": "^2.0.10", "joi@^17": "^17.13.4", - "js-yaml@^3": "^3.15.0", - "js-yaml@^4": "^4.2.0", + "js-yaml@^3": "^3.15.1", + "js-yaml@^4": "^4.3.1", "launch-editor@^2": "^2.14.1", "lodash@^4": "^4.18.0", "lodash-es@^4": "^4.18.0", "mdast-util-to-hast@^13": "^13.2.1", - "mermaid@^11": "^11.15.0", + "mermaid@^11": "^11.16.1", "minimatch@^9": "^9.0.7", "nanoid@3.3.3": "^3.3.15", "nanoid@4.0.2": "^5.1.16", diff --git a/website/pnpm-lock.yaml b/website/pnpm-lock.yaml index 1d4fcff751c..0afb78f9b3e 100644 --- a/website/pnpm-lock.yaml +++ b/website/pnpm-lock.yaml @@ -15,13 +15,13 @@ overrides: follow-redirects@^1: ^1.16.0 http-proxy-middleware@^2: ^2.0.10 joi@^17: ^17.13.4 - js-yaml@^3: ^3.15.0 - js-yaml@^4: ^4.2.0 + js-yaml@^3: ^3.15.1 + js-yaml@^4: ^4.3.1 launch-editor@^2: ^2.14.1 lodash@^4: ^4.18.0 lodash-es@^4: ^4.18.0 mdast-util-to-hast@^13: ^13.2.1 - mermaid@^11: ^11.15.0 + mermaid@^11: ^11.16.1 minimatch@^9: ^9.0.7 nanoid@3.3.3: ^3.3.15 nanoid@4.0.2: ^5.1.16 @@ -4263,12 +4263,12 @@ packages: js-tokens@4.0.0: resolution: {integrity: sha512-RdJUflcE3cUzKiMqQgsCu06FPu9UdIJO0beYbPhHN4k6apgJtifcoCtT9bcxOpYBtpD2kCM6Sbzg4CausW/PKQ==} - js-yaml@3.15.0: - resolution: {integrity: sha512-ttBQIIQPDeLjpPOohtUdXuXUVoA2uIB6fEH9HyJ7234s5mBJ5wTx20njxplLZQgLaOfpmPQA7X2t5AX6tIPbog==} + js-yaml@3.15.1: + resolution: {integrity: sha512-S99WuO3HlhO3XN41EtYUNl9zzXjoJx7QvmipxsJVxtCBT0YHEFy+iOJhjSvrmV12nYhWpZaM8lPHkJm0yUMbag==} hasBin: true - js-yaml@4.3.0: - resolution: {integrity: sha512-1td788aAnnZ5qs7V2QIRl1owjtYpbKt749Y3xauqQgwIIGF/xXWz1wMTEBx5O3LK3lXLVuqXPdPxj2BoFHaW9Q==} + js-yaml@4.3.1: + resolution: {integrity: sha512-CY6crGq313MX8GkwvB7tzgp99vjQxY1++5y10/BKN/GUfHqWaOGQMNZkBvqSzsZKWk/ijwHlWzzkLulsGHhjWQ==} hasBin: true jsesc@3.1.0: @@ -4503,8 +4503,8 @@ packages: resolution: {integrity: sha512-8q7VEgMJW4J8tcfVPy8g09NcQwZdbwFEqhe/WZkoIzjn/3TGDwtOCYtXGxA3O8tPzpczCCDgv+P2P5y00ZJOOg==} engines: {node: '>= 8'} - mermaid@11.16.0: - resolution: {integrity: sha512-Zvm3kbstgdpvIJPPItlL7fppIZ3kibvc1oZIGxdvk9t6UFz6flv+Jw7FtRGKwfcI8OckmH04LqG6LlS6X4B1pA==} + mermaid@11.16.1: + resolution: {integrity: sha512-TQsq6u22fAn3rek5VOubrhKPo1g5hwC3FXUN9hiyupTckcYiGuuKGkNQrKYwGJkXUxZdojwRG46gsSCFZMDp4g==} methods@1.1.2: resolution: {integrity: sha512-iclAHeNqNm68zFtnZ0e+1L2yUIdvzNoauKU4WBA3VvH/vPFieF7qfRlwUZU+DA9P9bPXIS90ulxoUoCH23sV2w==} @@ -6634,7 +6634,7 @@ snapshots: '@11ty/gray-matter@1.0.0': dependencies: - js-yaml: 4.3.0 + js-yaml: 4.3.1 kind-of: 6.0.3 section-matter: 1.0.0 strip-bom-string: 1.0.0 @@ -8248,7 +8248,7 @@ snapshots: '@types/react-router-config': 5.0.11 combine-promises: 1.2.0 fs-extra: 11.3.2 - js-yaml: 4.3.0 + js-yaml: 4.3.1 lodash: 4.18.1 react: 18.3.1 react-dom: 18.3.1(react@18.3.1) @@ -8696,7 +8696,7 @@ snapshots: '@docusaurus/theme-common': 3.10.2(@docusaurus/plugin-content-docs@3.10.2(@mdx-js/react@3.1.1(@types/react@19.2.2)(react@18.3.1))(postcss@8.5.23)(react-dom@18.3.1(react@18.3.1))(react@18.3.1))(postcss@8.5.23)(react-dom@18.3.1(react@18.3.1))(react@18.3.1) '@docusaurus/types': 3.10.2(clean-css@5.3.3)(cssnano@6.1.2(postcss@8.5.23))(html-minifier-terser@7.2.0)(postcss@8.5.23)(react-dom@18.3.1(react@18.3.1))(react@18.3.1) '@docusaurus/utils-validation': 3.10.2(postcss@8.5.23)(react-dom@18.3.1(react@18.3.1))(react@18.3.1) - mermaid: 11.16.0 + mermaid: 11.16.1 react: 18.3.1 react-dom: 18.3.1(react@18.3.1) tslib: 2.8.1 @@ -8837,7 +8837,7 @@ snapshots: '@docusaurus/utils-common': 3.10.2(clean-css@5.3.3)(cssnano@6.1.2(postcss@8.5.23))(html-minifier-terser@7.2.0)(postcss@8.5.23)(react-dom@18.3.1(react@18.3.1))(react@18.3.1) fs-extra: 11.3.2 joi: 17.13.4 - js-yaml: 4.3.0 + js-yaml: 4.3.1 lodash: 4.18.1 tslib: 2.8.1 transitivePeerDependencies: @@ -8871,7 +8871,7 @@ snapshots: github-slugger: 1.5.0 globby: 11.1.0 jiti: 1.21.7 - js-yaml: 4.3.0 + js-yaml: 4.3.1 lodash: 4.18.1 micromatch: 4.0.8 p-queue: 6.6.2 @@ -8948,7 +8948,7 @@ snapshots: dependencies: '@excalidraw/markdown-to-text': 0.1.2 '@mermaid-js/parser': 0.6.3 - mermaid: 11.16.0 + mermaid: 11.16.1 nanoid: 5.1.16 transitivePeerDependencies: - supports-color @@ -10630,7 +10630,7 @@ snapshots: cosmiconfig@8.3.6: dependencies: import-fresh: 3.3.1 - js-yaml: 4.3.0 + js-yaml: 4.3.1 parse-json: 5.2.0 path-type: 4.0.0 @@ -11536,7 +11536,7 @@ snapshots: gray-matter@4.0.3: dependencies: - js-yaml: 3.15.0 + js-yaml: 3.15.1 kind-of: 6.0.3 section-matter: 1.0.0 strip-bom-string: 1.0.0 @@ -12010,12 +12010,12 @@ snapshots: js-tokens@4.0.0: {} - js-yaml@3.15.0: + js-yaml@3.15.1: dependencies: argparse: 1.0.10 esprima: 4.0.1 - js-yaml@4.3.0: + js-yaml@4.3.1: dependencies: argparse: 2.0.1 @@ -12351,7 +12351,7 @@ snapshots: merge2@1.4.1: {} - mermaid@11.16.0: + mermaid@11.16.1: dependencies: '@braintree/sanitize-url': 7.1.2 '@iconify/utils': 3.0.2 From d8485c4511b69dc3c0c9434b95d4c9bb449bef67 Mon Sep 17 00:00:00 2001 From: Erik Osterman Date: Fri, 7 Aug 2026 10:38:13 -0500 Subject: [PATCH 11/17] fix(io): preserve unwritten suffix on partial LinePrefixWriter writes Write and stdio.WriteString can return n > 0 with an error; the prior code ignored n and restored the entire raw line on retry, so bytes the underlying writer already accepted (including the prefix) could be resent. A nil-error short write (n < len(payload)) also silently dropped the unwritten tail. Track the encoded pending payload and retry only its unwritten suffix, converting a nil-error short write into io.ErrShortWrite. Co-Authored-By: Claude Sonnet 5 --- pkg/io/line_prefix_writer.go | 51 +++++++++++++++++++++++++------ pkg/io/line_prefix_writer_test.go | 50 ++++++++++++++++++++++++++++++ 2 files changed, 92 insertions(+), 9 deletions(-) diff --git a/pkg/io/line_prefix_writer.go b/pkg/io/line_prefix_writer.go index 2e6ab56dbca..f7872cb8b7c 100644 --- a/pkg/io/line_prefix_writer.go +++ b/pkg/io/line_prefix_writer.go @@ -27,6 +27,12 @@ type LinePrefixWriter struct { prefix string w stdio.Writer buffer []byte + // pending holds the already-prefixed, already-normalized suffix of a line + // that a prior write to w left unwritten (a short write or a write error + // after n > 0 bytes). It is retried byte-for-byte before any new line, so + // bytes w already accepted are never re-sent and the line's prefix is + // never re-applied. + pending []byte } // NewLinePrefixWriter creates a writer that prefixes every rendered line with @@ -78,7 +84,7 @@ func (w *LinePrefixWriter) Flush() error { w.mu.Lock() defer w.mu.Unlock() - if len(w.buffer) == 0 { + if len(w.buffer) == 0 && len(w.pending) == 0 { return nil } lines, rest := splitBufferedLines(w.buffer) @@ -121,9 +127,12 @@ func splitBufferedLines(buf []byte) (lines [][]byte, rest []byte) { // so the whole batch lands on the shared underlying writer as one contiguous // block instead of being interleaved line-by-line with a concurrent writer's // own batch -- e.g. a \r-terminated segment held back by a prior Write plus -// the line that completes it in the next Write. On error, the line that -// failed and everything after it, plus rest, are restored to w.buffer so a -// later Write or Flush can retry them; nothing already written is repeated. +// the line that completes it in the next Write. Any pending suffix left over +// from a prior short or failed write is retried first. On error, only the +// unwritten suffix of the line in progress is kept (in w.pending, already +// encoded); the untouched lines after it, plus rest, are restored to +// w.buffer so a later Write or Flush can retry them. Nothing already +// accepted by w is repeated, and a failed line's prefix is never re-applied. // The caller must already hold w.mu. func (w *LinePrefixWriter) writeLinesLocked(lines [][]byte, rest []byte) error { if w.w == nil { @@ -133,18 +142,24 @@ func (w *LinePrefixWriter) writeLinesLocked(lines [][]byte, rest []byte) error { w.writeMu.Lock() defer w.writeMu.Unlock() + if len(w.pending) > 0 { + if err := w.writePendingLocked(); err != nil { + w.buffer = append(joinLines(lines), rest...) + return err + } + } + for i, line := range lines { normalized := bytes.ReplaceAll(line, crlfBytes, lfBytes) normalized = bytes.ReplaceAll(normalized, crBytes, lfBytes) - var err error if w.prefix == "" { - _, err = w.w.Write(normalized) + w.pending = normalized } else { - _, err = stdio.WriteString(w.w, w.prefix+string(normalized)) + w.pending = append([]byte(w.prefix), normalized...) } - if err != nil { - w.buffer = append(joinLines(lines[i:]), rest...) + if err := w.writePendingLocked(); err != nil { + w.buffer = append(joinLines(lines[i+1:]), rest...) return err } } @@ -152,6 +167,24 @@ func (w *LinePrefixWriter) writeLinesLocked(lines [][]byte, rest []byte) error { return nil } +// writePendingLocked writes w.pending to the underlying writer, retaining +// only the unwritten suffix if the write is short or fails. A nil-error +// short write (n < len(w.pending) with err == nil) is treated as +// stdio.ErrShortWrite so callers still see and retry it. The caller must +// already hold w.writeMu. +func (w *LinePrefixWriter) writePendingLocked() error { + n, err := w.w.Write(w.pending) + if err == nil && n < len(w.pending) { + err = stdio.ErrShortWrite + } + if err != nil { + w.pending = append([]byte(nil), w.pending[n:]...) + return err + } + w.pending = nil + return nil +} + // joinLines concatenates lines back into a single buffer, for restoring // unwritten output after a partial-batch write failure. func joinLines(lines [][]byte) []byte { diff --git a/pkg/io/line_prefix_writer_test.go b/pkg/io/line_prefix_writer_test.go index fa245157787..2bbcb40a66a 100644 --- a/pkg/io/line_prefix_writer_test.go +++ b/pkg/io/line_prefix_writer_test.go @@ -4,6 +4,7 @@ import ( "bytes" "errors" "fmt" + stdio "io" "strings" "sync" "testing" @@ -226,6 +227,31 @@ func TestLinePrefixWriterKeepsBufferedOutputAfterWriteErrors(t *testing.T) { require.Equal(t, "[node] partial", target.out.String()) } +func TestLinePrefixWriterRetriesNilErrorShortWriteWithoutDuplicating(t *testing.T) { + target := &linePrefixPartialWriter{acceptOnFirst: 6} + writer := NewLinePrefixWriter("node", target, nil) + + _, err := writer.Write([]byte("line\n")) + require.ErrorIs(t, err, stdio.ErrShortWrite) + require.Equal(t, "[node]", target.out.String()) + + require.NoError(t, writer.Flush()) + require.Equal(t, "[node] line\n", target.out.String()) +} + +func TestLinePrefixWriterRetriesPartialFailedWriteWithoutDuplicating(t *testing.T) { + expectedErr := errors.New("write failed") + target := &linePrefixPartialWriter{acceptOnFirst: 6, errOnFirst: expectedErr} + writer := NewLinePrefixWriter("node", target, nil) + + _, err := writer.Write([]byte("line\n")) + require.ErrorIs(t, err, expectedErr) + require.Equal(t, "[node]", target.out.String()) + + require.NoError(t, writer.Flush()) + require.Equal(t, "[node] line\n", target.out.String()) +} + type linePrefixErrorWriter struct { err error } @@ -247,3 +273,27 @@ func (w *linePrefixFailOnceWriter) Write(p []byte) (int, error) { } return w.out.Write(p) } + +// linePrefixPartialWriter accepts only acceptOnFirst bytes of the first Write +// call (optionally paired with errOnFirst), then accepts everything on later +// calls. It simulates a real io.Writer's short-write and partial-failure +// behavior, where n > 0 bytes reach the underlying sink before an error. +type linePrefixPartialWriter struct { + acceptOnFirst int + errOnFirst error + failed bool + out bytes.Buffer +} + +func (w *linePrefixPartialWriter) Write(p []byte) (int, error) { + if !w.failed { + w.failed = true + n := w.acceptOnFirst + if n > len(p) { + n = len(p) + } + w.out.Write(p[:n]) + return n, w.errOnFirst + } + return w.out.Write(p) +} From 47a247f8c551c39d2be0547fdbd69345cd2c5288 Mon Sep 17 00:00:00 2001 From: Erik Osterman Date: Mon, 10 Aug 2026 20:14:45 -0500 Subject: [PATCH 12/17] fix(website): repair broken pnpm lockfile and migration doc links website-deploy-preview failed on pnpm install --frozen-lockfile because a duplicate nanoid@^3.3.16 override (added independently by two commits and merged from main) produced a duplicate YAML key in pnpm-lock.yaml. Also fix 5 new migration docs linking to the nonexistent /ai/agent-skills route instead of /ai/skills, which broke the docusaurus build once the lockfile issue was resolved. Co-Authored-By: Claude Sonnet 5 --- website/docs/migration/justfile.mdx | 2 +- website/docs/migration/makefile.mdx | 2 +- website/docs/migration/native-terraform.mdx | 2 +- website/docs/migration/taskfile.mdx | 2 +- website/docs/migration/terraform-workspaces.mdx | 2 +- website/package.json | 1 - website/pnpm-lock.yaml | 1 - 7 files changed, 5 insertions(+), 7 deletions(-) diff --git a/website/docs/migration/justfile.mdx b/website/docs/migration/justfile.mdx index 3b59fc49c1f..31d100660e8 100644 --- a/website/docs/migration/justfile.mdx +++ b/website/docs/migration/justfile.mdx @@ -29,7 +29,7 @@ assistants can apply this guide directly to your repository: atmos ai skill install atmos-migration ``` -See [AI Agent Skills](/ai/agent-skills) for details. +See [AI Agent Skills](/ai/skills) for details. ::: ## Why Some Teams Choose Just Over Make diff --git a/website/docs/migration/makefile.mdx b/website/docs/migration/makefile.mdx index ae6c89897fe..f1f998da1c9 100644 --- a/website/docs/migration/makefile.mdx +++ b/website/docs/migration/makefile.mdx @@ -29,7 +29,7 @@ assistants can apply this guide directly to your repository: atmos ai skill install atmos-migration ``` -See [AI Agent Skills](/ai/agent-skills) for details. +See [AI Agent Skills](/ai/skills) for details. ::: ## Key Concepts at a Glance diff --git a/website/docs/migration/native-terraform.mdx b/website/docs/migration/native-terraform.mdx index 12a494c1128..ec995d7ace1 100644 --- a/website/docs/migration/native-terraform.mdx +++ b/website/docs/migration/native-terraform.mdx @@ -27,7 +27,7 @@ assistants can apply this guide directly to your repository: atmos ai skill install atmos-migration ``` -See [AI Agent Skills](/ai/agent-skills) for details. +See [AI Agent Skills](/ai/skills) for details. ::: ## Why This Guide? diff --git a/website/docs/migration/taskfile.mdx b/website/docs/migration/taskfile.mdx index fe56632c947..2b8fabe1263 100644 --- a/website/docs/migration/taskfile.mdx +++ b/website/docs/migration/taskfile.mdx @@ -29,7 +29,7 @@ assistants can apply this guide directly to your repository: atmos ai skill install atmos-migration ``` -See [AI Agent Skills](/ai/agent-skills) for details. +See [AI Agent Skills](/ai/skills) for details. ::: ## Key Concepts at a Glance diff --git a/website/docs/migration/terraform-workspaces.mdx b/website/docs/migration/terraform-workspaces.mdx index 4e7cab6da55..98a9524e4f1 100644 --- a/website/docs/migration/terraform-workspaces.mdx +++ b/website/docs/migration/terraform-workspaces.mdx @@ -23,7 +23,7 @@ assistants can apply this guide directly to your repository: atmos ai skill install atmos-migration ``` -See [AI Agent Skills](/ai/agent-skills) for details. +See [AI Agent Skills](/ai/skills) for details. ::: ## How Workspaces Fit With Atmos diff --git a/website/package.json b/website/package.json index 3871380aa88..ead3c9ee106 100644 --- a/website/package.json +++ b/website/package.json @@ -122,7 +122,6 @@ "nanoid@3.3.3": "^3.3.17", "nanoid@^3.3.16": "^3.3.17", "nanoid@4.0.2": "^5.1.16", - "nanoid@^3.3.16": "^3.3.17", "node-forge@^1": "^1.4.0", "path-to-regexp@^0.1": "^0.1.13", "picomatch@^2": "^2.3.2", diff --git a/website/pnpm-lock.yaml b/website/pnpm-lock.yaml index 1140630f0dd..9ec60177a97 100644 --- a/website/pnpm-lock.yaml +++ b/website/pnpm-lock.yaml @@ -26,7 +26,6 @@ overrides: nanoid@3.3.3: ^3.3.17 nanoid@^3.3.16: ^3.3.17 nanoid@4.0.2: ^5.1.16 - nanoid@^3.3.16: ^3.3.17 node-forge@^1: ^1.4.0 path-to-regexp@^0.1: ^0.1.13 picomatch@^2: ^2.3.2 From 9f8cb2b56a7201deba8c5a0d91ad2a3da947606d Mon Sep 17 00:00:00 2001 From: Erik Osterman Date: Tue, 11 Aug 2026 13:37:07 -0500 Subject: [PATCH 13/17] docs(commands): qualify that internal commands don't appear in help Addresses CodeRabbit review comment: the `name` field description said names unconditionally appear in `atmos help`, contradicting the `internal` field's documented exclusion. Co-Authored-By: Claude Sonnet 5 --- website/docs/cli/configuration/commands/command/index.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/cli/configuration/commands/command/index.mdx b/website/docs/cli/configuration/commands/command/index.mdx index 008b9f29d59..0ebd2a8a0c7 100644 --- a/website/docs/cli/configuration/commands/command/index.mdx +++ b/website/docs/cli/configuration/commands/command/index.mdx @@ -48,7 +48,7 @@ atmos deploy vpc -s plat-ue2-prod
`name`
-
Required. The command name as typed after `atmos` (for example, `deploy`). For nested commands, this is the subcommand name. Names appear in `atmos help`.
+
Required. The command name as typed after `atmos` (for example, `deploy`). For nested commands, this is the subcommand name. Names appear in `atmos help`, unless the command is marked `internal`.
`description`
Required. Help text shown in `atmos help` and `atmos --help`. Supports multi-line YAML for usage examples.
From 70732ad0332bddbfa0edd50c3acb295c2fc2dc5a Mon Sep 17 00:00:00 2001 From: Erik Osterman Date: Tue, 11 Aug 2026 16:58:08 -0500 Subject: [PATCH 14/17] docs(migration): fix stale no-parity claims for deps/freshness now shipped #2882 (already merged into this branch) added dependencies.commands/ dependencies.workflows and step-level inputs/artifacts, giving Atmos direct parity with Task's deps:/sources:/generates: and Make's dependency ordering and file-timestamp caching. This branch's own migration guides -- the subject of this PR -- still declared those exact features unsupported gaps, written before #2882 landed. - taskfile.mdx / from-taskfile.md: rewrite "parallel-by-default" and "sources/generates gap" sections to document dependencies.commands and inputs/artifacts as the direct matches, including the automatic dedup behavior a hand-built parallel step doesn't provide. - makefile.mdx / from-makefile.md: document dependencies.commands for target chains with a shared prerequisite, and inputs/artifacts (with timestamp.changed for make's exact mtime semantics) for file-timestamp targets. - SKILL.md: fix the same false claims in the top-level "Common Problems" summary agents read before the per-tool reference files. Co-Authored-By: Claude Sonnet 5 --- agent-skills/skills/atmos-migration/SKILL.md | 20 +-- .../references/from-makefile.md | 50 ++++---- .../references/from-taskfile.md | 93 ++++++++------ website/docs/migration/makefile.mdx | 91 +++++++++++--- website/docs/migration/taskfile.mdx | 114 +++++++++++------- 5 files changed, 243 insertions(+), 125 deletions(-) diff --git a/agent-skills/skills/atmos-migration/SKILL.md b/agent-skills/skills/atmos-migration/SKILL.md index 4642b153ee8..5c7536c9bf7 100644 --- a/agent-skills/skills/atmos-migration/SKILL.md +++ b/agent-skills/skills/atmos-migration/SKILL.md @@ -101,13 +101,19 @@ These behaviors apply to every task runner. Check them before you open a referen - **The default order can change.** Task runs `deps:` at the same time by default. Make and Just run dependencies one after another, unless the user adds a flag such as `make -j`. Atmos steps - always run one after another, unless you put them inside a `parallel` or `matrix` step. Check - the source tool's real default. Do not assume the step order stays the same when you move it to - Atmos. -- **Atmos has no file-freshness cache.** Task's `sources:`/`generates:` fields and non-`.PHONY` - Make targets both skip work when a file has not changed. Atmos steps always run. The - `require`/`assert` step type does not replace this. It only checks that a file exists. It does - not check if the file is new. Tell the user this directly. + always run one after another, unless you put them inside a `parallel` or `matrix` step. A named + task/target/recipe dependency (Task's `deps:`, Make's `target: dep1 dep2`) maps to command-level + `dependencies.commands`/`dependencies.workflows`, not to plain steps or a hand-built `parallel` + step -- it runs concurrently by default and dedups a dependency shared by more than one caller + to a single run, matching Task's/Make's own behavior. Check the source tool's real default. Do + not assume the step order stays the same when you move it to Atmos. +- **Freshness checks map to `inputs`/`artifacts`, not to plain steps.** Task's `sources:`/ + `generates:` fields and non-`.PHONY` Make targets both skip work when a file has not changed. + Atmos's step-level `inputs.sources`/`artifacts.paths` fields are the direct match: with no + explicit `when:`, declaring them implicitly means `when: checksum.changed`, and the step is + skipped when nothing has changed since its last successful run. This does not carry over on its + own -- add `inputs`/`artifacts` to the migrated step yourself. The `require`/`assert` step type + does not replace this. It only checks that a file exists, not whether it is fresh. - **`workflows.base_path` needs to be set explicitly once the user has their own `atmos.yaml`.** A target chain becomes an Atmos workflow (Principle 7), but `atmos workflow ` fails with `'workflows.base_path' must be configured in 'atmos.yaml'` until you add it (for example, diff --git a/agent-skills/skills/atmos-migration/references/from-makefile.md b/agent-skills/skills/atmos-migration/references/from-makefile.md index b6f82218bf4..e13a11b032d 100644 --- a/agent-skills/skills/atmos-migration/references/from-makefile.md +++ b/agent-skills/skills/atmos-migration/references/from-makefile.md @@ -102,18 +102,19 @@ deploy: build test ## Plan and apply the given ENV (default: dev) **Steps:** 1. Turn `ENV ?= dev` into a command `flags:` entry with `default: "dev"`. -2. Turn the target order (`deploy: build test`) into steps that run in the same order. In this - example, the steps call the Shape A commands, one after the other. -3. Check if the prerequisites are truly independent. In this example, `build` must finish before - `test` runs, but nothing else depends on their order relative to each other. When two - prerequisites do not depend on each other, use a `parallel` step with `needs:` instead of - listing them one after the other. See [Shape C](#shape-c-recursive-or-parallel-make) for the - general `parallel`/`matrix` pattern. -4. Move the Terraform-specific line, `terraform apply -var-file=envs/$(ENV).tfvars`, to +2. Turn the target list (`deploy: build test`) into command-level `dependencies.commands: [build, + test]`. This is the direct match, not a workaround: it resolves through the same DAG scheduler + as `parallel`/`matrix` `needs:`, runs concurrently by default -- `make` itself does not + guarantee prerequisite order without `-j` either -- and dedups a dependency shared by more + than one target to a single run, the same guarantee `make` already gives for free. Do not turn + this into plain sequential steps unless one prerequisite genuinely must finish before another + starts; if so, declare that dependency directly on the later one's own `dependencies.commands` + instead of ordering a flat list. +3. Move the Terraform-specific line, `terraform apply -var-file=envs/$(ENV).tfvars`, to [from-native-terraform.md Shape B](from-native-terraform.md#shape-b-single-dir-with--var-file-from-a-makefile). That guide shows how the Terraform side maps to stacks. Here, the line becomes a single `type: atmos` step, because `terraform apply` is a native Atmos verb. -5. Turn `ifeq ($(ENV),prod)` conditionals into a Go template conditional inside a custom command: +4. Turn `ifeq ($(ENV),prod)` conditionals into a Go template conditional inside a custom command: `{{ if eq .Flags.env "prod" }}...{{ end }}`. This is the same pattern used for `--verbose` and other boolean flags. Inside a workflow, use `when: !cel 'stack == "prod"'` on the step instead. @@ -126,11 +127,9 @@ commands: - name: env shorthand: e default: "dev" + dependencies: + commands: [build, test] steps: - - type: shell - command: atmos build - - type: shell - command: atmos test - type: atmos command: terraform apply infra -s {{ .Flags.env }} ``` @@ -182,10 +181,15 @@ commands: Atmos steps have no tab requirement. Do not confuse `.PHONY` with a caching feature. If a Makefile target is not `.PHONY` and uses file timestamps to skip work when inputs have not -changed, that caching behavior has no equivalent in Atmos. Atmos steps always run. Tell the user -this directly. Do not imply that the behavior carries over. Task's `sources:`/`generates:` -feature has the same problem. See -[from-taskfile.md](from-taskfile.md#the-sourcesgenerates-gap) for more detail. +changed, turn it into step `inputs.sources`/`artifacts.paths` -- with no explicit `when:`, that +implicitly means `when: checksum.changed`, and the step is skipped when nothing has changed since +its last successful run. It does not carry over automatically; tell the user to add +`inputs`/`artifacts` to the migrated step themselves. Content hashing (the default) is a +deliberate upgrade over Make's own mtime comparison -- a fresh `git clone`/CI checkout resets +every file's mtime, which makes Make think everything changed even when it didn't; use +`when: timestamp.changed` instead for Make's exact mtime semantics. Task's `sources:`/`generates:` +feature maps to the same fields. See +[from-taskfile.md](from-taskfile.md#sourcesgenerates-becomes-inputsartifacts) for more detail. ### Silent recipes and command echo @@ -210,16 +214,20 @@ language. ## What Not To Do -- Do not build file-timestamp or `.PHONY` caching as an Atmos feature. It does not exist. State - this directly instead of dropping the behavior without comment. +- Do not confuse `.PHONY` with a caching feature -- it is not one. Do not drop file-timestamp + caching without adding the matching `inputs`/`artifacts` fields to the migrated step; it is a + direct match, not a gap, but it does not carry over on its own. +- Do not turn `target: dep1 dep2` into plain sequential steps, or into a hand-built `parallel` + step, without first considering command-level `dependencies.commands` -- it runs concurrently + by default and dedups a dependency shared by more than one target, the way `make` already does. - Do not turn every private or helper target into its own discoverable command by default. If the helper is called from only one recipe, put its logic in a step inside the command or workflow that needs it. If it needs to be called from more than one recipe, or invoked directly for debugging, make it a custom command with `internal: true` instead -- it stays runnable but is excluded from `atmos --help` listings and completion suggestions. - Do not treat "wrap `atmos` commands in the Makefile" as the final state. It is a valid bridge - during early migration, as shown in Shape B, step 4. Leaf targets should become custom - commands. Target chains should become workflows. + during early migration. Leaf targets should become custom commands. Target chains should become + workflows. - Do not invent `when:` conditions that check flag values on workflow steps. The `when:` field checks CEL context values, such as `stack`, `ci`, and `local`. Flag-based conditionals belong in the custom command's own Go templates. diff --git a/agent-skills/skills/atmos-migration/references/from-taskfile.md b/agent-skills/skills/atmos-migration/references/from-taskfile.md index 250809d67ce..563362ca683 100644 --- a/agent-skills/skills/atmos-migration/references/from-taskfile.md +++ b/agent-skills/skills/atmos-migration/references/from-taskfile.md @@ -18,6 +18,11 @@ Terraform-specific steps. | `vars:`/`env:` and `sources:`/`generates:` | [Shape C](#shape-c-variables-and-up-to-date-checks) | | `includes:` (multi-file composition) | see [Common Problems](#includes-multi-file-composition) | +`deps:` maps to command-level `dependencies.commands` (concurrent by default, deduped -- the +direct match). `sources:`/`generates:` maps to step-level `inputs.sources`/`artifacts.paths` +(implicit `when: checksum.changed` -- the direct match). Neither is a gap; both need the user to +add the matching field during migration, since neither carries over automatically. + ## Shape A: Simple Tasks **Before:** @@ -82,28 +87,19 @@ tasks: - terraform -chdir=terraform apply -var-file=envs/dev.tfvars ``` -This is the most important problem in this migration. Task runs `deps:` at the same time by -default. Atmos custom-command and workflow steps run one after another by default. This is the -opposite default. If you turn `deps: [test, lint]` into two plain steps that run one after -another, the command becomes slower. It also changes what happens when one task fails. To keep -Task's default behavior, put the dependency tasks inside a `parallel` step: +Task runs `deps:` at the same time by default. Atmos custom-command and workflow steps run one +after another by default -- so a `deps:` entry is not a step and never becomes one. It maps to +the command-level `dependencies.commands` field, which resolves through the same DAG scheduler as +`parallel`/`matrix` `needs:` and runs concurrently by default -- matching Task's `deps:` behavior +directly, not working around it with a hand-built `parallel` step: ```yaml commands: - name: deploy description: Plan and apply the given environment + dependencies: + commands: [test, lint] steps: - - name: checks - type: parallel - fail: - mode: wait_all - steps: - - name: test - type: shell - command: atmos test - - name: lint - type: shell - command: atmos lint - type: atmos command: terraform apply infra -s dev ``` @@ -113,9 +109,17 @@ task's Terraform code to `components/terraform/infra/` (the default `components.terraform.base_path` is `components/terraform`), then swap `infra` for whatever the user actually names the component. -If the Taskfile's `deps:` list needs its own internal order, add `needs:` to the steps inside the -`parallel` block. Do not assume the tasks should run one after another just because that is -Atmos's default for steps outside a `parallel` or `matrix` block. +`dependencies.commands` also matches a behavior Task itself has that a hand-rolled `parallel` +step does not: if two commands both depend on the same one -- for example both `test` and `lint` +depending on `build` -- Atmos runs `build` exactly once and dedups it, the same as Task's own +`deps:` graph. A `parallel` step calling `atmos build` from two different places would run it +twice. If one dependency itself depends on another (`lint` depends on `build`, and `deploy` +depends on `test` and `lint`), declare that directly on `lint`'s own `dependencies.commands` -- +the scheduler resolves the whole transitive graph itself, still deduping `build` to a single run. + +Reach for a `parallel` step instead of `dependencies.commands` only for concurrency inside a +single command's own steps, not between named commands -- for example, running several shell +commands side by side that were never their own Task tasks to begin with. ## Shape C: Variables and Up-to-Date Checks @@ -141,19 +145,33 @@ tasks: `default: "dev"`. Task's Sprig `default` filter becomes the plain `default:` field. - Turn `env:` into an `env:` map. The two are almost identical. -### The `sources`/`generates` gap +### `sources:`/`generates:` becomes `inputs`/`artifacts` -Task skips a task's `cmds:` when its `sources:` files match its `generates:` outputs. It checks -this with a file hash. Atmos steps always run. There is no built-in check for whether a file is -up to date. The `require`/`assert` step type does not fix this. It only checks that a file, tool, -or directory exists. It does not compare hashes or timestamps. +Task skips a task's `cmds:` when its `sources:` files match its `generates:` outputs, checked by +default with a content hash (Task also supports `method: timestamp` for an mtime-based check). +The step-level `inputs.sources` and `artifacts.paths` fields are the direct match, with the same +checksum-by-default/timestamp-as-an-option choice: -If the user depends on `sources:`/`generates:` to skip a slow step, such as code generation, tell -them plainly that this behavior does not carry over. Then offer two honest choices: +```yaml +commands: + - name: build + description: Compile the deployable artifact + steps: + - type: shell + command: go build -o bin/handler ./cmd/handler + inputs: + sources: ["cmd/**/*.go"] + artifacts: + paths: ["bin/handler"] +``` -1. Accept that the step always runs. This is correct for most fast build steps. -2. Add a hash or timestamp check inside the shell step itself. This is a script the user - maintains. It is not a built-in Atmos feature. +With no explicit `when:`, declaring `inputs`/`artifacts` on a step is enough -- it implicitly +means `when: checksum.changed`, and the step is skipped when the hash of the matched source files +matches the hash recorded after the last successful run. This does not carry over on its own -- +add `inputs`/`artifacts` to the migrated step yourself, matching the Taskfile's own +`sources:`/`generates:` lists. The `require`/`assert` step type is a different, older step type -- +it only checks that a file, tool, or directory exists, not whether it is fresh, so it does not +replace `inputs`/`artifacts`. ## Common Problems @@ -173,17 +191,20 @@ Pick the one that fits the content being split: "stacks/workflows"`) the first time the user's migration reaches a workflow, or `atmos workflow ` fails with `'workflows.base_path' must be configured in 'atmos.yaml'`. -### `sources`/`generates` has no built-in match +### `sources`/`generates` maps to a different field than `steps` -See [Shape C](#the-sourcesgenerates-gap) above. This is the largest real gap in this migration. -State it directly. Do not gloss over it. +See [Shape C](#sourcesgenerates-becomes-inputsartifacts) above. It does not carry over +automatically -- the user must add `inputs`/`artifacts` to the migrated step themselves. State +that directly. Do not gloss over it, and do not claim it "just works" without the field. ## What Not To Do -- Do not drop `sources:`/`generates:` caching without comment. State the change directly. Let - the user decide how, or whether, to replace it. -- Do not turn `deps:` into plain sequential steps without warning the user about the change in - default concurrency. +- Do not drop `sources:`/`generates:` without adding the matching `inputs`/`artifacts` fields to + the migrated step. It is a direct match, not a gap, but it does not carry over on its own. +- Do not turn `deps:` into plain sequential steps, or into a hand-built `parallel` step, without + first considering command-level `dependencies.commands` -- it is the direct match: concurrent + by default, and it dedups a dependency shared by more than one command the same way Task's own + `deps:` graph does. - Do not describe `require`/`assert` as a freshness or caching check. It only checks that something exists. - Do not turn every `internal: true` task into its own discoverable command by default. If it is diff --git a/website/docs/migration/makefile.mdx b/website/docs/migration/makefile.mdx index f1f998da1c9..406fa834301 100644 --- a/website/docs/migration/makefile.mdx +++ b/website/docs/migration/makefile.mdx @@ -38,7 +38,7 @@ See [AI Agent Skills](/ai/skills) for details. |--------------------------------------------------|------------------------------------------------------------| | `.PHONY` target, no dependencies | [Custom command](/cli/configuration/commands) | | `VAR ?= default`, `$(VAR)` | Command `flags:` with a `default:` | -| `target: dep1 dep2` (dependency order) | Steps in order, or a `parallel` step with `needs:` | +| `target: dep1 dep2` (dependency order) | Steps in order for a single chain, or command [`dependencies.commands`](/cli/configuration/commands/dependencies#named-command-and-workflow-dependencies) when the same prerequisite is shared by more than one target | | Recipe shell lines (`@`-silenced, tab-indented) | Steps (`type: shell`, `output: none`) | | `ifeq (...)` conditional | Go template `{{ if }}` in a command, or `when:` in a workflow | | `$(MAKE) -j`, `$(MAKE) -C dir` | `parallel` / `matrix` steps | @@ -80,9 +80,9 @@ See [AI Agent Skills](/ai/skills) for details. - name: test description: Run unit tests + dependencies: + commands: [build] steps: - - type: shell - command: atmos build - type: shell command: go test ./... @@ -105,11 +105,9 @@ See [AI Agent Skills](/ai/skills) for details. - name: env shorthand: e default: "dev" + dependencies: + commands: [build, test] steps: - - type: shell - command: atmos build - - type: shell - command: atmos test - type: atmos command: terraform apply infra -s {{ .Flags.env }} ``` @@ -154,6 +152,38 @@ from another command or workflow step, prefer `type: atmos` with `command: build `type: shell` with `command: atmos build`: `type: atmos` auto-applies `-s ` from the step's `stack:` field and renders through Atmos's structured step output. +For a simple, single-caller chain such as `deploy: build test`, steps calling each prerequisite in +order are enough. But `make` itself only builds a shared prerequisite once per invocation, even +when more than one target depends on it — plain steps lose that property, since each caller's +`type: atmos`/`type: shell` step runs it again. Command-level +[`dependencies.commands`](/cli/configuration/commands/dependencies#named-command-and-workflow-dependencies) +is the direct match for that case: it resolves through the same DAG scheduler as +[`parallel`](/workflows/steps/type/parallel)/[`matrix`](/workflows/steps/type/matrix) `needs:`, +runs concurrently by default, and dedups a dependency shared by more than one command to a single +run — the same guarantee `make` gives you for free: + +```yaml +commands: + - name: build + steps: + - type: shell + command: go build ./... + + - name: test + dependencies: + commands: [build] + steps: + - type: shell + command: go test ./... + + - name: deploy + dependencies: + commands: [build, test] + steps: + - type: atmos + command: terraform apply infra -s dev +``` + If the Makefile's `deploy` target also selects a Terraform environment, for example through `-var-file=envs/$(ENV).tfvars`, that part is a Terraform migration step, not a Make migration step. See [Migrating from Native Terraform](/migration/native-terraform) for how the Terraform @@ -182,6 +212,35 @@ commands: ``` +## File-Timestamp Targets Become `inputs`/`artifacts` + +Make skips a target's recipe when every prerequisite (the target's own `inputs`) is older than +the target file it already produced (the target's own `artifacts`) — comparing mtimes. The +step-level [`inputs.sources`](/workflows/steps/inputs) and +[`artifacts.paths`](/workflows/steps/artifacts) fields are the direct match: + +```yaml +commands: + - name: build + description: Compile the deployable artifact + steps: + - type: shell + command: go build -o bin/handler ./cmd/handler + inputs: + sources: ["cmd/**/*.go"] + artifacts: + paths: ["bin/handler"] +``` + +With no explicit `when:`, declaring `inputs`/`artifacts` on a step is enough — it implicitly means +`when: checksum.changed`, and the step is skipped when the hash of the matched source files +matches the hash recorded after the last successful run. That default is a deliberate upgrade +over Make's own mtime comparison: a fresh `git clone`/CI checkout resets every file's mtime, +which makes Make (and the mtime-based `timestamp.changed` fact) think everything changed even +when it didn't. Use `when: timestamp.changed` instead if you want Make's exact mtime semantics. +See [inputs](/workflows/steps/inputs) for the full field reference. It does not carry over on its +own — add `inputs`/`artifacts` to the migrated step yourself. + ## What Stays the Same - **Your Terraform code, scripts, and CLIs.** Atmos wraps them. It does not replace them. @@ -190,10 +249,6 @@ commands: your team is ready. Then remove it, once each leaf target and chain has become a command or a workflow. -One gap to state directly: a target that skips work based on file timestamps has no match in -Atmos. Atmos steps always run. If this caching matters to your build, plan for it. Do not assume -it carries over on its own. - ## What You Gain - **One command for discovery.** Run `atmos help` to list every command. Run @@ -218,11 +273,11 @@ it carries over on its own. - [ ] List every Makefile target. Mark each one as independent, or part of a chain - [ ] Turn independent targets into custom commands in `atmos.yaml` - [ ] Turn `VAR ?= default` variables into command `flags:` with defaults -- [ ] Turn dependency chains into steps in order, or into a workflow +- [ ] Turn dependency chains into steps in order, or `dependencies.commands` when a prerequisite is shared by more than one target - [ ] Replace `$(MAKE) -j`/`-C` with `parallel`/`matrix` steps - [ ] Delete any hand-written `help` target. `atmos help` replaces it - [ ] Keep the Makefile as a thin wrapper during the transition, if you want one -- [ ] Confirm no target depends on file-timestamp caching before you remove it +- [ ] Turn file-timestamp targets into step `inputs.sources`/`artifacts.paths` ## Common Questions @@ -234,14 +289,18 @@ workflow. There is no fixed deadline for this. -There is no direct match in Atmos. Atmos steps always run. If a target relies on file timestamps -to skip slow work, plan for that change directly. Do not assume the behavior stays the same. +Turn them into step `inputs.sources`/`artifacts.paths` -- with no explicit `when:`, that +implicitly skips the step when nothing has changed since its last successful run. It does not +carry over automatically; you add it to the migrated step yourself. See +[File-Timestamp Targets Become inputs/artifacts](#file-timestamp-targets-become-inputsartifacts). Yes, through either step type: `type: atmos` with `command: `, or `type: shell` with `command: atmos `. Prefer `type: atmos` — it auto-applies -`-s ` and gets Atmos's structured step output, which `type: shell` does not. +`-s ` and gets Atmos's structured step output, which `type: shell` does not. If the same +prerequisite is shared by more than one target, prefer `dependencies.commands` instead — it dedups +a shared dependency to a single run, the way `make` already does. diff --git a/website/docs/migration/taskfile.mdx b/website/docs/migration/taskfile.mdx index 4368ba78f98..a4e69803da6 100644 --- a/website/docs/migration/taskfile.mdx +++ b/website/docs/migration/taskfile.mdx @@ -17,8 +17,8 @@ import TabItem from '@theme/TabItem' A Taskfile gives your team an ad-hoc way to run build, test, and deploy tasks, defined in YAML. Atmos gives you the same capability as a native, documented feature: custom commands and workflows. Task and Atmos both use declarative YAML, so most of the mapping from one field to -another is direct. Two real differences in behavior still need attention first, covered below. -Your Terraform code and scripts do not need to change. +another is direct. Two Task features map to a dedicated field rather than plain steps, covered +below. Your Terraform code and scripts do not need to change. :::tip Using an AI Coding Assistant? @@ -38,19 +38,21 @@ See [AI Agent Skills](/ai/skills) for details. |--------------------------------------------------|------------------------------------------------------------------| | `desc:` | Command `description:` | | `cmds:` (list of shell commands) | `steps:` (`type: shell`, or `type: atmos` for any atmos command) | -| `deps:` (runs at the same time by default) | A `parallel` step. Plain steps in order change the default | +| `deps:` (runs at the same time by default) | Command [`dependencies.commands`](/cli/configuration/commands/dependencies#named-command-and-workflow-dependencies), concurrent by default | | `vars:` / `env:` | Command `flags:` (with `default:`) / `env:` map | -| `sources:` / `generates:` (freshness check) | No direct match. Steps always run | +| `sources:` / `generates:` (freshness check) | Step [`inputs.sources`](/workflows/steps/inputs) / [`artifacts.paths`](/workflows/steps/artifacts) | | `includes:` (multi-file composition) | Auto-discovered `atmos.d/*.yaml`, or separate workflow files | | `internal: true` task | Command `internal: true` | -## The Parallel-by-Default Difference +## `deps:` Becomes `dependencies.commands` -This is the most important behavior change in this migration. Task runs `deps:` at the same time -by default. This is the opposite of Atmos, where custom-command and workflow steps run one after -another by default. If you turn `deps: [test, lint]` into two plain steps in order, the migrated -command becomes slower. The failure behavior also changes. To keep Task's default, put the -dependency tasks inside a [`parallel`](/workflows/steps/type/parallel) step: +Task runs `deps:` at the same time by default. Atmos custom-command and workflow steps, in +contrast, run one after another by default — so a `deps:` entry is not a step and never becomes +one. It maps to the command-level +[`dependencies.commands`](/cli/configuration/commands/dependencies#named-command-and-workflow-dependencies) +field, which resolves through the same DAG scheduler as +[`parallel`](/workflows/steps/type/parallel)/[`matrix`](/workflows/steps/type/matrix) `needs:` and +runs concurrently by default — matching Task's `deps:` behavior directly, not working around it: @@ -68,18 +70,9 @@ dependency tasks inside a [`parallel`](/workflows/steps/type/parallel) step: commands: - name: deploy description: Plan and apply the given environment + dependencies: + commands: [test, lint] steps: - - name: checks - type: parallel - fail: - mode: wait_all - steps: - - name: test - type: shell - command: atmos test - - name: lint - type: shell - command: atmos lint - type: atmos command: terraform apply infra -s dev ``` @@ -91,24 +84,48 @@ dependency tasks inside a [`parallel`](/workflows/steps/type/parallel) step: `components.terraform.base_path` is `components/terraform`), then swap `infra` for whatever you actually name the component. -If a Taskfile's `deps:` list needs an order between its own entries, add `needs:` to the steps -inside the `parallel` block. Do not default to sequential steps just because that is Atmos's own -default outside a `parallel` or `matrix` block. +`dependencies.commands` also matches a behavior Task itself has that a hand-rolled `parallel` +step does not: if two commands both depend on the same one — for example both `test` and `lint` +depending on `build` — Atmos runs `build` exactly once and dedups it, the same as Task's own +`deps:` graph. A `parallel` step calling `atmos build` from two different places would run it +twice. If one dependency itself depends on another (`lint` depends on `build`, and `deploy` +depends on `test` and `lint`), declare that directly on `lint`'s own `dependencies.commands` — the +scheduler resolves the whole transitive graph itself, still deduping `build` to a single run. -## The `sources`/`generates` Gap +Reach for a `parallel` step instead of `dependencies.commands` only for concurrency inside a +single command's own steps, not between named commands — for example, running several shell +commands side by side that were never their own Task tasks to begin with. -Task skips a task's `cmds:` when its `sources:` files match its `generates:` outputs, through a -file-hash check. Atmos steps always run. There is no built-in match for this. -[`require`/`assert`](/workflows/steps/type/require) does not replace it. That step type only -checks that a file, tool, or directory exists. It does not check freshness. +## `sources:`/`generates:` Becomes `inputs`/`artifacts` -If you rely on `sources:`/`generates:` for caching, for example to skip a slow codegen step, -plan for this directly. It is a real gap, not something Atmos preserves on its own. Two honest -choices: +Task skips a task's `cmds:` when its `sources:` files match its `generates:` outputs, checked by +default with a content hash (Task also supports `method: timestamp` for an mtime-based check). +The step-level [`inputs.sources`](/workflows/steps/inputs) and +[`artifacts.paths`](/workflows/steps/artifacts) fields are the direct match, with the same +checksum-by-default/timestamp-as-an-option choice: -1. Accept that the step always runs. This is correct for most fast build steps. -2. Add a hash or timestamp check inside the shell step itself. This is a script you write and - maintain, not a built-in Atmos feature. +```yaml +commands: + - name: build + description: Compile the deployable artifact + steps: + - type: shell + command: go build -o bin/handler ./cmd/handler + inputs: + sources: ["cmd/**/*.go"] + artifacts: + paths: ["bin/handler"] +``` + +With no explicit `when:`, declaring `inputs`/`artifacts` on a step is enough — it implicitly means +`when: checksum.changed`, and the step is skipped when the hash of the matched source files +matches the hash recorded after the last successful run. Run the same command twice in a row and +the second run skips `build` entirely; edit a matched source file and the next run executes it +again. See [inputs](/workflows/steps/inputs) for the full field reference, including the +`timestamp.changed` fact for an mtime-based check matching Task's `method: timestamp` mode. +[`require`/`assert`](/workflows/steps/type/require) is a different, older step type — it only +checks that a file, tool, or directory exists, not whether it is fresh, so it does not replace +`inputs`/`artifacts`. ## `internal: true` Tasks @@ -158,12 +175,16 @@ single-caller and has no reason to be invoked on its own. steps: - type: shell command: go build -o bin/handler ./cmd/handler + inputs: + sources: ["cmd/**/*.go"] + artifacts: + paths: ["bin/handler"] - name: test description: Run unit tests + dependencies: + commands: [build] steps: - - type: shell - command: atmos build - type: shell command: go test ./... @@ -229,9 +250,9 @@ Choose the one that fits what you are splitting: - [ ] List every task. Mark each one as independent, or part of a `deps:` chain - [ ] Turn `desc:`/`cmds:` into command `description:`/`steps:` -- [ ] Put any `deps:` chain in a `parallel` step, to keep Task's default behavior +- [ ] Turn any `deps:` chain into `dependencies.commands`, to keep Task's concurrent-by-default behavior - [ ] Turn `vars:`/`env:` into command `flags:` (with defaults) and `env:` maps -- [ ] Decide how to handle `sources:`/`generates:` freshness checks. Do not assume parity +- [ ] Turn `sources:`/`generates:` into step `inputs.sources`/`artifacts.paths` - [ ] Split `includes:` into auto-discovered `atmos.d/*.yaml` files, separate workflow files, or both - [ ] Mark `internal: true` tasks as `internal: true` custom commands @@ -239,15 +260,18 @@ Choose the one that fits what you are splitting: ## Common Questions -No. Atmos steps always run. There is no built-in file-hash freshness check. -`require`/`assert` only checks that something exists, not whether it is fresh. If this caching -matters to you, plan for it directly. See [The sources/generates Gap](#the-sourcesgenerates-gap). +Yes, once you migrate it. Turn `sources:`/`generates:` into step `inputs.sources`/`artifacts.paths` +— with no explicit `when:`, that implicitly skips the step when nothing has changed since its last +successful run, the same as Task's own check. It does not carry over automatically; you add it to +the migrated step yourself. See +[sources:/generates: Becomes inputs/artifacts](#sourcesgenerates-becomes-inputsartifacts). -A `parallel` step. Task runs `deps:` at the same time by default. Atmos steps run one after -another by default. Put the dependency tasks in a `parallel` step to keep the original behavior, -instead of running them one after another without meaning to. +Command-level `dependencies.commands`. It resolves through the same DAG scheduler as `parallel`/ +`matrix` `needs:` and runs concurrently by default, matching Task's `deps:` behavior directly — +including deduping a dependency shared by more than one command to a single run. Reach for a +`parallel` step only for concurrency inside one command's own steps, not between named commands. From 34a145c532d834cbd29208634ad17bcc12d6f99b Mon Sep 17 00:00:00 2001 From: Erik Osterman Date: Tue, 11 Aug 2026 18:24:12 -0500 Subject: [PATCH 15/17] docs(migration): fix reviewer feedback on dependencies/freshness guidance Local review comments: - Recommend `atmos --help` (scriptable, direct) over the interactive `atmos help` (pages the same listing) for command discovery, across all three migration guides and their skill mirrors. - Reframe makefile.mdx around "Atmos is the front door either way": a custom command can call `make ` as its one step, permanently if desired, rather than treating full migration into native `steps:` as the only end state. CodeRabbit findings, verified against actual behavior before fixing: - GNU Make's default is to build a target's prerequisites one at a time, in listed order -- `-j` is required for concurrency. `dependencies.commands` runs concurrently by default, so presenting it as Make's/Just's "direct match" changes behavior and can race prerequisites that were only ever sequential by accident. Ordered steps are now the default-preserving match; `dependencies.commands` is reserved for a shared prerequisite (dedup, independent of concurrency), genuine independence, or an explicit `-j` source. Fixed in SKILL.md, makefile.mdx, and from-makefile.md, including a corrected Shape B example showing how to keep `build` ordered ahead of `test` even under the concurrent scheduler. - Confirmed in cmd/cmd_utils.go/internal/exec/workflow_utils.go that a skipped step just `continue`s the loop: inputs/artifacts freshness is evaluated and recorded per step, unlike Task's/Make's whole-recipe/task scope. Documented this across taskfile.mdx, from-makefile.md, and SKILL.md, with guidance to combine multiple commands into one step when a single freshness decision must gate all of them. - Removed the stale "target chains become workflows" claim, which contradicted the dependencies.commands guidance it now sits next to; workflows are reserved for fixed, multi-step orchestration across more than one component. Co-Authored-By: Claude Sonnet 5 --- agent-skills/skills/atmos-migration/SKILL.md | 47 +++++++---- .../references/from-justfile.md | 2 +- .../references/from-makefile.md | 59 +++++++++---- website/docs/migration/justfile.mdx | 2 +- website/docs/migration/makefile.mdx | 83 ++++++++++++------- website/docs/migration/taskfile.mdx | 16 +++- 6 files changed, 141 insertions(+), 68 deletions(-) diff --git a/agent-skills/skills/atmos-migration/SKILL.md b/agent-skills/skills/atmos-migration/SKILL.md index 5c7536c9bf7..e2283be6ebc 100644 --- a/agent-skills/skills/atmos-migration/SKILL.md +++ b/agent-skills/skills/atmos-migration/SKILL.md @@ -72,8 +72,10 @@ user's repository. targets, recipes, and tasks that Make, Just, and Task provide. This doesn't have to happen all at once — a Makefile, Justfile, or Taskfile can stay as a thin wrapper around `atmos` commands during migration, the same incremental approach described in Principle 6. The end state turns - each leaf target into a custom command and each target chain into a workflow, reached step by - step. + each leaf target into a custom command; a target chain usually stays a custom command too, + using `dependencies.commands`/`dependencies.workflows` for its prerequisites. Reserve + workflows for fixed, multi-step orchestration across more than one component — not every + dependency chain needs one. ## Decide the Migration Shape First @@ -99,23 +101,32 @@ the user has existing Terraform state that a new Atmos component must read. These behaviors apply to every task runner. Check them before you open a reference file: -- **The default order can change.** Task runs `deps:` at the same time by default. Make and Just - run dependencies one after another, unless the user adds a flag such as `make -j`. Atmos steps - always run one after another, unless you put them inside a `parallel` or `matrix` step. A named - task/target/recipe dependency (Task's `deps:`, Make's `target: dep1 dep2`) maps to command-level - `dependencies.commands`/`dependencies.workflows`, not to plain steps or a hand-built `parallel` - step -- it runs concurrently by default and dedups a dependency shared by more than one caller - to a single run, matching Task's/Make's own behavior. Check the source tool's real default. Do - not assume the step order stays the same when you move it to Atmos. -- **Freshness checks map to `inputs`/`artifacts`, not to plain steps.** Task's `sources:`/ - `generates:` fields and non-`.PHONY` Make targets both skip work when a file has not changed. - Atmos's step-level `inputs.sources`/`artifacts.paths` fields are the direct match: with no - explicit `when:`, declaring them implicitly means `when: checksum.changed`, and the step is - skipped when nothing has changed since its last successful run. This does not carry over on its - own -- add `inputs`/`artifacts` to the migrated step yourself. The `require`/`assert` step type - does not replace this. It only checks that a file exists, not whether it is fresh. +- **The default order can change, and it differs by source tool.** Task runs `deps:` at the same + time by default, so command-level `dependencies.commands`/`dependencies.workflows` -- also + concurrent by default -- is its direct match. Make and Just run dependencies one after another + by default; `make -j` is required for concurrency. Do not describe `dependencies.commands` as + matching Make's/Just's *default* -- it changes the order, and can introduce a race between + prerequisites that were only ever sequential by accident, not by a declared dependency. For an + ordinary Make/Just chain, ordered steps preserve the default; reach for `dependencies.commands` + there only when the source used `-j`, the prerequisites are genuinely independent, or a + prerequisite is shared by more than one caller (it dedups a shared dependency to a single run + regardless of concurrency -- true for every one of these tools). Check the source tool's real + default before you move it. +- **Freshness checks map to `inputs`/`artifacts`, not to plain steps -- and the scope is per + step.** Task's `sources:`/`generates:` fields and non-`.PHONY` Make targets both skip the + *entire* recipe/task when a file has not changed. Atmos's step-level `inputs.sources`/ + `artifacts.paths` fields are the direct match: with no explicit `when:`, declaring them + implicitly means `when: checksum.changed`, and *that one step* is skipped when nothing has + changed since its last successful run -- later steps in the same command still run regardless. + If the source recipe/task runs more than one command and the freshness decision must gate all + of them together, combine them into a single `shell`/`script` step rather than spreading + `inputs`/`artifacts` across several steps. This does not carry over on its own -- add + `inputs`/`artifacts` to the migrated step yourself. The `require`/`assert` step type does not + replace this. It only checks that a file exists, not whether it is fresh. - **`workflows.base_path` needs to be set explicitly once the user has their own `atmos.yaml`.** - A target chain becomes an Atmos workflow (Principle 7), but `atmos workflow ` fails with + Only fixed, multi-step orchestration across more than one component becomes an Atmos workflow + (Principle 7) -- most target chains stay a custom command with `dependencies.commands` instead. + `atmos workflow ` fails with `'workflows.base_path' must be configured in 'atmos.yaml'` until you add it (for example, `workflows.base_path: "stacks/workflows"`). None of this skill's `atmos.yaml` snippets show it by default -- add it the moment the user's migration reaches its first workflow. diff --git a/agent-skills/skills/atmos-migration/references/from-justfile.md b/agent-skills/skills/atmos-migration/references/from-justfile.md index b5a9e6ff264..b4c8bc2492b 100644 --- a/agent-skills/skills/atmos-migration/references/from-justfile.md +++ b/agent-skills/skills/atmos-migration/references/from-justfile.md @@ -38,7 +38,7 @@ _clean: **Steps:** 1. Turn the `# comment` above a recipe into the command's `description:` field. Atmos shows this - text in `atmos help` and `atmos --help`. This replaces `just --list`. + text in `atmos --help` and `atmos --help`. This replaces `just --list`. 2. Turn a recipe's named parameter with a default value, such as `deploy env='dev':`, into a command `flags:` entry with a matching `default:` value. Inside a step, read the value as `{{ .Flags.env }}`. Do not use Just's own `{{env}}` syntax. See diff --git a/agent-skills/skills/atmos-migration/references/from-makefile.md b/agent-skills/skills/atmos-migration/references/from-makefile.md index e13a11b032d..80b0b808dc9 100644 --- a/agent-skills/skills/atmos-migration/references/from-makefile.md +++ b/agent-skills/skills/atmos-migration/references/from-makefile.md @@ -51,7 +51,7 @@ clean: ## Remove build artifacts [Split commands across files](#split-commands-across-files) below. 2. Turn the silent-recipe `@` prefix into the step field `output: none`. 3. Delete the `help` target. Atmos generates the same information from each command's - `description:` field. Run `atmos help` or `atmos --help` to see it. + `description:` field. Run `atmos --help` or `atmos --help` to see it. 4. When one target depends on another leaf target, such as `test: build`, add a step that runs the dependency's command. Do not use a `type: atmos` step to call a custom command. That step type is only for native Atmos verbs, such as `terraform plan`. Use a `type: shell` step with @@ -102,14 +102,19 @@ deploy: build test ## Plan and apply the given ENV (default: dev) **Steps:** 1. Turn `ENV ?= dev` into a command `flags:` entry with `default: "dev"`. -2. Turn the target list (`deploy: build test`) into command-level `dependencies.commands: [build, - test]`. This is the direct match, not a workaround: it resolves through the same DAG scheduler - as `parallel`/`matrix` `needs:`, runs concurrently by default -- `make` itself does not - guarantee prerequisite order without `-j` either -- and dedups a dependency shared by more - than one target to a single run, the same guarantee `make` already gives for free. Do not turn - this into plain sequential steps unless one prerequisite genuinely must finish before another - starts; if so, declare that dependency directly on the later one's own `dependencies.commands` - instead of ordering a flat list. +2. GNU Make's own default is to build a target's prerequisites one at a time, in the order + listed, not concurrently (`-j` is required for that) -- so ordered steps, not + `dependencies.commands`, are the default-preserving match for a plain target list. Reach for + command-level `dependencies.commands: [build, test]` instead only when: a prerequisite is + shared by more than one target (it dedups a shared dependency to a single run, the same + guarantee `make` already gives for free, independent of concurrency), the prerequisites are + genuinely independent, or the source target actually used `-j`. Here, `build` is shared -- + Shape A's `test` already depends on it via its own `atmos build` step -- so + `dependencies.commands` is the right call for `deploy`. Convert Shape A's `test` from its + `atmos build` step to `dependencies.commands: [build]` at the same time, so the scheduler + still orders `build` before `test` even though both now run through the same + concurrent-by-default mechanism, instead of listing `build` and `test` as a flat, unordered + sibling list on `deploy`. 3. Move the Terraform-specific line, `terraform apply -var-file=envs/$(ENV).tfvars`, to [from-native-terraform.md Shape B](from-native-terraform.md#shape-b-single-dir-with--var-file-from-a-makefile). That guide shows how the Terraform side maps to stacks. Here, the line becomes a single @@ -121,6 +126,14 @@ deploy: build test ## Plan and apply the given ENV (default: dev) ```yaml commands: + - name: test + description: Run unit tests + dependencies: + commands: [build] + steps: + - type: shell + command: go test ./... + - name: deploy description: Plan and apply the given environment (default dev) flags: @@ -134,6 +147,9 @@ commands: command: terraform apply infra -s {{ .Flags.env }} ``` +`build` still runs exactly once for the whole `atmos deploy` invocation -- `test`'s own edge on +`build` orders it correctly ahead of `test`, and `deploy`'s own steps wait for both to finish. + `infra` is a placeholder Atmos component name, not the `terraform` verb repeated. Move the target's Terraform code to `components/terraform/infra/` (the default `components.terraform.base_path` is `components/terraform`), then swap `infra` for whatever the @@ -190,6 +206,11 @@ every file's mtime, which makes Make think everything changed even when it didn' `when: timestamp.changed` instead for Make's exact mtime semantics. Task's `sources:`/`generates:` feature maps to the same fields. See [from-taskfile.md](from-taskfile.md#sourcesgenerates-becomes-inputsartifacts) for more detail. +The scope is different, too: Make's freshness check gates the target's entire recipe, but +`inputs`/`artifacts` are declared per step -- skipping one step does not stop later steps in the +same command from running. If a target's recipe has more than one command line and the freshness +decision must gate all of them together, combine them into a single `shell`/`script` step rather +than spreading `inputs`/`artifacts` across several steps. ### Silent recipes and command echo @@ -216,18 +237,26 @@ language. - Do not confuse `.PHONY` with a caching feature -- it is not one. Do not drop file-timestamp caching without adding the matching `inputs`/`artifacts` fields to the migrated step; it is a - direct match, not a gap, but it does not carry over on its own. -- Do not turn `target: dep1 dep2` into plain sequential steps, or into a hand-built `parallel` - step, without first considering command-level `dependencies.commands` -- it runs concurrently - by default and dedups a dependency shared by more than one target, the way `make` already does. + direct match, not a gap, but it does not carry over on its own. Remember the scope difference + too: it gates one step, not the whole recipe -- combine multiple command lines into a single + step if the freshness decision must cover all of them. +- Do not describe `dependencies.commands` as matching Make's *default* prerequisite order -- + `make` builds prerequisites one at a time, in the order listed, unless `-j` is given. + `dependencies.commands` runs concurrently by default, which changes that order. Ordered steps + are the default-preserving match for a plain `target: dep1 dep2`; reach for + `dependencies.commands` only when a prerequisite is shared by more than one target (it dedups a + shared dependency to a single run regardless of concurrency, the way `make` already does), the + prerequisites are genuinely independent, or the source used `-j`. - Do not turn every private or helper target into its own discoverable command by default. If the helper is called from only one recipe, put its logic in a step inside the command or workflow that needs it. If it needs to be called from more than one recipe, or invoked directly for debugging, make it a custom command with `internal: true` instead -- it stays runnable but is excluded from `atmos --help` listings and completion suggestions. - Do not treat "wrap `atmos` commands in the Makefile" as the final state. It is a valid bridge - during early migration. Leaf targets should become custom commands. Target chains should become - workflows. + during early migration. Leaf targets should become custom commands. A target chain usually stays + a custom command too, using `dependencies.commands` for its prerequisites -- do not prescribe a + workflow for every dependency chain. Reserve workflows for fixed, multi-step orchestration + across more than one component. - Do not invent `when:` conditions that check flag values on workflow steps. The `when:` field checks CEL context values, such as `stack`, `ci`, and `local`. Flag-based conditionals belong in the custom command's own Go templates. diff --git a/website/docs/migration/justfile.mdx b/website/docs/migration/justfile.mdx index 22f824d1d17..3b2cd736c60 100644 --- a/website/docs/migration/justfile.mdx +++ b/website/docs/migration/justfile.mdx @@ -203,7 +203,7 @@ integration instead of a plaintext `.env` file. - [ ] Turn `export VAR := value` into `env:` maps - [ ] Replace `set dotenv-load` with `env: !include .env` (or a store/secrets integration for secret values) - [ ] Mark `[private]` recipes as `internal: true` custom commands -- [ ] Stop relying on `just --list`. `atmos help` replaces it +- [ ] Stop relying on `just --list`. `atmos --help` replaces it ## Common Questions diff --git a/website/docs/migration/makefile.mdx b/website/docs/migration/makefile.mdx index 406fa834301..386661345ea 100644 --- a/website/docs/migration/makefile.mdx +++ b/website/docs/migration/makefile.mdx @@ -16,9 +16,10 @@ import TabItem from '@theme/TabItem' A Makefile gives your team an ad-hoc way to run build, test, and deploy tasks. Atmos gives you the same capability as a native, documented feature: custom commands and workflows. Your -Terraform code, scripts, and other tools do not need to change. Atmos replaces only the -Makefile's targets and recipes. You can keep the Makefile as a thin wrapper around `atmos` -commands while you migrate. +Terraform code, scripts, and other tools do not need to change, and neither does your Makefile. +Atmos is the front door either way: if you want, a custom command can simply call `make ` +as its one step, making Atmos a thin wrapper around the Makefile you already have, not the other +way around. Moving a target's own recipe logic into native `steps:` is worthwhile, but optional. :::tip Using an AI Coding Assistant? @@ -38,11 +39,11 @@ See [AI Agent Skills](/ai/skills) for details. |--------------------------------------------------|------------------------------------------------------------| | `.PHONY` target, no dependencies | [Custom command](/cli/configuration/commands) | | `VAR ?= default`, `$(VAR)` | Command `flags:` with a `default:` | -| `target: dep1 dep2` (dependency order) | Steps in order for a single chain, or command [`dependencies.commands`](/cli/configuration/commands/dependencies#named-command-and-workflow-dependencies) when the same prerequisite is shared by more than one target | +| `target: dep1 dep2` (dependency order) | Steps in order (`make`'s own default is sequential); command [`dependencies.commands`](/cli/configuration/commands/dependencies#named-command-and-workflow-dependencies) only when a prerequisite is shared by more than one target, or the source used `-j` | | Recipe shell lines (`@`-silenced, tab-indented) | Steps (`type: shell`, `output: none`) | | `ifeq (...)` conditional | Go template `{{ if }}` in a command, or `when:` in a workflow | | `$(MAKE) -j`, `$(MAKE) -C dir` | `parallel` / `matrix` steps | -| Self-documenting `help` target (`awk`-parsed) | Free: `atmos help` and each command's `description:` | +| Self-documenting `help` target (`awk`-parsed) | Free: `atmos --help` and each command's `description:` | ## Before and After @@ -123,7 +124,7 @@ actually name the component. Turn each `.PHONY` target that has no dependency into its own [custom command](/cli/configuration/commands). In the example above, these are `build`, `test`, -`lint`, and `clean`. Give each command a `description:`. Then `atmos help` shows the same +`lint`, and `clean`. Give each command a `description:`. Then `atmos --help` shows the same information as a hand-written, `awk`-parsed `help` target, so you can delete that target. Turn `VAR ?= default` into a `flags:` entry with a matching `default:`. @@ -152,15 +153,22 @@ from another command or workflow step, prefer `type: atmos` with `command: build `type: shell` with `command: atmos build`: `type: atmos` auto-applies `-s ` from the step's `stack:` field and renders through Atmos's structured step output. -For a simple, single-caller chain such as `deploy: build test`, steps calling each prerequisite in -order are enough. But `make` itself only builds a shared prerequisite once per invocation, even -when more than one target depends on it — plain steps lose that property, since each caller's -`type: atmos`/`type: shell` step runs it again. Command-level +For a chain such as `deploy: build test`, steps calling each prerequisite in order is the +default-preserving match — GNU Make's own default is to build a target's prerequisites one at a +time, in the order listed, not concurrently (`-j` is required for that). Do not reach for +`dependencies.commands` here just because it exists; it runs concurrently by default, which +changes the order unless you've confirmed the prerequisites are genuinely independent. + +`dependencies.commands` earns its place for a different reason: `make` only builds a shared +prerequisite once per invocation, even when more than one target depends on it — plain steps +lose that property, since each caller's `type: atmos`/`type: shell` step runs it again. [`dependencies.commands`](/cli/configuration/commands/dependencies#named-command-and-workflow-dependencies) -is the direct match for that case: it resolves through the same DAG scheduler as -[`parallel`](/workflows/steps/type/parallel)/[`matrix`](/workflows/steps/type/matrix) `needs:`, -runs concurrently by default, and dedups a dependency shared by more than one command to a single -run — the same guarantee `make` gives you for free: +dedups a dependency shared by more than one command to a single run, the same guarantee `make` +gives you for free, independent of whether you also want the concurrency. If order still matters +between the shared dependents (`test` must still finish before `deploy`'s own steps run, and +`build` before `test`), declare that edge directly on the later command's own +`dependencies.commands`, rather than listing every prerequisite as a flat, sibling list on the +caller — the scheduler then waits for it, concurrency aside: ```yaml commands: @@ -184,6 +192,9 @@ commands: command: terraform apply infra -s dev ``` +`build` still runs exactly once for the whole `atmos deploy` invocation — `test`'s own edge on +`build` orders it correctly ahead of `test`, and `deploy`'s own steps wait for both to finish. + If the Makefile's `deploy` target also selects a Terraform environment, for example through `-var-file=envs/$(ENV).tfvars`, that part is a Terraform migration step, not a Make migration step. See [Migrating from Native Terraform](/migration/native-terraform) for how the Terraform @@ -241,19 +252,28 @@ when it didn't. Use `when: timestamp.changed` instead if you want Make's exact m See [inputs](/workflows/steps/inputs) for the full field reference. It does not carry over on its own — add `inputs`/`artifacts` to the migrated step yourself. +The scope is different, too: Make's freshness check gates the target's *entire* recipe, but +`inputs`/`artifacts` are declared per step — skipping one step does not stop later steps in the +same command from running. If a target's recipe has more than one command line and the freshness +decision must gate all of them together, combine them into a single `shell`/`script` step rather +than spreading `inputs`/`artifacts` across several steps. + ## What Stays the Same - **Your Terraform code, scripts, and CLIs.** Atmos wraps them. It does not replace them. - **Environment variables you already export.** Move them into a command or step `env:` map. -- **The Makefile itself, for a while.** Keep it as a thin wrapper around `atmos` commands until - your team is ready. Then remove it, once each leaf target and chain has become a command or a - workflow. +- **The Makefile itself, for as long as you want.** A custom command can call `make ` + directly as its one step — Atmos becomes a consistent facade over the tools you already have, + not a mandatory rewrite of them. Move a target's own recipe logic into native `steps:` only when + you want something `make` doesn't give you: richer step types, freshness checks, or interactive + prompts. ## What You Gain -- **One command for discovery.** Run `atmos help` to list every command. Run - `atmos --help` to see its flags and arguments. A Makefile has no built-in match for - this. It needs a hand-written, `awk`-parsed `help` target instead. +- **One command for discovery.** Run `atmos --help` to list every command (recommended over the + interactive `atmos help`, which pages the same listing). Run `atmos --help` to see its + flags and arguments. A Makefile has no built-in match for this. It needs a hand-written, + `awk`-parsed `help` target instead. - **One consistent interface.** Every custom command uses the same flag and argument syntax. A Makefile has no fixed convention. Each target can parse its own variables in its own way. - **More than 30 built-in step types.** Atmos ships step types for shell commands, orchestration @@ -273,19 +293,23 @@ own — add `inputs`/`artifacts` to the migrated step yourself. - [ ] List every Makefile target. Mark each one as independent, or part of a chain - [ ] Turn independent targets into custom commands in `atmos.yaml` - [ ] Turn `VAR ?= default` variables into command `flags:` with defaults -- [ ] Turn dependency chains into steps in order, or `dependencies.commands` when a prerequisite is shared by more than one target +- [ ] Turn dependency chains into steps in order (`make`'s own default) -- reach for `dependencies.commands` only when a prerequisite is shared by more than one target, or the source used `-j` - [ ] Replace `$(MAKE) -j`/`-C` with `parallel`/`matrix` steps -- [ ] Delete any hand-written `help` target. `atmos help` replaces it -- [ ] Keep the Makefile as a thin wrapper during the transition, if you want one +- [ ] Delete any hand-written `help` target. `atmos --help` replaces it +- [ ] Keep the Makefile for as long as you want -- call `make ` from a step, or migrate the logic into `steps:`, whichever fits - [ ] Turn file-timestamp targets into step `inputs.sources`/`artifacts.paths` ## Common Questions -No. It can stay as a thin wrapper around `atmos` commands during a step-by-step migration. Each -leaf target should become a custom command over time. Each dependency chain should become a -workflow. There is no fixed deadline for this. +No, and not every target has to move into Atmos either. A custom command can call `make ` +as its one step (`command: make build`), which gives you `atmos build`'s consistent flags, +`atmos --help` discovery, and tool-dependency management as a facade over the Makefile you already +have — the point of migrating is a consistent interface over every tool, not necessarily rewriting +every tool. Move a target's logic into native `steps:` when you want something `make` doesn't +have: richer step types, freshness checks, or interactive prompts. Neither is required, and there +is no fixed deadline for either. @@ -298,9 +322,10 @@ carry over automatically; you add it to the migrated step yourself. See Yes, through either step type: `type: atmos` with `command: `, or `type: shell` with `command: atmos `. Prefer `type: atmos` — it auto-applies -`-s ` and gets Atmos's structured step output, which `type: shell` does not. If the same -prerequisite is shared by more than one target, prefer `dependencies.commands` instead — it dedups -a shared dependency to a single run, the way `make` already does. +`-s ` and gets Atmos's structured step output, which `type: shell` does not. This preserves +`make`'s own sequential default. Reach for `dependencies.commands` instead only when the same +prerequisite is shared by more than one target — it dedups a shared dependency to a single run, +the way `make` already does — or when the source target actually used `-j`. diff --git a/website/docs/migration/taskfile.mdx b/website/docs/migration/taskfile.mdx index a4e69803da6..d3316737fa0 100644 --- a/website/docs/migration/taskfile.mdx +++ b/website/docs/migration/taskfile.mdx @@ -127,6 +127,12 @@ again. See [inputs](/workflows/steps/inputs) for the full field reference, inclu checks that a file, tool, or directory exists, not whether it is fresh, so it does not replace `inputs`/`artifacts`. +The scope is different, too: Task's `sources:`/`generates:` gates the task's *entire* `cmds:` +list, but `inputs`/`artifacts` are declared per step — skipping one step does not stop later +steps in the same command from running. If a task has more than one `cmds:` entry and the +freshness decision must gate all of them together, combine them into a single `shell`/`script` +step rather than spreading `inputs`/`artifacts` across several migrated steps. + ## `internal: true` Tasks Set `internal: true` on the custom command. The command still runs (`atmos ...`, or as a @@ -243,7 +249,7 @@ Choose the one that fits what you are splitting: [Toolchain Configuration](/cli/configuration/toolchain). Task has no built-in match for this. - **One interface across every command.** `task --list` shows task names and descriptions. Atmos adds `atmos --help` with the same flag and argument format for every command, - in addition to the CLI-wide `atmos help`. + in addition to the CLI-wide `atmos --help`. ## Migration Checklist @@ -252,7 +258,7 @@ Choose the one that fits what you are splitting: - [ ] Turn `desc:`/`cmds:` into command `description:`/`steps:` - [ ] Turn any `deps:` chain into `dependencies.commands`, to keep Task's concurrent-by-default behavior - [ ] Turn `vars:`/`env:` into command `flags:` (with defaults) and `env:` maps -- [ ] Turn `sources:`/`generates:` into step `inputs.sources`/`artifacts.paths` +- [ ] Turn `sources:`/`generates:` into step `inputs.sources`/`artifacts.paths` -- combine multiple `cmds:` entries into one step if the freshness decision must gate all of them together - [ ] Split `includes:` into auto-discovered `atmos.d/*.yaml` files, separate workflow files, or both - [ ] Mark `internal: true` tasks as `internal: true` custom commands @@ -262,8 +268,10 @@ Choose the one that fits what you are splitting: Yes, once you migrate it. Turn `sources:`/`generates:` into step `inputs.sources`/`artifacts.paths` — with no explicit `when:`, that implicitly skips the step when nothing has changed since its last -successful run, the same as Task's own check. It does not carry over automatically; you add it to -the migrated step yourself. See +successful run. Unlike Task's own check, the scope is per step, not per task — if the task has +more than one `cmds:` entry, combine them into a single step so the freshness decision still gates +all of them together. It does not carry over automatically; you add it to the migrated step +yourself. See [sources:/generates: Becomes inputs/artifacts](#sourcesgenerates-becomes-inputsartifacts). From fc3f4a447428461e7d7ffa7b9094efa51d7af4c2 Mon Sep 17 00:00:00 2001 From: Erik Osterman Date: Wed, 19 Aug 2026 20:55:41 -0500 Subject: [PATCH 16/17] chore(gitignore): ignore cached tools/gomodcheck binary Mirrors the existing tools/lintroller/.lintroller entry for the mage lint helper's cached build output. --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index 9f67a775bb3..b043e2f3783 100644 --- a/.gitignore +++ b/.gitignore @@ -134,6 +134,7 @@ TEST_QUALITY_*.md tools/lintroller/.lintroller .golangci/lintroller/lintroller /custom-gcl +tools/gomodcheck/.gomodcheck # golangci-lint per-worktree cache + lock isolation (see # magefiles/mage_lint_golangci_run.go). The cache is large (100MB+); .golangci-tmp From d17c6e44a91bdb2f221d5fc084fa393cb027bb18 Mon Sep 17 00:00:00 2001 From: Erik Osterman Date: Mon, 31 Aug 2026 21:45:39 -0500 Subject: [PATCH 17/17] fix(docs): address CodeRabbit findings on atmos-migration skill docs Corrects eight documentation-accuracy issues flagged by CodeRabbit on the atmos-migration skill: prefer `type: atmos` over `type: shell` for calling another custom command (preserves stack context and structured output); map Make's `@` prefix to `show: { command: false }` instead of `output: none` (which discards stdout/stderr entirely); describe `build-all`'s `-j4` loop as sequential, not parallel; define a proper per-service `build-service` command for the Shape C matrix example; document the `metadata.component` no-move option for mapping a stack component onto an existing directory; scope "target chains become workflows" guidance to ordered custom-command steps instead; and fix the justfile.mdx examples to carry environment variables and per-environment Terraform vars across build/test/deploy. Co-Authored-By: Claude Sonnet 5 --- agent-skills/skills/atmos-migration/SKILL.md | 8 ++- .../references/from-justfile.md | 15 ++--- .../references/from-makefile.md | 56 ++++++++++++++----- .../references/from-native-terraform.md | 31 ++++++---- website/docs/migration/justfile.mdx | 42 +++++++++++--- website/docs/migration/makefile.mdx | 2 +- 6 files changed, 114 insertions(+), 40 deletions(-) diff --git a/agent-skills/skills/atmos-migration/SKILL.md b/agent-skills/skills/atmos-migration/SKILL.md index b9217bbed2f..c8418ea7a65 100644 --- a/agent-skills/skills/atmos-migration/SKILL.md +++ b/agent-skills/skills/atmos-migration/SKILL.md @@ -247,8 +247,12 @@ Push back if a user or another agent proposes one of these methods during migrat with one stack file. - **"Wrap atmos commands in a Makefile, Justfile, or Taskfile forever."** This is false. A wrapper is a good bridge while the user builds trust in Atmos. But it is not the final state. - Change each leaf target to a custom command. Change each dependency chain to a workflow, once - the team is ready. + Change each leaf target to a custom command. An ordinary Make or Just dependency chain (for + example, `deploy: build test`) stays a custom command with ordered steps or + `dependencies.commands` -- it does not need a workflow. A Taskfile's `deps:` maps the same way, + onto ordered steps or `dependencies.commands` on the custom command. Reserve workflows for + fixed, multi-step orchestration across more than one component, not for an ordinary target + chain. ## Additional Resources diff --git a/agent-skills/skills/atmos-migration/references/from-justfile.md b/agent-skills/skills/atmos-migration/references/from-justfile.md index b4c8bc2492b..8c446f10b0e 100644 --- a/agent-skills/skills/atmos-migration/references/from-justfile.md +++ b/agent-skills/skills/atmos-migration/references/from-justfile.md @@ -77,17 +77,18 @@ deploy env='dev': build test ``` **Steps:** use the same method as Make's dependency chains. See -[from-makefile.md Shape B](from-makefile.md#shape-b-target-chains-with-dependencies). Do not use -a `type: atmos` step to call another custom command. That step type is only for native Atmos -verbs. Call the dependency with a `type: shell` step and `command: atmos build`. +[from-makefile.md Shape B](from-makefile.md#shape-b-target-chains-with-dependencies). Use a +`type: atmos` step with `command: build` to call another custom command -- `type: atmos` preserves +step-level stack context and structured output handling, which a `type: shell` step running +`atmos build` does not. ```yaml commands: - name: test description: Run tests (builds first) steps: - - type: shell - command: atmos build + - type: atmos + command: build - type: shell command: go test ./... @@ -98,8 +99,8 @@ commands: shorthand: e default: "dev" steps: - - type: shell - command: atmos test + - type: atmos + command: test - type: atmos command: terraform apply infra -s {{ .Flags.env }} ``` diff --git a/agent-skills/skills/atmos-migration/references/from-makefile.md b/agent-skills/skills/atmos-migration/references/from-makefile.md index 80b0b808dc9..d21eaf4c70b 100644 --- a/agent-skills/skills/atmos-migration/references/from-makefile.md +++ b/agent-skills/skills/atmos-migration/references/from-makefile.md @@ -49,13 +49,18 @@ clean: ## Remove build artifacts 1. Turn each leaf target into a custom command in `atmos.yaml`. If the Makefile is large, put the commands in a separate file. See [Split commands across files](#split-commands-across-files) below. -2. Turn the silent-recipe `@` prefix into the step field `output: none`. +2. Turn the silent-recipe `@` prefix into `show: { command: false }`. Make's `@` only suppresses + the echoed command line -- the recipe's own stdout/stderr still prints. `output: none` is not + the same thing: it sends the step's stdout *and* stderr to the void, which would hide real + command output (for example, `go test` diagnostics). Reserve `output: none` for a step whose + output genuinely needs to be discarded entirely. 3. Delete the `help` target. Atmos generates the same information from each command's `description:` field. Run `atmos --help` or `atmos --help` to see it. 4. When one target depends on another leaf target, such as `test: build`, add a step that runs - the dependency's command. Do not use a `type: atmos` step to call a custom command. That step - type is only for native Atmos verbs, such as `terraform plan`. Use a `type: shell` step with - `command: atmos build` instead. + the dependency's command. Use a `type: atmos` step with `command: build` to call the other + custom command -- `type: atmos` preserves step-level stack context and gives you structured + output handling (captured stdout/stderr, exit code) that a `type: shell` step invoking + `atmos build` does not. ```yaml commands: @@ -68,8 +73,8 @@ commands: - name: test description: Run unit tests steps: - - type: shell - command: atmos build + - type: atmos + command: build - type: shell command: go test ./... @@ -84,7 +89,8 @@ commands: steps: - type: shell command: rm -rf bin/ - output: none + show: + command: false ``` ## Shape B: Target Chains with Dependencies @@ -172,12 +178,34 @@ build-parallel: **Steps:** -1. Turn `$(MAKE) -j` into a `parallel` step. Use `max_concurrency` to set the fan-out width. +1. `build-all`'s recipe is a shell `for` loop, not multiple Make targets -- `make -j` only + parallelizes independent targets within a single `make` invocation, it does not parallelize + commands inside one recipe's shell script. So `$(MAKE) -j4 build-all` still runs `vpc`, `eks`, + and `rds` one at a time, in order, exactly like plain `build-all` would. `-j4` here does + nothing. 2. Turn `$(MAKE) -C dir target` recursion over a fixed set of directories into a `matrix` step. - Define a `service` axis, and call the per-service command once for each value. + Define a `service` axis, and call the per-service command once for each value. Using + `max_concurrency` on the matrix step is a deliberate upgrade over the sequential source + behavior, not a literal translation of `-j4` -- call it out to whoever is reviewing the + migration, since it changes execution behavior (all services build concurrently instead of + one at a time). +3. The per-directory `build` target here is a different recipe than Shape A's top-level `build` + command -- it lives in each `services//` directory, not at the repo root. Give it its own + custom command, `build-service`, with a `service` flag/argument that maps into the + `services/` path. Do not reuse Shape A's `build` command as-is; it has no way to receive + a per-service value. ```yaml commands: + - name: build-service + description: Build a single service + flags: + - name: service + required: true + steps: + - type: shell + command: go build -o bin/{{ .Flags.service }} ./services/{{ .Flags.service }} + - name: build-all description: Build every service steps: @@ -187,8 +215,8 @@ commands: service: [vpc, eks, rds] max_concurrency: 4 steps: - - type: shell - command: atmos build --service {{ .matrix.service }} + - type: atmos + command: build-service --service {{ .matrix.service }} ``` ## Common Problems @@ -214,8 +242,10 @@ than spreading `inputs`/`artifacts` across several steps. ### Silent recipes and command echo -`@command` suppresses the echo of one command line. It maps to the step field `output: none` on -that one step. It does not mean you should add `output: none` to every step. +`@command` suppresses the echo of one command line -- it does not touch the recipe's own output. +It maps to `show: { command: false }` on that one step, not `output: none`. `output: none` +discards the step's stdout and stderr entirely, which is a much bigger change in behavior than +Make's `@` prefix ever was; reserve it for steps whose output genuinely needs to be thrown away. ### Split commands across files diff --git a/agent-skills/skills/atmos-migration/references/from-native-terraform.md b/agent-skills/skills/atmos-migration/references/from-native-terraform.md index aa79b438dd0..55bc209e4fa 100644 --- a/agent-skills/skills/atmos-migration/references/from-native-terraform.md +++ b/agent-skills/skills/atmos-migration/references/from-native-terraform.md @@ -100,10 +100,10 @@ With a Makefile like `terraform plan -var-file=envs/$(ENV).tfvars`. included_paths: - "**/*" ``` -2. **The component name must match the physical directory name** -- Atmos resolves a component - to `/`, so with `base_path: "."` the - component name has to be `terraform` (the real directory), not an invented name like `infra`. - Renaming the component in Atmos config does not rename the directory on disk: +2. **The component name must match the physical directory name, unless you set + `metadata.component`** -- Atmos resolves a component to + `/`, so with `base_path: "."` a stack + component named `terraform` resolves to the real directory with no extra configuration: ```yaml # stacks/dev.yaml components: @@ -111,12 +111,23 @@ With a Makefile like `terraform plan -var-file=envs/$(ENV).tfvars`. terraform: vars: !include ../terraform/envs/dev.tfvars ``` - If the user wants a friendlier component name without moving files, rename the directory - itself (e.g. `terraform/` to `infra/`) rather than trying to alias it in `atmos.yaml` -- there - is no `metadata.component` override needed here since this is a single-component repo, and - `metadata.component` is for pointing multiple stack instances at one shared component (see - [remote-state-bridge.md](remote-state-bridge.md)), not for renaming a component's own - directory. + If the user wants a friendlier stack component name, such as `infra`, without moving or + renaming the directory, set `metadata.component` to the physical directory name instead: + ```yaml + # stacks/dev.yaml + components: + terraform: + infra: + metadata: + component: terraform # points at the existing `terraform/` directory + vars: !include ../terraform/envs/dev.tfvars + ``` + `metadata.component` is a pointer to the physical directory, resolved the same way the + component name normally is (relative to `components.terraform.base_path`, or absolute). It is + also how you point multiple stack instances at one shared component (see + [remote-state-bridge.md](remote-state-bridge.md)) -- the no-move rename above is the same + mechanism applied to a single-component repo. Only rename the directory on disk (e.g. + `terraform/` to `infra/`) if the user prefers a physical rename over an alias. 3. The Makefile can stay as a thin wrapper around `atmos terraform plan terraform -s dev` during transition, then be deleted. diff --git a/website/docs/migration/justfile.mdx b/website/docs/migration/justfile.mdx index 3b2cd736c60..d4ab3086e85 100644 --- a/website/docs/migration/justfile.mdx +++ b/website/docs/migration/justfile.mdx @@ -80,11 +80,18 @@ only cares what the recipe does. ```yaml + # AWS_REGION (and anything else in .env) applies to every Atmos operation, not just one + # command's own steps -- this matches the source Justfile's `set dotenv-load`/`export`, + # which apply to every recipe. A command-level `env:` map only reaches that command's own + # steps, so `test` invoking `atmos build` (a separate process) and `deploy` invoking + # `atmos test` would not inherit it. + env: + <<: !include .env + AWS_REGION: us-east-1 + commands: - name: build description: Build the deployable artifact - env: - AWS_REGION: us-east-1 steps: - type: shell command: go build -o bin/handler ./cmd/handler @@ -92,8 +99,8 @@ only cares what the recipe does. - name: test description: Run tests (builds first) steps: - - type: shell - command: atmos build + - type: atmos + command: build - type: shell command: go test ./... @@ -104,8 +111,8 @@ only cares what the recipe does. shorthand: e default: "dev" steps: - - type: shell - command: atmos test + - type: atmos + command: test - type: atmos command: terraform apply infra -s {{ .Flags.env }} ``` @@ -115,7 +122,28 @@ only cares what the recipe does. `infra` is a placeholder Atmos component name, not the `terraform` verb repeated. Move the old `terraform/` directory's `.tf` files to `components/terraform/infra/` (the default `components.terraform.base_path` is `components/terraform`), then swap `infra` for whatever you -actually name the component. +actually name the component. Alternatively, keep the existing `terraform/` directory where it is: +set `components.terraform.base_path: "."` and add `metadata.component: terraform` on the `infra` +stack component -- `metadata.component` points the stack component at the physical directory, so +no files need to move. + +`-s {{ .Flags.env }}` only selects *which stack* runs, the way `terraform.apply` picks a stack by +name (`dev`, `staging`, `prod`); it does not, by itself, load that environment's Terraform +variables the way the source `-var-file=envs/{{env}}.tfvars` did. Bring the per-environment +`.tfvars` files in through each stack file instead, one per environment (`stacks/dev.yaml`, +`stacks/staging.yaml`, `stacks/prod.yaml`), each pointing at its own file: + + +```yaml +components: + terraform: + infra: + vars: !include ../components/terraform/infra/envs/dev.tfvars +``` + + +See [Migrating from Native Terraform](/migration/native-terraform) for the full `.tfvars`/stack +mapping. ## Named Parameters Become Flags and Arguments diff --git a/website/docs/migration/makefile.mdx b/website/docs/migration/makefile.mdx index 386661345ea..f06f2f31e38 100644 --- a/website/docs/migration/makefile.mdx +++ b/website/docs/migration/makefile.mdx @@ -145,7 +145,7 @@ atmos build ``` -## Target Chains Become Workflows +## Target Chains Become Ordered Custom-Command Steps A `type: atmos` step runs the `atmos` binary itself, so it can call any atmos command, a native verb such as `terraform apply` or another custom command such as `build`. To reuse a command