AWS infra build-image is accepted, then both Lambda MicroVM builds fail without diagnostics

[abemcintosh]

Summary

On the AWS target, qm infra build-image successfully uploads its source ZIP and AWS accepts CreateMicrovmImage, but both server-side builds fail with only:

The container image build failed.

The dedicated CloudWatch log group exists but contains no log streams, and GetMicrovmImageBuild exposes no more detailed failure reason.

This is a hard deployment blocker, not a cosmetic sandbox warning. In QM 0.1.4, plan, up, and check --live all require the configured MicroVM image version to be SUCCESSFUL and ACTIVE, even with HARNESS: "mock". Terraform otherwise applied successfully and provisioned 58 resources, but the deployment cannot reach its pre-up plan while the image is failed.

Environment

• QM: @yc-software/qm@0.1.4
• Target: aws
• Region: us-east-1
• Services: core, slack
• Harness: mock
• Node.js: 24.19.0
• npm: 11.17.0
• AWS CLI: 2.36.19 with lambda-microvms commands
• Terraform: 1.15.8
• Terraform providers: hashicorp/aws 5.100.0, hashicorp/random 3.9.0
• Operator OS: Amazon Linux 2023, arm64

Reproduction and observed result

Exact QM invocation:

npm exec qm -- infra build-image

Verbatim QM output:

· uploaded s3://<BUCKET>/deployment/microvm-images/59bc8eef058e526fd1b9fcd9efedf2590e5822a50cfd022ae2143770a3596919.zip
· creating tangle-qm-sandbox
error: MicroVM image version 1.0 failed

CreateMicrovmImage was authorized and returned the image in CREATING; it did not return an IAM or API error. The image later became FAILED / INACTIVE.

AWS reported two failed builds:

• Build 79c17703-809c-49a0-9b42-2125f0de682b: ARM_64, GRAVITON, generation 4; reason: The container image build failed.
• Build aa07df2b-670a-43cc-9bfb-dda4b980bac4: ARM_64, GRAVITON, generation 3; reason: The container image build failed.

The expected log group /aws/lambda/microvms/tangle-qm-sandbox exists but has zero log streams. Both ListMicrovmImageBuilds and GetMicrovmImageBuild return only the generic reason above.

Generated image request and Dockerfile from QM 0.1.4

Image request, with account and bucket redacted:

name: tangle-qm-sandbox
imageArn: arn:aws:lambda:us-east-1:<ACCOUNT_ID>:microvm-image:tangle-qm-sandbox
imageVersion: "1.0"
baseImageArn: arn:aws:lambda:us-east-1:aws:microvm-image:al2023-1
buildRoleName: tangle-qm-microvm-build
codeArtifact:
  uri: s3://<BUCKET>/deployment/microvm-images/59bc8eef058e526fd1b9fcd9efedf2590e5822a50cfd022ae2143770a3596919.zip
egressNetworkConnectors:
  - arn:aws:lambda:us-east-1:aws:network-connector:aws-network-connector:INTERNET_EGRESS
resources:
  - minimumMemoryInMiB: 2048

The uploaded ZIP contains this Dockerfile and agent.mjs:

FROM public.ecr.aws/lambda/microvms:al2023-minimal@sha256:05cb9b38d841e7ff1b693dc9e894909612f340bf99ec97d426e8000a5bbe96c3

RUN dnf install -y \
      curl-minimal findutils tar git wget jq unzip \
      python3 python3-pip nodejs24 nodejs24-npm \
  && dnf clean all \
  && rpm -q openssl-snapsafe-libs \
  && curl --version \
  && node --version \
  && npm --version \
  && npx --version \
  && setsid --version \
  && seq 1 1

RUN curl -fsSL https://cli.github.com/packages/rpm/gh-cli.repo \
       -o /etc/yum.repos.d/gh-cli.repo \
  && dnf install -y gh-2.96.0-1 \
  && dnf clean all \
  && gh --version

RUN curl -fsSL "https://awscli.amazonaws.com/awscli-exe-linux-aarch64-2.36.5.zip" -o /tmp/awscliv2.zip \
  && echo "2662774cf3e70e2203ca1e6b85b84ff8438f6638523ea9d474f9731472740d85  /tmp/awscliv2.zip" | sha256sum -c - \
  && unzip -q /tmp/awscliv2.zip -d /tmp \
  && /tmp/aws/install \
  && rm -rf /tmp/aws /tmp/awscliv2.zip \
  && aws --version

RUN ARCH="$(uname -m)" && case "$ARCH" in \
      x86_64) LS_ARCH=amd64; LS_SHA=eb75a3de5cab03875cdae9f5f539e6aedadd66607003d9b1e7a9077948818ba0 ;; \
      aarch64) LS_ARCH=arm64; LS_SHA=9585f5a508516bd66af2b2376bab4de256a5ef8e2b73ec760559e679628f2d59 ;; \
      *) echo "unsupported arch $ARCH" && exit 1 ;; esac \
  && curl -fsSL "https://github.com/benbjohnson/litestream/releases/download/v0.3.13/litestream-v0.3.13-linux-${LS_ARCH}.tar.gz" -o /tmp/litestream.tgz \
  && echo "${LS_SHA}  /tmp/litestream.tgz" | sha256sum -c - \
  && tar -xzf /tmp/litestream.tgz -C /usr/local/bin litestream \
  && rm /tmp/litestream.tgz \
  && litestream version

RUN python3 -m venv /opt/agent-venv \
  && /opt/agent-venv/bin/pip install --no-cache-dir --upgrade pip
ENV PATH="/opt/agent-venv/bin:${PATH}"
ENV VIRTUAL_ENV=/opt/agent-venv

COPY agent.mjs /opt/microvm-agent/agent.mjs

ENV HOME=/root
WORKDIR /root

CMD ["node", "/opt/microvm-agent/agent.mjs"]

Checks already completed

IAM is no longer the failure. CloudTrail confirms that CreateMicrovmImage now authorizes cleanly and accepts the request using the narrowly scoped build role tangle-qm-microvm-build. The failure occurs later in the AWS-managed image build.

The operator host did not leak its architecture into the artifact. QM builds a ZIP containing the Dockerfile and agent.mjs in Node, uploads it, and asks AWS to build it server-side. It does not invoke local Docker or Buildx. The exact pinned al2023-minimal manifest resolves only to linux/arm64, and the packaged Dockerfile deliberately downloads the aarch64 AWS CLI. The ARM64/Graviton builds therefore appear intentional; I did not attempt an incorrect amd64 build.

There is no unpublished fix on upstream main. I compared v0.1.4 (7f2c916) with main (0f0e0ad, 28 commits ahead at the time of testing). None of those commits touches aws/microvm-agent/ or cli/. The Dockerfile and agent.mjs are byte-identical across the tag, main, and installed npm package, and npm latest remains 0.1.4.

PassRole documentation note

One separate integration gotcha may be worth documenting. CreateMicrovmImage requires iam:PassRole, but its authorization context supplies an iam:PassedToService value that is not lambda.amazonaws.com. The value is redacted from CloudTrail.

Because the key is present with a non-matching value, both StringEquals and StringEqualsIfExists conditioned on lambda.amazonaws.com deny the call; IfExists does not help. Since operators cannot observe the supplied value, they cannot write a condition that matches it. We ultimately used a separate unconditioned iam:PassRole statement scoped only to the exact tangle-qm-microvm-build role, while retaining service-conditioned PassRole for ordinary ECS/Lambda roles.

Questions

1. Where can operators retrieve the service-side container-build diagnostics for a failed Lambda MicroVM image build when GetMicrovmImageBuild gives only The container image build failed and the configured log group has no streams?
2. Are there documented compatibility or packaging requirements for the source ZIP consumed by CreateMicrovmImage, beyond containing this Dockerfile and agent.mjs?
3. Is the pinned public.ecr.aws/lambda/microvms:al2023-minimal@sha256:05cb9b38... base image still valid for QM 0.1.4 in us-east-1?
4. Does Lambda MicroVM image building require any account-level enablement, preview access, or service quota beyond successful authorization of CreateMicrovmImage and its build role?