Unable to get module to parse cross-account assumed role within AWS

[westinn]

Currently I am running this module within an EC2 server as a Caddy Docker container within a developer AWS account (ACCOUNT_A). This EC2 machine has access to a set of Route53 resources on another account (ACCOUNT_B) via an attached IAM Role. The trust policies are in place cross-account, and when assigning proper AWS creds via a Bash script to query metadata and setting ENV variables for testing, this module is able to assume the role fine.

When I forgo this more manual environment setting script and allow the Caddy container to assume the roles automatically, I am getting failures. Is there a way to tell this module to attempt to assume a role? I was reading that the AWS SDK allows for this.

Logs:

Within the logs I see explicit failures to access the resources that worked previously, and it seems to be because it is using the developer accounts role, rather than assuming a role cross-account as it should. I have read that passing the target AWS_ROLE_ARN (as a Docker env variable) should work, as the AWS SDK will attempt to reach it. For example within my Docker Compose: AWS_ROLE_ARN: arn:aws:iam::ACCOUNT_B:role/dev-machine-role.

Here is my error with some info omitted. I'd expect to see the correct AWS Account ID at the very least since I'm specifying a target ARN as stated above.

2025/07/15 21:53:24.558 ERROR   tls.obtain      could not get certificate from issuer   {"identifier": "*.subdomain.test.com", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "[*.subdomain.test.com] solving challenges: presenting for challenge: adding temporary record for zone \"test.com.\": operation error Route 53: ListResourceRecordSets, https response error StatusCode: 403, api error AccessDenied: User: arn:aws:sts::ACCOUNT_A:assumed-role/dev-machine-role/i-123 is not authorized to access this resource"}

Does this error signify that the modules SDK is picking up role information incorrectly? Or is it something on my end?

[AndrianBdn]

This case is complex, and we currently lack contributors to investigate it in depth.

Significant core updates, including AWS SDK changes, were made recently, which might indirectly resolve the issue. Please test with the latest beta versions, noting that they require Caddy 2.10.

If the problem persists, try using AWS access keys as a temporary workaround - it's not ideal but likely functional.

[HelixSpiral]

Hey @westinn - just checking in to see if this was resolved or not.

In the case updating to the latest version didn't indirectly resolve your issue I have something for you to try.

Create a profile in ~/.aws/config with something like the following:

[profile cross-account-route53]
role_arn=arn:aws:iam::ACCOUNT_B:role/your-role-for-route53`
credential_source=Ec2InstanceMetadata
region=us-east-1

Then configure Caddy with

tls {
  dns route53 {
    profile "cross-account-route53"
    region "us-east-1"
  }
}

What I think is happening is you're providing AWS_ROLE_ARN without providing the credentials for it to use to assume that role, so it falls back to other methods in the default provider chain (like your EC2 instance profile), and then fails to do the cross-account route53 actions with that assumed role.

Docs: https://docs.aws.amazon.com/cli/v1/userguide/cli-configure-files.html#cli-configure-files-format-profile (Amazon EC2 instance metadata credentials tab)

[westinn]

I am no longer using this environment setup, so I can't confirm whether or not an update would fix it. I figure I'll add some additional info in case this comes up in the future. This is also based on my memory, so take it with a grain of salt.

Regarding the proposed cause/solution: providing AWS_ROLE_ARN without providing the credentials to use it:

I believe I tried the caddy module without the AWS_ROLE_ARN at first, so I think the SDK should be going to my instance profile (that is correctly set up) regardless. It didn't work with or without this additional env variable.

To confirm the role adoption issue, I saw different behavior when using Caddy vs when running manual commands on AWS CLI in the terminal. Running AWS CLI commands, I was able to query AWS resources on ACCOUNT_B from a dev machine living in ACCOUNT_A using a cross-account instance role assigned within the AWS GUI console. This was with no further environment variable changes or manual specification.

I would assume that the SDK and all subsequent libraries/modules (caddy-dns for example) would be able to utilize the same credentials and role info. Maybe since I was running Caddy within Docker on that EC2 machine, I would need to do more to provide more info within the container. I had read that the AWS SDK's default credentials provider chain would be able to handle this automatically without me intervening though. I also want to say that I had the server metadata hops configured correctly, so this should've worked within the container (if not, that would be an immediate issue).

[AndrianBdn]

Sorry this has been open a while. If you (or anyone else still hitting this) want to try again — v1.6.1 just added an optional debug_logging true setting. Add it inside your dns route53 { ... } block, and set Caddy's log level to debug. The module will then write events to Caddy's log, including which AWS identity it's using when it calls Route53. That should make it clear whether the SDK is picking up your cross-account role, or falling back to the EC2 instance role.