feed.xml

<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.9.0">Jekyll</generator><link href="https://thirstydeveloper.io/feed.xml" rel="self" type="application/atom+xml" /><link href="https://thirstydeveloper.io/" rel="alternate" type="text/html" /><updated>2022-01-02T12:12:45+00:00</updated><id>https://thirstydeveloper.io/feed.xml</id><title type="html">thirstydeveloper</title><subtitle>Infrastructure-as-Code-as-a-Team</subtitle><author><name>Chris Kent</name></author><entry><title type="html">Terraform Skeleton Part 6: Protecting State</title><link href="https://thirstydeveloper.io/tf-skeleton/2021/02/25/part-6-protecting-state.html" rel="alternate" type="text/html" title="Terraform Skeleton Part 6: Protecting State" /><published>2021-02-25T19:00:00+00:00</published><updated>2021-02-25T19:00:00+00:00</updated><id>https://thirstydeveloper.io/tf-skeleton/2021/02/25/part-6-protecting-state</id><content type="html" xml:base="https://thirstydeveloper.io/tf-skeleton/2021/02/25/part-6-protecting-state.html">&lt;p&gt;&lt;a href=&quot;/tf-skeleton/2021/02/17/part-5-cfn-terraform-state.html&quot;&gt;Part 5&lt;/a&gt; moved Terraform’s operational infrastructure &lt;a href=&quot;https://github.com/thirstydeveloper/terraform-terragrunt-skeleton/compare/release/1.3...release/1.4#diff-cc93827adede14cb1e69dde0ed0769a47014bbb00f7ec444c61b927f3901c35a&quot;&gt;into a CloudFormation stack&lt;/a&gt; so that Terragrunt no longer manages the state bucket, lock table, and log bucket. Doing so offers us the opportunity to protect these resources in ways not supported by Terragrunt. We will capitalize on that opportunity today.&lt;/p&gt;

&lt;p&gt;We’ll cover adding a bucket policy to our CloudFormation template to restrict access to Terraform’s state, leveraging the backend role and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Terraformer&lt;/code&gt; principal tag introduced in &lt;a href=&quot;/tf-skeleton/2021/02/10/part-4-backend-role.html&quot;&gt;Part 4&lt;/a&gt;. I hope my example will ease your navigation of IAM policy syntax and save you the pain of decrypting unhelpful IAM error messages.&lt;/p&gt;

&lt;p&gt;We’ll also add some basic protections to the log bucket that Terragrunt fails to implement out of the box.&lt;/p&gt;

&lt;h1 id=&quot;goals&quot;&gt;Goals&lt;/h1&gt;

&lt;ul&gt;
  &lt;li&gt;Grant only authorized principals access to Terraform state&lt;/li&gt;
  &lt;li&gt;Protect the log bucket in a similar fashion to the state bucket&lt;/li&gt;
  &lt;li&gt;Remove unnecessary permissions from the backend role&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you prefer to jump to the end, the code implementing this post’s final result is available on branch &lt;a href=&quot;https://github.com/thirstydeveloper/terraform-terragrunt-skeleton/tree/release/1.5&quot;&gt;release/1.5&lt;/a&gt; on GitHub. Additionally, you can &lt;a href=&quot;https://github.com/thirstydeveloper/terraform-terragrunt-skeleton/compare/release/1.4...release/1.5&quot;&gt;view the diffs from part 5&lt;/a&gt;, if that’s more your speed.&lt;/p&gt;

&lt;h1 id=&quot;access-roles&quot;&gt;Access Roles&lt;/h1&gt;

&lt;p&gt;There are three types of authorized access to Terraform state we will cover today.&lt;/p&gt;

&lt;p&gt;First is our backend role, which the Terragrunt configures Terraform to use for all state operations via the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;root.hcl&lt;/code&gt; from &lt;a href=&quot;/tf-skeleton/2021/02/10/part-4-backend-role.html&quot;&gt;Part 4&lt;/a&gt;, reprinted here:&lt;/p&gt;

&lt;div class=&quot;language-hcl highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;nx&quot;&gt;remote_state&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
  &lt;span class=&quot;nx&quot;&gt;backend&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;s3&quot;&lt;/span&gt;
  &lt;span class=&quot;nx&quot;&gt;generate&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
    &lt;span class=&quot;nx&quot;&gt;path&lt;/span&gt;      &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;backend.tf&quot;&lt;/span&gt;
    &lt;span class=&quot;nx&quot;&gt;if_exists&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;overwrite&quot;&lt;/span&gt;
  &lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
  &lt;span class=&quot;nx&quot;&gt;config&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
    &lt;span class=&quot;nx&quot;&gt;bucket&lt;/span&gt;   &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;terraform-skeleton-state&quot;&lt;/span&gt;
    &lt;span class=&quot;nx&quot;&gt;region&lt;/span&gt;   &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;us-east-1&quot;&lt;/span&gt;
    &lt;span class=&quot;nx&quot;&gt;encrypt&lt;/span&gt;  &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;kc&quot;&gt;true&lt;/span&gt;
    &lt;span class=&quot;nx&quot;&gt;role_arn&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;arn:aws:iam::${get_aws_account_id()}:role/terraform/TerraformBackend&quot;&lt;/span&gt;

    &lt;span class=&quot;nx&quot;&gt;key&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;${dirname(local.relative_deployment_path)}/${local.stack}.tfstate&quot;&lt;/span&gt;

    &lt;span class=&quot;nx&quot;&gt;dynamodb_table&lt;/span&gt;            &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;terraform-skeleton-state-locks&quot;&lt;/span&gt;
    &lt;span class=&quot;nx&quot;&gt;accesslogging_bucket_name&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;terraform-skeleton-state-logs&quot;&lt;/span&gt;
  &lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Second are humans, who may wish to interact with the state for any number of reasons. One example is changing the name of an existing stack (which in our setup will change the name of the state file).&lt;/p&gt;

&lt;p&gt;I like to differentiate between developer-level and administrative-level access to Terraform state, the primary difference being that only administrators should be allowed to delete state files &lt;em&gt;permanently&lt;/em&gt;. Deleting a state file is not something to take lightly. If you delete one you still need by accident, and you have no backups, your only choice will be to reconstruct it by hand.&lt;/p&gt;

&lt;p&gt;The policy we implement today will grant:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Full access to administrative IAM users, including the AWS account’s root user&lt;/li&gt;
  &lt;li&gt;Limited access to developer IAM users and the backend role&lt;/li&gt;
  &lt;li&gt;Deny access to anyone else&lt;/li&gt;
&lt;/ul&gt;

&lt;h1 id=&quot;state-bucket-policy&quot;&gt;State Bucket Policy&lt;/h1&gt;

&lt;p&gt;We will use an S3 bucket policy to implement the above restrictions. An S3 bucket policy is an ideal choice for two reasons. First, it directly applies permissions to the resource we want to protect (the state bucket). Second, developers can likely create it themselves, unlike adding permissions to IAM users, something typically controlled by an enterprise team.&lt;/p&gt;

&lt;p&gt;We’ll add the bucket policy as a resource to our &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;init-admin-account.cf.yml&lt;/code&gt; CloudFormation template. We’ll break the policy down statement by statement.&lt;/p&gt;

&lt;p&gt;The first statement requires TLS encryption for any requests accessing Terraform state. This policy Terragrunt creates for you; we preserve it here.&lt;/p&gt;

&lt;div class=&quot;language-yaml highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;  &lt;span class=&quot;na&quot;&gt;TerraformStateBucketPolicy&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
    &lt;span class=&quot;na&quot;&gt;Type&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;AWS::S3::BucketPolicy'&lt;/span&gt;
    &lt;span class=&quot;na&quot;&gt;DeletionPolicy&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;Retain&lt;/span&gt;
    &lt;span class=&quot;na&quot;&gt;UpdateReplacePolicy&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;Retain&lt;/span&gt;
    &lt;span class=&quot;na&quot;&gt;Properties&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
      &lt;span class=&quot;na&quot;&gt;Bucket&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;!Ref&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;TerraformStateBucket&lt;/span&gt;
      &lt;span class=&quot;na&quot;&gt;PolicyDocument&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
        &lt;span class=&quot;na&quot;&gt;Version&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;2012-10-17'&lt;/span&gt;
        &lt;span class=&quot;na&quot;&gt;Statement&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
          &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;na&quot;&gt;Sid&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;AllowTLSRequestsOnly'&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Principal&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;*'&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Condition&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
              &lt;span class=&quot;na&quot;&gt;Bool&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
                &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;aws:SecureTransport'&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;no&quot;&gt;false&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Effect&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;Deny&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Action&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;*'&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Resource&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;!GetAtt&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;TerraformStateBucket.Arn&quot;&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;!Sub&lt;/span&gt;
                &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;${Bucket}/*&quot;&lt;/span&gt;
                &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;na&quot;&gt;Bucket&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;!GetAtt&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;TerraformStateBucket.Arn&quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;The second statement uses the &lt;a href=&quot;https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_variables.html#principaltable&quot;&gt;aws:PrincipalType&lt;/a&gt; and &lt;a href=&quot;https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-principaltag&quot;&gt;aws:PrincipalTag&lt;/a&gt; condition keys to deny access to any IAM users lacking the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Terraformer&lt;/code&gt; principal tag we introduced in &lt;a href=&quot;/tf-skeleton/2021/02/10/part-4-backend-role.html&quot;&gt;Part 4&lt;/a&gt;:&lt;/p&gt;

&lt;div class=&quot;language-yaml highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;          &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;na&quot;&gt;Sid&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;DenyNonTerraformerUsers&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Principal&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;*&quot;&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Condition&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
              &lt;span class=&quot;na&quot;&gt;StringEquals&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
                &lt;span class=&quot;s&quot;&gt;aws:PrincipalType: User&lt;/span&gt;
              &lt;span class=&quot;na&quot;&gt;StringNotLike&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
                &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;aws:PrincipalTag/Terraformer'&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;*'&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Effect&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;Deny&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Action&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;*'&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Resource&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;!GetAtt&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;TerraformStateBucket.Arn&quot;&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;!Sub&lt;/span&gt;
                &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;${Bucket}/*&quot;&lt;/span&gt;
                &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;na&quot;&gt;Bucket&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;!GetAtt&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;TerraformStateBucket.Arn&quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;The third statement begins differentiating between administrative and non-administrative access to the state. We do so again using the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Terraformer&lt;/code&gt; tag, this time inspecting its value.&lt;/p&gt;

&lt;div class=&quot;language-yaml highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;          &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;na&quot;&gt;Sid&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;RestrictTerraformNonAdmins&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Principal&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;*&quot;&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Condition&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
              &lt;span class=&quot;na&quot;&gt;StringEquals&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
                &lt;span class=&quot;s&quot;&gt;aws:PrincipalType: User&lt;/span&gt;
              &lt;span class=&quot;na&quot;&gt;StringLike&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
                &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;aws:PrincipalTag/Terraformer'&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;*'&lt;/span&gt;
              &lt;span class=&quot;na&quot;&gt;StringNotEquals&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
                &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;aws:PrincipalTag/Terraformer'&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;Admin'&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Effect&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;Deny&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;NotAction&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;s3:List*'&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;s3:Get*'&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;s3:Describe*'&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;s3:PutObject'&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;s3:DeleteObject'&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Resource&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;!GetAtt&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;TerraformStateBucket.Arn&quot;&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;!Sub&lt;/span&gt;
                &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;${Bucket}/*&quot;&lt;/span&gt;
                &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;na&quot;&gt;Bucket&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;!GetAtt&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;TerraformStateBucket.Arn&quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;If the IAM user has the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Terraformer&lt;/code&gt; tag, but its value is not &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Admin&lt;/code&gt;, we grant non-administrative access to that user. We use IAM’s &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;NotAction&lt;/code&gt; to whitelist the permitted actions.&lt;/p&gt;

&lt;p&gt;Notably, non-administrative access permits &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;s3:DeleteObject&lt;/code&gt; but &lt;em&gt;not&lt;/em&gt; &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;s3:DeleteObjectVersion&lt;/code&gt;. Since our state bucket is versioned (see &lt;a href=&quot;/tf-skeleton/2021/02/17/part-5-cfn-terraform-state.html#define-operational-infrastructure-with-cloudformation&quot;&gt;Part 5&lt;/a&gt;), granting &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;s3:DeleteObject&lt;/code&gt; is not inherently dangerous because all it does is add a delete marker to the object; you can always restore the version before the delete marker. Granting developers the ability to add delete markers aids state migration, so we do so here.&lt;/p&gt;

&lt;p&gt;The fourth statement denies access to all IAM roles other than our backend role:&lt;/p&gt;

&lt;div class=&quot;language-yaml highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;          &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;na&quot;&gt;Sid&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;DenyNonBackendRoles&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Principal&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;*&quot;&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Condition&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
              &lt;span class=&quot;na&quot;&gt;StringEquals&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
                &lt;span class=&quot;s&quot;&gt;aws:PrincipalType: AssumedRole&lt;/span&gt;
              &lt;span class=&quot;na&quot;&gt;StringNotLike&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
                &lt;span class=&quot;s&quot;&gt;aws:userId:&lt;/span&gt;
                  &lt;span class=&quot;s&quot;&gt;- !Sub&lt;/span&gt;
                    &lt;span class=&quot;s&quot;&gt;- &quot;${TerraformBackendRoleId}:*&quot;&lt;/span&gt;
                    &lt;span class=&quot;s&quot;&gt;- TerraformBackendRoleId&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;!GetAtt&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;TerraformBackendRole.RoleId&quot;&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Effect&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;Deny&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Action&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;*'&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Resource&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;!GetAtt&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;TerraformStateBucket.Arn&quot;&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;!Sub&lt;/span&gt;
                &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;${Bucket}/*&quot;&lt;/span&gt;
                &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;na&quot;&gt;Bucket&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;!GetAtt&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;TerraformStateBucket.Arn&quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;I did not find the syntax intuitive for restricting access to a specific IAM role. Specifying the role ARN in the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Principal&lt;/code&gt; property does not work. &lt;a href=&quot;https://aws.amazon.com/blogs/security/how-to-restrict-amazon-s3-bucket-access-to-a-specific-iam-role/&quot;&gt;This AWS post explains why&lt;/a&gt; and demonstrates using the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;StringNotLike&lt;/code&gt; and &lt;a href=&quot;https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_variables.html#principaltable&quot;&gt;aws:userId&lt;/a&gt; combination I use here.&lt;/p&gt;

&lt;p&gt;The fifth statement grants the backend role access:&lt;/p&gt;

&lt;div class=&quot;language-yaml highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;          &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;na&quot;&gt;Sid&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;ResrictBackendRoleToReadWrite&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Principal&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;*&quot;&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Condition&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
              &lt;span class=&quot;na&quot;&gt;StringEquals&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
                &lt;span class=&quot;s&quot;&gt;aws:PrincipalType: AssumedRole&lt;/span&gt;
              &lt;span class=&quot;na&quot;&gt;StringLike&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
                &lt;span class=&quot;s&quot;&gt;aws:userId:&lt;/span&gt;
                  &lt;span class=&quot;s&quot;&gt;- !Sub&lt;/span&gt;
                    &lt;span class=&quot;s&quot;&gt;- &quot;${TerraformBackendRoleId}:*&quot;&lt;/span&gt;
                    &lt;span class=&quot;s&quot;&gt;- TerraformBackendRoleId&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;!GetAtt&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;TerraformBackendRole.RoleId&quot;&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Effect&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;Deny&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;NotAction&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;s3:ListBucket'&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;s3:GetBucketVersioning'&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;s3:GetObject'&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;s3:PutObject'&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Resource&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;!GetAtt&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;TerraformStateBucket.Arn&quot;&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;!Sub&lt;/span&gt;
                &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;${Bucket}/*&quot;&lt;/span&gt;
                &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;na&quot;&gt;Bucket&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;!GetAtt&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;TerraformStateBucket.Arn&quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;And our final statement denies access to any other principal types (e.g., FederatedUsers), as we’re not considering those as part of this skeleton.&lt;/p&gt;

&lt;div class=&quot;language-yaml highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;          &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;na&quot;&gt;Sid&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;DenyAllOtherPrincipals&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Principal&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;*&quot;&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Condition&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
              &lt;span class=&quot;na&quot;&gt;StringNotEquals&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
                &lt;span class=&quot;s&quot;&gt;aws:PrincipalType:&lt;/span&gt;
                  &lt;span class=&quot;s&quot;&gt;- AssumedRole&lt;/span&gt;
                  &lt;span class=&quot;s&quot;&gt;- Account&lt;/span&gt;
                  &lt;span class=&quot;s&quot;&gt;- User&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Effect&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;Deny&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Action&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;*'&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Resource&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;!GetAtt&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;TerraformStateBucket.Arn&quot;&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;!Sub&lt;/span&gt;
                &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;${Bucket}/*&quot;&lt;/span&gt;
                &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;na&quot;&gt;Bucket&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;!GetAtt&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;TerraformStateBucket.Arn&quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;A summary of all changes is available by &lt;a href=&quot;https://github.com/thirstydeveloper/terraform-terragrunt-skeleton/compare/release/1.4...release/1.5&quot;&gt;viewing the diffs from part 5&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;There are three last items to note about this policy.&lt;/p&gt;

&lt;p&gt;First, the bucket policy does not contain any permissions for users who have the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Terraformer&lt;/code&gt; tag set to &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Admin&lt;/code&gt;. The lack of permissions means such users will have whatever access the IAM policy attached to their IAM user grants, presumably full access to S3.&lt;/p&gt;

&lt;p&gt;Second, the bucket policy does not explicitly grant the AWS account’s root user access. &lt;a href=&quot;https://aws.amazon.com/premiumsupport/knowledge-center/s3-access-denied-bucket-policy/&quot;&gt;AWS always allows the root user access&lt;/a&gt; to remove or modify bucket policies of buckets owned by that root user’s account, making it unnecessary to specify here.&lt;/p&gt;

&lt;p&gt;Finally, none of the permissions in the policy grant either adding or modifying the bucket policy, which means that aside from the root user, only &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Admin&lt;/code&gt; Terraformers can do so, assuming those admins have the requisite permissions on their IAM user).&lt;/p&gt;

&lt;h1 id=&quot;applying-the-state-bucket-policy&quot;&gt;Applying the State Bucket Policy&lt;/h1&gt;

&lt;p&gt;We’re now ready to deploy the bucket policy.&lt;/p&gt;

&lt;p&gt;First, change the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Terraformer&lt;/code&gt; tag on your IAM user to have a value of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Admin&lt;/code&gt;.&lt;/p&gt;

&lt;div class=&quot;language-bash highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;aws iam tag-user &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
  &lt;span class=&quot;nt&quot;&gt;--user-name&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;${&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;IAM_USER&lt;/span&gt;&lt;span class=&quot;k&quot;&gt;}&lt;/span&gt; &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
  &lt;span class=&quot;nt&quot;&gt;--tags&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'{
    &quot;Key&quot;: &quot;Terraformer&quot;,
    &quot;Value&quot;: &quot;Admin&quot;
  }'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Second, if Terragrunt created the state bucket for you, it may already have a bucket policy attached to it. Delete it using the following CLI command, replacing the bucket name as appropriate:&lt;/p&gt;

&lt;div class=&quot;language-bash highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;aws s3api delete-bucket-policy &lt;span class=&quot;nt&quot;&gt;--bucket&lt;/span&gt; terraform-skeleton-state
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Deploy the updated CloudFormation template using the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;init-admin&lt;/code&gt; Makefile target we added in &lt;a href=&quot;/tf-skeleton/2021/02/10/part-4-backend-role.html&quot;&gt;Part 4&lt;/a&gt;.&lt;sup id=&quot;fnref:1&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:1&quot; class=&quot;footnote&quot;&gt;1&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

&lt;div class=&quot;language-bash highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;make init-admin
aws cloudformation deploy &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nt&quot;&gt;--template-file&lt;/span&gt; init/admin/init-admin-account.cf.yml &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nt&quot;&gt;--stack-name&lt;/span&gt; tf-admin-init &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nt&quot;&gt;--capabilities&lt;/span&gt; CAPABILITY_NAMED_IAM &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nt&quot;&gt;--parameter-overrides&lt;/span&gt; &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
			&lt;span class=&quot;nv&quot;&gt;AdminAccountId&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&amp;lt;omitted&amp;gt; &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
			&lt;span class=&quot;nv&quot;&gt;StateBucketName&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;terraform-skeleton-state &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
			&lt;span class=&quot;nv&quot;&gt;StateLogBucketName&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;terraform-skeleton-state-logs &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
			&lt;span class=&quot;nv&quot;&gt;LockTableName&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;terraform-skeleton-state-locks

Waiting &lt;span class=&quot;k&quot;&gt;for &lt;/span&gt;changeset to be created..
Waiting &lt;span class=&quot;k&quot;&gt;for &lt;/span&gt;stack create/update to &lt;span class=&quot;nb&quot;&gt;complete
&lt;/span&gt;Successfully created/updated stack - tf-admin-init
aws cloudformation update-termination-protection &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nt&quot;&gt;--stack-name&lt;/span&gt; tf-admin-init &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nt&quot;&gt;--enable-termination-protection&lt;/span&gt;
&lt;span class=&quot;o&quot;&gt;{&lt;/span&gt;
    &lt;span class=&quot;s2&quot;&gt;&quot;StackId&quot;&lt;/span&gt;: &lt;span class=&quot;s2&quot;&gt;&quot;arn:aws:cloudformation:us-east-1:&amp;lt;omitted&amp;gt;:stack/tf-admin-init/8704b070-5f61-11eb-9ff1-0eea077046db&quot;&lt;/span&gt;
&lt;span class=&quot;o&quot;&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Let’s see if it works as expected.&lt;/p&gt;

&lt;h1 id=&quot;testing-the-state-bucket-policy&quot;&gt;Testing the State Bucket Policy&lt;/h1&gt;

&lt;p&gt;First, we can quickly verify the backend role has the access it requires by running &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;terragrunt apply-all&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Verifying user-level access requires manipulating the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Terraformer&lt;/code&gt; tag. Ideally, we’d have these tests automated and run as part of a CI pipeline. Perhaps we’ll get to that in another post.&lt;/p&gt;

&lt;p&gt;Remove the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Terraformer&lt;/code&gt; tag from your user altogether:&lt;/p&gt;

&lt;div class=&quot;language-bash highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;aws iam untag-user &lt;span class=&quot;nt&quot;&gt;--user-name&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;${&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;IAM_USER&lt;/span&gt;&lt;span class=&quot;k&quot;&gt;}&lt;/span&gt; &lt;span class=&quot;nt&quot;&gt;--tag-keys&lt;/span&gt; Terraformer
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;and verify you can’t even list the state bucket now:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;aws s3 ls s3://terraform-skeleton-state

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Next, set the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Terraformer&lt;/code&gt; tag on your IAM user to something other than &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Admin&lt;/code&gt;. For instance:&lt;/p&gt;

&lt;div class=&quot;language-bash highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;aws iam tag-user &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
  &lt;span class=&quot;nt&quot;&gt;--user-name&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;${&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;IAM_USER&lt;/span&gt;&lt;span class=&quot;k&quot;&gt;}&lt;/span&gt; &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
  &lt;span class=&quot;nt&quot;&gt;--tags&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'{
    &quot;Key&quot;: &quot;Terraformer&quot;,
    &quot;Value&quot;: &quot;User&quot;
  }'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Then verify you cannot update the state bucket’s policy:&lt;/p&gt;

&lt;div class=&quot;language-bash highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;aws s3api put-bucket-policy &lt;span class=&quot;nt&quot;&gt;--bucket&lt;/span&gt; terraform-skeleton-state &lt;span class=&quot;nt&quot;&gt;--policy&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;&quot;&lt;/span&gt;

An error occurred &lt;span class=&quot;o&quot;&gt;(&lt;/span&gt;AccessDenied&lt;span class=&quot;o&quot;&gt;)&lt;/span&gt; when calling the PutBucketPolicy operation: Access Denied
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;or delete it:&lt;/p&gt;

&lt;div class=&quot;language-bash highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;aws s3api delete-bucket-policy &lt;span class=&quot;nt&quot;&gt;--bucket&lt;/span&gt; terraform-skeleton-state

An error occurred &lt;span class=&quot;o&quot;&gt;(&lt;/span&gt;AccessDenied&lt;span class=&quot;o&quot;&gt;)&lt;/span&gt; when calling the DeleteBucketPolicy operation: Access Denied
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Also, verify you &lt;em&gt;can&lt;/em&gt; delete a state object:&lt;/p&gt;

&lt;div class=&quot;language-bash highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;aws s3 &lt;span class=&quot;nb&quot;&gt;rm &lt;/span&gt;s3://terraform-skeleton-state/app/dev/test-stack.tfstate
delete: s3://terraform-skeleton-state/app/dev/test-stack.tfstate
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;but can’t delete an object version:&lt;/p&gt;

&lt;div class=&quot;language-bash highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;nv&quot;&gt;DELETE_MARKER_VERSION&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;si&quot;&gt;$(&lt;/span&gt;aws s3api list-object-versions &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
  &lt;span class=&quot;nt&quot;&gt;--bucket&lt;/span&gt; terraform-skeleton-state &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
  &lt;span class=&quot;nt&quot;&gt;--prefix&lt;/span&gt; app/dev/test-stack.tfstate &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
  &lt;span class=&quot;nt&quot;&gt;--query&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'DeleteMarkers[?IsLatest==`true`].VersionId'&lt;/span&gt; | jq &lt;span class=&quot;nt&quot;&gt;-r&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'.[0]'&lt;/span&gt; &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
&lt;span class=&quot;si&quot;&gt;)&lt;/span&gt;

aws s3api delete-object &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
  &lt;span class=&quot;nt&quot;&gt;--bucket&lt;/span&gt; terraform-skeleton-state &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
  &lt;span class=&quot;nt&quot;&gt;--key&lt;/span&gt; app/dev/test-stack.tfstate &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
  &lt;span class=&quot;nt&quot;&gt;--version-id&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;${&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;DELETE_MARKER_VERSION&lt;/span&gt;&lt;span class=&quot;k&quot;&gt;}&lt;/span&gt;

An error occurred &lt;span class=&quot;o&quot;&gt;(&lt;/span&gt;AccessDenied&lt;span class=&quot;o&quot;&gt;)&lt;/span&gt; when calling the DeleteObject operation: Access Denied
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;The inability to remove the delete marker does introduce a hurdle users will have to clear to restore state files, but restoration is still possible. See the footnotes for more information.&lt;sup id=&quot;fnref:2&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:2&quot; class=&quot;footnote&quot;&gt;2&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

&lt;p&gt;Change the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Terraformer&lt;/code&gt; tag to &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Admin&lt;/code&gt; again:&lt;/p&gt;

&lt;div class=&quot;language-bash highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;aws iam tag-user &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
  &lt;span class=&quot;nt&quot;&gt;--user-name&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;${&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;IAM_USER&lt;/span&gt;&lt;span class=&quot;k&quot;&gt;}&lt;/span&gt; &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
  &lt;span class=&quot;nt&quot;&gt;--tags&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'{
    &quot;Key&quot;: &quot;Terraformer&quot;,
    &quot;Value&quot;: &quot;Admin&quot;
  }'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;and delete the delete marker again, verifying it works this time.&lt;/p&gt;

&lt;div class=&quot;language-bash highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;aws s3api delete-object &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
  &lt;span class=&quot;nt&quot;&gt;--bucket&lt;/span&gt; terraform-skeleton-state &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
  &lt;span class=&quot;nt&quot;&gt;--key&lt;/span&gt; app/dev/test-stack.tfstate &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
  &lt;span class=&quot;nt&quot;&gt;--version-id&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;${&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;DELETE_MARKER_VERSION&lt;/span&gt;&lt;span class=&quot;k&quot;&gt;}&lt;/span&gt;

&lt;span class=&quot;o&quot;&gt;{&lt;/span&gt;
    &lt;span class=&quot;s2&quot;&gt;&quot;DeleteMarker&quot;&lt;/span&gt;: &lt;span class=&quot;nb&quot;&gt;true&lt;/span&gt;,
    &lt;span class=&quot;s2&quot;&gt;&quot;VersionId&quot;&lt;/span&gt;: &lt;span class=&quot;s2&quot;&gt;&quot;...&quot;&lt;/span&gt;
&lt;span class=&quot;o&quot;&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;That about covers it. We now have a bucket policy restricting access to Terraform’s state to only authorized principals.&lt;/p&gt;

&lt;h1 id=&quot;cleanup&quot;&gt;Cleanup&lt;/h1&gt;

&lt;p&gt;With the bucket policy in place, let’s turn our attention to a couple of other hardening items we can tackle in the CloudFormation template: protecting the logs bucket and removing unnecessary permissions from the backend role.&lt;/p&gt;

&lt;h2 id=&quot;protecting-the-logs-bucket&quot;&gt;Protecting the Logs Bucket&lt;/h2&gt;

&lt;p&gt;As discussed in &lt;a href=&quot;/tf-skeleton/2021/01/28/part-3-aws-backend.html#limitations&quot;&gt;Part 3&lt;/a&gt;, when Terragrunt creates the log bucket for you, it does not enable encryption or explicitly block public access. We can rectify both of those issues now that the bucket is under CloudFormation’s control.&lt;/p&gt;

&lt;p&gt;Using our &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;TerraformStateBucket&lt;/code&gt; resource as a template, add &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;BucketEncryption&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;PublicAccessBlockConfiguration&lt;/code&gt; properties to the log bucket:&lt;/p&gt;

&lt;div class=&quot;language-yaml highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;  &lt;span class=&quot;na&quot;&gt;TerraformStateLogBucket&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
    &lt;span class=&quot;na&quot;&gt;Type&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;AWS::S3::Bucket'&lt;/span&gt;
    &lt;span class=&quot;na&quot;&gt;DeletionPolicy&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;Retain&lt;/span&gt;
    &lt;span class=&quot;na&quot;&gt;UpdateReplacePolicy&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;Retain&lt;/span&gt;
    &lt;span class=&quot;na&quot;&gt;Properties&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
      &lt;span class=&quot;na&quot;&gt;BucketName&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;!Ref&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;StateLogBucketName&lt;/span&gt;
      &lt;span class=&quot;na&quot;&gt;AccessControl&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;LogDeliveryWrite&lt;/span&gt;
      &lt;span class=&quot;na&quot;&gt;BucketEncryption&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
        &lt;span class=&quot;na&quot;&gt;ServerSideEncryptionConfiguration&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
          &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;na&quot;&gt;ServerSideEncryptionByDefault&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
              &lt;span class=&quot;na&quot;&gt;SSEAlgorithm&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;aws:kms&lt;/span&gt;
      &lt;span class=&quot;na&quot;&gt;PublicAccessBlockConfiguration&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
        &lt;span class=&quot;na&quot;&gt;BlockPublicAcls&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;True&lt;/span&gt;
        &lt;span class=&quot;na&quot;&gt;BlockPublicPolicy&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;True&lt;/span&gt;
        &lt;span class=&quot;na&quot;&gt;IgnorePublicAcls&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;True&lt;/span&gt;
        &lt;span class=&quot;na&quot;&gt;RestrictPublicBuckets&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;True&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Run &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;make init-admin&lt;/code&gt; to deploy.&lt;/p&gt;

&lt;h2 id=&quot;backend-role-permissions&quot;&gt;Backend Role Permissions&lt;/h2&gt;

&lt;p&gt;When we created the backend role in &lt;a href=&quot;/tf-skeleton/2021/02/10/part-4-backend-role.html#the-backend-role&quot;&gt;Part 4&lt;/a&gt;, we granted it permissions to create S3 buckets and DynamoDB tables because Terragrunt managed our state bucket and lock table. We can remove those permissions now that CloudFormation deploys both:&lt;/p&gt;

&lt;div class=&quot;language-diff highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;gh&quot;&gt;diff --git a/init/admin/init-admin-account.cf.yml b/init/admin/init-admin-account.cf.yml
index b8dcda0..3aed8e5 100644
&lt;/span&gt;&lt;span class=&quot;gd&quot;&gt;--- a/init/admin/init-admin-account.cf.yml
&lt;/span&gt;&lt;span class=&quot;gi&quot;&gt;+++ b/init/admin/init-admin-account.cf.yml
&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;@@ -122,27 +122,6 @@&lt;/span&gt; Resources:
               - 'dynamodb:PutItem'
               - 'dynamodb:DeleteItem'
             Resource: !Sub &quot;arn:aws:dynamodb:${AWS::Region}:${AWS::AccountId}:table/${LockTableName}&quot;
&lt;span class=&quot;gd&quot;&gt;-          - Sid: AllowStateBucketCreation
-            Effect: Allow
-            Action:
-              - 's3:GetBucketAcl'
-              - 's3:GetBucketLogging'
-              - 's3:CreateBucket'
-              - 's3:PutBucketPublicAccessBlock'
-              - 's3:PutBucketTagging'
-              - 's3:PutBucketPolicy'
-              - 's3:PutBucketVersioning'
-              - 's3:PutEncryptionConfiguration'
-              - 's3:PutBucketAcl'
-              - 's3:PutBucketLogging'
-            Resource:
-              - !Sub &quot;arn:aws:s3:::${StateBucketName}&quot;
-              - !Sub &quot;arn:aws:s3:::${StateLogBucketName}&quot;
-          - Sid: AllowLockTableCreation
-            Effect: Allow
-            Action:
-              - 'dynamodb:CreateTable'
-            Resource: !Sub &quot;arn:aws:dynamodb:${AWS::Region}:${AWS::AccountId}:table/${LockTableName}&quot;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Run a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;terragrunt apply-all&lt;/code&gt; to verify all’s well.&lt;/p&gt;

&lt;h1 id=&quot;whats-next&quot;&gt;What’s Next?&lt;/h1&gt;

&lt;p&gt;In this post, we’ve significantly enhanced the protections surrounding the Terraform state. Looking back at these first six entries, we’ve covered a lot of ground, and I think it will be worthwhile to take a step back and summarize what we’ve done so far. After a recap, I’d like to start covering continuous integration with Terraform and Terragrunt. We’ll see where that takes us.&lt;/p&gt;

&lt;h1 id=&quot;footnotes&quot;&gt;Footnotes&lt;/h1&gt;

&lt;div class=&quot;footnotes&quot; role=&quot;doc-endnotes&quot;&gt;
  &lt;ol&gt;
    &lt;li id=&quot;fn:1&quot; role=&quot;doc-endnote&quot;&gt;

      &lt;p&gt;I hope that you don’t encounter any errors deploying the bucket policy. If you do, AWS is often not much help diagnosing what went wrong. Here are some pointers.&lt;/p&gt;

      &lt;p&gt;If &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;make init-admin&lt;/code&gt; fails, it will likely say:&lt;/p&gt;

      &lt;div class=&quot;language-bash highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;Failed to create/update the stack. Run the following &lt;span class=&quot;nb&quot;&gt;command &lt;/span&gt;to fetch the
list of events leading up to the failure
aws cloudformation describe-stack-events &lt;span class=&quot;nt&quot;&gt;--stack-name&lt;/span&gt; tf-admin-init
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;      &lt;/div&gt;

      &lt;p&gt;If you do so, you’ll get a JSON dump of events. Since you presumably failed when creating the bucket policy, look for an event with a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;ResourceStatus&lt;/code&gt; field set to &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;CREATE_FAILED&lt;/code&gt;. Here’s an abbreviated example:&lt;/p&gt;

      &lt;div class=&quot;language-json highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;w&quot;&gt;  
    &lt;/span&gt;&lt;span class=&quot;nl&quot;&gt;&quot;StackName&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;w&quot;&gt; &lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;tf-admin-init&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;w&quot;&gt;
    &lt;/span&gt;&lt;span class=&quot;nl&quot;&gt;&quot;ResourceStatus&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;w&quot;&gt; &lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;CREATE_FAILED&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;w&quot;&gt;
    &lt;/span&gt;&lt;span class=&quot;nl&quot;&gt;&quot;ResourceStatusReason&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;w&quot;&gt; &lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;Invalid policy syntax. (Service: Amazon S3; Status Code: 400; Error Code: MalformedPolicy; Request ID: 9EB1BD50BCF1FCB7; S3 Extended Request ID: hHkqY/snGYyhc4paSxhBT1IzpmgoWjKvz5I/JYiYUKu3PLSn1CWuAceLU7QEckf/omDhF4ZdeGU=; Proxy: null)&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;,&lt;/span&gt;&lt;span class=&quot;w&quot;&gt;
    &lt;/span&gt;&lt;span class=&quot;nl&quot;&gt;&quot;ResourceProperties&quot;&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;w&quot;&gt; &lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;{&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;Bucket&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;terraform-skeleton-state&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;...&quot;&lt;/span&gt;&lt;span class=&quot;w&quot;&gt;
&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;w&quot;&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;      &lt;/div&gt;

      &lt;p&gt;If you have an &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Invalid policy syntax&lt;/code&gt; error, I recommend removing statements until the policy works, then adding them back in one-by-one until you find the problem. &lt;a href=&quot;#fnref:1&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:2&quot; role=&quot;doc-endnote&quot;&gt;

      &lt;p&gt;Since users cannot delete object versions, they cannot restore a state file by deleting its delete marker since it is itself an object version.&lt;/p&gt;

      &lt;p&gt;As discussed in the AWS docs &lt;a href=&quot;https://aws.amazon.com/premiumsupport/knowledge-center/s3-undelete-configuration/&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://docs.aws.amazon.com/AmazonS3/latest/userguide/RestoringPreviousVersions.html&quot;&gt;here&lt;/a&gt;, users can still restore state files by copying a previous version to become the latest. Here’s an example of how:&lt;/p&gt;

      &lt;p&gt;Step 1: Find version to restore (e.g., using the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;LastModified&lt;/code&gt; field to limit the search area).&lt;/p&gt;

      &lt;div class=&quot;language-bash highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;aws s3api list-object-versions &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
  &lt;span class=&quot;nt&quot;&gt;--bucket&lt;/span&gt; terraform-skeleton-state &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
  &lt;span class=&quot;nt&quot;&gt;--key&lt;/span&gt; app/dev/test-stack.tfstate &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
  &lt;span class=&quot;nt&quot;&gt;--query&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'Versions[?contains(LastModified, `'&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;2021-02-20&quot;&lt;/span&gt;&lt;span class=&quot;s1&quot;&gt;'`)]'&lt;/span&gt; &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
  | jq &lt;span class=&quot;s1&quot;&gt;'.[] | { Key, VersionId, LastModified }'&lt;/span&gt;

&lt;span class=&quot;o&quot;&gt;{&lt;/span&gt;
  &lt;span class=&quot;s2&quot;&gt;&quot;Key&quot;&lt;/span&gt;: &lt;span class=&quot;s2&quot;&gt;&quot;app/stage/test-stack.tfstate&quot;&lt;/span&gt;,
  &lt;span class=&quot;s2&quot;&gt;&quot;VersionId&quot;&lt;/span&gt;: &lt;span class=&quot;s2&quot;&gt;&quot;lwMes1R1AfkEZ.lQ4U9d217yeU7rWbcj&quot;&lt;/span&gt;,
  &lt;span class=&quot;s2&quot;&gt;&quot;LastModified&quot;&lt;/span&gt;: &lt;span class=&quot;s2&quot;&gt;&quot;2021-02-20T13:03:50+00:00&quot;&lt;/span&gt;
&lt;span class=&quot;o&quot;&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;      &lt;/div&gt;

      &lt;p&gt;Step 2: Copy the desired object version to make it the new latest version, replacing &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;${VERSION_ID}&lt;/code&gt; as appropriate.&lt;/p&gt;

      &lt;div class=&quot;language-bash highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;nv&quot;&gt;BUCKET&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;terraform-skeleton-state
&lt;span class=&quot;nv&quot;&gt;PREFIX&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;app/dev/test-stack.tfstate
&lt;span class=&quot;nv&quot;&gt;VERSION&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;TgfUNdVuoZKwSOHF1QeGO_nB8iZGzE3f

aws s3api copy-object &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
  &lt;span class=&quot;nt&quot;&gt;--copy-source&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;k&quot;&gt;${&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;BUCKET&lt;/span&gt;&lt;span class=&quot;k&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;/&lt;/span&gt;&lt;span class=&quot;k&quot;&gt;${&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;PREFIX&lt;/span&gt;&lt;span class=&quot;k&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;?versionId=&lt;/span&gt;&lt;span class=&quot;k&quot;&gt;${&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;VERSION&lt;/span&gt;&lt;span class=&quot;k&quot;&gt;}&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt; &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
  &lt;span class=&quot;nt&quot;&gt;--key&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;${&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;PREFIX&lt;/span&gt;&lt;span class=&quot;k&quot;&gt;}&lt;/span&gt; &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
  &lt;span class=&quot;nt&quot;&gt;--bucket&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;${&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;BUCKET&lt;/span&gt;&lt;span class=&quot;k&quot;&gt;}&lt;/span&gt;

&lt;span class=&quot;o&quot;&gt;{&lt;/span&gt;  
    ...
    &lt;span class=&quot;s2&quot;&gt;&quot;CopyObjectResult&quot;&lt;/span&gt;: &lt;span class=&quot;o&quot;&gt;{&lt;/span&gt;
        &lt;span class=&quot;s2&quot;&gt;&quot;ETag&quot;&lt;/span&gt;: &lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;...&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;,
        &lt;span class=&quot;s2&quot;&gt;&quot;LastModified&quot;&lt;/span&gt;: &lt;span class=&quot;s2&quot;&gt;&quot;2021-02-20T13:36:47+00:00&quot;&lt;/span&gt;
    &lt;span class=&quot;o&quot;&gt;}&lt;/span&gt;
&lt;span class=&quot;o&quot;&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;      &lt;/div&gt;
      &lt;p&gt;&lt;a href=&quot;#fnref:2&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
  &lt;/ol&gt;
&lt;/div&gt;</content><author><name>Chris</name></author><category term="tf-skeleton" /><category term="terraform" /><category term="terragrunt" /><summary type="html">Part 5 moved Terraform’s operational infrastructure into a CloudFormation stack so that Terragrunt no longer manages the state bucket, lock table, and log bucket. Doing so offers us the opportunity to protect these resources in ways not supported by Terragrunt. We will capitalize on that opportunity today.</summary></entry><entry><title type="html">Terraform Skeleton Part 5: Remote State with CloudFormation</title><link href="https://thirstydeveloper.io/tf-skeleton/2021/02/17/part-5-cfn-terraform-state.html" rel="alternate" type="text/html" title="Terraform Skeleton Part 5: Remote State with CloudFormation" /><published>2021-02-17T12:20:00+00:00</published><updated>2021-02-17T12:20:00+00:00</updated><id>https://thirstydeveloper.io/tf-skeleton/2021/02/17/part-5-cfn-terraform-state</id><content type="html" xml:base="https://thirstydeveloper.io/tf-skeleton/2021/02/17/part-5-cfn-terraform-state.html">&lt;p&gt;&lt;a href=&quot;/tf-skeleton/2021/01/28/part-3-aws-backend.html&quot;&gt;Part 3&lt;/a&gt; showed us Terraform requires some infrastructure itself to store remote state and discussed some &lt;a href=&quot;/tf-skeleton/2021/01/28/part-3-aws-backend.html#limitations&quot;&gt;limitations&lt;/a&gt; of using Terragrunt to manage the creation of that infrastructure. In &lt;a href=&quot;/tf-skeleton/2021/02/10/part-4-backend-role.html&quot;&gt;part 4&lt;/a&gt;, we introduced more operational infrastructure for Terraform and &lt;a href=&quot;/tf-skeleton/2021/02/10/part-4-backend-role.html#admin-account-cloudformation-template&quot;&gt;began managing that infrastructure with CloudFormation&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Today, we’ll continue down the path of using CloudFormation to store operational Terraform infrastructure. Achieving a clean separation between the infrastructure Terraform needs to run and the infrastructure Terraform manages opens up additional possibilities for locking down the Terraform state.&lt;/p&gt;

&lt;p&gt;We’ll make use of &lt;a href=&quot;https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/resource-import.html&quot;&gt;CloudFormation’s import feature&lt;/a&gt; to bring the state bucket and lock table Terragrunt created for us under CloudFormation’s control for a seamless transition of ownership.&lt;/p&gt;

&lt;h1 id=&quot;goals&quot;&gt;Goals&lt;/h1&gt;

&lt;ol&gt;
  &lt;li&gt;Control all Terraform operational infrastructure with CloudFormation&lt;/li&gt;
  &lt;li&gt;Import existing operational infrastructure (buckets, tables) into CloudFormation&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;If you prefer to jump to the end, the code implementing this post’s final result is available on branch &lt;a href=&quot;https://github.com/thirstydeveloper/terraform-terragrunt-skeleton/tree/release/1.4&quot;&gt;release/1.4&lt;/a&gt; on GitHub. Additionally, you can &lt;a href=&quot;https://github.com/thirstydeveloper/terraform-terragrunt-skeleton/compare/release/1.3...release/1.4&quot;&gt;view the diffs from part 4&lt;/a&gt;, if that’s more your speed.&lt;/p&gt;

&lt;h1 id=&quot;define-operational-infrastructure-with-cloudformation&quot;&gt;Define Operational Infrastructure with CloudFormation&lt;/h1&gt;

&lt;p&gt;There are three items we need to import into CloudFormation:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;The state S3 bucket&lt;/li&gt;
  &lt;li&gt;The log S3 bucket&lt;/li&gt;
  &lt;li&gt;The DynamoDB lock table&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The first step is to define these resources in a CloudFormation template. We’ll add resource definitions to the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;init-admin-account.cf.yml&lt;/code&gt; template &lt;a href=&quot;/tf-skeleton/2021/02/10/part-4-backend-role.html#admin-account-cloudformation-template&quot;&gt;we created in part 4&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;If you prefer, you can &lt;a href=&quot;https://github.com/thirstydeveloper/terraform-terragrunt-skeleton/compare/release/1.3...release/1.4#diff-cc93827adede14cb1e69dde0ed0769a47014bbb00f7ec444c61b927f3901c35a&quot;&gt;view the diffs&lt;/a&gt; on the CloudFormation template from part 4.&lt;/p&gt;

&lt;p&gt;First up, the state bucket. Add an &lt;a href=&quot;https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket.html&quot;&gt;AWS::S3::Bucket&lt;/a&gt; resource to the CloudFormation template for our state bucket under the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Resources&lt;/code&gt; block:&lt;/p&gt;

&lt;div class=&quot;language-yaml highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;na&quot;&gt;TerraformStateBucket&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;Type&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;AWS::S3::Bucket'&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;DeletionPolicy&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;Retain&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;UpdateReplacePolicy&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;Retain&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;Properties&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
    &lt;span class=&quot;na&quot;&gt;BucketName&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;!Ref&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;StateBucketName&lt;/span&gt;
    &lt;span class=&quot;na&quot;&gt;BucketEncryption&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
      &lt;span class=&quot;na&quot;&gt;ServerSideEncryptionConfiguration&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
        &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;na&quot;&gt;ServerSideEncryptionByDefault&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;SSEAlgorithm&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;aws:kms&lt;/span&gt;
    &lt;span class=&quot;na&quot;&gt;LoggingConfiguration&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
      &lt;span class=&quot;na&quot;&gt;DestinationBucketName&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;!Ref&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;StateLogBucketName&lt;/span&gt;
      &lt;span class=&quot;na&quot;&gt;LogFilePrefix&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;TFStateLogs/&lt;/span&gt;
    &lt;span class=&quot;na&quot;&gt;PublicAccessBlockConfiguration&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
      &lt;span class=&quot;na&quot;&gt;BlockPublicAcls&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;True&lt;/span&gt;
      &lt;span class=&quot;na&quot;&gt;BlockPublicPolicy&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;True&lt;/span&gt;
      &lt;span class=&quot;na&quot;&gt;IgnorePublicAcls&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;True&lt;/span&gt;
      &lt;span class=&quot;na&quot;&gt;RestrictPublicBuckets&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;True&lt;/span&gt;
    &lt;span class=&quot;na&quot;&gt;VersioningConfiguration&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
      &lt;span class=&quot;na&quot;&gt;Status&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;Enabled&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;The properties above match what Terragrunt used &lt;a href=&quot;/tf-skeleton/2021/01/28/part-3-aws-backend.html#backend-configuration-with-terragrunt&quot;&gt;when it created the state bucket in part 3&lt;/a&gt;. Note that we’re also using the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;StateBucketName&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;StateLogBucketName&lt;/code&gt; &lt;a href=&quot;/tf-skeleton/2021/02/10/part-4-backend-role.html#admin-account-cloudformation-template&quot;&gt;parameters we added in part 4&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;CloudFormation &lt;a href=&quot;https://docs.amazonaws.cn/en_us/AWSCloudFormation/latest/UserGuide/resource-import-existing-stack.html&quot;&gt;requires the DeletionPolicy attribute&lt;/a&gt; for any resources it will import, and it is a good safety measure in general.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-updatereplacepolicy.html&quot;&gt;UpdateReplacePolicy&lt;/a&gt; is also set. Although not required, it is needed to make our &lt;a href=&quot;/tf-skeleton/2021/02/10/part-4-backend-role.html#preparation&quot;&gt;CloudFormation linter pre-commit hook&lt;/a&gt; happy.&lt;sup id=&quot;fnref:1&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:1&quot; class=&quot;footnote&quot;&gt;1&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

&lt;p&gt;Next, we’ll tackle the log bucket. Add the following to the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Resources&lt;/code&gt; block:&lt;/p&gt;

&lt;div class=&quot;language-yaml highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;na&quot;&gt;TerraformStateLogBucket&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;Type&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;AWS::S3::Bucket'&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;DeletionPolicy&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;Retain&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;UpdateReplacePolicy&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;Retain&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;Properties&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
    &lt;span class=&quot;na&quot;&gt;BucketName&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;!Ref&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;StateLogBucketName&lt;/span&gt;
    &lt;span class=&quot;na&quot;&gt;AccessControl&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;LogDeliveryWrite&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Finally, we have the lock table:&lt;/p&gt;

&lt;div class=&quot;language-yaml highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;na&quot;&gt;TerraformStateLockTable&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;Type&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;AWS::DynamoDB::Table'&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;DeletionPolicy&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;Retain&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;UpdateReplacePolicy&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;Retain&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;Properties&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
    &lt;span class=&quot;na&quot;&gt;TableName&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;!Ref&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;LockTableName&lt;/span&gt;
    &lt;span class=&quot;na&quot;&gt;AttributeDefinitions&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
      &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;na&quot;&gt;AttributeName&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;LockID&lt;/span&gt;
        &lt;span class=&quot;na&quot;&gt;AttributeType&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;S&lt;/span&gt;
    &lt;span class=&quot;na&quot;&gt;KeySchema&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
      &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;na&quot;&gt;AttributeName&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;LockID&lt;/span&gt;
        &lt;span class=&quot;na&quot;&gt;KeyType&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;HASH&lt;/span&gt;
    &lt;span class=&quot;na&quot;&gt;BillingMode&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;PAY_PER_REQUEST&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;If you’re starting fresh, and Terragrunt hasn’t already created the state bucket, log bucket, and lock table for you, you can skip the next section on importing and run the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;make init-admin&lt;/code&gt; target we created in &lt;a href=&quot;/tf-skeleton/2021/01/28/part-3-aws-backend.html&quot;&gt;part 3&lt;/a&gt; to deploy your stack. Otherwise, continue on to import the resources Terragrunt created for you.&lt;/p&gt;

&lt;h1 id=&quot;import-operational-infrastructure-into-cloudformation&quot;&gt;Import Operational Infrastructure into CloudFormation&lt;/h1&gt;

&lt;p&gt;While you can use the AWS management console to import the resources, I prefer to work from the command line, but the commands to do so aren’t straightforward. We’ll add some targets to our Makefile in an attempt to simplify.&lt;/p&gt;

&lt;p&gt;For a clean import, we need the following capabilities in our Makefile:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;Use &lt;a href=&quot;https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-stack-drift.html&quot;&gt;CloudFormation Drift Detection&lt;/a&gt; to ensure our stack is up-to-date and our resource definitions match what Terragrunt created&lt;/li&gt;
  &lt;li&gt;Create a &lt;a href=&quot;https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-changesets.html&quot;&gt;CloudFormation change set&lt;/a&gt; to import the resources and show us what CloudFormation intends to do&lt;/li&gt;
  &lt;li&gt;Execute the change set to import the resources&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;If you prefer to jump to the end, &lt;a href=&quot;https://github.com/thirstydeveloper/terraform-terragrunt-skeleton/blob/release/1.4/Makefile&quot;&gt;here’s the resulting Makefile&lt;/a&gt;, and here are &lt;a href=&quot;https://github.com/thirstydeveloper/terraform-terragrunt-skeleton/compare/release/1.3...release/1.4#diff-76ed074a9305c04054cdebb9e9aad2d818052b07091de1f20cad0bbac34ffb52&quot;&gt;the diffs from part 4&lt;/a&gt;.&lt;/p&gt;

&lt;h2 id=&quot;check-for-drift&quot;&gt;Check for Drift&lt;/h2&gt;

&lt;p&gt;The first step in the import process is to verify that the stack we will be importing into has no drift, i.e., it has no other unapplied changes. The CloudFormation API uses separate calls to &lt;a href=&quot;https://docs.aws.amazon.com/cli/latest/reference/cloudformation/detect-stack-drift.html&quot;&gt;start a drift detection job&lt;/a&gt; and &lt;a href=&quot;https://docs.aws.amazon.com/cli/latest/reference/cloudformation/describe-stack-drift-detection-status.html&quot;&gt;check the status of a drift detection job&lt;/a&gt;. We’ll add some helper functions to our Makefile to wrap these calls:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;define wait_cfn_drift_detect_job
	@while [[ \
		&quot;$$($(CFN_STATUS_DRIFT_DETECTION) $(1) | jq -r .DetectionStatus)&quot; == \
		&quot;DETECTION_IN_PROGRESS&quot; \
	]]; do \
		echo &quot;Detection in progress. Waiting 3 seconds...&quot;; \
		sleep 3; \
	done
endef

define show_cfn_drift
	$(eval DRIFT_ID=$(shell $(CFN_START_DRIFT_DETECTION) $(1) \
		| jq -r .StackDriftDetectionId))
	$(call wait_cfn_drift_detect_job,${DRIFT_ID})
	@$(CFN_STATUS_DRIFT_DETECTION) $(DRIFT_ID) | jq '{ \
		DetectionStatus, \
		StackDriftStatus, \
		DriftedStackResourceCount \
	}'
endef
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Next, add a make target to perform the drift detection:&lt;/p&gt;

&lt;div class=&quot;language-make highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;nl&quot;&gt;.PHONY&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;check-init-admin-drift&lt;/span&gt;
&lt;span class=&quot;nl&quot;&gt;check-init-admin-drift&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;:&lt;/span&gt;
	&lt;span class=&quot;nf&quot;&gt;$(&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;call&lt;/span&gt; show_cfn_drift,&lt;span class=&quot;nv&quot;&gt;${ADMIN_INIT_STACK_NAME}&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Execute drift detection with:&lt;/p&gt;

&lt;div class=&quot;language-bash highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;➜ make check-init-admin-drift
Detection &lt;span class=&quot;k&quot;&gt;in &lt;/span&gt;progress. Waiting 3 seconds...
&lt;span class=&quot;o&quot;&gt;{&lt;/span&gt;
  &lt;span class=&quot;s2&quot;&gt;&quot;DetectionStatus&quot;&lt;/span&gt;: &lt;span class=&quot;s2&quot;&gt;&quot;DETECTION_COMPLETE&quot;&lt;/span&gt;,
  &lt;span class=&quot;s2&quot;&gt;&quot;StackDriftStatus&quot;&lt;/span&gt;: &lt;span class=&quot;s2&quot;&gt;&quot;IN_SYNC&quot;&lt;/span&gt;,
  &lt;span class=&quot;s2&quot;&gt;&quot;DriftedStackResourceCount&quot;&lt;/span&gt;: 0
&lt;span class=&quot;o&quot;&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;If you get a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;StackDriftStatus&lt;/code&gt; other than &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;IN_SYNC&lt;/code&gt;, adjust your CloudFormation template to resolve, and verify using the drift check. Once you’re &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;IN_SYNC&lt;/code&gt;, create the import change set as follows.&lt;/p&gt;

&lt;h2 id=&quot;create-import-changeset&quot;&gt;Create Import Changeset&lt;/h2&gt;

&lt;p&gt;Creating the import change set requires a CloudFormation template and information about the resources to import, including:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;The resource type&lt;/li&gt;
  &lt;li&gt;The logical name of the resource in your template&lt;/li&gt;
  &lt;li&gt;The unique identifier for that resource in AWS&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;For instance, our CloudFormation template defines the state bucket with:&lt;/p&gt;

&lt;div class=&quot;language-yaml highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;na&quot;&gt;Resources&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;TerraformStateLogBucket&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
    &lt;span class=&quot;na&quot;&gt;Type&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;AWS::S3::Bucket'&lt;/span&gt;
    &lt;span class=&quot;s&quot;&gt;...&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;which means the resource type is &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;AWS::S3::Bucket&lt;/code&gt;, and the logical name is &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;TerraformStateLogBucket&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The unique identifier depends on the resource type. For an S3 bucket, it’s the bucket name. For a DynamoDB table, it’s the table name.&lt;/p&gt;

&lt;p&gt;Below is a make target for creating the import change set:&lt;/p&gt;

&lt;div class=&quot;language-make highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;nl&quot;&gt;import-terragrunt-changeset.json&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;:&lt;/span&gt;
	&lt;span class=&quot;p&quot;&gt;@&lt;/span&gt;aws cloudformation create-change-set &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nt&quot;&gt;--stack-name&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;${ADMIN_INIT_STACK_NAME}&lt;/span&gt; &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nt&quot;&gt;--change-set-name&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;${ADMIN_INIT_STACK_NAME}&lt;/span&gt;&lt;span class=&quot;nt&quot;&gt;-import-terragrunt&lt;/span&gt; &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nt&quot;&gt;--change-set-type&lt;/span&gt; IMPORT &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nt&quot;&gt;--template-body&lt;/span&gt; file://init/admin/init-admin-account.cf.yml &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nt&quot;&gt;--capabilities&lt;/span&gt; CAPABILITY_NAMED_IAM &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nt&quot;&gt;--parameters&lt;/span&gt; &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
			&lt;span class=&quot;nv&quot;&gt;ParameterKey&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;AdminAccountId,UsePreviousValue&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;True &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
			&lt;span class=&quot;nv&quot;&gt;ParameterKey&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;StateBucketName,UsePreviousValue&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;True &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
			&lt;span class=&quot;nv&quot;&gt;ParameterKey&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;StateLogBucketName,UsePreviousValue&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;True &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
			&lt;span class=&quot;nv&quot;&gt;ParameterKey&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;LockTableName,UsePreviousValue&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;True &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nt&quot;&gt;--resources-to-import&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;[ &lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;
			{ &lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;
				&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;ResourceType&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;AWS::S3::Bucket&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;, &lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;
				&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;LogicalResourceId&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;TerraformStateBucket&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;, &lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;
				&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;ResourceIdentifier&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;: { &lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;
					&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;BucketName&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;: &lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;${STATE_BUCKET_NAME}&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt; &lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;
				} &lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;
			}, &lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;
			{ &lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;
				&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;ResourceType&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;AWS::S3::Bucket&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;, &lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;
				&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;LogicalResourceId&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;TerraformStateLogBucket&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;, &lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;
				&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;ResourceIdentifier&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;: { &lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;
					&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;BucketName&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;: &lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;${STATE_LOG_BUCKET_NAME}&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt; &lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;
				} &lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;
			}, &lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;
			{ &lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;
			&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;ResourceType&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;AWS::DynamoDB::Table&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;, &lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;
				&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;LogicalResourceId&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;:&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;TerraformStateLockTable&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;, &lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;
				&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;ResourceIdentifier&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;: { &lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;
					&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;TableName&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;: &lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;${LOCK_TABLE_NAME}&lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&quot;&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt; &lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;
				} &lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;
			} &lt;/span&gt;&lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;&lt;span class=&quot;s2&quot;&gt;
		]&quot;&lt;/span&gt; | &lt;span class=&quot;nb&quot;&gt;tee &lt;/span&gt;import-terragrunt-changeset.json
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;CloudFormation assigns each change set a unique ID needed for describing, executing, or discarding the change set with subsequent API calls. The &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;import-terragrunt-changeset.json&lt;/code&gt; stores the change set identifier in a JSON file for our other make targets to consume.&lt;/p&gt;

&lt;p&gt;Next, we want a target for describing the created change set so we can see what modifications it will make:&lt;/p&gt;

&lt;div class=&quot;language-make highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;nl&quot;&gt;.PHONY&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;prepare-cfn-import-terragrunt&lt;/span&gt;
&lt;span class=&quot;nl&quot;&gt;prepare-cfn-import-terragrunt&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;import-terragrunt-changeset.json&lt;/span&gt;
	&lt;span class=&quot;nf&quot;&gt;$(&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;eval&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;CHANGE_SET_ID&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;$(&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;shell&lt;/span&gt; jq &lt;span class=&quot;nt&quot;&gt;-r&lt;/span&gt; .Id import-terragrunt-changeset.json&lt;span class=&quot;nf&quot;&gt;))&lt;/span&gt;
	aws cloudformation &lt;span class=&quot;nb&quot;&gt;wait &lt;/span&gt;change-set-create-complete &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nt&quot;&gt;--change-set-name&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;${CHANGE_SET_ID}&lt;/span&gt; &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nt&quot;&gt;--stack-name&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;${ADMIN_INIT_STACK_NAME}&lt;/span&gt;
	&lt;span class=&quot;p&quot;&gt;@&lt;/span&gt;aws cloudformation describe-change-set &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nt&quot;&gt;--change-set-name&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;${CHANGE_SET_ID}&lt;/span&gt; &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nt&quot;&gt;--stack-name&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;${ADMIN_INIT_STACK_NAME}&lt;/span&gt; &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		| jq &lt;span class=&quot;s1&quot;&gt;'{ Changes, Status, StatusReason }'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;prepare-cfn-import-terragrunt&lt;/code&gt; target depends on &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;import-terragrunt-changeset.json&lt;/code&gt; to create the change set and communicate its unique id, then describes it once the change set finishes creating.&lt;/p&gt;

&lt;p&gt;At this point, we’ll add another target for discarding the change set, in case we don’t like what we see with &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;prepare-cfn-import-terragrunt&lt;/code&gt;:&lt;/p&gt;

&lt;div class=&quot;language-make highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;nl&quot;&gt;.PHONY&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;discard-cfn-import-terragrunt&lt;/span&gt;
&lt;span class=&quot;nl&quot;&gt;discard-cfn-import-terragrunt&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;import-terragrunt-changeset.json&lt;/span&gt;
	&lt;span class=&quot;nf&quot;&gt;$(&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;eval&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;CHANGE_SET_ID&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;$(&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;shell&lt;/span&gt; jq &lt;span class=&quot;nt&quot;&gt;-r&lt;/span&gt; .Id import-terragrunt-changeset.json&lt;span class=&quot;nf&quot;&gt;))&lt;/span&gt;
	aws cloudformation delete-change-set &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nt&quot;&gt;--change-set-name&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;${CHANGE_SET_ID}&lt;/span&gt; &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nt&quot;&gt;--stack-name&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;${ADMIN_INIT_STACK_NAME}&lt;/span&gt;
	&lt;span class=&quot;p&quot;&gt;@&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;rm &lt;/span&gt;import-terragrunt-changeset.json
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Let’s add a conventional &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;clean&lt;/code&gt; target too:&lt;/p&gt;

&lt;div class=&quot;language-make highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;nl&quot;&gt;.PHONY&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;clean&lt;/span&gt;
&lt;span class=&quot;nl&quot;&gt;clean&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;:&lt;/span&gt;
	&lt;span class=&quot;nb&quot;&gt;rm &lt;/span&gt;import-terragrunt-changeset.json
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Now, let’s use our targets to create and describe the change set:&lt;/p&gt;

&lt;div class=&quot;language-bash highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;➜ make prepare-cfn-import-terragrunt
&lt;span class=&quot;o&quot;&gt;{&lt;/span&gt;
    &lt;span class=&quot;s2&quot;&gt;&quot;Id&quot;&lt;/span&gt;: &lt;span class=&quot;s2&quot;&gt;&quot;arn:aws:cloudformation:us-east-1:&amp;lt;omitted&amp;gt;:changeSet/tf-admin-init-import-terragrunt/179b1efd-2961-48af-95c2-6f45b96a9925&quot;&lt;/span&gt;,
    &lt;span class=&quot;s2&quot;&gt;&quot;StackId&quot;&lt;/span&gt;: &lt;span class=&quot;s2&quot;&gt;&quot;arn:aws:cloudformation:us-east-1:&amp;lt;omitted&amp;gt;:stack/tf-admin-init/8704b070-5f61-11eb-9ff1-0eea077046db&quot;&lt;/span&gt;
&lt;span class=&quot;o&quot;&gt;}&lt;/span&gt;
aws cloudformation &lt;span class=&quot;nb&quot;&gt;wait &lt;/span&gt;change-set-create-complete &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nt&quot;&gt;--change-set-name&lt;/span&gt; arn:aws:cloudformation:us-east-1:&amp;lt;omitted&amp;gt;:changeSet/tf-admin-init-import-terragrunt/179b1efd-2961-48af-95c2-6f45b96a9925 &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nt&quot;&gt;--stack-name&lt;/span&gt; tf-admin-init
&lt;span class=&quot;o&quot;&gt;{&lt;/span&gt;
  &lt;span class=&quot;s2&quot;&gt;&quot;Changes&quot;&lt;/span&gt;: &lt;span class=&quot;o&quot;&gt;[&lt;/span&gt;
    &lt;span class=&quot;o&quot;&gt;{&lt;/span&gt;
      &lt;span class=&quot;s2&quot;&gt;&quot;Type&quot;&lt;/span&gt;: &lt;span class=&quot;s2&quot;&gt;&quot;Resource&quot;&lt;/span&gt;,
      &lt;span class=&quot;s2&quot;&gt;&quot;ResourceChange&quot;&lt;/span&gt;: &lt;span class=&quot;o&quot;&gt;{&lt;/span&gt;
        &lt;span class=&quot;s2&quot;&gt;&quot;Action&quot;&lt;/span&gt;: &lt;span class=&quot;s2&quot;&gt;&quot;Import&quot;&lt;/span&gt;,
        &lt;span class=&quot;s2&quot;&gt;&quot;LogicalResourceId&quot;&lt;/span&gt;: &lt;span class=&quot;s2&quot;&gt;&quot;TerraformStateBucket&quot;&lt;/span&gt;,
        &lt;span class=&quot;s2&quot;&gt;&quot;PhysicalResourceId&quot;&lt;/span&gt;: &lt;span class=&quot;s2&quot;&gt;&quot;terraform-skeleton-state&quot;&lt;/span&gt;,
        &lt;span class=&quot;s2&quot;&gt;&quot;ResourceType&quot;&lt;/span&gt;: &lt;span class=&quot;s2&quot;&gt;&quot;AWS::S3::Bucket&quot;&lt;/span&gt;,
        &lt;span class=&quot;s2&quot;&gt;&quot;Scope&quot;&lt;/span&gt;: &lt;span class=&quot;o&quot;&gt;[]&lt;/span&gt;,
        &lt;span class=&quot;s2&quot;&gt;&quot;Details&quot;&lt;/span&gt;: &lt;span class=&quot;o&quot;&gt;[]&lt;/span&gt;
      &lt;span class=&quot;o&quot;&gt;}&lt;/span&gt;
    &lt;span class=&quot;o&quot;&gt;}&lt;/span&gt;,
    &lt;span class=&quot;o&quot;&gt;{&lt;/span&gt;
      &lt;span class=&quot;s2&quot;&gt;&quot;Type&quot;&lt;/span&gt;: &lt;span class=&quot;s2&quot;&gt;&quot;Resource&quot;&lt;/span&gt;,
      &lt;span class=&quot;s2&quot;&gt;&quot;ResourceChange&quot;&lt;/span&gt;: &lt;span class=&quot;o&quot;&gt;{&lt;/span&gt;
        &lt;span class=&quot;s2&quot;&gt;&quot;Action&quot;&lt;/span&gt;: &lt;span class=&quot;s2&quot;&gt;&quot;Import&quot;&lt;/span&gt;,
        &lt;span class=&quot;s2&quot;&gt;&quot;LogicalResourceId&quot;&lt;/span&gt;: &lt;span class=&quot;s2&quot;&gt;&quot;TerraformStateLockTable&quot;&lt;/span&gt;,
        &lt;span class=&quot;s2&quot;&gt;&quot;PhysicalResourceId&quot;&lt;/span&gt;: &lt;span class=&quot;s2&quot;&gt;&quot;terraform-skeleton-state-locks&quot;&lt;/span&gt;,
        &lt;span class=&quot;s2&quot;&gt;&quot;ResourceType&quot;&lt;/span&gt;: &lt;span class=&quot;s2&quot;&gt;&quot;AWS::DynamoDB::Table&quot;&lt;/span&gt;,
        &lt;span class=&quot;s2&quot;&gt;&quot;Scope&quot;&lt;/span&gt;: &lt;span class=&quot;o&quot;&gt;[]&lt;/span&gt;,
        &lt;span class=&quot;s2&quot;&gt;&quot;Details&quot;&lt;/span&gt;: &lt;span class=&quot;o&quot;&gt;[]&lt;/span&gt;
      &lt;span class=&quot;o&quot;&gt;}&lt;/span&gt;
    &lt;span class=&quot;o&quot;&gt;}&lt;/span&gt;,
    &lt;span class=&quot;o&quot;&gt;{&lt;/span&gt;
      &lt;span class=&quot;s2&quot;&gt;&quot;Type&quot;&lt;/span&gt;: &lt;span class=&quot;s2&quot;&gt;&quot;Resource&quot;&lt;/span&gt;,
      &lt;span class=&quot;s2&quot;&gt;&quot;ResourceChange&quot;&lt;/span&gt;: &lt;span class=&quot;o&quot;&gt;{&lt;/span&gt;
        &lt;span class=&quot;s2&quot;&gt;&quot;Action&quot;&lt;/span&gt;: &lt;span class=&quot;s2&quot;&gt;&quot;Import&quot;&lt;/span&gt;,
        &lt;span class=&quot;s2&quot;&gt;&quot;LogicalResourceId&quot;&lt;/span&gt;: &lt;span class=&quot;s2&quot;&gt;&quot;TerraformStateLogBucket&quot;&lt;/span&gt;,
        &lt;span class=&quot;s2&quot;&gt;&quot;PhysicalResourceId&quot;&lt;/span&gt;: &lt;span class=&quot;s2&quot;&gt;&quot;terraform-skeleton-state-logs&quot;&lt;/span&gt;,
        &lt;span class=&quot;s2&quot;&gt;&quot;ResourceType&quot;&lt;/span&gt;: &lt;span class=&quot;s2&quot;&gt;&quot;AWS::S3::Bucket&quot;&lt;/span&gt;,
        &lt;span class=&quot;s2&quot;&gt;&quot;Scope&quot;&lt;/span&gt;: &lt;span class=&quot;o&quot;&gt;[]&lt;/span&gt;,
        &lt;span class=&quot;s2&quot;&gt;&quot;Details&quot;&lt;/span&gt;: &lt;span class=&quot;o&quot;&gt;[]&lt;/span&gt;
      &lt;span class=&quot;o&quot;&gt;}&lt;/span&gt;
    &lt;span class=&quot;o&quot;&gt;}&lt;/span&gt;
  &lt;span class=&quot;o&quot;&gt;]&lt;/span&gt;,
  &lt;span class=&quot;s2&quot;&gt;&quot;Status&quot;&lt;/span&gt;: &lt;span class=&quot;s2&quot;&gt;&quot;CREATE_COMPLETE&quot;&lt;/span&gt;,
  &lt;span class=&quot;s2&quot;&gt;&quot;StatusReason&quot;&lt;/span&gt;: null
&lt;span class=&quot;o&quot;&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Our describe command should show three &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Import&lt;/code&gt; actions occurring in the change list with no other changes. If you see any other actions, it means your template contains unapplied changes, and you’ll want to address those first.&lt;/p&gt;

&lt;h2 id=&quot;execute-import&quot;&gt;Execute Import&lt;/h2&gt;

&lt;p&gt;Having created the import change set and verified the actions it staged, we’ll now add a target for executing it:&lt;/p&gt;

&lt;div class=&quot;language-make highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;nl&quot;&gt;.PHONY&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;cfn-import-terragrunt&lt;/span&gt;
&lt;span class=&quot;nl&quot;&gt;cfn-import-terragrunt&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;import-terragrunt-changeset.json&lt;/span&gt;
	&lt;span class=&quot;nf&quot;&gt;$(&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;eval&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;CHANGE_SET_ID&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;$(&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;shell&lt;/span&gt; jq &lt;span class=&quot;nt&quot;&gt;-r&lt;/span&gt; .Id import-terragrunt-changeset.json&lt;span class=&quot;nf&quot;&gt;))&lt;/span&gt;
	aws cloudformation &lt;span class=&quot;nb&quot;&gt;wait &lt;/span&gt;change-set-create-complete &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nt&quot;&gt;--change-set-name&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;${CHANGE_SET_ID}&lt;/span&gt; &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nt&quot;&gt;--stack-name&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;${ADMIN_INIT_STACK_NAME}&lt;/span&gt;
	aws cloudformation execute-change-set &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nt&quot;&gt;--change-set-name&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;${CHANGE_SET_ID}&lt;/span&gt; &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nt&quot;&gt;--stack-name&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;${ADMIN_INIT_STACK_NAME}&lt;/span&gt;
	&lt;span class=&quot;p&quot;&gt;@&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;rm &lt;/span&gt;import-terragrunt-changeset.json
	aws cloudformation &lt;span class=&quot;nb&quot;&gt;wait &lt;/span&gt;stack-import-complete &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nt&quot;&gt;--stack-name&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;${ADMIN_INIT_STACK_NAME}&lt;/span&gt;
	&lt;span class=&quot;nf&quot;&gt;$(&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;call&lt;/span&gt; show_cfn_drift,&lt;span class=&quot;nv&quot;&gt;${ADMIN_INIT_STACK_NAME}&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Execute the import with:&lt;/p&gt;

&lt;div class=&quot;language-bash highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;➜ make cfn-import-terragrunt
aws cloudformation &lt;span class=&quot;nb&quot;&gt;wait &lt;/span&gt;change-set-create-complete &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nt&quot;&gt;--change-set-name&lt;/span&gt; arn:aws:cloudformation:us-east-1:&amp;lt;omitted&amp;gt;:changeSet/tf-admin-init-import-terragrunt/179b1efd-2961-48af-95c2-6f45b96a9925 &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nt&quot;&gt;--stack-name&lt;/span&gt; tf-admin-init
aws cloudformation execute-change-set &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nt&quot;&gt;--change-set-name&lt;/span&gt; arn:aws:cloudformation:us-east-1:&amp;lt;omitted&amp;gt;:changeSet/tf-admin-init-import-terragrunt/179b1efd-2961-48af-95c2-6f45b96a9925 &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nt&quot;&gt;--stack-name&lt;/span&gt; tf-admin-init
aws cloudformation &lt;span class=&quot;nb&quot;&gt;wait &lt;/span&gt;stack-import-complete &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nt&quot;&gt;--stack-name&lt;/span&gt; tf-admin-init
&lt;span class=&quot;o&quot;&gt;{&lt;/span&gt;
  &lt;span class=&quot;s2&quot;&gt;&quot;DetectionStatus&quot;&lt;/span&gt;: &lt;span class=&quot;s2&quot;&gt;&quot;DETECTION_COMPLETE&quot;&lt;/span&gt;,
  &lt;span class=&quot;s2&quot;&gt;&quot;StackDriftStatus&quot;&lt;/span&gt;: &lt;span class=&quot;s2&quot;&gt;&quot;IN_SYNC&quot;&lt;/span&gt;,
  &lt;span class=&quot;s2&quot;&gt;&quot;DriftedStackResourceCount&quot;&lt;/span&gt;: 0
&lt;span class=&quot;o&quot;&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;At the end of the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;cfn-import-terragrunt&lt;/code&gt; target, we call our &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;show_cfn_drift&lt;/code&gt; helper function to verify that the properties of the resources we imported match what we specified in our template. You should see &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;StackDriftStatus&lt;/code&gt; is &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;IN_SYNC&lt;/code&gt;, which means we’ve successfully imported the resources and the definitions for those resources in our template match reality.&lt;sup id=&quot;fnref:2&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:2&quot; class=&quot;footnote&quot;&gt;2&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

&lt;h1 id=&quot;whats-next&quot;&gt;What’s Next&lt;/h1&gt;

&lt;p&gt;We’ve now fully separated the creation of operational infrastructure required to run Terraform from the infrastructure Terraform manages. Creating the operational infrastructure with CloudFormation has numerous benefits. We can now harden our state bucket, log bucket, and lock table in ways that were not available to us with Terragrunt managing their creation. We’ll tackle such hardening in upcoming posts.&lt;/p&gt;

&lt;h1 id=&quot;footnotes&quot;&gt;Footnotes&lt;/h1&gt;

&lt;div class=&quot;footnotes&quot; role=&quot;doc-endnotes&quot;&gt;
  &lt;ol&gt;
    &lt;li id=&quot;fn:1&quot; role=&quot;doc-endnote&quot;&gt;

      &lt;p&gt;If you omit &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;UpdateReplacePolicy&lt;/code&gt;, the linter will report something like:&lt;/p&gt;

      &lt;blockquote&gt;
        &lt;p&gt;W3011 Both UpdateReplacePolicy and DeletionPolicy are needed to protect Resources/TerraformStateBucket from deletion&lt;/p&gt;
      &lt;/blockquote&gt;
      &lt;p&gt;&lt;a href=&quot;#fnref:1&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:2&quot; role=&quot;doc-endnote&quot;&gt;

      &lt;p&gt;If you see that there is drift after executing the import change set, the resources you imported have a different configuration than what you specified in your template. For instance, maybe your template specified a bucket is KMS encrypted when in reality, the bucket is AES-256 encrypted. Drift detection will report such differences. To resolve the drift:&lt;/p&gt;

      &lt;ol&gt;
        &lt;li&gt;Decide whether the drift is appropriate and should be retained&lt;/li&gt;
        &lt;li&gt;Modify the template to match any appropriate drift&lt;/li&gt;
        &lt;li&gt;Create a change set on the stack with the updated template&lt;/li&gt;
        &lt;li&gt;Verify the change set reports it will discard the inappropriate drift (or have no changes if there was no inappropriate drift)&lt;/li&gt;
        &lt;li&gt;Execute the change set&lt;/li&gt;
      &lt;/ol&gt;
      &lt;p&gt;&lt;a href=&quot;#fnref:2&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
  &lt;/ol&gt;
&lt;/div&gt;</content><author><name>Chris</name></author><category term="tf-skeleton" /><category term="terraform" /><category term="terragrunt" /><summary type="html">Part 3 showed us Terraform requires some infrastructure itself to store remote state and discussed some limitations of using Terragrunt to manage the creation of that infrastructure. In part 4, we introduced more operational infrastructure for Terraform and began managing that infrastructure with CloudFormation.</summary></entry><entry><title type="html">Terraform Skeleton Part 4: Backend Role</title><link href="https://thirstydeveloper.io/tf-skeleton/2021/02/10/part-4-backend-role.html" rel="alternate" type="text/html" title="Terraform Skeleton Part 4: Backend Role" /><published>2021-02-10T22:30:00+00:00</published><updated>2021-02-10T22:30:00+00:00</updated><id>https://thirstydeveloper.io/tf-skeleton/2021/02/10/part-4-backend-role</id><content type="html" xml:base="https://thirstydeveloper.io/tf-skeleton/2021/02/10/part-4-backend-role.html">&lt;p&gt;The &lt;a href=&quot;/tf-skeleton/2021/01/28/part-3-aws-backend.html&quot;&gt;previous entry&lt;/a&gt; enhanced the &lt;a href=&quot;/series/tf-skeleton.html&quot;&gt;terraform skeleton&lt;/a&gt; with remote state storage using AWS S3 and DynamoDB. Access to the state was granted based on whatever AWS credentials were configured in the shell at the time terraform was executed.&lt;/p&gt;

&lt;p&gt;Terraform state is a highly sensitive resource. It is likely to contain lots of sensitive information including passwords and access tokens. Additionally, recovering from a lost state file means either recreating all the infrastructure that was in it, or spending some quality time running &lt;a href=&quot;https://www.terraform.io/docs/cli/import/index.html&quot;&gt;terraform import&lt;/a&gt; commands for resources that support it, and hand modifying state files for those that do not.&lt;/p&gt;

&lt;p&gt;On projects I’ve supported, developers have had near-administrative level permissions on their cloud accounts. Typically, these are the credentials they will use when running terraform, which also requires near-administrative level permissions to manage the wide variety of cloud resources a project is likely to need. Unless instructed otherwise, terraform will use those administrative credentials for accessing the state, and that opens up the potential for disaster (e.g., a developer mistakenly deleting a state bucket).&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/tf-skeleton/part-4/admin-state-access.png&quot; alt=&quot;Accessing terraform state with administrative IAM user credentials&quot; /&gt;&lt;/p&gt;

&lt;p&gt;In this post, we’ll create a dedicated IAM role for backend access so that every developer on our team working with terraform will access state with the same permissions, and those permissions are scoped to just what is needed to read and write state.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/tf-skeleton/part-4/backend-role-state-access.png&quot; alt=&quot;Accessing terraform state with least-privilege IAM role credentials&quot; /&gt;&lt;/p&gt;

&lt;p&gt;Using a dedicated backend role for state access will help avoid “works on my machine problems” and allow us, in future entries, to lock down access to terraform state due to its sensitive nature.&lt;/p&gt;

&lt;h1 id=&quot;goals&quot;&gt;Goals&lt;/h1&gt;

&lt;ol&gt;
  &lt;li&gt;The skeleton uses a dedicated IAM role for accessing terraform state such that everyone on the team accesses state with the same least-privilege permissions&lt;/li&gt;
  &lt;li&gt;The IAM role and its permissions are controlled using infrastructure-as-code&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;If you prefer to jump to the end, the code implementing the final result is available on &lt;a href=&quot;https://github.com/thirstydeveloper/terraform-terragrunt-skeleton/tree/release/1.3&quot;&gt;GitHub&lt;/a&gt;.&lt;/p&gt;

&lt;h1 id=&quot;setup&quot;&gt;Setup&lt;/h1&gt;

&lt;p&gt;You will need:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;An AWS account to serve as your terraform “admin” account, holding the state resources&lt;/li&gt;
  &lt;li&gt;The &lt;a href=&quot;https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html&quot;&gt;AWS CLI&lt;/a&gt; installed on your workstation&lt;/li&gt;
  &lt;li&gt;Credentials for that account configured in the terminal for running &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;aws&lt;/code&gt; CLI commands&lt;/li&gt;
&lt;/ol&gt;

&lt;h1 id=&quot;the-administrative-account&quot;&gt;The Administrative Account&lt;/h1&gt;

&lt;p&gt;Before we create our backend IAM role, it is worth discussing cloud account organization. Chances are strong that you’re going to have more than one account.&lt;sup id=&quot;fnref:1&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:1&quot; class=&quot;footnote&quot;&gt;1&lt;/a&gt;&lt;/sup&gt; Typically, one per application environment is recommended:&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/tf-skeleton/part-4/unmanaged-app-accounts.png&quot; alt=&quot;Separate AWS accounts for dev, test, and production app-tier environments&quot; /&gt;&lt;/p&gt;

&lt;p&gt;With multiple accounts, the question of where we put our terraform state arises. We could put the state for each tier-environment in its corresponding account (e.g., app-dev state in the app-dev account) but I don’t like doing that for several reasons:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;It assumes our tiers line up one-to-one with accounts, which may not always be true&lt;/li&gt;
  &lt;li&gt;I prefer to restrict administrative access to terraform state to a subset of those who have access to environment accounts.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Instead, I prefer to use a separate admin account to hold all the state, an approach recommended in &lt;a href=&quot;https://www.terraform.io/docs/backends/types/s3.html#multi-account-aws-architecture&quot;&gt;the documentation for the S3 backend&lt;/a&gt;:&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/assets/tf-skeleton/part-4/admin-managed-app-accounts.png&quot; alt=&quot;Admin AWS account managing app accounts&quot; /&gt;&lt;/p&gt;

&lt;p&gt;With this approach, our admin account contains:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;S3 bucket(s) for terraform state&lt;/li&gt;
  &lt;li&gt;S3 bucket(s) for state logs&lt;sup id=&quot;fnref:2&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:2&quot; class=&quot;footnote&quot;&gt;2&lt;/a&gt;&lt;/sup&gt;&lt;/li&gt;
  &lt;li&gt;DynamoDB table(s) for terraform locks&lt;/li&gt;
  &lt;li&gt;IAM roles for accessing the above&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The first three were created in the &lt;a href=&quot;/tf-skeleton/2021/01/28/part-3-aws-backend.html&quot;&gt;previous entry&lt;/a&gt;. Now, we’ll create our backend role.&lt;/p&gt;

&lt;h1 id=&quot;the-backend-role&quot;&gt;The Backend Role&lt;/h1&gt;

&lt;p&gt;We have a bit of a chicken and egg problem with the backend role. We want to control all infrastructure as code, the backend role included, but we can’t use the backend role for terraform until it is created. A similar problem presents itself if you want to self-manage the creation of your S3 state bucket and DynamoDB lock table.&lt;/p&gt;

&lt;p&gt;While it’s possible to use terraform to create these resources by having a stack that either stores state locally initially and/or uses user credentials instead of the backend role, I tend not to do that. I prefer to separate the infrastructure needed for terraform to run from the infrastructure terraform creates, and manage the former using &lt;a href=&quot;https://aws.amazon.com/cloudformation/&quot;&gt;CloudFormation&lt;/a&gt;. A simple CloudFormation stack is more than capable of managing the few resources needed to bootstrap an AWS account to serve as our admin account and means we offload the state management of that bootstrap infrastructure to AWS’s CloudFormation tool.&lt;/p&gt;

&lt;h2 id=&quot;preparation&quot;&gt;Preparation&lt;/h2&gt;

&lt;p&gt;Before adding CloudFormation templates to our infra repository, I like to add a &lt;a href=&quot;https://pre-commit.com/&quot;&gt;pre-commit&lt;/a&gt; hook that validates those templates. It’s easy to make syntax errors with CloudFormation; using a validation hook helps to shorten the feedback loop. I do the following:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;Store bootstrap CloudFormation templates under &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;init/&lt;/code&gt; in the infra repo&lt;/li&gt;
  &lt;li&gt;Use a file extension of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;.cf.yml&lt;/code&gt; to differentiate CloudFormation templates from regular YAML files&lt;/li&gt;
  &lt;li&gt;Use the &lt;a href=&quot;https://github.com/aws-cloudformation/cfn-python-lint&quot;&gt;cfn-python-lint&lt;/a&gt; pre-commit hook to validate the CloudFormation templates&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Let’s add the hook to our &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;.pre-commit-config.yaml&lt;/code&gt;:&lt;/p&gt;

&lt;div class=&quot;language-diff highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;gh&quot;&gt;diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index fa080cf..27217e9 100644
&lt;/span&gt;&lt;span class=&quot;gd&quot;&gt;--- a/.pre-commit-config.yaml
&lt;/span&gt;&lt;span class=&quot;gi&quot;&gt;+++ b/.pre-commit-config.yaml
&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;@@ -24,3 +25,9 @@&lt;/span&gt; repos:
   rev: v0.1.10
   hooks:
     - id: terragrunt-hclfmt
&lt;span class=&quot;gi&quot;&gt;+
+- repo: https://github.com/aws-cloudformation/cfn-python-lint
+  rev: v0.44.5
+  hooks:
+  -   id: cfn-python-lint
+      files: init/.*\.cf\.(yml|yaml)$
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;This configures the hook to scan all templates under the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;init/&lt;/code&gt; directory. We also need to tell our existing &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;check-yaml&lt;/code&gt; hook to ignore the CloudFormation templates because it will throw errors on CloudFormation’s interpolation syntax. We can do that with:&lt;/p&gt;

&lt;div class=&quot;language-diff highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;gh&quot;&gt;diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index fa080cf..27217e9 100644
&lt;/span&gt;&lt;span class=&quot;gd&quot;&gt;--- a/.pre-commit-config.yaml
&lt;/span&gt;&lt;span class=&quot;gi&quot;&gt;+++ b/.pre-commit-config.yaml
&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;@@ -8,6 +8,7 @@&lt;/span&gt; repos:
       args: [ --markdown-linebreak-ext=* ]
     - id: check-yaml
       args: [ --allow-multiple-documents ]
&lt;span class=&quot;gi&quot;&gt;+      exclude: .*\.cf\.(yml|yaml)$
&lt;/span&gt;     - id: check-json
     - id: check-merge-conflict
     - id: detect-aws-credentials
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Finally, run &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;pre-commit install&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;pre-commit run -a&lt;/code&gt; to make sure all hooks are currently passing.&lt;/p&gt;

&lt;h2 id=&quot;admin-account-cloudformation-template&quot;&gt;Admin Account CloudFormation Template&lt;/h2&gt;

&lt;p&gt;Create a new CloudFormation template at &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;init/admin/init-admin-account.cf.yml&lt;/code&gt;. You’ll need &lt;a href=&quot;https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/parameters-section-structure.html&quot;&gt;parameters&lt;/a&gt; for the admin account ID, state bucket, log bucket, and lock table names. I like to use &lt;a href=&quot;https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/metadata-section-structure.html&quot;&gt;CloudFormation’s metadata property&lt;/a&gt; to group the parameters sensibly, should you use the AWS Management Console to deploy the stack:&lt;/p&gt;

&lt;div class=&quot;language-yaml highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;nn&quot;&gt;---&lt;/span&gt;
&lt;span class=&quot;na&quot;&gt;AWSTemplateFormatVersion&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;2010-09-09'&lt;/span&gt;
&lt;span class=&quot;na&quot;&gt;Description&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;Initialize terraform admin account&lt;/span&gt;
&lt;span class=&quot;na&quot;&gt;Metadata&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
  &lt;span class=&quot;s&quot;&gt;AWS::CloudFormation::Interface:&lt;/span&gt;
    &lt;span class=&quot;s&quot;&gt;ParameterGroups&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
      &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;na&quot;&gt;Label&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
          &lt;span class=&quot;na&quot;&gt;default&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;Admin Account Config&lt;/span&gt;
        &lt;span class=&quot;na&quot;&gt;Parameters&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
          &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;AdminAccountId&lt;/span&gt;
      &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;na&quot;&gt;Label&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
          &lt;span class=&quot;na&quot;&gt;default&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;Terraform State Resources&lt;/span&gt;
        &lt;span class=&quot;na&quot;&gt;Parameters&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
          &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;StateBucketName&lt;/span&gt;
          &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;StateLogBucketName&lt;/span&gt;
          &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;LockTableName&lt;/span&gt;
&lt;span class=&quot;na&quot;&gt;Parameters&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;AdminAccountId&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
    &lt;span class=&quot;na&quot;&gt;Type&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;String&lt;/span&gt;
    &lt;span class=&quot;na&quot;&gt;Description&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;Account ID of the admin account to contain the state&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;StateBucketName&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
    &lt;span class=&quot;na&quot;&gt;Type&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;String&lt;/span&gt;
    &lt;span class=&quot;na&quot;&gt;Description&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;Name of the S3 bucket for terraform state&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;StateLogBucketName&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
    &lt;span class=&quot;na&quot;&gt;Type&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;String&lt;/span&gt;
    &lt;span class=&quot;na&quot;&gt;Description&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;Name of the S3 bucket for terraform state logs&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;LockTableName&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
    &lt;span class=&quot;na&quot;&gt;Type&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;String&lt;/span&gt;
    &lt;span class=&quot;na&quot;&gt;Description&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;Name of the terraform DynamoDB lock table&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Next, add a &lt;a href=&quot;https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/rules-section-structure.html&quot;&gt;CloudFormation rule&lt;/a&gt; to restrict deployment of the stack to only the specified account. This is to help prevent someone from deploying the stack to the wrong account.&lt;/p&gt;

&lt;div class=&quot;language-yaml highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;na&quot;&gt;Rules&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;EnsureDeployingToCorrectAccount&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
    &lt;span class=&quot;na&quot;&gt;Assertions&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
      &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;na&quot;&gt;Assert&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
        &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;Fn::Equals'&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
          &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;!Ref&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;AWS::AccountId&lt;/span&gt;
          &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;!Ref&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;AdminAccountId&lt;/span&gt;
        &lt;span class=&quot;na&quot;&gt;AssertDescription&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;Stack&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt; &lt;/span&gt;&lt;span class=&quot;s&quot;&gt;can&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt; &lt;/span&gt;&lt;span class=&quot;s&quot;&gt;only&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt; &lt;/span&gt;&lt;span class=&quot;s&quot;&gt;be&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt; &lt;/span&gt;&lt;span class=&quot;s&quot;&gt;deployed&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt; &lt;/span&gt;&lt;span class=&quot;s&quot;&gt;into&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt; &lt;/span&gt;&lt;span class=&quot;s&quot;&gt;the&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt; &lt;/span&gt;&lt;span class=&quot;s&quot;&gt;specified&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt; &lt;/span&gt;&lt;span class=&quot;s&quot;&gt;AdminAccountId'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Next, add a resource to create an IAM managed policy for read/write state access:&lt;/p&gt;

&lt;div class=&quot;language-yaml highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;na&quot;&gt;Resources&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
  &lt;span class=&quot;na&quot;&gt;TerraformStateReadWritePolicy&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
    &lt;span class=&quot;na&quot;&gt;Type&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;AWS::IAM::ManagedPolicy'&lt;/span&gt;
    &lt;span class=&quot;na&quot;&gt;Properties&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
      &lt;span class=&quot;na&quot;&gt;ManagedPolicyName&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;TerraformStateReadWrite&lt;/span&gt;
      &lt;span class=&quot;na&quot;&gt;Path&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;/terraform/&lt;/span&gt;
      &lt;span class=&quot;na&quot;&gt;Description&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;Read/write access to terraform state&lt;/span&gt;
      &lt;span class=&quot;na&quot;&gt;PolicyDocument&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
        &lt;span class=&quot;na&quot;&gt;Version&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;2012-10-17&lt;/span&gt;
        &lt;span class=&quot;c1&quot;&gt;# Permissions are based on:&lt;/span&gt;
        &lt;span class=&quot;c1&quot;&gt;# https://www.terraform.io/docs/backends/types/s3.html#example-configuration&lt;/span&gt;
        &lt;span class=&quot;c1&quot;&gt;# https://github.com/gruntwork-io/terragrunt/issues/919&lt;/span&gt;
        &lt;span class=&quot;na&quot;&gt;Statement&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
          &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;na&quot;&gt;Sid&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;AllowStateBucketList&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Effect&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;Allow&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Action&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;s3:ListBucket'&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;s3:GetBucketVersioning'&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Resource&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;!Sub&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;arn:aws:s3:::${StateBucketName}&quot;&lt;/span&gt;
          &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;na&quot;&gt;Sid&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;AllowStateReadWrite&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Effect&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;Allow&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Action&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;s3:GetObject'&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;s3:PutObject'&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Resource&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;!Sub&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;arn:aws:s3:::${StateBucketName}/*&quot;&lt;/span&gt;
          &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;na&quot;&gt;Sid&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;AllowStateLockReadWrite&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Effect&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;Allow&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Action&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;dynamodb:DescribeTable'&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;dynamodb:GetItem'&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;dynamodb:PutItem'&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;dynamodb:DeleteItem'&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Resource&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;!Sub&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;arn:aws:dynamodb:${AWS::Region}:${AWS::AccountId}:table/${LockTableName}&quot;&lt;/span&gt;
          &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;na&quot;&gt;Sid&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;AllowStateBucketCreation&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Effect&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;Allow&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Action&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;s3:GetBucketAcl'&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;s3:GetBucketLogging'&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;s3:CreateBucket'&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;s3:PutBucketPublicAccessBlock'&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;s3:PutBucketTagging'&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;s3:PutBucketPolicy'&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;s3:PutBucketVersioning'&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;s3:PutEncryptionConfiguration'&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;s3:PutBucketAcl'&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;s3:PutBucketLogging'&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Resource&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;!Sub&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;arn:aws:s3:::${StateBucketName}&quot;&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;!Sub&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;arn:aws:s3:::${StateLogBucketName}&quot;&lt;/span&gt;
          &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;na&quot;&gt;Sid&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;AllowLockTableCreation&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Effect&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;Allow&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Action&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;dynamodb:CreateTable'&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Resource&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;!Sub&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;arn:aws:dynamodb:${AWS::Region}:${AWS::AccountId}:table/${LockTableName}&quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;This policy grants everything needed by the S3 backend to manage the state. It’s worth noting that &lt;a href=&quot;https://github.com/gruntwork-io/terragrunt/issues/919&quot;&gt;terragrunt requires additional permissions&lt;/a&gt; beyond what &lt;a href=&quot;https://www.terraform.io/docs/backends/types/s3.html#example-configuration&quot;&gt;terraform specifies&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;In particular, this policy &lt;em&gt;does not&lt;/em&gt; grant delete access to the terraform state. As already discussed, losing your state file can create a tremendous amount of pain. Using a versioned S3 bucket helps, but you can still get yourself into trouble if you delete the bucket itself. Since the backend role does not require delete access, it does not get it.&lt;/p&gt;

&lt;p&gt;Furthermore, the policy is restricted to creating only the specified bucket and lock table names. Given that terragrunt will auto-create a bucket and lock table if they do not exist, this restriction helps avoid unintended auto-creations due to developer errors.&lt;/p&gt;

&lt;p&gt;Next, we’ll create the backend role, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;TerraformBackend&lt;/code&gt;, and attach the policy to it:&lt;/p&gt;

&lt;div class=&quot;language-yaml highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;  &lt;span class=&quot;na&quot;&gt;TerraformBackendRole&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
    &lt;span class=&quot;na&quot;&gt;Type&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;AWS::IAM::Role'&lt;/span&gt;
    &lt;span class=&quot;na&quot;&gt;Properties&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
      &lt;span class=&quot;na&quot;&gt;AssumeRolePolicyDocument&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
        &lt;span class=&quot;na&quot;&gt;Version&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;2012-10-17&lt;/span&gt;
        &lt;span class=&quot;na&quot;&gt;Statement&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
          &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;na&quot;&gt;Effect&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;Allow&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Principal&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
              &lt;span class=&quot;na&quot;&gt;AWS&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;!Ref&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;AWS::AccountId&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Action&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
              &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;sts:AssumeRole'&lt;/span&gt;
            &lt;span class=&quot;na&quot;&gt;Condition&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
              &lt;span class=&quot;na&quot;&gt;StringEquals&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
                &lt;span class=&quot;s&quot;&gt;aws:PrincipalType: User&lt;/span&gt;
              &lt;span class=&quot;na&quot;&gt;StringLike&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
                &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;aws:PrincipalTag/Terraformer'&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s1&quot;&gt;'&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;*'&lt;/span&gt;
      &lt;span class=&quot;na&quot;&gt;RoleName&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;TerraformBackend&lt;/span&gt;
      &lt;span class=&quot;na&quot;&gt;Path&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;/terraform/&lt;/span&gt;
      &lt;span class=&quot;na&quot;&gt;ManagedPolicyArns&lt;/span&gt;&lt;span class=&quot;pi&quot;&gt;:&lt;/span&gt;
        &lt;span class=&quot;pi&quot;&gt;-&lt;/span&gt; &lt;span class=&quot;kt&quot;&gt;!Ref&lt;/span&gt; &lt;span class=&quot;s&quot;&gt;TerraformStateReadWritePolicy&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;In the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;AssumeRolePolicyDocument&lt;/code&gt;, I’m specifying who can assume the backend role. There are several ways to go about this, depending on what type of principals are doing the assuming. You could use IAM groups if the principals are IAM users, or SAML context keys if dealing with federated users. I’m going to use an &lt;a href=&quot;https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction_attribute-based-access-control.html&quot;&gt;attribute-based access control (ABAC) approach&lt;/a&gt; that will grant access if the principal has a tag of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Terraformer&lt;/code&gt;. I’m also going to restrict access down to IAM users since that’s what I’m working with. The same approach should work for other principal types as well.&lt;/p&gt;

&lt;p&gt;Finally, let’s add a Makefile to script deployment of our CloudFormation stack:&lt;/p&gt;

&lt;div class=&quot;language-make highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;nv&quot;&gt;ADMIN_INIT_STACK_NAME&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;:=&lt;/span&gt; tf-admin-init
&lt;span class=&quot;nv&quot;&gt;STATE_BUCKET_NAME&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;:=&lt;/span&gt; terraform-skeleton-state
&lt;span class=&quot;nv&quot;&gt;STATE_LOG_BUCKET_NAME&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;:=&lt;/span&gt; terraform-skeleton-state-logs
&lt;span class=&quot;nv&quot;&gt;LOCK_TABLE_NAME&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;:=&lt;/span&gt; terraform-skeleton-state-locks

&lt;span class=&quot;c&quot;&gt;# Use a known profile to ensure account ID is correct
&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;ADMIN_ACCOUNT_ID&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;:=&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;$(&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;shell&lt;/span&gt; &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
	aws &lt;span class=&quot;nt&quot;&gt;--profile&lt;/span&gt; tf-admin-account sts get-caller-identity | jq &lt;span class=&quot;nt&quot;&gt;-r&lt;/span&gt; .Account &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
&lt;span class=&quot;nf&quot;&gt;)&lt;/span&gt;

&lt;span class=&quot;nv&quot;&gt;BACKEND_ROLE_PATH&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;:=&lt;/span&gt; terraform/TerraformBackend
&lt;span class=&quot;nv&quot;&gt;BACKEND_ROLE_ARN&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;:=&lt;/span&gt; arn:aws:iam::&lt;span class=&quot;nv&quot;&gt;${ADMIN_ACCOUNT_ID}&lt;/span&gt;:role/&lt;span class=&quot;nv&quot;&gt;${BACKEND_ROLE_PATH}&lt;/span&gt;

&lt;span class=&quot;nv&quot;&gt;DEPLOYMENT_DIRS&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;:=&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;$(&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;shell&lt;/span&gt; find deployments &lt;span class=&quot;nt&quot;&gt;-name&lt;/span&gt; terragrunt.hcl &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
	&lt;span class=&quot;nt&quot;&gt;-not&lt;/span&gt; &lt;span class=&quot;nt&quot;&gt;-path&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;*&lt;/span&gt;/.terragrunt-cache/&lt;span class=&quot;k&quot;&gt;*&lt;/span&gt; &lt;span class=&quot;nt&quot;&gt;-exec&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;dirname&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;{&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;}&lt;/span&gt; &lt;span class=&quot;se&quot;&gt;\;&lt;/span&gt; &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
&lt;span class=&quot;o&quot;&gt;)&lt;/span&gt;

&lt;span class=&quot;nl&quot;&gt;.PHONY&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;init-admin&lt;/span&gt;
&lt;span class=&quot;nl&quot;&gt;init-admin&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;:&lt;/span&gt;
	aws cloudformation deploy &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nt&quot;&gt;--template-file&lt;/span&gt; init/admin/init-admin-account.cf.yml &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nt&quot;&gt;--stack-name&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;${ADMIN_INIT_STACK_NAME}&lt;/span&gt; &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nt&quot;&gt;--capabilities&lt;/span&gt; CAPABILITY_NAMED_IAM &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nt&quot;&gt;--parameter-overrides&lt;/span&gt; &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
			&lt;span class=&quot;nv&quot;&gt;AdminAccountId&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;${ADMIN_ACCOUNT_ID}&lt;/span&gt; &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
			&lt;span class=&quot;nv&quot;&gt;StateBucketName&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;${STATE_BUCKET_NAME}&lt;/span&gt; &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
			&lt;span class=&quot;nv&quot;&gt;StateLogBucketName&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;${STATE_LOG_BUCKET_NAME}&lt;/span&gt; &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
			&lt;span class=&quot;nv&quot;&gt;LockTableName&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;nv&quot;&gt;${LOCK_TABLE_NAME}&lt;/span&gt;
	aws cloudformation update-termination-protection &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nt&quot;&gt;--stack-name&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;${ADMIN_INIT_STACK_NAME}&lt;/span&gt; &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nt&quot;&gt;--enable-termination-protection&lt;/span&gt;

&lt;span class=&quot;nl&quot;&gt;.PHONY&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;test-backend-assume&lt;/span&gt;
&lt;span class=&quot;nl&quot;&gt;test-backend-assume&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;:&lt;/span&gt;
	aws sts assume-role &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nt&quot;&gt;--role-arn&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;${BACKEND_ROLE_ARN}&lt;/span&gt; &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nt&quot;&gt;--role-session-name&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;$(&lt;/span&gt;&lt;span class=&quot;nb&quot;&gt;shell&lt;/span&gt; &lt;span class=&quot;nb&quot;&gt;whoami&lt;/span&gt;&lt;span class=&quot;nf&quot;&gt;)&lt;/span&gt;

&lt;span class=&quot;nl&quot;&gt;.PHONY&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;:&lt;/span&gt; &lt;span class=&quot;nf&quot;&gt;init-all&lt;/span&gt;
&lt;span class=&quot;nl&quot;&gt;init-all&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;:&lt;/span&gt;
	&lt;span class=&quot;k&quot;&gt;for &lt;/span&gt;d &lt;span class=&quot;k&quot;&gt;in&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;${DEPLOYMENT_DIRS}&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;do&lt;/span&gt; &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nb&quot;&gt;pushd&lt;/span&gt; &lt;span class=&quot;nv&quot;&gt;$$&lt;/span&gt;d&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt; &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		terragrunt init&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt; &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
		&lt;span class=&quot;nb&quot;&gt;popd&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;;&lt;/span&gt; &lt;span class=&quot;se&quot;&gt;\&lt;/span&gt;
	&lt;span class=&quot;k&quot;&gt;done&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;The Makefile:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;Uses the &lt;a href=&quot;https://awscli.amazonaws.com/v2/documentation/api/latest/reference/cloudformation/deploy/index.html&quot;&gt;aws cloudformation deploy&lt;/a&gt; command, which will both create the stack if it does not exist and update it if it does.&lt;/li&gt;
  &lt;li&gt;Turns on termination protection as a basic safety measure on our stack.&lt;/li&gt;
  &lt;li&gt;Provides a test command to verify our ability to assume the backend role.&lt;/li&gt;
  &lt;li&gt;Provides an init-all command, which will be needed when we change out deployments to use the new backend role.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;It’s time to deploy the backend role and use it. To do so:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;Open a terminal and ensure you have AWS credentials to deploy the stack loaded.&lt;/li&gt;
  &lt;li&gt;Run &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;make init-admin&lt;/code&gt; to deploy the CloudFormation stack.&lt;/li&gt;
  &lt;li&gt;Add the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;Terraformer&lt;/code&gt; tag to the appropriate IAM users.&lt;sup id=&quot;fnref:3&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:3&quot; class=&quot;footnote&quot;&gt;3&lt;/a&gt;&lt;/sup&gt;&lt;/li&gt;
  &lt;li&gt;Ensure you can assume the new backend role with &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;make test-backend-assume&lt;/code&gt;.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Next, we need to tell terraform to use the backend role for remote state access. Do so by updating the remote_state block inside &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;root.hcl&lt;/code&gt; to include the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;role_arn&lt;/code&gt; property:&lt;/p&gt;

&lt;div class=&quot;language-diff highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;gh&quot;&gt;diff --git a/deployments/root.hcl b/deployments/root.hcl
index eed54e8..33091af 100644
&lt;/span&gt;&lt;span class=&quot;gd&quot;&gt;--- a/deployments/root.hcl
&lt;/span&gt;&lt;span class=&quot;gi&quot;&gt;+++ b/deployments/root.hcl
&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;@@ -46,9 +46,10 @@&lt;/span&gt; remote_state {
     if_exists = &quot;overwrite&quot;
   }
   config = {
     bucket   = &quot;terraform-skeleton-state&quot;
     region   = &quot;us-east-1&quot;
     encrypt  = true
&lt;span class=&quot;gi&quot;&gt;+    role_arn = &quot;arn:aws:iam::YOUR_ADMIN_ACCOUNT_ID:role/terraform/TerraformBackend&quot;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;To finish switching the deployments over to the new backend configuration we must re-initialize them. Do so with &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;make init-all&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Finally, run &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;terragrunt apply-all&lt;/code&gt; from the repository root. Everything should work.&lt;/p&gt;

&lt;h1 id=&quot;whats-next&quot;&gt;What’s Next?&lt;/h1&gt;

&lt;p&gt;Although this was a fair amount of work, it is important. We now have a single role that gets used for all state manipulation by terraform. That role can be assumed by any of the developers working with terraform and, in future entries, can also be assumed by CI/CD jobs such that all terraform operations use a consistent set of permissions.&lt;/p&gt;

&lt;p&gt;We also began creating infrastructure required to run terraform itself outside of terraform, relying on CloudFormation instead. Upcoming entries will continue this trend, which, along with the backend role we’ve created, will allow us to lock down access to our state bucket.&lt;/p&gt;

&lt;h1 id=&quot;footnotes&quot;&gt;Footnotes&lt;/h1&gt;

&lt;div class=&quot;footnotes&quot; role=&quot;doc-endnotes&quot;&gt;
  &lt;ol&gt;
    &lt;li id=&quot;fn:1&quot; role=&quot;doc-endnote&quot;&gt;
      &lt;p&gt;I’m using the AWS term &lt;em&gt;account&lt;/em&gt; here because out of the cloud providers, my teams use AWS the most heavily. Equivalents include Google Cloud Platform (GCP) &lt;em&gt;projects&lt;/em&gt; and Microsoft Azure &lt;em&gt;resource groups&lt;/em&gt;. &lt;a href=&quot;#fnref:1&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:2&quot; role=&quot;doc-endnote&quot;&gt;
      &lt;p&gt;Another option is to place the logs bucket in a dedicated information security account. For more on this, I recommend videos from re:invent covering multi-account architectures. Here’s &lt;a href=&quot;https://www.youtube.com/watch?reload=9&amp;amp;v=fxo67UeeN1A&quot;&gt;one&lt;/a&gt; from 2019. &lt;a href=&quot;#fnref:2&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:3&quot; role=&quot;doc-endnote&quot;&gt;
      &lt;p&gt;Although I don’t talk about it here, you would want to have tight control over who can add the Terraformer tag. Similarly, you need tight control over the ability to attach terraform IAM policies. The control mechanisms to use depend on how your organization manages user access to cloud accounts and is beyond the scope of this post. &lt;a href=&quot;#fnref:3&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
  &lt;/ol&gt;
&lt;/div&gt;</content><author><name>Chris</name></author><category term="tf-skeleton" /><category term="terraform" /><category term="terragrunt" /><summary type="html">The previous entry enhanced the terraform skeleton with remote state storage using AWS S3 and DynamoDB. Access to the state was granted based on whatever AWS credentials were configured in the shell at the time terraform was executed.</summary></entry><entry><title type="html">Terraform Skeleton Part 3: AWS Backend</title><link href="https://thirstydeveloper.io/tf-skeleton/2021/01/28/part-3-aws-backend.html" rel="alternate" type="text/html" title="Terraform Skeleton Part 3: AWS Backend" /><published>2021-01-28T11:00:00+00:00</published><updated>2021-01-28T11:00:00+00:00</updated><id>https://thirstydeveloper.io/tf-skeleton/2021/01/28/part-3-aws-backend</id><content type="html" xml:base="https://thirstydeveloper.io/tf-skeleton/2021/01/28/part-3-aws-backend.html">&lt;p&gt;Terraform uses &lt;a href=&quot;https://www.terraform.io/docs/state/index.html&quot;&gt;state&lt;/a&gt; files to track the resources it creates back to resource definitions in your &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;*.tf&lt;/code&gt; files. Each deployment has its own state. State is stored according to the &lt;a href=&quot;https://www.terraform.io/docs/configuration/blocks/backends/index.html&quot;&gt;backend&lt;/a&gt; configured for the deployment. Terraform uses a &lt;a href=&quot;https://www.terraform.io/docs/backends/types/local.html&quot;&gt;local backend&lt;/a&gt; for storing state on the local filesystem by default, which is what we’ve been using for &lt;a href=&quot;/2021/01/17/part-1-organizing-terragrunt.html&quot;&gt;part 1&lt;/a&gt; and &lt;a href=&quot;/tf-skeleton/2021/01/23/part-2-variables.html&quot;&gt;part 2&lt;/a&gt; of the &lt;a href=&quot;/series/tf-skeleton.html&quot;&gt;terraform skeleton series&lt;/a&gt;.&lt;sup id=&quot;fnref:1&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:1&quot; class=&quot;footnote&quot;&gt;1&lt;/a&gt;&lt;/sup&gt; This works fine for a simple demonstration but is insufficient for production use because:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;You can only run terraform commands on your deployments from that one machine&lt;/li&gt;
  &lt;li&gt;There are no built-in backups on your state files; if they’re lost, you’re going to have a bad day&lt;/li&gt;
  &lt;li&gt;There’s no built-in versioning of your state files, which can be handy for recovering from more advanced terraform operations gone wrong&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;While you can commit the state files into your git repository for achieving distribution across team members, backups, and versioning, that isn’t a good idea because:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;State files can include sensitive information you might not want in version control&lt;/li&gt;
  &lt;li&gt;If multiple team members make changes affecting the same state file, you’ll wind up with a mess of multiple incomplete state files needing to be merged&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;a href=&quot;https://www.terraform.io/docs/state/remote.html&quot;&gt;Remote state&lt;/a&gt; storage is built in to terraform and solves these problems, making it easy for teams to collaborate on the same deployments. Terragrunt provides further enhancements that make working with remote state even easier. Today, we’ll add remote state to the skeleton using terragrunt, &lt;a href=&quot;https://aws.amazon.com/s3/&quot;&gt;S3&lt;/a&gt;, and &lt;a href=&quot;https://aws.amazon.com/dynamodb/&quot;&gt;DynamoDB&lt;/a&gt;. This will give us a simple foundation to build on.&lt;/p&gt;

&lt;h1 id=&quot;goals&quot;&gt;Goals&lt;/h1&gt;

&lt;ol&gt;
  &lt;li&gt;Our skeleton stores state remotely such that multiple teammates can run terraform commands&lt;/li&gt;
  &lt;li&gt;State files are stored in a directory structure matching our deployments&lt;/li&gt;
  &lt;li&gt;Locks are in place around state manipulation to avoid corruption due to concurrent terraform command executions&lt;/li&gt;
  &lt;li&gt;State is stored securely, with encryption, versioning, and logging enabled&lt;/li&gt;
  &lt;li&gt;Remote state configuration is DRY - defined once and reused across deployments&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;If you prefer to jump to the end, the code implementing the final result is available on &lt;a href=&quot;https://github.com/thirstydeveloper/terraform-terragrunt-skeleton/tree/release/1.2&quot;&gt;GitHub&lt;/a&gt;.&lt;/p&gt;

&lt;h1 id=&quot;setup&quot;&gt;Setup&lt;/h1&gt;

&lt;p&gt;You will need:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;An AWS account&lt;/li&gt;
  &lt;li&gt;Credentials for that account configured in the terminal used for running &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;terraform&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;terragrunt&lt;/code&gt; commands&lt;/li&gt;
&lt;/ol&gt;

&lt;h1 id=&quot;the-s3-backend&quot;&gt;The S3 Backend&lt;/h1&gt;

&lt;p&gt;Remote state can be accomplished in many different ways with terraform. The teams I support tend to have an Amazon Web Services footprint already, and therefore I typically use terraform’s &lt;a href=&quot;https://www.terraform.io/docs/backends/types/s3.html&quot;&gt;S3 backend&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;The S3 backend stores your state files in S3 and retrieves them for stateful terraform commands. This meets the distribution, versioning, and encryption requirements we require. To avoid corruption from concurrent terraform commands, the S3 backend uses a DynamoDB table to manage lock files. Stateful terraform commands first obtain a lock from the DynamoDB table, effectively single-threading commands operating on the same state file.&lt;/p&gt;

&lt;p&gt;We’ll focus on using the S3 backend today.&lt;/p&gt;

&lt;h1 id=&quot;backend-configuration-with-terraform&quot;&gt;Backend Configuration with Terraform&lt;/h1&gt;

&lt;p&gt;Terraform backends are configured using a block in the stack’s &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;*.tf&lt;/code&gt; files:&lt;/p&gt;

&lt;pre&gt;&lt;code class=&quot;language-HCL&quot;&gt;terraform {
  backend &quot;s3&quot; {
    bucket = &quot;terraform-skeleton-state&quot;
    key    = &quot;something/unique/to/the/stack&quot;
    region = &quot;us-east-1&quot;
  }
}
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;This inline configuration is less than ideal. First, you have to configure the backend for every stack definition (i.e., in every directory under our &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;modules/stacks&lt;/code&gt;), leading to a lot of duplication. Second, you cannot use interpolation in the backend block to configure it using variables, locals, or data source attributes. A lack of interpolation can be problematic if you want to use different buckets for different environments for example, since we have multiple deployments (e.g., &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;deployments/app/dev/test-stack&lt;/code&gt;, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;deployments/app/test/test-stack&lt;/code&gt;) sharing the same &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;modules/stack/app/test-stack&lt;/code&gt; files.&lt;/p&gt;

&lt;p&gt;Terraform provides several ways to work around these limitations through &lt;a href=&quot;https://www.terraform.io/docs/configuration/backend.html#partial-configuration&quot;&gt;partial configuration&lt;/a&gt;. This allows you to omit properties from the backend block and instead provide them through an out-of-band mechanism such as a configuration file, a command-line option, or interactively on the command-line. With partial configuration, you could, for example, omit the bucket property in the configuration above, and instead pass it through one of the mentioned mechanisms to achieve different buckets for different environments.&lt;/p&gt;

&lt;p&gt;Partial configuration, although a step in the right direction, still requires you to figure out how you’re going to manage the backend configuration properties. If using configuration files, where will those be stored and distributed to the team? How will you ensure every terraform command receives the necessary command-line properties with the correct values? This more or less forces you to use a build tool for your terraform commands (e.g., a makefile) to achieve consistency/reproducibility. While a build tool isn’t necessarily a bad thing, it strikes me as unusual to require it for remote state, a seemingly basic and fundamental capability.&lt;/p&gt;

&lt;p&gt;Fortunately, terragrunt gives us a way to make remote state usage easy for anyone on the team.&lt;/p&gt;

&lt;h1 id=&quot;backend-configuration-with-terragrunt&quot;&gt;Backend Configuration with Terragrunt&lt;/h1&gt;

&lt;p&gt;With terragrunt, you can configure your backend within a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;terragrunt.hcl&lt;/code&gt;, or in our case &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;root.hcl&lt;/code&gt;. Terragrunt then generates the necessary terraform backend configuration based on what you specify. Configuring remote state inside &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;root.hcl&lt;/code&gt; means every deployment will receive the backend configuration by way of terragrunt’s include mechanism. This removes all duplication of remote state configuration from the stacks.&lt;/p&gt;

&lt;p&gt;Additionally, the remote state block supports interpolation, allowing each deployment to share common parts of the remote state configuration while having other parts be unique. For instance, every deployment can share the same bucket for their state files but use a different prefix within that bucket.&lt;/p&gt;

&lt;p&gt;Using terragrunt’s remote state configuration is &lt;a href=&quot;https://terragrunt.gruntwork.io/docs/features/keep-your-remote-state-configuration-dry/&quot;&gt;well documented by Gruntwork&lt;/a&gt;. Here’s how we can add it to our skeleton.&lt;/p&gt;

&lt;h2 id=&quot;implementation&quot;&gt;Implementation&lt;/h2&gt;

&lt;p&gt;Add a &lt;a href=&quot;https://terragrunt.gruntwork.io/docs/reference/config-blocks-and-attributes/#remote_state&quot;&gt;remote_state&lt;/a&gt; block to our &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;root.hcl&lt;/code&gt; to generate an s3 backend configuration:&lt;/p&gt;

&lt;div class=&quot;language-diff highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;gd&quot;&gt;--- a/deployments/root.hcl
&lt;/span&gt;&lt;span class=&quot;gi&quot;&gt;+++ b/deployments/root.hcl
&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;@@ -39,6 +40,24 @@&lt;/span&gt; locals {
 # environment variables
 inputs = local.merged_config

+remote_state {
&lt;span class=&quot;gi&quot;&gt;+  backend = &quot;s3&quot;
+  generate = {
+    path      = &quot;backend.tf&quot;
+    if_exists = &quot;overwrite&quot;
+  }
+  config = {
+    bucket  = &quot;terraform-skeleton-state&quot;
+    region  = &quot;us-east-1&quot;
+    encrypt = true
+
+    key = &quot;${dirname(local.relative_deployment_path)}/${local.stack}.tfstate&quot;
+
+    dynamodb_table            = &quot;terraform-skeleton-state-locks&quot;
+    accesslogging_bucket_name = &quot;terraform-skeleton-state-logs&quot;
+  }
+}
+
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;This approach uses:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;A single bucket, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;terraform-skeleton-state&lt;/code&gt;, for all deployment state files&lt;/li&gt;
  &lt;li&gt;A key/prefix unique to each deployment based on the relative path from the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;root.hcl&lt;/code&gt; file&lt;/li&gt;
  &lt;li&gt;A corresponding DynamoDB lock table, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;terraform-skeleton-state-locks&lt;/code&gt;&lt;/li&gt;
  &lt;li&gt;A logging bucket, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;terraform-skeleton-state-logs&lt;/code&gt;, for logging all S3 access requests to the state bucket&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;how-this-works&quot;&gt;How This Works&lt;/h2&gt;

&lt;p&gt;Terragrunt executes terraform commands from a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;.terragrunt-cache&lt;/code&gt; directory. Before executing terraform, terragrunt populates the cache with:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;Any files in the current deployment directory (location of your &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;terragrunt.hcl&lt;/code&gt; file)&lt;/li&gt;
  &lt;li&gt;The stack files in the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;terraform.source&lt;/code&gt; directory specified in our &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;root.hcl&lt;/code&gt;&lt;/li&gt;
  &lt;li&gt;Files from &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;generate&lt;/code&gt; blocks defined in the HCL files&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The last step above is what translates the generate block defined in &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;root.hcl&lt;/code&gt; to a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;backend.tf&lt;/code&gt; file for each stack containing the appropriate terraform backend configuration.&lt;sup id=&quot;fnref:2&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:2&quot; class=&quot;footnote&quot;&gt;2&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

&lt;h2 id=&quot;buckettable-initialization&quot;&gt;Bucket/Table Initialization&lt;/h2&gt;

&lt;p&gt;By using the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;remote_state&lt;/code&gt; block in our HCL files, we allow terragrunt to manage the creation of the S3 bucket and DynamoDB table for us.&lt;/p&gt;

&lt;p&gt;Terragrunt will:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Automatically create both buckets and/or lock table for you if it does not exist&lt;/li&gt;
  &lt;li&gt;Configure the state bucket (but not the logs bucket) with
    &lt;ul&gt;
      &lt;li&gt;Versioning and encryption enabled&lt;/li&gt;
      &lt;li&gt;Public access disabled&lt;/li&gt;
      &lt;li&gt;Enforcing TLS-only access&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
  &lt;li&gt;Enable encryption on the lock table&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&quot;activating-remote-state&quot;&gt;Activating Remote State&lt;/h2&gt;

&lt;p&gt;Switching from local state to remote state requires running &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;terragrunt init&lt;/code&gt; on each deployment.&lt;/p&gt;

&lt;p&gt;The first init will prompt you to create the buckets and/or dynamo table if they don’t exist:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;Initializing remote state for the s3 backend
Remote state S3 bucket terraform-skeleton-state does not
exist or you don't have permissions to access it.
Would you like Terragrunt to create it? (y/n)
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Every init will then prompt you to copy your existing local state to the new remote backend:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;[terragrunt] 2020/12/13 10:04:56 Initializing remote state for the s3 backend
[terragrunt] 2020/12/13 10:04:57 Running command: terraform init

Initializing the backend...
Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous &quot;local&quot; backend to the
  newly configured &quot;s3&quot; backend. No existing state was found in the newly
  configured &quot;s3&quot; backend. Do you want to copy this state to the new &quot;s3&quot;
  backend? Enter &quot;yes&quot; to copy and &quot;no&quot; to start with an empty state.

  Enter a value:
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Finally, although this shouldn’t be necessary, I’ve had to &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;rm -rf&lt;/code&gt; the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;.terragrunt-cache&lt;/code&gt; directory inside each deployment for terragrunt commands to work after the migration.&lt;/p&gt;

&lt;p&gt;After activating remote state for all deployments, we can list our state bucket and see the state files nicely organized:&lt;/p&gt;

&lt;div class=&quot;language-shell highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;➜ aws s3 &lt;span class=&quot;nb&quot;&gt;ls&lt;/span&gt; &lt;span class=&quot;nt&quot;&gt;--recursive&lt;/span&gt; terraform-skeleton-state
2020-12-12 07:38:39       1154 app/dev/test-stack.tfstate
2020-12-13 10:22:49       1148 app/prod/test-stack.tfstate
2020-12-13 10:22:34       1154 app/stage/test-stack.tfstate
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;h2 id=&quot;limitations&quot;&gt;Limitations&lt;/h2&gt;

&lt;p&gt;Using terragrunt’s &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;remote_state&lt;/code&gt; block has several advantages:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;It’s easy&lt;/li&gt;
  &lt;li&gt;It includes sensible security defaults for the state bucket and lock table&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;While it is good enough for today’s skeleton, there are some limitations of the terragrunt approach worth covering:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;
    &lt;p&gt;Terragrunt lacks security defaults on the log bucket&lt;/p&gt;

    &lt;p&gt;If terragrunt creates the log bucket, it will not have encryption enabled and it will not have public access explicitly blocked. For this reason, you should strongly consider self-management of the log bucket. I tend to do this with a CloudFormation stack.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Disabling auto-creation of the state bucket and lock table is broken&lt;sup id=&quot;fnref:3&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:3&quot; class=&quot;footnote&quot;&gt;3&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

    &lt;p&gt;For a single team owning all infrastructure auto-creation is likely safe enough. Once I start distributing ownership of pieces of infrastructure to different teams, I want to control where state files are stored as much as possible, which means avoiding a situation where a team accidentally creates a new state bucket or lock table. Consistent state file storage makes it easier to audit infrastructure for compliance with team and organization standards using tools like &lt;a href=&quot;https://terraform-compliance.com/&quot;&gt;terraform-compliance&lt;/a&gt;. Easier auditing in turn makes it more palatable to push down infrastructure ownership responsibilities.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Terragrunt doesn’t offer full control over the credentials used to access the terraform state&lt;/p&gt;

    &lt;p&gt;The &lt;a href=&quot;https://terragrunt.gruntwork.io/docs/reference/config-blocks-and-attributes/#remote_state&quot;&gt;remote_state&lt;/a&gt; block has a few fields for specifying a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;role_arn&lt;/code&gt; or AWS &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;profile&lt;/code&gt; to use for remote state access, but you can’t control things like &lt;a href=&quot;https://docs.aws.amazon.com/IAM/latest/UserGuide/id_session-tags.html&quot;&gt;IAM session tags&lt;/a&gt;, transitive tag usage, or assume role policies.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Terragrunt doesn’t offer full control over all fields on the buckets and table&lt;/p&gt;

    &lt;p&gt;While terragrunt applies sensible security defaults, you can’t control everything using its &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;remote_state&lt;/code&gt; block. For example, you’ll need to self-manage if you need to specify specific KMS keys for encryption of the bucket or table. Similarly, if you want to set up replication of your terraform state bucket, self-management is the way to go.&lt;/p&gt;
  &lt;/li&gt;
&lt;/ol&gt;

&lt;h1 id=&quot;whats-next&quot;&gt;What’s next?&lt;/h1&gt;

&lt;p&gt;The skeleton now supports multiple developers working on the infrastructure as a team, sharing a state file stored in S3 with contention resolved through a DynamoDB lock table. State is encrypted and versioned, and access to it is logged. Remote state is configured once, in the root.hcl, and reused across our stacks.&lt;/p&gt;

&lt;p&gt;Thus far, terragrunt has been running with whatever AWS credentials were configured in the shell at the time of execution.&lt;/p&gt;

&lt;p&gt;This isn’t ideal in a team setting because:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;It could lead to “works on my machine problems” if developers have different configurations&lt;/li&gt;
  &lt;li&gt;It encourages running terraform with administrative-level permissions all the time for all developers working with it&lt;/li&gt;
  &lt;li&gt;Terraform state is sensitive and should be protected from modification except by a well-defined set of roles&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The next entry in the &lt;a href=&quot;/series/tf-skeleton.html&quot;&gt;terraform skeleton series&lt;/a&gt; will take a first step towards addressing these issues.&lt;/p&gt;

&lt;h1 id=&quot;footnotes&quot;&gt;Footnotes&lt;/h1&gt;

&lt;div class=&quot;footnotes&quot; role=&quot;doc-endnotes&quot;&gt;
  &lt;ol&gt;
    &lt;li id=&quot;fn:1&quot; role=&quot;doc-endnote&quot;&gt;
      &lt;p&gt;Terragrunt stores local state in a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;terraform.tfstate&lt;/code&gt; file located underneath the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;.terragrunt-cache&lt;/code&gt; directory it creates within our deployment directory. &lt;a href=&quot;#fnref:1&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:2&quot; role=&quot;doc-endnote&quot;&gt;

      &lt;p&gt;The terragrunt remote state approach we cover here uses the newer &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;generate&lt;/code&gt; option, which was &lt;a href=&quot;https://github.com/gruntwork-io/terragrunt/pull/1050&quot;&gt;added&lt;/a&gt; in terragrunt &lt;a href=&quot;https://github.com/gruntwork-io/terragrunt/releases/tag/v0.22.0&quot;&gt;v0.22.0&lt;/a&gt;. Before that version, terragrunt wouldn’t generate a backend.tf file but would instead pass CLI arguments to the terraform command, making use of terraform’s partial configuration implementation. The generate approach has several advantages over the old method. The most notable is that previously, you had to include an empty backend configuration block in every stack, whereas with the generate approach you do not. &lt;a href=&quot;#fnref:2&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:3&quot; role=&quot;doc-endnote&quot;&gt;
      &lt;p&gt;Terragrunt has a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;disable_init&lt;/code&gt; attribute on the &lt;a href=&quot;https://terragrunt.gruntwork.io/docs/reference/config-blocks-and-attributes/#remote_state&quot;&gt;remote_state&lt;/a&gt; block, which will prevent auto-creation but, as described by this &lt;a href=&quot;https://github.com/gruntwork-io/terragrunt/issues/1422&quot;&gt;open issue&lt;/a&gt;, also completely disables backend initialization. &lt;a href=&quot;#fnref:3&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
  &lt;/ol&gt;
&lt;/div&gt;</content><author><name>Chris</name></author><category term="tf-skeleton" /><category term="terraform" /><category term="terragrunt" /><summary type="html">Terraform uses state files to track the resources it creates back to resource definitions in your *.tf files. Each deployment has its own state. State is stored according to the backend configured for the deployment. Terraform uses a local backend for storing state on the local filesystem by default, which is what we’ve been using for part 1 and part 2 of the terraform skeleton series.1 This works fine for a simple demonstration but is insufficient for production use because: Terragrunt stores local state in a terraform.tfstate file located underneath the .terragrunt-cache directory it creates within our deployment directory. &amp;#8617;</summary></entry><entry><title type="html">Terraform Skeleton Part 2: Variables</title><link href="https://thirstydeveloper.io/tf-skeleton/2021/01/23/part-2-variables.html" rel="alternate" type="text/html" title="Terraform Skeleton Part 2: Variables" /><published>2021-01-23T10:53:09+00:00</published><updated>2021-01-23T10:53:09+00:00</updated><id>https://thirstydeveloper.io/tf-skeleton/2021/01/23/part-2-variables</id><content type="html" xml:base="https://thirstydeveloper.io/tf-skeleton/2021/01/23/part-2-variables.html">&lt;p&gt;In &lt;a href=&quot;/2021/01/17/part-1-organizing-terragrunt.html&quot;&gt;part 1&lt;/a&gt; of the &lt;a href=&quot;/series/tf-skeleton.html&quot;&gt;terraform skeleton series&lt;/a&gt;, we set up a &lt;a href=&quot;https://github.com/thirstydeveloper/terraform-terragrunt-skeleton/tree/release/1.0&quot;&gt;terraform repository&lt;/a&gt; that allows the team to apply infrastructure at any level: from individual stacks to entire environments. We build on that foundation in this post, adding a variable hierarchy that similarly allows definition and overriding of variables at each level of the infrastructure.&lt;/p&gt;

&lt;h1 id=&quot;goals&quot;&gt;Goals&lt;/h1&gt;

&lt;p&gt;As a refresher, our infrastructure is organized by:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Tier&lt;/li&gt;
  &lt;li&gt;Environment&lt;/li&gt;
  &lt;li&gt;Layer (Optional)&lt;/li&gt;
  &lt;li&gt;Stack&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A &lt;strong&gt;deployment&lt;/strong&gt; is an instantiation of a stack for a tier-environment.&lt;/p&gt;

&lt;p&gt;Today’s goals are to:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Allow defining variable values in files at each level of the infrastructure&lt;/li&gt;
  &lt;li&gt;Have lower-level variable definitions override higher levels&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you prefer to jump to the end, the code implementing the final result is available on &lt;a href=&quot;https://github.com/thirstydeveloper/terraform-terragrunt-skeleton/tree/release/1.1&quot;&gt;GitHub&lt;/a&gt;.&lt;/p&gt;

&lt;h1 id=&quot;setup&quot;&gt;Setup&lt;/h1&gt;

&lt;p&gt;Let’s add to our test-stack some variables to be defined at each level of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;deployments/&lt;/code&gt;:&lt;/p&gt;

&lt;div class=&quot;language-terraform highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;c1&quot;&gt;# modules/stacks/app/test-stack/variables.tf&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;variable&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;global_var&quot;&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
  &lt;span class=&quot;nx&quot;&gt;default&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;unset&quot;&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;

&lt;span class=&quot;k&quot;&gt;variable&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;tier_var&quot;&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
  &lt;span class=&quot;nx&quot;&gt;default&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;unset&quot;&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;

&lt;span class=&quot;k&quot;&gt;variable&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;env_var&quot;&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
  &lt;span class=&quot;nx&quot;&gt;default&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;unset&quot;&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;

&lt;span class=&quot;k&quot;&gt;variable&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;layer_var&quot;&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
  &lt;span class=&quot;nx&quot;&gt;default&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;unset&quot;&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;

&lt;span class=&quot;k&quot;&gt;variable&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;stack_var&quot;&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
  &lt;span class=&quot;nx&quot;&gt;default&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;unset&quot;&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Let’s also add outputs to the stack so we can see what the variables are set to:&lt;/p&gt;

&lt;div class=&quot;language-terraform highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;c1&quot;&gt;# modules/stacks/app/test-stack/outputs.tf&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;output&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;pet&quot;&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
  &lt;span class=&quot;nx&quot;&gt;value&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;nx&quot;&gt;random_pet&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nx&quot;&gt;pet&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nx&quot;&gt;id&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;

&lt;span class=&quot;k&quot;&gt;output&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;global_var&quot;&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
  &lt;span class=&quot;nx&quot;&gt;value&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;kd&quot;&gt;var&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nx&quot;&gt;global_var&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;

&lt;span class=&quot;k&quot;&gt;output&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;tier_var&quot;&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
  &lt;span class=&quot;nx&quot;&gt;value&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;kd&quot;&gt;var&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nx&quot;&gt;tier_var&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;

&lt;span class=&quot;k&quot;&gt;output&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;env_var&quot;&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
  &lt;span class=&quot;nx&quot;&gt;value&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;kd&quot;&gt;var&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nx&quot;&gt;env_var&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;

&lt;span class=&quot;k&quot;&gt;output&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;layer_var&quot;&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
  &lt;span class=&quot;nx&quot;&gt;value&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;kd&quot;&gt;var&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nx&quot;&gt;layer_var&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;

&lt;span class=&quot;k&quot;&gt;output&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;stack_var&quot;&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
  &lt;span class=&quot;nx&quot;&gt;value&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;kd&quot;&gt;var&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nx&quot;&gt;stack_var&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Next, we’ll cover implementing the variable hierarchy in two ways. The first works for terraform pre-0.12. The second slightly modifies our first approach to work for 0.12+.&lt;/p&gt;

&lt;h1 id=&quot;variables-via-tfvars-pre-012&quot;&gt;Variables via tfvars (pre-0.12)&lt;/h1&gt;

&lt;p&gt;&lt;a href=&quot;https://www.terraform.io/docs/configuration/variables.html#variable-definitions-tfvars-files&quot;&gt;tfvars files&lt;/a&gt; are a standard way of defining variable values for terraform. Terragrunt allows you to define &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;required_var_files&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;optional_var_files&lt;/code&gt; within the &lt;a href=&quot;https://terragrunt.gruntwork.io/docs/reference/config-blocks-and-attributes/#terraform&quot;&gt;terraform block&lt;/a&gt; of your &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;terragrunt.hcl&lt;/code&gt;, as covered by this &lt;a href=&quot;https://blog.gruntwork.io/terragrunt-how-to-keep-your-terraform-code-dry-and-maintainable-f61ae06959d8#d7f6&quot;&gt;Gruntwork post&lt;/a&gt;. Terragrunt then passes these files to terraform using its &lt;a href=&quot;https://www.terraform.io/docs/commands/plan.html#var-file-foo&quot;&gt;-var-file&lt;/a&gt; option. Terraform loads the files and sets the variable values, with later files overriding previous files.&lt;/p&gt;

&lt;p&gt;We can use this to implement a tfvars hierarchy. The first step is defining a tfvars file in each level of our deployments directory structure:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;deployments/
    app/
        dev/
            test-stack/
                terraform.tfvars
                terragrunt.hcl
        stage/
            test-stack/
                terraform.tfvars
                terragrunt.hcl
        prod/
            test-stack/
                terraform.tfvars
                terragrunt.hcl
            terraform.tfvars
        terraform.tfvars
    terraform.tfvars
    root.hcl
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Next, define values at each level:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;# deployments/terraform.tfvars
global_var = &quot;set in deployments/&quot;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;# deployments/app/terraform.tfvars
tier_var = &quot;set in deployments/app/&quot;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;# deployments/app/dev/terraform.tfvars
env_var = &quot;set in deployments/app/dev&quot;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;# deployments/app/dev/test-stack/terraform.tfvars
stack_var = &quot;set in deployments/app/dev/test-stack/&quot;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;and so on for &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;deployments/app/test/**&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;deployments/app/prod/**&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Next, we modify our &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;root.hcl&lt;/code&gt; file to load all the tfvars files. We make use of the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;deployment_path_components&lt;/code&gt; local we defined in &lt;a href=&quot;/2021/01/17/part-1-organizing-terragrunt.html&quot;&gt;part 1&lt;/a&gt; to generate a list of all possible tfvar locations:&lt;/p&gt;

&lt;pre&gt;&lt;code class=&quot;language-HCL&quot;&gt;# root.hcl
locals {
  relative_deployment_path   = path_relative_to_include()
  deployment_path_components = compact(
    split(&quot;/&quot;, local.relative_deployment_path)
  )

  ...

  # Get a list of every possible tfvars path between root_deployments_directory
  # and the path of the deployment
  possible_config_locations = [
    for i in range(0, length(local.deployment_path_components) + 1) :
      join(&quot;/&quot;, concat(
        [local.root_deployments_dir],
        slice(local.deployment_path_components, 0, i),
        [&quot;terraform.tfvars&quot;]
      ))
  ]
}
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;Finally, instruct terragrunt to pass the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;possible_config_locations&lt;/code&gt; as &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;optional_var_files&lt;/code&gt; to terraform under an &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;extra_arguments&lt;/code&gt; block:&lt;/p&gt;

&lt;pre&gt;&lt;code class=&quot;language-HCL&quot;&gt;# root.hcl
...

terraform {
  ...
  extra_arguments &quot;load_config_files&quot; {
    commands = get_terraform_commands_that_need_vars()
    optional_var_files = local.possible_config_locations
  }
}
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;Having done this, we can run plan and apply and see the variables are loaded. For instance, running &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;terragrunt apply&lt;/code&gt; from &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;deployments/app/dev/test-stack&lt;/code&gt; produces:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

env_var = set in deployments/app/dev
global_var = overridden in deployments/app
layer_var = unset
stack_var = set in deployments/app/dev/test-stack/
tier_var = set in deployments/app/
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;We can also override values in lower levels. For instance, changing the value of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;global_var&lt;/code&gt; in &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;deployments/app/terraform.tfvars&lt;/code&gt;:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;# deployments/app/terraform.tfvars
global_var = &quot;overridden in deployments/app&quot;
tier_var   = &quot;set in deployments/app/&quot;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;and running &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;terragrunt output global_var&lt;/code&gt; in &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;deployments/app/terraform.tfvars&lt;/code&gt; produces:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;overridden in deployments/app
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;h1 id=&quot;where-tfvars-fail&quot;&gt;Where tfvars fail&lt;/h1&gt;

&lt;p&gt;The above worked well until terraform 0.12, which introduced a &lt;a href=&quot;https://github.com/hashicorp/terraform/issues/22004&quot;&gt;controversial feature&lt;/a&gt; to print warnings when values are specified for undefined variables. For instance, if we add an unused variable to &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;deployments/terraform.tfvars&lt;/code&gt;:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;# deployments/terraform.tfvars
global_var = &quot;set-in-deployments/&quot;
unused     = true
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;any stack we &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;plan&lt;/code&gt; or &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;apply&lt;/code&gt; now prints:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;Warning: Value for undeclared variable

The root module does not declare a variable named &quot;unused&quot;
but a value was found in file
&quot;/Users/td/code/td/terraform-skeleton/deployments/terraform.tfvars&quot;.
To use this value, add a &quot;variable&quot; block to the configuration.
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;The warnings are annoying enough in that they clutter the plan output, but worse still the warning states:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;Using a variables file to set an undeclared variable is deprecated and will
become an error in a future release.
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;That means we can’t reliably use tfvar files as our source of a variable hierarchy for the long term, at least if you have any variables defined in tfvars that are loaded and go unused by any stacks.&lt;/p&gt;

&lt;p&gt;You might wonder if this is actually a problem. &lt;a href=&quot;https://github.com/hashicorp/terraform/blob/v0.12/CHANGELOG.md#incompatibilities-and-notes&quot;&gt;Hashicorp states&lt;/a&gt; the change was made “to give better feedback about mistakes”, so maybe we should just avoid having variables defined in tfvars files that are loaded by stacks that do not use them.&lt;/p&gt;

&lt;p&gt;Allow me to present an example where unused variables are helpful.&lt;/p&gt;

&lt;h2 id=&quot;unused-variables-a-use-case&quot;&gt;Unused variables: a use case&lt;/h2&gt;

&lt;p&gt;My team has several global flags for our infrastructure. An example is one used to conditionally enable or disable access to our system for a third-party team. Any stack that must make changes to facilitate that access defines an &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;enable_third_party_access&lt;/code&gt; variable in its &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;variables.tf&lt;/code&gt; file. If true, that stack makes the necessary changes.&lt;/p&gt;

&lt;p&gt;Multiple stacks defining this variable poses a challenge. When third-party access is requested, we don’t want to have to find every deployment of every stack with this variable and set its value to &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;true&lt;/code&gt;. We want to define the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;enable_third_party_access&lt;/code&gt; once, either across all environments or for specific environments, so we don’t forget to enable or disable individual flags. Furthermore, not every stack needs to enable third-party access, and therefore not every stack needs this variable.&lt;/p&gt;

&lt;p&gt;Leveraging unused variables means we can set the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;enable_third_party_access&lt;/code&gt; value in fewer places, leading to fewer mistakes.&lt;/p&gt;

&lt;p&gt;There are plenty more examples of how unused variables are useful in the &lt;a href=&quot;https://github.com/hashicorp/terraform/issues/22004&quot;&gt;Hashicorp issue discussion&lt;/a&gt;. So how do we support unused variables in 0.12 and beyond?&lt;/p&gt;

&lt;h1 id=&quot;variables-via-yaml-012&quot;&gt;Variables via YAML (0.12+)&lt;/h1&gt;

&lt;p&gt;Fortunately, Hashicorp does give you a workaround for defining values for unused variables, also displayed in the warning messages they print:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;If you wish to provide certain &quot;global&quot; settings to all configurations in
your organization, use TF_VAR_... environment variables to set these instead.
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;So instead of using tfvar files, we need to define environment variables. But how can we do that easily and  reproducibly across systems? Terragrunt again comes to the rescue.&lt;/p&gt;

&lt;p&gt;We can pass an HCL map to an &lt;a href=&quot;https://terragrunt.gruntwork.io/docs/features/inputs/&quot;&gt;inputs&lt;/a&gt; attribute in our &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;root.hcl&lt;/code&gt;, which terragrunt converts to TF_VAR environment variables passed to terraform. Now we just need a map of our variables and their values. There’s no easy way to generate such a map from tfvars files, but terragrunt &lt;a href=&quot;https://terragrunt.gruntwork.io/docs/reference/built-in-functions/#terraform-built-in-functions&quot;&gt;provides access&lt;/a&gt; to all of &lt;a href=&quot;https://www.terraform.io/docs/configuration/functions.html&quot;&gt;terraform’s functions&lt;/a&gt; in the HCL file. This gives us the building blocks we need to use YAML files instead of tfvars to build our variable hierarchy. Specifically, we’ll use:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;a href=&quot;https://www.terraform.io/docs/configuration/functions/fileexists.html&quot;&gt;fileexists&lt;/a&gt; - for checking if a file exists before attempting to load it&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;https://www.terraform.io/docs/configuration/functions/file.html&quot;&gt;file&lt;/a&gt; - for loading file contents to a string&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;https://www.terraform.io/docs/configuration/functions/yamldecode.html&quot;&gt;yamldecode&lt;/a&gt; - for converting a YAML string to an HCL map&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;https://www.terraform.io/docs/configuration/functions/merge.html&quot;&gt;merge&lt;/a&gt; - for merging multiple HCL maps&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Start by converting the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;terraform.tfvars&lt;/code&gt; files to &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;config.yml&lt;/code&gt; files&lt;sup id=&quot;fnref:1&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:1&quot; class=&quot;footnote&quot;&gt;1&lt;/a&gt;&lt;/sup&gt;. For instance, convert:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;# deployments/terraform.tfvars
global_var = &quot;set-in-deployments/&quot;
unused     = true
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;to:&lt;/p&gt;

&lt;pre&gt;&lt;code class=&quot;language-YAML&quot;&gt;# deployments/config.yml
---
global_var: &quot;set-in-deployments/&quot;
unused: true
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;Next, modify &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;root.hcl&lt;/code&gt; to load the YAML files, first by removing the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;extra_arguments&lt;/code&gt; block we used for loading the tfvars:&lt;/p&gt;

&lt;pre&gt;&lt;code class=&quot;language-HCL&quot;&gt;# root.hcl
...
terraform {
  source = &quot;${local.root_deployments_dir}/../modules/stacks/${local.tier}/${local.stack}&quot;
  # no extra_arguments block here
}
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;then by changing the locals to load the YAML files and pass them into the terragrunt inputs:&lt;/p&gt;

&lt;pre&gt;&lt;code class=&quot;language-HCL&quot;&gt;# root.hcl
locals {
  ...

  # Get a list of every path between root_deployments_directory and the path of
  # the deployment
  possible_config_dirs = [
    for i in range(0, length(local.deployment_path_components) + 1) :
      join(&quot;/&quot;, concat(
        [local.root_deployments_dir],
        slice(local.deployment_path_components, 0, i)
      ))
  ]

  # Generate a list of possible config files at every possible_config_dir
  # (support both .yml and .yaml)
  possible_config_paths = flatten([
    for dir in local.possible_config_dirs : [
      &quot;${dir}/config.yml&quot;,
      &quot;${dir}/config.yaml&quot;
    ]
  ])

  # Load every YAML config file that exists into an HCL map
  file_configs = [
    for path in local.possible_config_paths :
      yamldecode(file(path)) if fileexists(path)
  ]

  # Merge the maps together, with deeper configs overriding higher configs
  merged_config = merge(local.file_configs...)
}

# Pass the merged config to terraform as variable values using TF_VAR_
# environment variables
inputs = local.merged_config

...
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;And that’s it. We have a YAML-based variable hierarchy that supports overriding and unused variables. The final result is available on &lt;a href=&quot;https://github.com/thirstydeveloper/terraform-terragrunt-skeleton/tree/release/1.1&quot;&gt;GitHub&lt;/a&gt;&lt;/p&gt;

&lt;h1 id=&quot;whats-next&quot;&gt;What’s next?&lt;/h1&gt;

&lt;p&gt;So far, the skeleton has been using the default &lt;a href=&quot;https://www.terraform.io/docs/backends/types/local.html&quot;&gt;local backend&lt;/a&gt; for state storage. For this project to work in a team setting with multiple developers and/or CI/CD performing terraform commands, we will need to store terraform state remotely with locking to avoid concurrent executions from stepping on each other. Setting up remote state storage will be the subject of the next post.&lt;/p&gt;

&lt;h1 id=&quot;footnotes&quot;&gt;Footnotes&lt;/h1&gt;

&lt;div class=&quot;footnotes&quot; role=&quot;doc-endnotes&quot;&gt;
  &lt;ol&gt;
    &lt;li id=&quot;fn:1&quot; role=&quot;doc-endnote&quot;&gt;

      &lt;p&gt;The YAML loading doesn’t play nice with config.yml files that are empty or contain just a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;---&lt;/code&gt; start of document marker. You can delete the empty &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;config.yml&lt;/code&gt; file and everything will work, due to the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;fileexists&lt;/code&gt; check. If you prefer to keep the empty config.yml, the most minimal contents required are:&lt;/p&gt;

      &lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;---
{}
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;      &lt;/div&gt;
      &lt;p&gt;&lt;a href=&quot;#fnref:1&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
  &lt;/ol&gt;
&lt;/div&gt;</content><author><name>Chris</name></author><category term="tf-skeleton" /><category term="terraform" /><category term="terragrunt" /><summary type="html">In part 1 of the terraform skeleton series, we set up a terraform repository that allows the team to apply infrastructure at any level: from individual stacks to entire environments. We build on that foundation in this post, adding a variable hierarchy that similarly allows definition and overriding of variables at each level of the infrastructure.</summary></entry><entry><title type="html">Terraform Skeleton Part 1: Organizing Terragrunt</title><link href="https://thirstydeveloper.io/2021/01/17/part-1-organizing-terragrunt.html" rel="alternate" type="text/html" title="Terraform Skeleton Part 1: Organizing Terragrunt" /><published>2021-01-17T12:23:00+00:00</published><updated>2021-01-17T12:23:00+00:00</updated><id>https://thirstydeveloper.io/2021/01/17/part-1-organizing-terragrunt</id><content type="html" xml:base="https://thirstydeveloper.io/2021/01/17/part-1-organizing-terragrunt.html">&lt;p&gt;&lt;a href=&quot;https://www.terraform.io/&quot;&gt;Terraform&lt;/a&gt; is my go-to infrastructure definition tool. I love that it enables declarative management of different Cloud, PaaS, and SaaS platforms with its unified HCL language and provider model. At times, however, I’ve wished it was easier to start a project off on the right foot.&lt;/p&gt;

&lt;p&gt;When I was getting started, it wasn’t obvious to me how to organize a terraform project in a way that makes “the easy things easy and the hard things possible” and would scale well as a team grows. &lt;a href=&quot;https://terragrunt.gruntwork.io/&quot;&gt;Terragrunt&lt;/a&gt; helps a great deal, but there are many configuration options, adding to the organizational complexity.&lt;/p&gt;

&lt;p&gt;This blog series takes lessons learned from using terraform and terragrunt in production across multiple teams and, over many posts, condenses those lessons into a &lt;a href=&quot;https://github.com/thirstydeveloper/terraform-terragrunt-skeleton&quot;&gt;walking skeleton repo&lt;/a&gt; to serve as a starting point for teams working with terraform and terragrunt. I’ve used this skeleton’s approach for a moderately complex suite of applications spanning multiple clouds and multiple Kubernetes clusters. I’m also using it with a smaller team running a much simpler monolith application. I believe it is a solid starting point, worth sharing with others for consideration.&lt;/p&gt;

&lt;p&gt;I’m using semantic versioning to track changes to the skeleton repo. This post covers major-minor version 1.0. The code this post arrives at is available on the release branch for that version &lt;a href=&quot;https://github.com/thirstydeveloper/terraform-terragrunt-skeleton/tree/release/1.0&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;

&lt;h1 id=&quot;goals&quot;&gt;Goals&lt;/h1&gt;

&lt;ol&gt;
  &lt;li&gt;Version control all infrastructure as code&lt;/li&gt;
  &lt;li&gt;Provide a basic safety net of explicit tool versions and pre-commit checks&lt;/li&gt;
  &lt;li&gt;Re-use the same infrastructure definition files across multiple environments&lt;/li&gt;
  &lt;li&gt;Have multiple levels of blast radiuses; allow the engineer to easily apply a small stack of infrastructure or an entire environment&lt;/li&gt;
  &lt;li&gt;Focus on manual infrastructure deployment for now. Continuous Integration will follow. Continuous Delivery will be case-by-case.&lt;sup id=&quot;fnref:1&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:1&quot; class=&quot;footnote&quot;&gt;1&lt;/a&gt;&lt;/sup&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;h1 id=&quot;setup&quot;&gt;Setup&lt;/h1&gt;

&lt;p&gt;You’ll need the following installed on your workstation:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;a href=&quot;https://git-scm.com/&quot;&gt;git&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;https://github.com/tfutils/tfenv&quot;&gt;tfenv&lt;/a&gt; for managing terraform versions&lt;sup id=&quot;fnref:2&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:2&quot; class=&quot;footnote&quot;&gt;2&lt;/a&gt;&lt;/sup&gt;&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;https://github.com/cunymatthieu/tgenv&quot;&gt;tgenv&lt;/a&gt; for managing terragrunt versions&lt;/li&gt;
  &lt;li&gt;&lt;a href=&quot;https://pre-commit.com/&quot;&gt;pre-commit&lt;/a&gt; for running syntax, semantic, and style checks on &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;git commit&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h1 id=&quot;the-infra-repo&quot;&gt;The ‘infra’ repo&lt;/h1&gt;

&lt;p&gt;I currently start teams off with a monorepo structure for terraform infrastructure. I admit that having one or more separate modules repositories, as &lt;a href=&quot;https://www.terraformupandrunning.com/&quot;&gt;Terraform Up &amp;amp; Running&lt;/a&gt; advises, is a good destination, but my teams have found it easier to get started making changes within a single repository. The organization that follows works both with mono and distributed repositories, however. Teams I support have made the jump from monorepo to distributed, and it was not painful.&lt;sup id=&quot;fnref:3&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:3&quot; class=&quot;footnote&quot;&gt;3&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

&lt;p&gt;Our team calls our monorepo &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;infra&lt;/code&gt;, and that’s the name I will use here.&lt;/p&gt;

&lt;h2 id=&quot;step-1-boilerplate&quot;&gt;Step 1: Boilerplate&lt;/h2&gt;

&lt;p&gt;After installing the tools listed above and creating a new &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;infra&lt;/code&gt; repository:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;
    &lt;p&gt;Create a CHANGELOG.md&lt;/p&gt;

    &lt;p&gt;I like the style of &lt;a href=&quot;https://keepachangelog.com/en/1.0.0/&quot;&gt;keepachangelog.com&lt;/a&gt;.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Add a .gitignore&lt;/p&gt;

    &lt;p&gt;I use &lt;a href=&quot;https://www.toptal.com/developers/gitignore&quot;&gt;gitignore.io&lt;/a&gt; as a &lt;a href=&quot;https://www.toptal.com/developers/gitignore/api/terraform,terragrunt&quot;&gt;starting point&lt;/a&gt;.&lt;/p&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Set tool versions&lt;/p&gt;

    &lt;p&gt;For instance, if using terraform 0.13.5 and terragrunt 0.26.4 set the versions with:&lt;/p&gt;

    &lt;div class=&quot;language-bash highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt; &lt;span class=&quot;nb&quot;&gt;echo&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;0.13.5&quot;&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;&amp;gt;&lt;/span&gt; .terraform-version
 &lt;span class=&quot;nb&quot;&gt;echo&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;0.26.4&quot;&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;&amp;gt;&lt;/span&gt; .terragrunt-version
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;    &lt;/div&gt;

    &lt;p&gt;Then install the versions with:&lt;/p&gt;

    &lt;div class=&quot;language-bash highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt; tfenv &lt;span class=&quot;nb&quot;&gt;install
 &lt;/span&gt;tgenv &lt;span class=&quot;nb&quot;&gt;install&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;    &lt;/div&gt;

    &lt;p&gt;Commit and update your changelog accordingly e.g.,&lt;/p&gt;

    &lt;div class=&quot;language-markdown highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt; ## [Unreleased]

 ### Added
&lt;span class=&quot;p&quot;&gt; -&lt;/span&gt; terraform version set to 0.13.5
&lt;span class=&quot;p&quot;&gt; -&lt;/span&gt; terragrunt version set to 0.26.4
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;    &lt;/div&gt;
  &lt;/li&gt;
  &lt;li&gt;
    &lt;p&gt;Install pre-commit hooks&lt;/p&gt;

    &lt;p&gt;These hooks provide a safety net that gets executed on each &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;git commit&lt;/code&gt;. I’d start with at least the following:&lt;sup id=&quot;fnref:4&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:4&quot; class=&quot;footnote&quot;&gt;4&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

    &lt;ul&gt;
      &lt;li&gt;&lt;a href=&quot;https://github.com/antonbabenko/pre-commit-terraform&quot;&gt;terraform_fmt hook&lt;/a&gt; formats *.tf files using the &lt;a href=&quot;https://www.terraform.io/docs/commands/fmt.html&quot;&gt;terraform fmt&lt;/a&gt; command&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;https://github.com/antonbabenko/pre-commit-terraform&quot;&gt;terraform_validate&lt;/a&gt; similarly runs &lt;a href=&quot;https://www.terraform.io/docs/commands/validate.html&quot;&gt;terraform validate&lt;/a&gt;&lt;/li&gt;
      &lt;li&gt;&lt;a href=&quot;https://github.com/gruntwork-io/pre-commit&quot;&gt;terragrunt-hclfmt&lt;/a&gt; runs &lt;a href=&quot;https://terragrunt.gruntwork.io/docs/reference/cli-options/#hclfmt&quot;&gt;terragrunt hclfmt&lt;/a&gt; on terragrunt.hcl files&lt;/li&gt;
    &lt;/ul&gt;

    &lt;p&gt;&lt;br /&gt;You specify these in a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;.pre-commit-config.yaml&lt;/code&gt; file. &lt;a href=&quot;https://github.com/thirstydeveloper/terraform-terragrunt-skeleton/blob/release/1.0/.pre-commit-config.yaml&quot;&gt;Here’s&lt;/a&gt; what it looks like.&lt;/p&gt;

    &lt;p&gt;Next, run &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;pre-commit install&lt;/code&gt; to install the hooks.&lt;/p&gt;
  &lt;/li&gt;
&lt;/ol&gt;

&lt;h2 id=&quot;step-2-directory-structure&quot;&gt;Step 2: Directory Structure&lt;/h2&gt;

&lt;p&gt;Covering the directory structure introduces a lot of terms. I start with two top-level directories:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;deployments/
modules/
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;h3 id=&quot;deployments&quot;&gt;Deployments&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;deployments&lt;/strong&gt; are instantiations of infrastructure. Directories under deployments contain &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;terragrunt.hcl&lt;/code&gt; files. You run &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;terragrunt&lt;/code&gt; commands inside deployment directories.&lt;sup id=&quot;fnref:5&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:5&quot; class=&quot;footnote&quot;&gt;5&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

&lt;p&gt;Deployments are organized in a directory tree. The first level in the tree is the infrastructure &lt;strong&gt;tiers&lt;/strong&gt;. Small projects may just have a single tier of infrastructure. For larger projects, there are often multiple tiers relying or building on each other. Sometimes these tiers are managed by different teams. For instance, my team has a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;foundation&lt;/code&gt; tier containing shared services like our docker image registry and CI/CD tooling, and a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;service&lt;/code&gt; tier containing application infrastructure:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;deployments/
    foundation/
    service/
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Each tier consists of one or more instantiations called &lt;strong&gt;environments&lt;/strong&gt;. For instance: development, staging, and production.&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;deployments/
    foundation/
        dev/
        stage/
        prod/
    service/
        dev/
        stage/
        prod/
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Underneath each environment, you have &lt;strong&gt;stacks&lt;/strong&gt; and/or &lt;strong&gt;layers&lt;/strong&gt; of stacks that make up that tier-environment. Layers are optional directories grouping stacks together. Stacks are bundles of infrastructure that are managed as a unit using terragrunt commands.&lt;/p&gt;

&lt;p&gt;For instance, our foundation tier might consist of a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;network&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;k8s&lt;/code&gt; stack, and a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;jenkins&lt;/code&gt; stack underneath an &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;apps&lt;/code&gt; layer:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;deployments/
    foundation/
        dev/
            network/
                terragrunt.hcl
            k8s/
                terragrunt.hcl
            apps/
                jenkins/
                    terragrunt.hcl
    root.hcl
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Bringing it all together, each &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;deployment&lt;/code&gt; is an instantiation of a stack for a tier-environment. For example, the jenkins stack for the foundation tier’s dev environment, which my team would summarize as foundation-dev-jenkins. To manage foundation-dev-jenkins, you run &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;terragrunt&lt;/code&gt; commands from the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;deployments/foundation/dev/apps/jenkins&lt;/code&gt; directory.&lt;/p&gt;

&lt;p&gt;Deployment directories do not contain the infrastructure definition (.tf) files themselves, however. Instead, each deployment references a stack underneath &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;modules/stacks/&lt;/code&gt;. This allows us to have a single stack definition deployed multiple times - across multiple environments for example.&lt;/p&gt;

&lt;h3 id=&quot;modules&quot;&gt;Modules&lt;/h3&gt;

&lt;p&gt;Expanding modules I have:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;deployments/
modules/
    components/
    stacks/
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;stacks&lt;/strong&gt; are root &lt;a href=&quot;https://www.terraform.io/docs/configuration/modules.html&quot;&gt;terraform modules&lt;/a&gt; and group together infrastructure that is managed as a unit. Stacks are defined as directories under &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;modules/stacks/&lt;/code&gt;, organized by tier, and contain the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;*.tf&lt;/code&gt; files defining the stack’s infrastructure. Using our foundation example from above, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;modules/stacks&lt;/code&gt; looks like this:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;modules/
    stacks/
        foundation/
            network/
                ...
            k8s/
                ...
            jenkins/
                main.tf
                outputs.tf
                providers.tf
                variables.tf
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Stacks may contain terraform provider resources directly (e.g., &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;aws_s3_bucket&lt;/code&gt;), but often will contain child terraform modules, which I call &lt;strong&gt;components&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Any components specific to this &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;infra&lt;/code&gt; repository reside in directories under &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;modules/components&lt;/code&gt;, and those directories similarly contain &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;*.tf&lt;/code&gt; files defining the infrastructure belonging to the component. For example:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;modules/
    components/
        some-component/
            main.tf
            variables.tf
            outputs.tf
        another-component/
            ...
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Summarized, each &lt;strong&gt;deployment&lt;/strong&gt; references a &lt;strong&gt;stack&lt;/strong&gt;, and each &lt;strong&gt;stack&lt;/strong&gt; may instantiate any number of &lt;strong&gt;components&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;You probably noticed the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;terragrunt.hcl&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;root.hcl&lt;/code&gt; files in the example above. Let’s talk about those next.&lt;/p&gt;

&lt;h2 id=&quot;step-3-hcl-files&quot;&gt;Step 3: HCL files&lt;/h2&gt;

&lt;p&gt;Each deployment contains a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;terragrunt.hcl&lt;/code&gt; file, marking it as a stack to be deployed with terragrunt. They are usually extremely simple, only containing an include to the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;root.hcl&lt;/code&gt; file.&lt;sup id=&quot;fnref:6&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:6&quot; class=&quot;footnote&quot;&gt;6&lt;/a&gt;&lt;/sup&gt;&lt;/p&gt;

&lt;pre&gt;&lt;code class=&quot;language-HCL&quot;&gt;# terragrunt.hcl
include {
  path = find_in_parent_folders(&quot;root.hcl&quot;)
}
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;The &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;root.hcl&lt;/code&gt; file contains all the terragrunt configuration. A simple starting point is:&lt;/p&gt;

&lt;pre&gt;&lt;code class=&quot;language-HCL&quot;&gt;# root.hcl
locals {
  root_deployments_dir       = get_parent_terragrunt_dir()
  relative_deployment_path   = path_relative_to_include()
  deployment_path_components = compact(split(&quot;/&quot;, local.relative_deployment_path))

  tier  = local.deployment_path_components[0]
  stack = reverse(local.deployment_path_components)[0]
}

# Default the stack each deployment deploys based on its directory structure
# Can be overridden by redefining this block in a child terragrunt.hcl
terraform {
  source = &quot;${local.root_deployments_dir}/../modules/stacks/${local.tier}/${local.stack}&quot;
}
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;This makes it so each deployment directory, by default, deploys the stack under &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;modules/stacks/&amp;lt;tier&amp;gt;&lt;/code&gt; with the same name as the deployment directory. For instance, &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;deployments/foundation/dev/apps/jenkins/&lt;/code&gt; deploys &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;modules/stacks/foundation/jenkins&lt;/code&gt;.&lt;sup id=&quot;fnref:7&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:7&quot; class=&quot;footnote&quot;&gt;7&lt;/a&gt;&lt;/sup&gt; I haven’t decided if I like making the layer part of that path or not, so thus far have left it out.&lt;/p&gt;

&lt;p&gt;With an understanding of the tools, directory structure, and HCL files, we can put together a minimal skeleton repo.&lt;/p&gt;

&lt;h2 id=&quot;step-4-the-skeleton&quot;&gt;Step 4: The Skeleton&lt;/h2&gt;

&lt;p&gt;Here’s a minimal skeleton I would start with, not including boilerplate files:&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;deployments/
    app/
        dev/
            test-stack/
                terragrunt.hcl
        stage/
            test-stack/
                terragrunt.hcl
        prod/
            test-stack/
                terragrunt.hcl
    root.hcl
modules/
    components/
    stacks/
        app/
            test-stack/
                main.tf
                outputs.tf
                providers.tf
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Our basic test-stack could use the &lt;a href=&quot;https://registry.terraform.io/providers/hashicorp/random/3.0.0&quot;&gt;random provider&lt;/a&gt; to create a single resource and output:&lt;/p&gt;

&lt;div class=&quot;language-terraform highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;c1&quot;&gt;# main.tf&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;resource&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;random_pet&quot;&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;pet&quot;&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;div class=&quot;language-terraform highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;c1&quot;&gt;# outputs.tf&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;output&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;pet&quot;&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
  &lt;span class=&quot;nx&quot;&gt;value&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;nx&quot;&gt;random_pet&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nx&quot;&gt;pet&lt;/span&gt;&lt;span class=&quot;p&quot;&gt;.&lt;/span&gt;&lt;span class=&quot;nx&quot;&gt;id&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;div class=&quot;language-terraform highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;&lt;span class=&quot;c1&quot;&gt;# providers.tf&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;terraform&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
  &lt;span class=&quot;nx&quot;&gt;required_providers&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
    &lt;span class=&quot;nx&quot;&gt;random&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;{&lt;/span&gt;
      &lt;span class=&quot;nx&quot;&gt;source&lt;/span&gt;  &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;hashicorp/random&quot;&lt;/span&gt;
      &lt;span class=&quot;nx&quot;&gt;version&lt;/span&gt; &lt;span class=&quot;p&quot;&gt;=&lt;/span&gt; &lt;span class=&quot;s2&quot;&gt;&quot;~&amp;gt; 3.0.0&quot;&lt;/span&gt;
    &lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
  &lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
&lt;span class=&quot;p&quot;&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;With this in place, we can run &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;terragrunt plan&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;terragrunt apply&lt;/code&gt; from any directory under &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;deployments/&lt;/code&gt;. We can also run &lt;a href=&quot;https://terragrunt.gruntwork.io/docs/reference/cli-options/#plan-all&quot;&gt;plan-all&lt;/a&gt; and &lt;a href=&quot;https://terragrunt.gruntwork.io/docs/reference/cli-options/#apply-all&quot;&gt;apply-all&lt;/a&gt; from:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;The root of the repository to affect all tiers and all environments&lt;/li&gt;
  &lt;li&gt;Any directory within deployments to affect everything underneath that directory&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;We can also use &lt;a href=&quot;https://terragrunt.gruntwork.io/docs/reference/cli-options/#terragrunt-exclude-dir&quot;&gt;–terragrunt-exclude-dir&lt;/a&gt; and &lt;a href=&quot;https://terragrunt.gruntwork.io/docs/reference/cli-options/#terragrunt-include-dir&quot;&gt;–terragrunt-include-dir&lt;/a&gt; to target &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;*-all&lt;/code&gt; commands. For example:&lt;/p&gt;

&lt;p&gt;&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;terragrunt apply-all --terragrunt-exclude-dir deployments/*/prod/**&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;would apply all non-production environments across all tiers.&lt;/p&gt;

&lt;h1 id=&quot;whats-next&quot;&gt;What’s next?&lt;/h1&gt;

&lt;p&gt;This skeleton is quite minimal. We haven’t covered variables, backends, RBAC, external modules, build tooling, continuous integration, and so on.&lt;/p&gt;

&lt;p&gt;The next post will dive deeper into what we can do with &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;root.hcl&lt;/code&gt;, including introducing a variable loading hierarchy that makes our deployments easier to work with.&lt;/p&gt;

&lt;h1 id=&quot;footnotes&quot;&gt;Footnotes&lt;/h1&gt;

&lt;div class=&quot;footnotes&quot; role=&quot;doc-endnotes&quot;&gt;
  &lt;ol&gt;
    &lt;li id=&quot;fn:1&quot; role=&quot;doc-endnote&quot;&gt;
      &lt;p&gt;In short, CI/CD with terraform is tricky. The authors of terragrunt discuss some of the challenges &lt;a href=&quot;https://github.com/gruntwork-io/terragrunt/issues/720#issuecomment-497888756&quot;&gt;here&lt;/a&gt;. I’m a proponent of pushing CI/CD for infrastructure, but doing so carefully and expecting times of manual intervention. Some things are difficult to automate out of the box - state migrations (e.g., &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;terraform state mv&lt;/code&gt;) being one example. I’ve trended towards starting teams off with CI and manual applies of infrastructure and working towards CD as the team becomes more comfortable. More on CI/CD in future posts. &lt;a href=&quot;#fnref:1&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:2&quot; role=&quot;doc-endnote&quot;&gt;
      &lt;p&gt;Terraform stores its state in state files and those state files contain the version of terraform last used. The version specified in the state file is auto-bumped whenever a newer version of terraform touches that state file. Once that happens, you may force anyone working on that infrastructure, &lt;em&gt;or relying on outputs&lt;/em&gt; from your state file via &lt;a href=&quot;https://www.terraform.io/docs/providers/terraform/d/remote_state.html&quot;&gt;terraform_remote_state&lt;/a&gt; data sources to upgrade. The second case is the most damaging because it can affect teams outside of your own, depending on who’s using your outputs. &lt;a href=&quot;https://github.com/hashicorp/terraform/blob/v0.14/CHANGELOG.md&quot;&gt;terraform 0.14&lt;/a&gt; addresses this, making state files backward and forward compatible across terraform versions as much as possible, but anyone using older versions of terraform should be particularly careful with terraform versioning. &lt;a href=&quot;#fnref:2&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:3&quot; role=&quot;doc-endnote&quot;&gt;
      &lt;p&gt;We’ve since split our infrastructure across multiple repositories, but only after demonstrated needs: distributed ownership, differing change/release rates, and re-use across teams. Much like &lt;a href=&quot;https://martinfowler.com/bliki/MonolithFirst.html&quot;&gt;starting with a monolith&lt;/a&gt; before evolving to microservices, starting with an infra monorepo avoids bloat slowing your team unnecessarily. &lt;a href=&quot;#fnref:3&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:4&quot; role=&quot;doc-endnote&quot;&gt;

      &lt;p&gt;There are lots of other tools/hooks that are worth looking at that I haven’t had time to yet. Some on my list to explore include:&lt;/p&gt;

      &lt;ul&gt;
        &lt;li&gt;&lt;a href=&quot;https://github.com/tfsec/tfsec&quot;&gt;tfsec&lt;/a&gt; - example &lt;a href=&quot;https://github.com/antonbabenko/pre-commit-terraform&quot;&gt;hook&lt;/a&gt;&lt;/li&gt;
        &lt;li&gt;&lt;a href=&quot;https://github.com/terraform-docs/terraform-docs&quot;&gt;terraform docs&lt;/a&gt; - example &lt;a href=&quot;https://github.com/antonbabenko/pre-commit-terraform&quot;&gt;hook&lt;/a&gt;&lt;/li&gt;
        &lt;li&gt;&lt;a href=&quot;https://github.com/terraform-linters/tflint&quot;&gt;tflint&lt;/a&gt; - example &lt;a href=&quot;https://github.com/antonbabenko/pre-commit-terraform&quot;&gt;hook&lt;/a&gt;&lt;/li&gt;
        &lt;li&gt;&lt;a href=&quot;https://www.checkov.io/&quot;&gt;checkov&lt;/a&gt; - official &lt;a href=&quot;https://www.checkov.io/4.Integrations/pre-commit.html&quot;&gt;hook&lt;/a&gt;&lt;/li&gt;
      &lt;/ul&gt;
      &lt;p&gt;&lt;a href=&quot;#fnref:4&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:5&quot; role=&quot;doc-endnote&quot;&gt;
      &lt;p&gt;If you’ve read &lt;a href=&quot;https://www.terraformupandrunning.com/&quot;&gt;Terraform Up &amp;amp; Running&lt;/a&gt;, my &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;deployments/&lt;/code&gt; directory equates to Yevgeniy’s &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;live/&lt;/code&gt; directory. My team just found the name deployments to be more understandable. &lt;a href=&quot;#fnref:5&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:6&quot; role=&quot;doc-endnote&quot;&gt;
      &lt;p&gt;I use the name &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;root.hcl&lt;/code&gt; instead of &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;terragrunt.hcl&lt;/code&gt; because the latter causes errors running &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;plan-all&lt;/code&gt; and &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;apply-all&lt;/code&gt; from the root or &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;deployments/&lt;/code&gt; directory. Terragrunt seems to treat the parent hcl file as a stack to be deployed and errors out. I tried terragrunt’s &lt;a href=&quot;https://terragrunt.gruntwork.io/docs/reference/config-blocks-and-attributes/#skip&quot;&gt;skip option&lt;/a&gt; but to no avail, at least as of version 0.26.2 &lt;a href=&quot;#fnref:6&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
    &lt;li id=&quot;fn:7&quot; role=&quot;doc-endnote&quot;&gt;
      &lt;p&gt;Terragrunt glues the deployment to the stack definition by copying everything in the deployment directory and everything in the referenced stack directory (containing the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;*.tf&lt;/code&gt; files) into a &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;.terragrunt-cache&lt;/code&gt; directory and then executing commands from that directory. By default, the &lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;.terragrunt-cache&lt;/code&gt; directory lives in your deployment directory. &lt;a href=&quot;#fnref:7&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
  &lt;/ol&gt;
&lt;/div&gt;</content><author><name>Chris</name></author><category term="tf-skeleton" /><category term="terraform" /><category term="terragrunt" /><summary type="html">Terraform is my go-to infrastructure definition tool. I love that it enables declarative management of different Cloud, PaaS, and SaaS platforms with its unified HCL language and provider model. At times, however, I’ve wished it was easier to start a project off on the right foot.</summary></entry></feed>