Terraform file structure and documentation

[alanlima]

Hello guys,

Since that we started to develop and add some terraform code into our project, I think that it might be a good time for us to discuss the file structure and some base guidelines to help us out.

There are two start points that I would like to discuss here and be free to add something else:

1. File structure using terraform modules.
2. Documentation (using terraform-docs, I will explain it).

File structure

TL;DR; this is the file structure proposed:

root
|-- terraform -> root directory for our terraform code
    | README.md -> index file with the list of all modules and reference for their docs. Also any extra info and guidance
    |-- modules -> root directory for all modules created (VPC, ECR, SSM, etc.)
        |-- module1
            |-- README.md -> each module should have it - how to use it, what are the inputs and outputs variables
            |-- main.tf
        |-- module2
            |-- README.md
            |-- main.tf
    |-- examples -> root directory (or playground directory) to demonstrate how the modules are used which also serves for testing
        |-- module1
            |-- main.tf
        |-- module2
            |-- main.tf
    |-- prod -> root directory for our production environment, here will use the modules to create our cloud environment
        |-- main.tf

@mfreitassm used this file structure in her PR #17, and it is also proposed by the terraform guidelines, so no, I don't have any credits for it. 😆

Right, now give me some reason. 😒

• We will have some tasks which depend on other tasks. Like, we will need to create the SSM parameters and ECR to do the task WordPress - ECS Task and Service. So we will need to look at our peer work and play with that to conclude our task. Of course, we could annoy them every time but they might be busy with something else or might not have time to help you at the moment.

• It is modular, so it is replicable, we could create different environments for testing, staging, production. Just add a new folder and start to pick up your modules as you pick up your favourite fruits (or candy) in your grocery store. (Also we will comply with one of our project requirements from our lovely project owner).

• The example folder is extra, but I like that idea to have some examples there to how to use the code for our teammates have to look at it. It will also be useful for testing, if you add/update something, we could modify the example one without need to set all the parameters again and again.

That is it, for now, I will repeat, I will love to hear your thoughts about it.

Documentation

Short story: So here I am thinking of how I can be more lazy I can make my life easier and automate some boring but essential task (DOCUMENTATION). So, surfing 🏄 in the internet, terraform-docs comes to rescue me and generate amazing markdown documents with all the input/output variables. On top of that, it is customizable and can be automated.

If you read the short story above, great, if not. I would like to present terraform-docs. It is a simple tool that you can install in your system and from the terraform code location, and it will read your files and generate documentation in various output formats.

It can be automated with a short script to generate/update the documents for all our modules at a glance. We could add it in the GitHub actions in the future. I actually wrote a small script (more like copy from the terraform-docs page) and create a root makefile to call the script. I will send a PR late if everyone agrees with that.

I just used it in my recent PR, so you guys can have a lot at its output. It is simple to use, I can explain it better if you want, but you should have a try. If you don't like to install, just run it as a container to check it out ;)

Give me your feedback

So, guys, I would like to know what do you think about that. Give any answer between:

• Fantastic, let's do it
• Hmm, I like it, but maybe we could express your opinion here
• I hate it !
• You are crazy !!
• What hell are you talking about ??

[mfreitassm]

Fantastic, let's do it

[fabioacp]

Man, you should write a book. Thanks for bringing this topic up in a such funny way. :)

Fantastic, let's use terraform-docs!

About the file structure, I think modules are a great idea. I'd just remove the "examples" folder. I see the example directory more suitable for a product or tool when you have to explain how to use it like a js library, CLI tool, etc. We can use the dev env for testing.

Here are some small changes I'd suggest:

root
  |-- docs
  |-- images
  |-- src
      |-- infra 
          |-- docker 
          |-- terraform 
              | README.md 
              |-- modules
                  |-- module1
                      |-- README.md
                      |-- main.tf
                  |-- module2
                      |-- README.md
                      |-- main.tf
              |-- env
                  |-- dev
                      |-- main.tf
                  |-- prod
                      |-- main.tf
      |-- www
  |-- scripts
  |-- .gitignore
  |-- LICENSE
  |-- README.md

[alanlima]

Thanks @fabioacp / @mfreitassm

It is a long text, so it is good to make fun of it to be easy to digest. 😅😅

terraform-docs: I will just wait for the other guys to vote and I will push the small script with make instruction to update the terraform-docs.

file structure purposed: I am 🆗 with those changes.

About the file structure, I think modules are a great idea. I'd just remove the "examples" folder. I see the example directory more suitable for a product or tool when you have to explain how to use it like a js library, CLI tool, etc. We can use the dev env for testing.

I agree with you that the examples folder would be more suitable for products or tools. I also agree with the dev env for testing.
However, I would prefer to have those examples folder which will behaviour like unit testing for our modules (not automated but we would play with that and check if it is working or not). I would let the dev environment for the integrated test sides, so we can check with that the modules are working well in group.

As an example, I am doing the ECR Registry at the moment, and one of the test that I am doing is to check if the EC2 instances can access and pull the image. This test don't belong to env folder, but it is useful for myself and for other which would like to test/check that the EC2 + ECR is working along without need to read a doc and create a terraform script for it.

This is just my point of view, I am ok if you guys decide to drop it as well. I would keep doing that for my testing to ensure that the modules that I created is working as expected but I will not push the testing code to the repo.

Final point, maybe the name examples is not the best one, maybe tests would be more suitable.

[feernandobraga]

Good job Alan... sorry for not providing the feedback before, but I had already checked and even used the recommended structure for my module.
I initially thought that having everything in one single .tf file would make it easier for the newbie here, and then we would re-factor at the end.

[alanlima]

Good job Alan... sorry for not providing the feedback before, but I had already checked and even used the recommended structure for my module.
I initially thought that having everything in one single .tf file would make it easier for the newbie here, and then we would re-factor at the end.

Thanks @feernandobraga

Starting small and splitting (refactoring) at the end might be the best way to go until we get used to it.
It is no ideal to create a single file for only 3 lines of code.. But it could be good to put a group of code in different files, like for IAM that usually might envolve more than one resource.

I also added the script to generate the docs... Be free to check it out and give some feedbacks ;)