Managing State¶

Before we even consider running Terraform, inside or outside of a CI pipeline, we need to discuss and understand what a Terraform state file is and how we can store it.

State¶

When Terraform is creating and managing resources it has to save what it's doing, and what IDs it got back from AWS, so that it can calculate differences in the future.

For example if Terraform creates an EC2 Instance that is a t2.small and you change the Terraform code later on to redefine the EC2 Instance to be a t2.large, Terraform has to calculate the difference between the state that represents what you currently have and the code, which is what you want to have. After it has calculated the change it can make it live, save the results, and now your state file is updated to reflect this new change. The cycle then repeats as you make future changes to the code or the remote state of things changes and no longer matches what you have in your state file.

This is a very overly simplified visual representation of this cycle:

graph LR
    a[Initial Code] --> b[Terraform Run]
    b --> c[Manage Changes]
    c --> d[Update State]
    d --> e[Future Change]
    e --> b

The question we now have to answer is: how do we store this state file and keep it safe?

State Storage¶

When Terraform completes creating, deleting, and just generally managing resources for you, it stores everything it's done inside of a file called terraform.tfstate (by default.) This file is simply a JSON file (.json) and is human and machine readable.

There are two ways we can store this state file:

1. Locally
2. Remotely

Let's discuss each of these in a bit of detail.

Local State Storage¶

When you run terraform commands such as apply, you end up with the terraform.tfstate file being written to the local disk in the same directory you executed Terraform inside of. This is usually the same directory all your .tf files are inside of.

The general flow of local execution looks like this:

graph LR
    A[terraform apply] -- 1 --> B[AWS API]
    B -. 2 .-> C[Resource IDs]
    C -. 3 .-> D[terraform.tfstate]
    D -. 4 .-> A

1. Terraform makes the call to the AWS API via HTTPS
2. The call to the AWS API results in resources being created with unique IDs
3. These IDs are then stored in the terraform.tfstate, which Terraform creates for us, allowing us to manage them in the future
4. The state file is then managed by Terraform and used for future calls and work

The state file contains everything Terraform knows about your infrastructure and without it, Terraform cannot manage any resources it has created for you in the past (and stored in the file.) It can also contain sensitive information such as passwords and security architecture that an attacker can use to bypass your security measures. After Terraform has written the state file to disk, you must ensure you protect it from being deleted, edited directly (by you or anyone else), and even read by other people.

Advantages¶

There are some advantages to using a local state storage.

1. You can use a local state file to work quickly and test ideas
2. You can work with a local state if you just want to terraform validate your work

Outside of these use cases, there isn't much going for a local state file.

Disadvantages¶

There are (big) disadvantage to storing the state file locally.

1. The state file cannot be worked on by multiple engineers easily
2. There is no back up of the state file if you're storing it locally
3. Storing it in a Git repository, which is tempting, exposes the file and its contents to exploitation
4. When working on a locally managed state file the lock is also local, something others cannot work with
   1. This potentially introduces race conditions
5. You have to make sure you keep a solid back up of this file because losing it means Terraform can no longer manage the infrastructure it created

I recommend you avoid using local state storage whenever you can. Use a remote storage option instead.

Remote State Storage¶

To help us better protect and back up the state file, we can use a remote state storage mechanism, such as GitLab's Terraform support, an AWS S3 Bucket, or one of many other options. When we use a remote state storage mechanism, like GitLab, we're securely storing the state, ensuring it's backed up, and allowing multiple people to work on it.

graph LR
    a[terraform apply] -- 1 --> b[Lock State]
    b -- 2 --> c[Download State]
    c -- 3 --> d[Manage Changes]
    d -- 4 --> e[Update State]
    e -- 5 --> f[Upload State]
    f -- 6 --> g[Unlock State]
    g -- 7 --> a

1. We lock the state to tell others we're changing it
2. Now download the state from the remote storage mechanism
3. Terraform computes and manages the differences between state and remote (AWS) API
4. It updates the state with any changes that have been made
5. We then upload the state to the remote storage back end
6. Now we unlock our state, allowing others to work on it
7. And we repeat this process later on with another change

Advantages¶

1. Your state file is more likely to be backed up a lot using GitLab's Terraform support versus doing it yourself
2. Multiple engineers can work on the infrastructure over its life time (but not at the same time)
3. You can issue a lock when you're running Terraform, stopping others from running Terraform at the same time

And more.

Disadvantages¶

1. It's a bit more work to get set up
2. You'll need additional remote resources configured ahead of time to store the remote state file
3. You have to trust the hosting provider behind that state storage mechanism - are they backing your state file up and securing it?
4. You can technically create the remote state storage using Terraform, but you cannot store the resultant state file in the state storage, because one has to come before the other (chicken and egg problem)

The short of it is: use a remote storage back end unless you want difficulties in the future.