Improve your Infrastructure Automation with Terraform Cloud
An overview about the different operational modes of Terraform
Hashicorp’s Terraform offering has passed quite a distance since the standalone binary was introduced some years ago.
While the Terraform Enterprise edition has offered a web-based UI and advanced operational features for some years now, community users had to deal with the management of operational artifacts on the command line level.
This changed now with the Terraform Cloud service being available as a community and paid offering. In this article want to highlight some of the patterns I experienced when working with Terraform in the last years. The evolution path from the standalone binary to a fully managed operational environment can be confusing, since there are so many options in different dimensions available. This is the first part of a mini series of lessons learnt.
Terraform Application
It all begins with the terraform executable that is meanwhile available in the 0.12.x release. When you start with Terraform today, be sure that you use that version since some notable syntax changes did occur in the foundational scripting language HCL.
Teraform CLI standalone
The terraform local binary that is typically used at the beginning of the learning curve, manages a terraform project within a file system directory. All *.tf files in that directory are processed together. The downloaded provider plugins are kept in the hidden .terraform directory and the state of the created infrastructure resides in the local .tfstate file. Especially the state file is important to keep since it reflects the active resource state. It’s loss would result in a decoupled infrastructure and the need to reimport resources into a new state file — a cumbersome procedure.
Terraform CLI Workspaces
The command line version introduced workspaces (formerly named “environments”) some years ago, after initial usage of the infrastructure as code system showed the need for better operational support.
A “workspace” in the command line version is basically a copy of the tfstate file that keeps shadow information of the created infrastructure. Developers can create a feature branch of commited terraform code and, in that branch, switch to another workspace to modify existing structures without compromising a production environment. Together with remote state and possibilities to react on the workspace name in the code, this opened options to manage different stages in a team of engineers and operators. Conceptually sound, but the wiring of this CLI feature combined with the usual git processes required some effort.
Terraform Remote State
Some additional options but also setup complexity creates the remote state/lock feature. Workspace state, and optionally a lock semaphore to control parallel access, can be moved away from the local execution machine to a remote storage like AWS S3, Hashicorp Consul, or Terraform Cloud in the initial release.
The benefit is that now multiple engineers can work with the same workspace without compromising the consistency of the state files.
In the code, the used backend must be referred to by a configuration stanza:
# AWS S3 state backend configuration sample
terraform {
backend "s3" {
bucket = "myorg-tf-status"
key = "dev-my-component"
region = "eu-west-1"
dynamodb_table = "myorg-tf-lock"
}
}
# Terraform Cloud state backend configuration sample
terraform {
backend "remote" {
organization = "my-organization"
workspaces {
name = "dev-my-component"
}
}
}
Locals allow the modification of configuration depending on state, as described here.
Terraform Cloud Workspaces
The workspace idea has evolved into Terraform Cloud, which is now available as a community version with limited users, and a paid option with extended user support and additional functionality. (Terraform Enterprise is now the on-premise release with additional features that hook into enterprises’ IT mangement).
Now, the backend not only keeps states and locks as appropriate, but also executes the Terraform binary. It can be raised manually but the best feature is the integration with CI/CD providers (as of now Gitlab, Github, Bitbucket and Azure DevOps). It needs some auth back-and-forth between the both backends until both sides can trust each other but when that is done, the usage is pretty straight-forward.
A commit or merge in the repository creates an event that starts the Terraform plan and apply process. This keeps the pipeline lean and removes the need for a separate runner setup in the CI/CD pipeline.
A manual approval can be included after the plan step, which might be a good idea for production environments. We will have to wait for a future Terraform Cloud release until this approval request can be emitted to external systems like Slack to create a seamless rollout workflow, but watching the Terraform evolution so far, I am sure this feature will be available soon.
More features are worth mentioning:
- Variables can be stored within the workplace, so there is no need for a separate management in
.tfvarsfiles any longer. Credentials can be marked as sensitive, so that no one can peek into the values after they have been stored — a must if you do not maintain a Key Management solution. - Secondly — Terraform has now an API and, even better, a Terraform Cloud provider! This allows to automate the creation of Terraform workspaces, which I will describe in a separate post.
- User and Role Management allow to manage access for Terraform users. This is essential in production environments, and is available with the paid version of Terraform Cloud.
- A private Module registry allows version control and an interface catalog even for your internally used modules. This is a great relief for all who had to manage module versions in their version control system beforehand.
I think Hashicorp made a big leap towards their vision — providing the foundation for the Cloud Operational model that is key for managing the agile infrastructures we face today.
