Co-authored with the amazing Daniel Lamando

Intro

Using modules to keep your Terraform code DRY is very common, but figuring out the best way to release and distribute these modules isn’t always easy. We at Forto quickly faced this struggle, as more of the organization adopted Terraform to describe their infrastructure with code. In fact, keen-eyed readers of our previous Terraform blog post might have noticed a small easter egg:
This image shows a code snippet within a Terraform configuration file, specifically focusing on an AWS SQS (Simple Queue Service) module. The file being edited is named sqs.tf, and the code snippet uses a module named sqs_example_queue. Key parameters are defined within the module, including source, queue_name, and dead_letter_queue_create. A notable part of the image is a highlighted warning: "../../../../modules/aws-sqs-queue@v2" # don't version your modules like this. This comment suggests that specifying module versions directly within the file path (such as @v2 in the directory structure) is considered a bad practice in Terraform. Proper module versioning should ideally be managed through Terraform’s versioning mechanisms rather than hardcoding the version within file paths. Below the code snippet, additional context shows that this code change is part of a pull request titled "DLQ FTW!!![production]" with a reviewer assigned and relevant labels for promotion and automation. This image highlights the importance of following best practices in Terraform module management to ensure code maintainability and avoid issues associated with improper versioning methods.

This snippet represented how we referenced Terraform modules at the time, and it exposed some significant technical debt: referencing unversioned Terraform modules by-path. This practice made every change to any module a risky endeavor. In some cases, we created extra copies of our modules to manage the risk level (leading to @v2 as seen in that screenshot).

When we realized the need to version our Terraform modules, I planned a hack day to find a solution. For context, Forto runs monthly one-day hackathons where engineers are encouraged to work on individual projects not related to their day-to-day work. The projects can range from machine learning to music synthesis. Still, sometimes we use those hackathons to prototype or implement some solution that is related to our work but somehow isn’t prioritized for the time being.
The versioning tool of choice for this first hack day was Versio. While our Versio implementation worked for incrementing version numbers and producing releases of our modules, we encountered a couple of papercuts, and as more engineers within the company started making module changes, the downsides of the added complexity became apparent. After a few months, we decided to take a second hack day to see if a simpler from-scratch solution could work more effectively.
Our new workflow still allows Terraform modules within a monorepo to be individually versioned. However, the release process is now tied closely to the Pull Request lifecycle. After a Pull Request is merged, a new version number is calculated and the produced artifacts are uploaded to an S3 bucket. Finally, the repository’s Wiki is updated with automatically generated module documentation along with a new changelog entry. This results in a simpler versioning workflow, while leveraging various Github features such as PR labels, Github Actions, and the GraphQL API.
In the following sections, we will explain why we version our modules, share the decisions we made and show an example flow of a module upgrade.
We also created a public reference repository github.com/freight-hub/terraform-modules-demo so anyone can easily implement this workflow on their own repositories.

(“Freighthub” is Forto’s previous brand name. Renaming our Github org is top on the Jira board, honestly!)

The Flow

Make a change in some module

You know, your job.
This image shows a code snippet from a Terraform file (variables.tf) used to define variables in an AWS VPC (Virtual Private Cloud) module. The file specifies Terraform variables that configure aspects of the VPC setup. Key Highlights: New Variable Addition: The green-highlighted code represents a newly added variable named example_variable. Attributes of example_variable: Description: "An example variable to showcase the module release process". Type: Set to bool (boolean). Default: Set to true. This variable is added to demonstrate how new variables can be incorporated in a module to support release or update processes. Existing Variable: Below the new addition, there is an existing variable named create_vpc. Description: "Controls if VPC should be created (it affects almost all resources)". Type: Boolean, indicating it toggles the creation of the VPC itself, impacting several dependent resources. This image serves as an example of how variables are structured in Terraform to control module behavior. By defining variables with clear descriptions, types, and default values, the module becomes more customizable and user-friendly for infrastructure management in AWS. This approach helps standardize variable handling in infrastructure-as-code practices.

Test the new module before release

If you want to test your code locally, you can use file system path-based module source reference, like:

source = “../../../modules-repos/name_of_module”

If you want to have the unreleased change tested by the CI/CD pipeline (see previous article), you can use git-based module source reference, such as:

source = “[email protected]:freight-hub/terraform-modules-demo.git?ref=BRANCH-NAME”

Open a PR for the change

Make sure to add the text intended for the changelog in the PR description. Providing a meaningful changelog text ensures that the release history will be accurate and useful.
This image depicts the GitHub interface for opening a pull request, used to propose changes to a code repository. In this example, a new branch named example_pr_minor_release has been created to introduce a minor change to a Terraform module. Key Details: Branch Information: The branch name, example_pr_minor_release, indicates this pull request is intended for a minor update to the module, likely adding or modifying a small feature or variable. Pull Request Title: The user has titled the pull request as "Add variable to showcase a minor release", signaling the specific purpose of the change. Placeholder Text for Change Log: In the pull request description box, there’s placeholder text reading: "" This reminder emphasizes the importance of providing a clear, meaningful description that will appear in the Terraform module's changelog. The final description should summarize the changes to inform users of the update’s impact. Merge Status: A green label indicating "Able to merge" signifies that the branch has no conflicts with the main branch, allowing for an automatic merge if approved. This image highlights a best practice in collaborative development: using clear descriptions and meaningful change logs in pull requests. This process helps team members and module users understand updates, improving module maintainability and transparency in version control.

Apply a release increment label to the PR

Based on Semantic Versioning guidelines.

Merging is blocked until this is done.
This image captures a GitHub pull request screen for a minor update to a Terraform module, specifically adding a new variable to showcase a minor release. The pull request is titled "Add variable to showcase a minor release #6" and aims to merge a change from the example_pr_minor_release branch into the main branch. The creator of the pull request, user “Oded-B,” added the "example_variable" to the variables.tf file, which is evident from the commit message and change summary in the pull request. In this interface, we see an interaction with labels that can categorize the pull request based on its scope or type of release. The arrow highlights the "Apply labels to this pull request" option, which includes labels such as "major," "minor," "no-release," and "patch." This feature enables the pull request creator or reviewers to tag the update appropriately, indicating that this particular change qualifies as a "minor" update due to its limited impact on the module. However, not all automated checks have passed successfully. There’s a red warning message reading "Some checks were not successful," which indicates one failed check and three skipped checks. Specifically, the failed check is related to "Monorepo Versioning / Detect pull request context," which implies an issue with context detection in the versioning workflow. Despite these check issues, a green message at the bottom confirms that "This branch has no conflicts with the base branch," meaning that, structurally, the changes can be merged without conflict. This pull request screen reflects typical collaborative workflows for maintaining Terraform modules. Automated checks verify coding standards, while labels help categorize the change’s impact, contributing to a streamlined and organized process in a shared repository. Proper use of labels, check results, and clear descriptions enhance clarity and transparency in version-controlled environments.

The pipeline will post a PR comment including current & new version numbers.
This image illustrates a release plan update generated by GitHub Actions in a pull request for a Terraform module, specifically showing version control adjustments as part of a minor update. In this pull request, the user "Oded-B" has applied the "minor" label to indicate that the proposed changes are incremental and non-breaking, aligned with semantic versioning practices for software releases. The release plan table displays essential information: Directory: Specifies the affected module or directory, in this case, terraform-aws-vpc. Previous Version: The last published version of this module, listed as 1.0.0. New Version: The planned version after this update, specified as 1.1.0. This version bump from 1.0.0 to 1.1.0 signals a minor release, indicating enhancements or new features that are backward-compatible without breaking changes. This helps users of the module understand that they can update to this version without needing significant adjustments. This structured release plan and label ensure that module users are well-informed about the nature and impact of the change, providing a clear preview of the updated version and helping maintain an organized changelog. This process of automated version management in a collaborative repository streamlines deployment and maintains consistency across releases, offering clarity for developers and users alike.
This image shows a terminal output from an automated GitHub Action that pushes recent changes to a GitHub repository’s wiki. The update here documents a new release version for a Terraform module, specifically updating the terraform-aws-vpc module to version 1.1.0. Key details in the output include: Git Configuration: The command Run git config --local user.email "github-actions[bot]@users.noreply.github.com" configures the GitHub bot's email for this commit, indicating that this change was automated through GitHub Actions rather than manually pushed by a user. Version Update: The line [master cecacc5] terraform-aws-vpc @ 1.1.0 specifies the module and the new version number, 1.1.0, reflecting the minor update made to the module. File Change Summary: The output shows that one file was changed, with 11 insertions and 4 deletions, suggesting modifications to the documentation or versioning file in the wiki. Push to Wiki Repository: The update is pushed to the URL https://github.com/freight-hub/terraform-modules-demo.wiki, which points to the wiki section of the terraform-modules-demo repository, indicating that documentation or release notes are being updated to reflect the latest changes. This automated update ensures that documentation in the GitHub Wiki is synchronized with the latest release, allowing users to access up-to-date information on module versions and changes. This process maintains consistency in version tracking and enhances transparency in the module’s development lifecycle by making sure that each release is well-documented and accessible directly from the wiki.
This image displays a terminal log from an automated process that prepares and uploads files related to a Terraform module release. The module in question is terraform-aws-vpc, and this log captures the steps involved in packaging and uploading the module files to an Amazon S3 storage location for distribution. Key Details in the Process: Module Directory: The process begins by locating the specific module directory, terraform-aws-vpc, within the workspace. It navigates through paths related to the project structure in a continuous integration/continuous deployment (CI/CD) environment. File List: Within the terraform-aws-vpc directory, the log lists several files related to the module: changelog.md: Likely contains a record of changes made in the new release. documentation.md: Provides detailed documentation for users of the module. new-version.txt and previous-version.txt: Text files that likely indicate the current and prior versions, which help track versioning within the CI/CD pipeline. terraform-module.zip: A compressed file containing the Terraform module files, prepared for upload. File Upload: The terraform-module.zip file, approximately 76 KB in size, is successfully uploaded to an Amazon S3 bucket at the path s3://forto-terraform-modules-demo/terraform-aws-vpc-1.1.0.zip. The version number (1.1.0) in the file path indicates that this is a minor release update for the module. Completion: The upload progress shows completion at a transfer rate of 72.7 KiB/s, with the file fully uploaded, indicating a successful deployment of the module. This automated process highlights how CI/CD pipelines can streamline the release and distribution of Terraform modules. By organizing files like changelogs and version tracking documents, compressing the module, and uploading it to a storage location, this system ensures that updated modules are readily available for users. This process helps maintain version consistency and facilitates smooth access to the latest module releases in shared storage, supporting efficient infrastructure management in cloud environments.

Here we see the new variable in the autogenerated wiki page:
The image shows a table listing Terraform module variables with three columns: the variable name, a brief description, and example values or use cases. Three variables are visible: enable_vpn_gateway - Described as a setting to enable a new VPN Gateway. example_variable - Highlighted, with a description indicating it’s an example variable to demonstrate module release. external_nat_ip_ids - Specifies a list of EIP IDs for NAT Gateways. Each row includes a variable name and its description.
The image shows a "Changelog" section for a Terraform module, listing two versions with release notes: Version 1.1.0 (2021-10-05): Linked to PR #6, which introduces a minor release by adding a new variable, example_variable. Version 1.0.0 (2021-10-05): Linked to PR #4, which includes a whitespace change to trigger a release after fixing CI/CD configuration issues. This entry also notes it as the initial release of the module, with a reference to the source GitHub repository link for terraform-aws-vpc. Each version includes a date, pull request reference, and a brief description of changes.

Private repos should use S3 URLs to allow Terraform to use AWS authentication and authorization, public repos can just use plain HTTP URLs

source = “s3:https://forto-terrform-modules-demo.s3.eu-west-1.amazonaws.com/terraform-aws-vpc-1.1.0.zip”

The Reasoning

Why version your modules at all?

Deterministic behavior — surprises are good for birthdays (maybe), but not infrastructure as code repos! Operators should expect to see changes in their infrastructure objects only when they change the code in their Terraform workspace, regardless of work done on the modules. When a new version for a module is desired (e.g. fixing a bug, adding functionality), an explicit change must be made to the Terraform workspace, thus ensuring ownership policies like Github CODEOWNER Required Reviews are kept.

Why Semver?

Consumers of the modules can easily tell if they need to look at the changelog before upgrading as major versions indicate large or breaking changes. Note that Patch versions don’t necessarily introduce less risk compared to minor or even major changes. Risk is managed by testing and code & plan reviews.

Some decisions we made

#1 Separate repo

The CI/CD and permission/ownership models are totally different compared to the other Terraform repos. Having modules and workspaces root code in the same repo would result in a highly complex repo configuration.

#2 Pull Request(PR) Based

We originally had a commit-based approach. Versio would check all the [conventional] commit messages in a merged PR, decide on a version bump, and release it.

While we liked the pure git approach, it resulted in some confusing user experiences. For example, commit messages from a “squash” merge might pop back into existence and fail the Versio release command. This error happens after the merge is complete which makes the damage annoying to repair.

And because GitHub does not provide centralized pre-commit hooks, we couldn’t easily enforce a policy on those commit messages, requiring users to carefully follow the repository instruction or be forced to re-write the PR’s git history.

For our from-scratch solution, we started leveraging the Github PR itself. A set of four PR labels (major, minor, patch, and no-release) replaced the complex scanning of commit messages for versioning markers such as “feat!” and “fix:”. With git history ignored, the version bumps became much more reliable and predictable.

It also allowed us to post the “planned” versioned in PR comment and disallow merges of PR with no label, ensuring attention was given to version bump size. This was implemented with our own code, in languages which we know (Typescript/YAML/Bash) and with about 95% fewer lines of code compared to Versio’s Rust codebase.

#3 Keep the version “state” in Git tags.

Our next steps

The big downside of strict versioning is version drift. Upgrades must be done explicitly, and if the upgrades aren’t done rigorously, previous versions of the modules will be kept in use long after new versions become available.We plan to introduce another job for the Terraform modules pipeline to automatically open PRs which upgrade all module references in the workspaces root repo to the latest version. For minor and patch version bumps, these PRs should be ready to merge as-is. Major version bumps would instead open draft PRs, as we assume the large or breaking changes will require more work before the PR is ready to merge.

That’s it

I hope you found this article relevant! Please checkout (pun very much intended) our reference repo: https://github.com/freight-hub/terraform-modules-demo.It should be possible to clone it and adjust the CI to your needs in a few hours. Finally, don’t forget to visit our careers page! We are doing all kinds of other cool stuff and most of it doesn’t end up in the blog.