TL;DR

Using Terraform to deploy and manage an Auto Scaling Group that utilises custom AMIs built by EC2 Image Builder and dealing with the limitations of EC2 Image Builder’s native capabilities for managing Launch Templates.

What were we asked?

I’ve recently worked on a project that required an Auto Scaling Group of EC2 instances that utilised custom-built AMIs produced by EC2 Image Builder. The requirement was to have an EC2 Image Builder pipeline produce weekly images based on Amazon Linux 2023, with various custom components applied.
It’s quite straightforward to do this in Terraform, so happy days!

If you’ve used Terraform to manage Auto Scaling Groups in AWS, you’ll know that this is relatively straightforward too. There are lots of examples and use cases out there to draw from.
So it should be super simple to marry those two concepts together into one simple, happy, solution you’d think.
Yeah right!

If you want to skip the dramatic retelling of my pain and just get the answer, skip to the solution

What’s the problem? Go!

The first two parts are straightforward - the ASG definition and EC2 Image Builder pipeline setup in Terraform. As is the rest of the Terraform really. Working with my buddy we quickly had IaC that would deploy the VPC that would host our Auto Scaling Group, the Launch Template that would complement that ASG, and the EC2 Image Builder components, recipe, pipeline, and distribution, etc. As well as all the trimmings, IAM policies, security groups, KMS keys, etc, etc.
During the initial deploy attempt we encountered our first red flag. We had valid Terraform code, but it threw up this error:

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: Your query returned no results. Please change your search criteria and try again.
│
│   with data.aws_ami.latest_custom,
│   on data.tf line 486, in data "aws_ami" "latest_custom":
│  486: data "aws_ami" "latest_custom" {
│

I’d used a data source to capture the ID of the latest image that EC2 Image Builder would produce, but of course that hadn’t run yet.
I thought I’d been smart, I had foreseen the bootstrapping issue, and thought I’d get around it by temporarily pointing the image_id at an AWS managed SSM parameter that stores the Amazon Linux 2023 AMI ID.

resource "aws_launch_template" "my-cool-project-resource" {
  name_prefix            = "my-cool-project-name-lt-"
  #image_id               = data.aws_ami.latest_custom.id
  image_id               = data.aws_ssm_parameter.al2023.id

This didn’t prevent the error above. To my shame (not really, I’m actually quite pragmatic) I commented out the data source for my latest custom image and was able to successfully deploy my infrastructure. I immediately reversed the commented out sections. data.aws_ami.latest_custom was reinstated, and the Launch Template was reverted back to image_id = data.aws_ami.latest_custom.id and all worked well. Sorted.

+5 influence points if you see where this is going

It wasn’t sorted. Not completely.
I could now successfully deploy the infrastructure reliably, I’d hacked my way past the bootstrapping issue, but comforted by my solemn vow to myself that I’d go back and fix that. Scout’s honour.
IdontBelieveYou

The situation was better but not quite what we needed. We had an EC2 Image Builder pipeline that ran to a schedule and worked as expected. Our Launch Template looked good but it wasn’t using the most recent image. It was using the most recent -1. Consider this sequence of events:

Terraform deploys IaC, creating ASG, Launch Template, and EC2 Image builder pipeline.
EC2 Image Builder was triggered manually to confirm it was working.
EC2 Image Builder produces a new version of the AMI.
Launch Template is still running the TF deployed AMI version and doesn’t know about the one EC2 Image Builder produced.

This surfaced what might have been obvious to you dear reader, that the Launch Template must be built before we’ve built any images.
Also that our Pipeline would run independently of our Terraform code - and so our Launch Template would never know about the latest image until we next ran Terraform to update the Launch Template. That’s not ideal as the intention was to set up a CI/CD pipeline to run the Terraform code upon merges to the main branch.

I was opposed to the idea of scheduling the Terraform code to run after the pipeline schedule. That wouldn’t be pragmatic, that would be messy.

Why don’t you just update the LT natively from EC2 Image Builder?

Reasons!
Mostly that I didn’t know that was a thing at this point. I started looking at how other people have solved this, because surely using custom-built AMIs in Auto Scaling Groups is super common? It kinda is, but so is this challenge.

In actual fact, the distribution settings within EC2 Image Builder allow for updating a Launch Template with the latest AMI… directly. It’s explained nicely here.

So that resulted in me adding this launch_template_configuration block to my distribution config:

resource "aws_imagebuilder_distribution_configuration" "dist" {
  name = "${var.project}-dist"

  distribution {
    region = var.region

    ami_distribution_configuration {
      name        = "${var.project}-{{imagebuilder:buildDate}}"
      description = "A descriptive description describing things"

      ami_tags = {
        OS       = "AmazonLinux2023"
        Hardened = "true"
        Name     = "${var.project}-{{imagebuilder:buildDate}}"
      }
    }
    launch_template_configuration {
      launch_template_id = aws_launch_template.my-cool-project-resource.id
      account_id         = local.account_id
    }
  }
}

My distribution would now update the Launch Templates with details of the new AMI ID.
And it did.

Half Baked Images!

This was a step forward, if you’d first pivoted 180 degrees.
While it did solve the problem of the Launch Template not having the latest AMI - it absolutely did have the latest AMI now - it came at a price.

So, for some reason the documentation gods appear not to have fully explained (please tell me on Linkedin if I’m wrong) EC2 Image Builder Distribution Configuration can only configure some elements of the Launch Template, not all!

And it doesn’t update the Launch Template in the sense that it retains previous config and simply updates the AMI ID. It updates in the sense that it creates a new version with the details you’ve told it to. In my case the Account ID and the Launch Template to update with the AMI ID. That makes sense when you step back and think about it, but it also felt unintuitive.

Why is this bad?

Because the updated version of the Launch Template no longer contains several key details about the rest of the required configuration, Instance profile, Security groups, key pair, etc.

I’ll save you the suspense - I checked with Amazon Q to point me to the explanation, and it had this to say:

“Unfortunately, EC2 Image Builder cannot preserve existing Launch Template settings when updating the AMI. When EC2 Image Builder creates a new Launch Template version through its distribution settings, it only copies the AMI ID to the new version - it doesn’t inherit or preserve other configuration settings like instance profiles, security groups, or other parameters from your existing Launch Template…
…the limitations are not explicitly stated in the official AWS docs. The documentation focuses on how to configure EC2 Image Builder to update Launch Templates, but it doesn’t clearly specify what happens to existing Launch Template settings when it creates a new version.”

The result of this was that my ASG was now spinning up instances with no instance profile, which wasn’t the only misconfiguration introduced, but this alone was a show stopper.

So what was the solution?

The solution I settled on was a two-parter, but let’s start by recapping what we want to happen:

Using Terraform: deploy & manage infrastructure that will create an ASG, Launch Templates, and an EC2 Image Builder pipeline that will update images frequently.
Shared custody of the latest AMI ID & Launch Template configuration between EC2 Image Builder and Terraform. By that I mean:
a. Terraform creates the infrastructure that holds the AMI details and doesn’t complain when…
b. The Launch Template is always referring to the latest AMI ID.
The Launch Template is updated as soon as an updated image is produced, without waiting for Terraform.
Terraform won’t freak out when it realises the Launch Template is using a newer AMI ID than it was expecting.

So practically how do we do this?

Step 1 - Get the IaC correct

First of all, Terraform needs to create an SSM Parameter as shown in this example:


data "aws_ssm_parameter" "al2023" {
  name = "/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-6.1-arm64"
}

resource "aws_ssm_parameter" "custom_built_image_id" {
  name        = "/imagebuilder/${var.project}/custom_id"
  description = "SSM Parameter for storing the AMI ID of the image built from EC2 Image Builder"
  type        = "String"
  data_type   = "aws:ec2:image"
  value       = data.aws_ssm_parameter.al2023.value

  lifecycle {
    ignore_changes = [
      value
    ]
  }
}

So what did we accomplish with the SSM parameter definition in Terraform?
This is our source of truth for our AMI image.

The Terraform definition for the Launch Template should retrieve the value stored here. Hence why it needs to be seeded with a real AMI ID to start with.
Something (no spoilers yet) needs to update the Launch Template with the latest custom built AMI ID. Hence why we have the lifecycle policy ignoring changes to the value.
Important 1 - The managed service role for EC2 Image Builder only has permissions to publish to items in Parameter Store that begin with the prefix /imagebuilder/ - see AWS docs on that point. i.e. You must use a name similar to the one in the example above.
Important 2 - data_type = "aws:ec2:image" must be set otherwise the Distribution Configuration in EC2 Image Builder will fail to update the parameter. (Ask me how I know this!)
Important 3 - Don’t forget to ignore changes to the value as shown above!

Now that we have somewhere to store the AMI ID, we need to make sure its referred to in our code, specifically in the Launch Template and the EC2 Image Builder distribution configs:

resource "aws_launch_template" "my_lt" {
  name_prefix            = "my-lt-"
  image_id               = aws_ssm_parameter.custom_built_image_id.value
  instance_type          = var.instance_type
  vpc_security_group_ids = [aws_security_group.private_sg.id]

  iam_instance_profile {
    name = aws_iam_instance_profile.custom_permissions.name
  }

  block_device_mappings {
    device_name = "/dev/xvda"
    ebs {
      volume_size = 20
      volume_type = "gp3"
      encrypted   = true
    }
  }

  tag_specifications {
    resource_type = "instance"
    tags = {
      Name    = "NamingIsHard"
      Project = var.project
    }
  }
}

resource "aws_imagebuilder_distribution_configuration" "dist" {
  name = "${var.project}-dist"

  distribution {
    region = var.region

    ami_distribution_configuration {
      name        = "${var.project}-{{imagebuilder:buildDate}}"
      description = "Custom built AL2023 image"

      ami_tags = {
        OS       = "AmazonLinux2023"
        Hardened = "true"
        Name     = "${var.project}-{{imagebuilder:buildDate}}"
      }
    }

    ssm_parameter_configuration {
      parameter_name = aws_ssm_parameter.custom_built_image_id.name
      ami_account_id = local.account_id
    }
  }
}

So now the Launch Template is happy, because even at boot time we’ve given it a valid AMI ID.
EC2 Image Builder is no longer sabotaging our Launch Template and is instead updating the SSM Parameter Store.

Step 2 - Join the dots

So, thus far we’ve completed our deployment, but we still need something to handle the Launch Template updates. Cue EventBridge & Lambda.
Whereas the native option in EC2 Image Builder would allow us to create a new version of the Launch Template with the new AMI ID, it didn’t allow us to specify the other important requirements like security group, profile etc.
A custom Lambda function allows us to do exactly that.

e.g. We can take this function, deploy it using Terraform…

import boto3
import os

ssm = boto3.client("ssm")
ec2 = boto3.client("ec2")

PARAM_NAME = os.environ["SSM_PARAM_NAME"]
LT_ID = os.environ["LAUNCH_TEMPLATE_ID"]
INSTANCE_TYPE = os.environ["INSTANCE_TYPE"]
SECURITY_GROUP_ID = os.environ["SECURITY_GROUP_ID"]
IAM_INSTANCE_PROFILE = os.environ["IAM_INSTANCE_PROFILE"]

def handler(event, context):
    # Get latest AMI from SSM
    response = ssm.get_parameter(Name=PARAM_NAME)
    ami_id = response["Parameter"]["Value"]

    # Create new Launch Template version
    ec2.create_launch_template_version(
        LaunchTemplateId=LT_ID,
        SourceVersion="$Default",
        LaunchTemplateData={
            "ImageId": ami_id,
            "InstanceType": INSTANCE_TYPE,
            "SecurityGroupIds": [SECURITY_GROUP_ID],
            "IamInstanceProfile": {"Name": IAM_INSTANCE_PROFILE}
        }
    )

    # Set the new version as default
    latest = ec2.describe_launch_template_versions(
        LaunchTemplateId=LT_ID,
        Versions=["$Latest"]
    )["LaunchTemplateVersions"][0]["VersionNumber"]

    ec2.modify_launch_template(
        LaunchTemplateId=LT_ID,
        DefaultVersion=str(latest)
    )

    print(f"Updated LT {LT_ID} to use {ami_id}")
    return {"statusCode": 200, "ami_id": ami_id}

What will this function do?

It takes five environment variables that are used to ensure all the required data for creating a useful new version of the Launch Template is included, but simply put it’s doing three things:

It obtains the latest AMI ID - that EC2 Image Builder’s Distribution config applied to the SSM Parameter.
Create a new version of the Launch Template, passing in the required details, image ID, instance type, security group(s), & Instance profile for permissions.
It updates the Launch Template to set the latest version it just created as the default version.

How do we trigger that function?

Now that we have the function to handle the updates, we just need to trigger it appropriately. We do that using EventBridge, and we can also deploy this from Terraform using something like this:

# This is used to trigger Lambda to update the ASG Launch Template when there's a new AMI available.

resource "aws_cloudwatch_event_rule" "imagebuilder_completed" {
  name          = "${var.project}-imagebuilder-completed"
  description   = "Trigger on successful image builds"
  event_pattern = <<EOF
{
  "source": ["aws.imagebuilder"],
  "detail-type": ["EC2 Image Builder Image State Change"],
  "detail": {
    "state": {
      "status": ["AVAILABLE"]
    }
  }
}
EOF
}

resource "aws_cloudwatch_event_target" "trigger_lambda" {
  rule      = aws_cloudwatch_event_rule.imagebuilder_completed.name
  target_id = "LaunchTemplateUpdater"
  arn       = aws_lambda_function.update_launch_template.arn
}

This creates an EventBridge rule that is triggered every time your EC2 Image Builder pipeline successfully produces an AMI with a status of ‘AVAILABLE’.

That rule in turn triggers the Lambda function shown above. So now we’ll have this sequence:

EC2 Image Builder runs and produces a new Image
It also updates the SSM Parameter store with the latest AMI ID
When the new AMI is updated to be ‘AVAILABLE’, our EventBridge Rule is triggered.
The EventBridge rule in turn runs the Lambda function to retrieve this new ID and create a new Launch Template version.

The result is that when the ASG next scales up, it’ll use the latest AMI ID.
It also means that on the next Terraform run it’ll pull the latest AMI ID from SSM and recognise that no change is required.

Conclusion

If you want to use EC2 Image Builder to manage AMIs for an ASG, all but the most simple use cases will require some extra plumbing when it comes to handling the Launch Template configuration.
In those more advanced scenarios a method as highlighted here will be effective in handling the synchronisation between Terraform’s view of the world and ensuring the Launch Template is always up to date (with a useful Instance Profile.)

Additional Resources

This post is part of my AWS technical blog series. Found this helpful? Connect with me on LinkedIn or check back here for more AWS content and discussions.

Half baked Images?

🔧 AWS Services Covered

TL;DR

What were we asked?

What’s the problem? Go!

+5 influence points if you see where this is going

Why don’t you just update the LT natively from EC2 Image Builder?

Half Baked Images!

Why is this bad?

So what was the solution?

Step 1 - Get the IaC correct

Step 2 - Join the dots

What will this function do?

How do we trigger that function?

Conclusion

Additional Resources

TL;DR#

What were we asked?#

What’s the problem? Go!#

+5 influence points if you see where this is going#

Why don’t you just update the LT natively from EC2 Image Builder?#

Half Baked Images!#

Why is this bad?#

So what was the solution?#

Step 1 - Get the IaC correct#

Step 2 - Join the dots#

What will this function do?#

How do we trigger that function?#

Conclusion#

Additional Resources#

TL;DR

What were we asked?

What’s the problem? Go!

+5 influence points if you see where this is going

Why don’t you just update the LT natively from EC2 Image Builder?

Half Baked Images!

Why is this bad?

So what was the solution?

Step 1 - Get the IaC correct

Step 2 - Join the dots

What will this function do?

How do we trigger that function?

Conclusion

Additional Resources