Remember that Lambda function I built to work around EC2 Image Builder’s Distribution Settings wiping your Launch Template config? You can bin it. AWS have updated the Distribution Settings behaviour and it now preserves your existing Launch Template settings when it creates a new version. The custom Lambda updater is no longer required.

A quick recap

In Half Baked Images I walked through the pain of integrating EC2 Image Builder with Terraform-managed Auto Scaling Groups. The core issue was that when EC2 Image Builder distributed a new AMI and updated your Launch Template, it created a new version containing only the AMI ID and account ID. Everything else, i.e. instance profile, security groups, key pair, block device mappings was gone.

The result? Your ASG would scale up instances with no instance profile, no security groups, and none of the configuration you’d carefully defined. Not ideal.

My solution was a two-parter:

1. Use an SSM Parameter as the source of truth for the AMI ID, updated by Image Builder’s Distribution Settings.
2. Deploy a custom Lambda function, triggered by EventBridge, that would read the latest AMI ID from SSM and create a proper new Launch Template version. One that actually included all the important bits.

It worked. It was reliable. But it was also plumbing that shouldn’t have been necessary.

What changed?

AWS have updated the EC2 Image Builder Distribution Settings behaviour. The documentation now states:

When launchTemplateConfigurations are present during the distribution process, Image Builder creates a new version of the launch template that includes all of the original settings from the template, and the new AMI ID from the build.

The important bit there: All of the original settings from the template.

That’s the bit that was missing before. Previously, Image Builder would create a new Launch Template version with only the AMI ID, effectively a blank template with a picture attached. Now it carries forward your existing configuration and simply swaps in the new AMI ID.

This is exactly the behaviour I expected when I first configured the launch_template_configuration block in my distribution config. It’s what most people would reasonably expect. And now it actually works that way.

What does this mean practically?

If you followed my original approach, you can simplify things significantly.

What you can remove

• The Lambda function that created new Launch Template versions with the full configuration.
• The EventBridge rule that triggered that Lambda on successful image builds.
• The associated IAM roles and policies for the Lambda function.
• Any CloudWatch log groups you set up for the Lambda.

That’s a decent chunk of infrastructure that existed purely to work around a limitation that no longer exists.

What you should keep

• The SSM Parameter approach is still a good pattern. Having a single source of truth for your AMI ID that both Terraform and Image Builder can reference is clean and avoids the bootstrapping headaches I described in the original post.
• The lifecycle ignore on the SSM Parameter value in Terraform, this is still needed so Terraform doesn’t fight with Image Builder over who owns that value.

What to add back

You can now use the launch_template_configuration block in your distribution config with confidence:

resource "aws_imagebuilder_distribution_configuration" "dist" {
  name = "${var.project}-dist"

  distribution {
    region = var.region

    ami_distribution_configuration {
      name        = "${var.project}-{{imagebuilder:buildDate}}"
      description = "Custom built AL2023 image"

      ami_tags = {
        OS       = "AmazonLinux2023"
        Hardened = "true"
        Name     = "${var.project}-{{imagebuilder:buildDate}}"
      }
    }

    launch_template_configuration {
      launch_template_id = aws_launch_template.my_lt.id
      account_id         = local.account_id
      set_default_version = true
    }
  }
}

Note the set_default_version = true… this tells Image Builder to set the newly created version as the default. Combined with the preserved settings, this means your ASG will pick up the new AMI on the next scale event without any Lambda glue in the middle.

Updated lab

I’ve updated the autoscaling_with_imagebuilder lab to reflect this. The Lambda updater and its associated EventBridge rule have been removed. The distribution config now handles the Launch Template update natively, as it always should have.

If you previously deployed the lab with the Lambda approach, you can safely remove those resources and update your distribution configuration. The SSM Parameter pattern remains, it’s still useful for the Terraform bootstrapping needs.

Thoughts

I’ll be honest, this is a satisfying update to write. The original solution worked, and I do think it was a reasonable approach given the constraints at the time. But it always felt like I was compensating for something that should have been a native capability.

The fact that AWS have addressed this is a good sign. It suggests they listened to the feedback (and I suspect I wasn’t the only one who ran into this). It also validates the frustration I had - if the fix was to make Distribution Settings preserve existing config, then the previous behaviour was the problem, not my expectations.

One less Lambda in the world. Now if AWS could just make their Lifecycle Policies stop nuking AMIs that are still in use, I might be able to retire the other Lambda too. A man can dream.

Additional Resources

• Half Baked Images - original blog post
• Half Baked Lifecycles - follow up blog post
• Updated Auto Scaling Group with EC2 Image Builder Lab
• EC2 Image Builder - Configure AMI distribution with a launch template

This post is part of my AWS technical blog series. Found this helpful? Connect with me on LinkedIn or check back here for more AWS content and discussions.

Using Terraform to deploy and manage an Auto Scaling Group that utilises custom AMIs built by EC2 Image Builder and dealing with the limitations of EC2 Image Builder’s native capabilities for managing Launch Templates.

What were we asked?

I’ve recently worked on a project that required an Auto Scaling Group of EC2 instances that utilised custom-built AMIs produced by EC2 Image Builder. The requirement was to have an EC2 Image Builder pipeline produce weekly images based on Amazon Linux 2023, with various custom components applied.
It’s quite straightforward to do this in Terraform, so happy days!

If you’ve used Terraform to manage Auto Scaling Groups in AWS, you’ll know that this is relatively straightforward too. There are lots of examples and use cases out there to draw from.
So it should be super simple to marry those two concepts together into one simple, happy, solution you’d think.
Yeah right!

If you want to skip the dramatic retelling of my pain and just get the answer, skip to the solution

What’s the problem? Go!

The first two parts are straightforward - the ASG definition and EC2 Image Builder pipeline setup in Terraform. As is the rest of the Terraform really. Working with my buddy we quickly had IaC that would deploy the VPC that would host our Auto Scaling Group, the Launch Template that would complement that ASG, and the EC2 Image Builder components, recipe, pipeline, and distribution, etc. As well as all the trimmings, IAM policies, security groups, KMS keys, etc, etc.
During the initial deploy attempt we encountered our first red flag. We had valid Terraform code, but it threw up this error:

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: Your query returned no results. Please change your search criteria and try again.
│
│   with data.aws_ami.latest_custom,
│   on data.tf line 486, in data "aws_ami" "latest_custom":
│  486: data "aws_ami" "latest_custom" {
│

I’d used a data source to capture the ID of the latest image that EC2 Image Builder would produce, but of course that hadn’t run yet.
I thought I’d been smart, I had foreseen the bootstrapping issue, and thought I’d get around it by temporarily pointing the image_id at an AWS managed SSM parameter that stores the Amazon Linux 2023 AMI ID.

resource "aws_launch_template" "my-cool-project-resource" {
  name_prefix            = "my-cool-project-name-lt-"
  #image_id               = data.aws_ami.latest_custom.id
  image_id               = data.aws_ssm_parameter.al2023.id

This didn’t prevent the error above. To my shame (not really, I’m actually quite pragmatic) I commented out the data source for my latest custom image and was able to successfully deploy my infrastructure. I immediately reversed the commented out sections. data.aws_ami.latest_custom was reinstated, and the Launch Template was reverted back to image_id = data.aws_ami.latest_custom.id and all worked well. Sorted.

+5 influence points if you see where this is going

It wasn’t sorted. Not completely.
I could now successfully deploy the infrastructure reliably, I’d hacked my way past the bootstrapping issue, but comforted by my solemn vow to myself that I’d go back and fix that. Scout’s honour.

The situation was better but not quite what we needed. We had an EC2 Image Builder pipeline that ran to a schedule and worked as expected. Our Launch Template looked good but it wasn’t using the most recent image. It was using the most recent -1. Consider this sequence of events:

1. Terraform deploys IaC, creating ASG, Launch Template, and EC2 Image builder pipeline.
2. EC2 Image Builder was triggered manually to confirm it was working.
3. EC2 Image Builder produces a new version of the AMI.
4. Launch Template is still running the TF deployed AMI version and doesn’t know about the one EC2 Image Builder produced.

This surfaced what might have been obvious to you dear reader, that the Launch Template must be built before we’ve built any images.
Also that our Pipeline would run independently of our Terraform code - and so our Launch Template would never know about the latest image until we next ran Terraform to update the Launch Template. That’s not ideal as the intention was to set up a CI/CD pipeline to run the Terraform code upon merges to the main branch.

I was opposed to the idea of scheduling the Terraform code to run after the pipeline schedule. That wouldn’t be pragmatic, that would be messy.

Why don’t you just update the LT natively from EC2 Image Builder?

Reasons!
Mostly that I didn’t know that was a thing at this point. I started looking at how other people have solved this, because surely using custom-built AMIs in Auto Scaling Groups is super common? It kinda is, but so is this challenge.

In actual fact, the distribution settings within EC2 Image Builder allow for updating a Launch Template with the latest AMI… directly. It’s explained nicely here.

So that resulted in me adding this launch_template_configuration block to my distribution config:

resource "aws_imagebuilder_distribution_configuration" "dist" {
  name = "${var.project}-dist"

  distribution {
    region = var.region

    ami_distribution_configuration {
      name        = "${var.project}-{{imagebuilder:buildDate}}"
      description = "A descriptive description describing things"

      ami_tags = {
        OS       = "AmazonLinux2023"
        Hardened = "true"
        Name     = "${var.project}-{{imagebuilder:buildDate}}"
      }
    }
    launch_template_configuration {
      launch_template_id = aws_launch_template.my-cool-project-resource.id
      account_id         = local.account_id
    }
  }
}

My distribution would now update the Launch Templates with details of the new AMI ID.
And it did.

Half Baked Images!

This was a step forward, if you’d first pivoted 180 degrees.
While it did solve the problem of the Launch Template not having the latest AMI - it absolutely did have the latest AMI now - it came at a price.

So, for some reason the documentation gods appear not to have fully explained (please tell me on Linkedin if I’m wrong) EC2 Image Builder Distribution Configuration can only configure some elements of the Launch Template, not all!

And it doesn’t update the Launch Template in the sense that it retains previous config and simply updates the AMI ID. It updates in the sense that it creates a new version with the details you’ve told it to. In my case the Account ID and the Launch Template to update with the AMI ID. That makes sense when you step back and think about it, but it also felt unintuitive.

Why is this bad?

Because the updated version of the Launch Template no longer contains several key details about the rest of the required configuration, Instance profile, Security groups, key pair, etc.

I’ll save you the suspense - I checked with Amazon Q to point me to the explanation, and it had this to say:

“Unfortunately, EC2 Image Builder cannot preserve existing Launch Template settings when updating the AMI. When EC2 Image Builder creates a new Launch Template version through its distribution settings, it only copies the AMI ID to the new version - it doesn’t inherit or preserve other configuration settings like instance profiles, security groups, or other parameters from your existing Launch Template…
…the limitations are not explicitly stated in the official AWS docs. The documentation focuses on how to configure EC2 Image Builder to update Launch Templates, but it doesn’t clearly specify what happens to existing Launch Template settings when it creates a new version.”

The result of this was that my ASG was now spinning up instances with no instance profile, which wasn’t the only misconfiguration introduced, but this alone was a show stopper.

So what was the solution?

The solution I settled on was a two-parter, but let’s start by recapping what we want to happen:

1. Using Terraform: deploy & manage infrastructure that will create an ASG, Launch Templates, and an EC2 Image Builder pipeline that will update images frequently.
2. Shared custody of the latest AMI ID & Launch Template configuration between EC2 Image Builder and Terraform. By that I mean:
   a. Terraform creates the infrastructure that holds the AMI details and doesn’t complain when…
   b. The Launch Template is always referring to the latest AMI ID.
3. The Launch Template is updated as soon as an updated image is produced, without waiting for Terraform.
4. Terraform won’t freak out when it realises the Launch Template is using a newer AMI ID than it was expecting.

So practically how do we do this?

Step 1 - Get the IaC correct

First of all, Terraform needs to create an SSM Parameter as shown in this example:

data "aws_ssm_parameter" "al2023" {
  name = "/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-6.1-arm64"
}

resource "aws_ssm_parameter" "custom_built_image_id" {
  name        = "/imagebuilder/${var.project}/custom_id"
  description = "SSM Parameter for storing the AMI ID of the image built from EC2 Image Builder"
  type        = "String"
  data_type   = "aws:ec2:image"
  value       = data.aws_ssm_parameter.al2023.value

  lifecycle {
    ignore_changes = [
      value
    ]
  }
}

So what did we accomplish with the SSM parameter definition in Terraform?
This is our source of truth for our AMI image.

• The Terraform definition for the Launch Template should retrieve the value stored here. Hence why it needs to be seeded with a real AMI ID to start with.
• Something (no spoilers yet) needs to update the Launch Template with the latest custom built AMI ID. Hence why we have the lifecycle policy ignoring changes to the value.
• Important 1 - The managed service role for EC2 Image Builder only has permissions to publish to items in Parameter Store that begin with the prefix /imagebuilder/ - see AWS docs on that point. i.e. You must use a name similar to the one in the example above.
• Important 2 - data_type = "aws:ec2:image" must be set otherwise the Distribution Configuration in EC2 Image Builder will fail to update the parameter. (Ask me how I know this!)
• Important 3 - Don’t forget to ignore changes to the value as shown above!

Now that we have somewhere to store the AMI ID, we need to make sure its referred to in our code, specifically in the Launch Template and the EC2 Image Builder distribution configs:

resource "aws_launch_template" "my_lt" {
  name_prefix            = "my-lt-"
  image_id               = aws_ssm_parameter.custom_built_image_id.value
  instance_type          = var.instance_type
  vpc_security_group_ids = [aws_security_group.private_sg.id]

  iam_instance_profile {
    name = aws_iam_instance_profile.custom_permissions.name
  }

  block_device_mappings {
    device_name = "/dev/xvda"
    ebs {
      volume_size = 20
      volume_type = "gp3"
      encrypted   = true
    }
  }

  tag_specifications {
    resource_type = "instance"
    tags = {
      Name    = "NamingIsHard"
      Project = var.project
    }
  }
}

resource "aws_imagebuilder_distribution_configuration" "dist" {
  name = "${var.project}-dist"

  distribution {
    region = var.region

    ami_distribution_configuration {
      name        = "${var.project}-{{imagebuilder:buildDate}}"
      description = "Custom built AL2023 image"

      ami_tags = {
        OS       = "AmazonLinux2023"
        Hardened = "true"
        Name     = "${var.project}-{{imagebuilder:buildDate}}"
      }
    }

    ssm_parameter_configuration {
      parameter_name = aws_ssm_parameter.custom_built_image_id.name
      ami_account_id = local.account_id
    }
  }
}

So now the Launch Template is happy, because even at boot time we’ve given it a valid AMI ID.
EC2 Image Builder is no longer sabotaging our Launch Template and is instead updating the SSM Parameter Store.

Step 2 - Join the dots

So, thus far we’ve completed our deployment, but we still need something to handle the Launch Template updates. Cue EventBridge & Lambda.
Whereas the native option in EC2 Image Builder would allow us to create a new version of the Launch Template with the new AMI ID, it didn’t allow us to specify the other important requirements like security group, profile etc.
A custom Lambda function allows us to do exactly that.

e.g. We can take this function, deploy it using Terraform…

import boto3
import os

ssm = boto3.client("ssm")
ec2 = boto3.client("ec2")

PARAM_NAME = os.environ["SSM_PARAM_NAME"]
LT_ID = os.environ["LAUNCH_TEMPLATE_ID"]
INSTANCE_TYPE = os.environ["INSTANCE_TYPE"]
SECURITY_GROUP_ID = os.environ["SECURITY_GROUP_ID"]
IAM_INSTANCE_PROFILE = os.environ["IAM_INSTANCE_PROFILE"]

def handler(event, context):
    # Get latest AMI from SSM
    response = ssm.get_parameter(Name=PARAM_NAME)
    ami_id = response["Parameter"]["Value"]

    # Create new Launch Template version
    ec2.create_launch_template_version(
        LaunchTemplateId=LT_ID,
        SourceVersion="$Default",
        LaunchTemplateData={
            "ImageId": ami_id,
            "InstanceType": INSTANCE_TYPE,
            "SecurityGroupIds": [SECURITY_GROUP_ID],
            "IamInstanceProfile": {"Name": IAM_INSTANCE_PROFILE}
        }
    )

    # Set the new version as default
    latest = ec2.describe_launch_template_versions(
        LaunchTemplateId=LT_ID,
        Versions=["$Latest"]
    )["LaunchTemplateVersions"][0]["VersionNumber"]

    ec2.modify_launch_template(
        LaunchTemplateId=LT_ID,
        DefaultVersion=str(latest)
    )

    print(f"Updated LT {LT_ID} to use {ami_id}")
    return {"statusCode": 200, "ami_id": ami_id}

What will this function do?

It takes five environment variables that are used to ensure all the required data for creating a useful new version of the Launch Template is included, but simply put it’s doing three things:

• It obtains the latest AMI ID - that EC2 Image Builder’s Distribution config applied to the SSM Parameter.
• Create a new version of the Launch Template, passing in the required details, image ID, instance type, security group(s), & Instance profile for permissions.
• It updates the Launch Template to set the latest version it just created as the default version.

How do we trigger that function?

Now that we have the function to handle the updates, we just need to trigger it appropriately. We do that using EventBridge, and we can also deploy this from Terraform using something like this:

# This is used to trigger Lambda to update the ASG Launch Template when there's a new AMI available.

resource "aws_cloudwatch_event_rule" "imagebuilder_completed" {
  name          = "${var.project}-imagebuilder-completed"
  description   = "Trigger on successful image builds"
  event_pattern = <<EOF
{
  "source": ["aws.imagebuilder"],
  "detail-type": ["EC2 Image Builder Image State Change"],
  "detail": {
    "state": {
      "status": ["AVAILABLE"]
    }
  }
}
EOF
}

resource "aws_cloudwatch_event_target" "trigger_lambda" {
  rule      = aws_cloudwatch_event_rule.imagebuilder_completed.name
  target_id = "LaunchTemplateUpdater"
  arn       = aws_lambda_function.update_launch_template.arn
}

This creates an EventBridge rule that is triggered every time your EC2 Image Builder pipeline successfully produces an AMI with a status of ‘AVAILABLE’.

That rule in turn triggers the Lambda function shown above. So now we’ll have this sequence:

1. EC2 Image Builder runs and produces a new Image
2. It also updates the SSM Parameter store with the latest AMI ID
3. When the new AMI is updated to be ‘AVAILABLE’, our EventBridge Rule is triggered.
4. The EventBridge rule in turn runs the Lambda function to retrieve this new ID and create a new Launch Template version.

The result is that when the ASG next scales up, it’ll use the latest AMI ID.
It also means that on the next Terraform run it’ll pull the latest AMI ID from SSM and recognise that no change is required.

Conclusion

If you want to use EC2 Image Builder to manage AMIs for an ASG, all but the most simple use cases will require some extra plumbing when it comes to handling the Launch Template configuration.
In those more advanced scenarios a method as highlighted here will be effective in handling the synchronisation between Terraform’s view of the world and ensuring the Launch Template is always up to date (with a useful Instance Profile.)

Additional Resources

• Complete Auto Scaling Group with EC2 Image Builder Example
• EC2 Image Builder & SSM

This post is part of my AWS technical blog series. Found this helpful? Connect with me on LinkedIn or check back here for more AWS content and discussions.