In my previous post, Half Baked Images I walked through building hardened AMIs using Amazon EC2 Image Builder, automating the pipeline, and wiring it into Launch Templates for Auto Scaling Groups. I touched on some aspects that felt almost, but not quite useful. In this post I’ll talk about my attempts to keep on top of the images produced and why (spoiler!) I ultimately chose Lambda over built-in Image Builder functionality.

Task: Stop AMIs and snapshots from piling up

AWS has an answer for that, of course: Image Builder lifecycle policies.
This should be easy (AKA Foreshadowing).

When asked “What does Lifecycle management in Image Builder do?”, Amazon Q gave this answer:

“The lifecycle manager implements automated rules to manage your image resources throughout their lifecycle, helping ensure image freshness while minimizing costs for underlying infrastructure like snapshot storage for AMIs or ECR repository storage for container images.”

I understand all those words so this is going to be easy peasy.

Lifecycle management made… confusing.

What I didn’t anticipate was how surprisingly awkward it would be to implement what felt like a very simple solution.

Unfortunately, the behaviour of those policies was less intuitive than the name suggested… to me at least.

This post is about the assumptions I made, why they were wrong, what Image Builder lifecycle actually does, and why I ultimately replaced it with a small Lambda function that behaved better for my use case.

Initial approach

I started with a simple goal:

Retain the last 5 AMIs produced by Image Builder.
Delete anything older, including associated snapshots.

Image Builder has a native lifecycle feature. Terraform exposes it cleanly. This felt so obvious that I didn’t question it. The console UI even said things like:

So I configured a lifecycle policy in terraform like this:

resource "aws_imagebuilder_lifecycle_policy" "retain_5_images" {
  name           = "${var.project}-lifecycle-policy"
  description    = "Retain only the 5 most recent AMIs created by Image Builder"
  execution_role = aws_iam_role.imagebuilder_role.arn

  resource_type = "AMI_IMAGE" 

  resource_selection {
    tag_map = {
      Project   = var.project
      ManagedBy = "AWSImageBuilder"
    }
  }

  policy_detail {
    action {
      type = "DELETE"
      include_resources {
        amis       = true
        containers = false
        snapshots  = true
      }
    }

    filter {
      type  = "COUNT" # or "COUNT"
      value = "5"
    }
  }
}

Note: This wasn’t my first attempt, this was the first that ran correctly.
It took me a week to get it to run. See Assumption #1.

Even now, recovering from the savage AMI cull of ‘25, I still read that code and think the policy should:

• Take the DELETE action for resources type “AMI_IMAGE”…
• also including resource types AMIs and snapshots, for those AMI_IMAGEs that matched…
• tags Project & ManagedBy, whilst retaining…
• Filter: COUNT = 5

Therefore I expected the outcome to be:

Find Images were Project = var.project and ManagedBy = "AWSImageBuilder"
If there are ≤5 images that match that criteria → delete nothing
If there are 6 → delete the oldest 1
If there are 18 → delete the oldest 13

etc. Simple. Predictable. Safe.

That is not what happened.
But what went wrong?

Assumption is the mother of all… fowl ups.

Assumption #1

My first assumption was that I would have some control over when the lifecycle management would run. Similar to controlling the pipeline schedule, I’d be able to control a lifecycle management schedule.

e.g. I’d be able to define a cron schedule or trigger it to run immediately after a new image became AVAILABLE. Nope.

Image Builder Lifecycle policies run on an internal AWS schedule, completely decoupled from the triggers or control I expected. There is also no “Run now” button to force an evaluation.

This had two consequences that significantly impacted my use case:

1. Testing is non-linear.
   Unsurprisingly my lifecycle didn’t work on the first try. So, when I attempted to fix it, I had to wait 24 hours to see if that had worked.
   Testing, that could have taken a couple of hours, took days to work through as a result of this approach.

2. Troubleshooting was complex and misleading
   I was working on a few aspects of the code simultaneously, so by the time my lifecycle ran each day I had usually made several changes.

   • Tags had been added or changed
   • IAM permissions were tightened
   • Images components were changing, meaning recipes had been updated.

The combination of those two realities made it very difficult to figure out cause and effect: Was lifecycle management failing because I’d borked the config? Was it related to my tagging changes? or the new recipe?
sigh.

Assumption #2

My second assumption was that the lifecycle policy in Image Builder relates to the images built by Image Builder… that is to say - the AMIs.

Sensible assumption? I thought so… and if you did too… join me in my pit of despair because, as we’ll get to in a moment, this was not the case.

Assumption #3

My third, and what proved to be my most costly assumption, was that I’d provided enough details to safely identify the images to be deleted and that only those Images would be deleted. That this turned out to be incorrect was the real sore point - this one wasn’t easy to predict or see coming until it went wrong.

So what actually happened?

When my lifecycle policy ran, it removed all AMIs. Everything.
Including AMIs still in use by my Auto Scaling Group’s Launch Template.
Nothing useful remained. Zero AMIs, zero snapshots.

Thankfully this occurred in a lab environment so wasn’t too impactful… but it did give me chills. I looked over the code, at the resources, and the lifecycle policy in the console, etc. It all looked correct. It seemed to make sense - but here I was with no useable images left.
And at the moment of discovering this issue, even from that point, it took a good few hours to understand why it had gone so badly wrong.

To say I was a bit surprised at the outcome is putting it mildly.
This could not be allowed to happen in an environment that matters to me.

Where did the assumptions fall down?

1. Not having control over the scheduling of the lifecycle policy was simultaneously a relatively minor issue and almost enough to be a deal breaker on its own. Who’s happy waiting 24 hours to test configuration changes in a lab? Fail fast isn’t an option, if you have to wait 24 hours to test each change.

2. My assumption that my lifecycle policy was targeting my AMI’s directly was incorrect. This one really mattered.
   EC2 Image Builder Lifecycle Policies target Image builder resources, e.g. things with arns like arn:aws:imagebuilder:eu-west-2:123456789012:image/custom-linux-build/1.0.6/3.
   This is a subtle but important distinction to understand when you’re thinking about how Image Builder Lifecycle policies identify targets. You can, as I did in the code above, state that it should include amis = true & snapshots = true. But, again, the AMIs and snapshots are secondary in this.

3. Assuming I’d provided the correct selection criteria was compounded by the previous assumption about what the lifecycle policies target.
   I mentioned that after my lifecycle policy ran I had zero AMI’s and zero snapshots.
   But I did still have 5 “images”. I still had 5 Image Builder Image’s retained. Although the AMI’s that they pointed to were gone.

Yup, I swore inwardly at this point.

Bringing it home

I asked Amazon Q why it would do this (to me), and in its reply it had this little unhelpful truth bomb:

“Even if the Image Builder Image resources were retained, if your policy had:"includeResources": { "amis": true, "snapshots": true } This would delete the underlying AMIs and snapshots while potentially keeping the Image Builder metadata.”

I swore outwardly at this point. Loudly.

This ia an awful failure condition by any stretch. I accept that this was my configuration, and I am responsible for this, it still feels like this was a booby trap waiting to go off rather than something you could reasonably foresee & avoid.

So what did I do about it

After a fair amount of frustration, confusion, and lost AMIs I decided to abandon the Image Builder lifecycle policy approach and turned to AWS Lambda - what can’t Lambda fix?

I created a small function that I deployed using Terraform, as follows:

import os
import json
import boto3
import logging
from datetime import datetime, timezone

logger = logging.getLogger()
logger.setLevel(logging.INFO)

ec2 = boto3.client("ec2")

RETAIN_COUNT = int(os.getenv("RETAIN_COUNT", "5"))
PROJECT_TAG_VALUE = os.getenv("PROJECT_TAG_VALUE", "labpoc")
MANAGED_BY_VALUE = os.getenv("MANAGED_BY_VALUE", "AWSImageBuilder")
LAUNCH_TEMPLATE_ID = os.getenv("LAUNCH_TEMPLATE_ID")  # required for protection
DRY_RUN = os.getenv("DRY_RUN", "true").lower() == "true"


def _parse_dt(s: str) -> datetime:
    return datetime.fromisoformat(s.replace("Z", "+00:00")).astimezone(timezone.utc)


def _list_matching_amis():
    filters = [
        {"Name": "tag:Project", "Values": [PROJECT_TAG_VALUE]},
        {"Name": "tag:ManagedBy", "Values": [MANAGED_BY_VALUE]},
        {"Name": "state", "Values": ["available"]},
    ]
    resp = ec2.describe_images(Owners=["self"], Filters=filters)
    images = resp.get("Images", [])
    images.sort(key=lambda i: _parse_dt(i["CreationDate"]), reverse=True)  # newest first
    return images


def _snapshots_for_ami(image: dict) -> list[str]:
    snaps = []
    for bdm in image.get("BlockDeviceMappings", []):
        ebs = bdm.get("Ebs")
        if ebs and ebs.get("SnapshotId"):
            snaps.append(ebs["SnapshotId"])
    return snaps


def _summarise_images(images: list[dict], max_items: int = 50) -> list[dict]:
    out = []
    for img in images[:max_items]:
        out.append({
            "ImageId": img.get("ImageId"),
            "CreationDate": img.get("CreationDate"),
            "Name": img.get("Name"),
        })
    if len(images) > max_items:
        out.append({"note": f"truncated: showing {max_items} of {len(images)}"})
    return out


def _amis_referenced_by_latest_lt_versions(launch_template_id: str, protect_versions: int) -> set[str]:
    """
    Protect AMIs referenced by the *latest N* launch template versions.
    """
    protected = set()

    # Fetch versions (we'll sort and take the latest N)
    versions = []
    paginator = ec2.get_paginator("describe_launch_template_versions")
    for page in paginator.paginate(LaunchTemplateId=launch_template_id):
        versions.extend(page.get("LaunchTemplateVersions", []))

    if not versions:
        return protected

    # Sort by VersionNumber descending and take latest N
    versions.sort(key=lambda v: v.get("VersionNumber", 0), reverse=True)
    latest = versions[:protect_versions]

    for v in latest:
        data = v.get("LaunchTemplateData", {})
        image_id = data.get("ImageId")
        if image_id:
            protected.add(image_id)

    return protected


def lambda_handler(event, context):
    logger.info(
        "Starting AMI retention. retain=%s project=%s managedby=%s dry_run=%s lt_id=%s",
        RETAIN_COUNT, PROJECT_TAG_VALUE, MANAGED_BY_VALUE, DRY_RUN, LAUNCH_TEMPLATE_ID
    )

    if RETAIN_COUNT < 1:
        raise ValueError("RETAIN_COUNT must be >= 1")

    images = _list_matching_amis()
    total = len(images)
    logger.info("Found %s matching AMIs.", total)
    logger.info("Matched AMIs (newest->oldest): %s", json.dumps(_summarise_images(images)))


    if total <= RETAIN_COUNT:
        logger.info("<= %s matching AMIs; deleting none.", RETAIN_COUNT)
        return {"statusCode": 200, "body": json.dumps({"matching": total, "deleted": 0})}

    protected = set()
    if LAUNCH_TEMPLATE_ID:
        protected = _amis_referenced_by_latest_lt_versions(LAUNCH_TEMPLATE_ID, RETAIN_COUNT)
        logger.info("Protected AMIs referenced by latest %s Launch Template versions: %s",
                RETAIN_COUNT, list(sorted(protected)))
    else:
        logger.warning("LAUNCH_TEMPLATE_ID not set; NOT protecting in-use AMIs (not recommended).")

    # Keep newest RETAIN_COUNT (always)
    keep_newest = {img["ImageId"] for img in images[:RETAIN_COUNT]}

    # Eligible = older than newest RETAIN_COUNT AND not protected by LT
    older = images[RETAIN_COUNT:]
    candidates = []
    for img in older:
        ami_id = img["ImageId"]
        if ami_id in protected:
            continue
        candidates.append(img)

    logger.info("Candidates for deletion (older than newest %s, not referenced by LT): %s",
                RETAIN_COUNT, len(candidates))

    deleted = []
    for img in candidates:
        ami_id = img["ImageId"]
        snaps = _snapshots_for_ami(img)

        logger.info("Deleting AMI %s (snapshots=%s)", ami_id, snaps)

        if not DRY_RUN:
            ec2.deregister_image(ImageId=ami_id)
            for snap_id in snaps:
                ec2.delete_snapshot(SnapshotId=snap_id)

        deleted.append({"ami": ami_id, "snapshots": snaps})

    logger.info("Done. Deleted %s AMIs.", len(deleted))
    return {"statusCode": 200, "body": json.dumps({"matching": total, "deleted": len(deleted), "items": deleted})}

Note: the environment vars, you can set these in terraform when you specify your aws_lambda_function

  environment {
    variables = {
      RETAIN_COUNT       = tostring(var.ami_retain_count)
      PROJECT_TAG_VALUE  = var.project
      MANAGED_BY_VALUE   = "AWSImageBuilder"
      DRY_RUN            = "true" # flip to false when confident it's working
      LAUNCH_TEMPLATE_ID = aws_launch_template.wafr_lt.id
    }
  }

This Lambda function is, in many ways, much simpler than what I thought I’d been doing with the Image Builder Lifecycle Manager… This function:

• Lists AMIs by tag (Project, ManagedBy)
• Sorts them by creation date
• Keeps the newest N (defined by Terraform variable ami_retain_count)
• Deregisters anything older
• Deletes the snapshots referenced by those AMIs
• Never deletes AMIs currently referenced by the latest N Launch Template (also defined by ami_retain_count.)

This runs on a schedule I control. And understand!
It’s observable, repeatable, and it can be run in dry-run mode while testing.

Most importantly, the behaviour can be explained in one sentence:

“We keep the 5 newest AMIs for this project, and we never delete one that’s in use.”

That is the rule I thought I was configuring when I attempted to enable lifecycle retention — but now it’s actually true!

Why a custom solution won (& why I’m okay with that)

After working through this in detail, I came to a conclusion that surprised me:
Image Builder lifecycle policies are not a good fit for intuitive AMI retention.

They are:

• metadata-centric
• recipe-scoped
• on their own schedule
• capable of deleting artifacts without leaving usable AMIs behind (talk about sharp edges!)

This doesn’t mean lifecycle policies are broken — but it does make them dangerous if your mental model is “Keep the last N usable AMIs.”

In my experience this model doesn’t map cleanly to how Image Builder lifecycle works.

By contrast my Lambda function is:

• explicit - I could set my own boundaries that did what I expected them to do.
• controlled - it runs when i want.
• able to add clear and simple guardrails - e.g. don’t delete any AMIs still in use, regardless of other filters.
• Auditable - see what’s going on easily in the execution logs.
• Test friendly. Flip “dry-run” to ’true’ to test what would be deleted before you take the training wheels off.

The result was a solution that worked as I expected, trimmed my AMIs and Snapshots to the number I wanted, and stopped my bill from growing forever with the ghost of a million unused snapshots, allowed me to safely test changes when I wanted. And it only cost only a tiny amount in Lambda usage. Worth it.

Final thoughts

EC2 Image Builder is excellent at building & customising images.
EventBridge and Lambda are excellent at reacting to them.

But AMI retention is an operational concern, not a build concern — and once I treated it that way, the solution became simpler, safer, and easier to understand.

If you’re currently using Image Builder lifecycle today and it works for you, that’s great.

If, however, your mental model is “keep the last N usable AMIs”, it’s worth pausing and double-checking that the system is actually doing what you think it is.

Sometimes the most robust solution is the one you can explain easily — and without waiting 24 hours to see what it decided to do.

Additional Resources

• Complete EC2 Image Builder with Lifecycle Management Example
• Amazon EC2 Image Builder
• AWS Lambda
• Half Baked Images - previous blog post

This post is part of my AWS technical blog series. Found this helpful? Connect with me on LinkedIn or check back here for more AWS content and discussions.

Using Terraform to deploy and manage an Auto Scaling Group that utilises custom AMIs built by EC2 Image Builder and dealing with the limitations of EC2 Image Builder’s native capabilities for managing Launch Templates.

What were we asked?

I’ve recently worked on a project that required an Auto Scaling Group of EC2 instances that utilised custom-built AMIs produced by EC2 Image Builder. The requirement was to have an EC2 Image Builder pipeline produce weekly images based on Amazon Linux 2023, with various custom components applied.
It’s quite straightforward to do this in Terraform, so happy days!

If you’ve used Terraform to manage Auto Scaling Groups in AWS, you’ll know that this is relatively straightforward too. There are lots of examples and use cases out there to draw from.
So it should be super simple to marry those two concepts together into one simple, happy, solution you’d think.
Yeah right!

If you want to skip the dramatic retelling of my pain and just get the answer, skip to the solution

What’s the problem? Go!

The first two parts are straightforward - the ASG definition and EC2 Image Builder pipeline setup in Terraform. As is the rest of the Terraform really. Working with my buddy we quickly had IaC that would deploy the VPC that would host our Auto Scaling Group, the Launch Template that would complement that ASG, and the EC2 Image Builder components, recipe, pipeline, and distribution, etc. As well as all the trimmings, IAM policies, security groups, KMS keys, etc, etc.
During the initial deploy attempt we encountered our first red flag. We had valid Terraform code, but it threw up this error:

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: Your query returned no results. Please change your search criteria and try again.
│
│   with data.aws_ami.latest_custom,
│   on data.tf line 486, in data "aws_ami" "latest_custom":
│  486: data "aws_ami" "latest_custom" {
│

I’d used a data source to capture the ID of the latest image that EC2 Image Builder would produce, but of course that hadn’t run yet.
I thought I’d been smart, I had foreseen the bootstrapping issue, and thought I’d get around it by temporarily pointing the image_id at an AWS managed SSM parameter that stores the Amazon Linux 2023 AMI ID.

resource "aws_launch_template" "my-cool-project-resource" {
  name_prefix            = "my-cool-project-name-lt-"
  #image_id               = data.aws_ami.latest_custom.id
  image_id               = data.aws_ssm_parameter.al2023.id

This didn’t prevent the error above. To my shame (not really, I’m actually quite pragmatic) I commented out the data source for my latest custom image and was able to successfully deploy my infrastructure. I immediately reversed the commented out sections. data.aws_ami.latest_custom was reinstated, and the Launch Template was reverted back to image_id = data.aws_ami.latest_custom.id and all worked well. Sorted.

+5 influence points if you see where this is going

It wasn’t sorted. Not completely.
I could now successfully deploy the infrastructure reliably, I’d hacked my way past the bootstrapping issue, but comforted by my solemn vow to myself that I’d go back and fix that. Scout’s honour.

The situation was better but not quite what we needed. We had an EC2 Image Builder pipeline that ran to a schedule and worked as expected. Our Launch Template looked good but it wasn’t using the most recent image. It was using the most recent -1. Consider this sequence of events:

1. Terraform deploys IaC, creating ASG, Launch Template, and EC2 Image builder pipeline.
2. EC2 Image Builder was triggered manually to confirm it was working.
3. EC2 Image Builder produces a new version of the AMI.
4. Launch Template is still running the TF deployed AMI version and doesn’t know about the one EC2 Image Builder produced.

This surfaced what might have been obvious to you dear reader, that the Launch Template must be built before we’ve built any images.
Also that our Pipeline would run independently of our Terraform code - and so our Launch Template would never know about the latest image until we next ran Terraform to update the Launch Template. That’s not ideal as the intention was to set up a CI/CD pipeline to run the Terraform code upon merges to the main branch.

I was opposed to the idea of scheduling the Terraform code to run after the pipeline schedule. That wouldn’t be pragmatic, that would be messy.

Why don’t you just update the LT natively from EC2 Image Builder?

Reasons!
Mostly that I didn’t know that was a thing at this point. I started looking at how other people have solved this, because surely using custom-built AMIs in Auto Scaling Groups is super common? It kinda is, but so is this challenge.

In actual fact, the distribution settings within EC2 Image Builder allow for updating a Launch Template with the latest AMI… directly. It’s explained nicely here.

So that resulted in me adding this launch_template_configuration block to my distribution config:

resource "aws_imagebuilder_distribution_configuration" "dist" {
  name = "${var.project}-dist"

  distribution {
    region = var.region

    ami_distribution_configuration {
      name        = "${var.project}-{{imagebuilder:buildDate}}"
      description = "A descriptive description describing things"

      ami_tags = {
        OS       = "AmazonLinux2023"
        Hardened = "true"
        Name     = "${var.project}-{{imagebuilder:buildDate}}"
      }
    }
    launch_template_configuration {
      launch_template_id = aws_launch_template.my-cool-project-resource.id
      account_id         = local.account_id
    }
  }
}

My distribution would now update the Launch Templates with details of the new AMI ID.
And it did.

Half Baked Images!

This was a step forward, if you’d first pivoted 180 degrees.
While it did solve the problem of the Launch Template not having the latest AMI - it absolutely did have the latest AMI now - it came at a price.

So, for some reason the documentation gods appear not to have fully explained (please tell me on Linkedin if I’m wrong) EC2 Image Builder Distribution Configuration can only configure some elements of the Launch Template, not all!

And it doesn’t update the Launch Template in the sense that it retains previous config and simply updates the AMI ID. It updates in the sense that it creates a new version with the details you’ve told it to. In my case the Account ID and the Launch Template to update with the AMI ID. That makes sense when you step back and think about it, but it also felt unintuitive.

Why is this bad?

Because the updated version of the Launch Template no longer contains several key details about the rest of the required configuration, Instance profile, Security groups, key pair, etc.

I’ll save you the suspense - I checked with Amazon Q to point me to the explanation, and it had this to say:

“Unfortunately, EC2 Image Builder cannot preserve existing Launch Template settings when updating the AMI. When EC2 Image Builder creates a new Launch Template version through its distribution settings, it only copies the AMI ID to the new version - it doesn’t inherit or preserve other configuration settings like instance profiles, security groups, or other parameters from your existing Launch Template…
…the limitations are not explicitly stated in the official AWS docs. The documentation focuses on how to configure EC2 Image Builder to update Launch Templates, but it doesn’t clearly specify what happens to existing Launch Template settings when it creates a new version.”

The result of this was that my ASG was now spinning up instances with no instance profile, which wasn’t the only misconfiguration introduced, but this alone was a show stopper.

So what was the solution?

The solution I settled on was a two-parter, but let’s start by recapping what we want to happen:

1. Using Terraform: deploy & manage infrastructure that will create an ASG, Launch Templates, and an EC2 Image Builder pipeline that will update images frequently.
2. Shared custody of the latest AMI ID & Launch Template configuration between EC2 Image Builder and Terraform. By that I mean:
   a. Terraform creates the infrastructure that holds the AMI details and doesn’t complain when…
   b. The Launch Template is always referring to the latest AMI ID.
3. The Launch Template is updated as soon as an updated image is produced, without waiting for Terraform.
4. Terraform won’t freak out when it realises the Launch Template is using a newer AMI ID than it was expecting.

So practically how do we do this?

Step 1 - Get the IaC correct

First of all, Terraform needs to create an SSM Parameter as shown in this example:

data "aws_ssm_parameter" "al2023" {
  name = "/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-6.1-arm64"
}

resource "aws_ssm_parameter" "custom_built_image_id" {
  name        = "/imagebuilder/${var.project}/custom_id"
  description = "SSM Parameter for storing the AMI ID of the image built from EC2 Image Builder"
  type        = "String"
  data_type   = "aws:ec2:image"
  value       = data.aws_ssm_parameter.al2023.value

  lifecycle {
    ignore_changes = [
      value
    ]
  }
}

So what did we accomplish with the SSM parameter definition in Terraform?
This is our source of truth for our AMI image.

• The Terraform definition for the Launch Template should retrieve the value stored here. Hence why it needs to be seeded with a real AMI ID to start with.
• Something (no spoilers yet) needs to update the Launch Template with the latest custom built AMI ID. Hence why we have the lifecycle policy ignoring changes to the value.
• Important 1 - The managed service role for EC2 Image Builder only has permissions to publish to items in Parameter Store that begin with the prefix /imagebuilder/ - see AWS docs on that point. i.e. You must use a name similar to the one in the example above.
• Important 2 - data_type = "aws:ec2:image" must be set otherwise the Distribution Configuration in EC2 Image Builder will fail to update the parameter. (Ask me how I know this!)
• Important 3 - Don’t forget to ignore changes to the value as shown above!

Now that we have somewhere to store the AMI ID, we need to make sure its referred to in our code, specifically in the Launch Template and the EC2 Image Builder distribution configs:

resource "aws_launch_template" "my_lt" {
  name_prefix            = "my-lt-"
  image_id               = aws_ssm_parameter.custom_built_image_id.value
  instance_type          = var.instance_type
  vpc_security_group_ids = [aws_security_group.private_sg.id]

  iam_instance_profile {
    name = aws_iam_instance_profile.custom_permissions.name
  }

  block_device_mappings {
    device_name = "/dev/xvda"
    ebs {
      volume_size = 20
      volume_type = "gp3"
      encrypted   = true
    }
  }

  tag_specifications {
    resource_type = "instance"
    tags = {
      Name    = "NamingIsHard"
      Project = var.project
    }
  }
}

resource "aws_imagebuilder_distribution_configuration" "dist" {
  name = "${var.project}-dist"

  distribution {
    region = var.region

    ami_distribution_configuration {
      name        = "${var.project}-{{imagebuilder:buildDate}}"
      description = "Custom built AL2023 image"

      ami_tags = {
        OS       = "AmazonLinux2023"
        Hardened = "true"
        Name     = "${var.project}-{{imagebuilder:buildDate}}"
      }
    }

    ssm_parameter_configuration {
      parameter_name = aws_ssm_parameter.custom_built_image_id.name
      ami_account_id = local.account_id
    }
  }
}

So now the Launch Template is happy, because even at boot time we’ve given it a valid AMI ID.
EC2 Image Builder is no longer sabotaging our Launch Template and is instead updating the SSM Parameter Store.

Step 2 - Join the dots

So, thus far we’ve completed our deployment, but we still need something to handle the Launch Template updates. Cue EventBridge & Lambda.
Whereas the native option in EC2 Image Builder would allow us to create a new version of the Launch Template with the new AMI ID, it didn’t allow us to specify the other important requirements like security group, profile etc.
A custom Lambda function allows us to do exactly that.

e.g. We can take this function, deploy it using Terraform…

import boto3
import os

ssm = boto3.client("ssm")
ec2 = boto3.client("ec2")

PARAM_NAME = os.environ["SSM_PARAM_NAME"]
LT_ID = os.environ["LAUNCH_TEMPLATE_ID"]
INSTANCE_TYPE = os.environ["INSTANCE_TYPE"]
SECURITY_GROUP_ID = os.environ["SECURITY_GROUP_ID"]
IAM_INSTANCE_PROFILE = os.environ["IAM_INSTANCE_PROFILE"]

def handler(event, context):
    # Get latest AMI from SSM
    response = ssm.get_parameter(Name=PARAM_NAME)
    ami_id = response["Parameter"]["Value"]

    # Create new Launch Template version
    ec2.create_launch_template_version(
        LaunchTemplateId=LT_ID,
        SourceVersion="$Default",
        LaunchTemplateData={
            "ImageId": ami_id,
            "InstanceType": INSTANCE_TYPE,
            "SecurityGroupIds": [SECURITY_GROUP_ID],
            "IamInstanceProfile": {"Name": IAM_INSTANCE_PROFILE}
        }
    )

    # Set the new version as default
    latest = ec2.describe_launch_template_versions(
        LaunchTemplateId=LT_ID,
        Versions=["$Latest"]
    )["LaunchTemplateVersions"][0]["VersionNumber"]

    ec2.modify_launch_template(
        LaunchTemplateId=LT_ID,
        DefaultVersion=str(latest)
    )

    print(f"Updated LT {LT_ID} to use {ami_id}")
    return {"statusCode": 200, "ami_id": ami_id}

What will this function do?

It takes five environment variables that are used to ensure all the required data for creating a useful new version of the Launch Template is included, but simply put it’s doing three things:

• It obtains the latest AMI ID - that EC2 Image Builder’s Distribution config applied to the SSM Parameter.
• Create a new version of the Launch Template, passing in the required details, image ID, instance type, security group(s), & Instance profile for permissions.
• It updates the Launch Template to set the latest version it just created as the default version.

How do we trigger that function?

Now that we have the function to handle the updates, we just need to trigger it appropriately. We do that using EventBridge, and we can also deploy this from Terraform using something like this:

# This is used to trigger Lambda to update the ASG Launch Template when there's a new AMI available.

resource "aws_cloudwatch_event_rule" "imagebuilder_completed" {
  name          = "${var.project}-imagebuilder-completed"
  description   = "Trigger on successful image builds"
  event_pattern = <<EOF
{
  "source": ["aws.imagebuilder"],
  "detail-type": ["EC2 Image Builder Image State Change"],
  "detail": {
    "state": {
      "status": ["AVAILABLE"]
    }
  }
}
EOF
}

resource "aws_cloudwatch_event_target" "trigger_lambda" {
  rule      = aws_cloudwatch_event_rule.imagebuilder_completed.name
  target_id = "LaunchTemplateUpdater"
  arn       = aws_lambda_function.update_launch_template.arn
}

This creates an EventBridge rule that is triggered every time your EC2 Image Builder pipeline successfully produces an AMI with a status of ‘AVAILABLE’.

That rule in turn triggers the Lambda function shown above. So now we’ll have this sequence:

1. EC2 Image Builder runs and produces a new Image
2. It also updates the SSM Parameter store with the latest AMI ID
3. When the new AMI is updated to be ‘AVAILABLE’, our EventBridge Rule is triggered.
4. The EventBridge rule in turn runs the Lambda function to retrieve this new ID and create a new Launch Template version.

The result is that when the ASG next scales up, it’ll use the latest AMI ID.
It also means that on the next Terraform run it’ll pull the latest AMI ID from SSM and recognise that no change is required.

Conclusion

If you want to use EC2 Image Builder to manage AMIs for an ASG, all but the most simple use cases will require some extra plumbing when it comes to handling the Launch Template configuration.
In those more advanced scenarios a method as highlighted here will be effective in handling the synchronisation between Terraform’s view of the world and ensuring the Launch Template is always up to date (with a useful Instance Profile.)

Additional Resources

• Complete Auto Scaling Group with EC2 Image Builder Example
• EC2 Image Builder & SSM

This post is part of my AWS technical blog series. Found this helpful? Connect with me on LinkedIn or check back here for more AWS content and discussions.