Using built-in CloudFormation macros to source AMI IDs by a friendly identifier

Using AWS::Include to perform lookups of AMI IDs by region in CloudFormation Templates

Table Of Contents

Today I Explained

CloudFormation doesn’t support any lookup mechanisms for Amazon Machine Images (AMIs), which is why you’ll often see AMIs for EC2s specified either as:

  • A Parameter to the CloudFormation Stack, sometimes defaulted to a Parameter Store entry
  • AMI public parameters for Amazon Linux (& some others) AMIs
  • Systems Manager Parameter pre-created in the AWS Account by some mechanism (post-publish step/cronjob)
  • An AMI by Region Mapping within the CloudFormation Template
  • Hardcoded AMI usage within the CloudFormation Template
  • A pre-deployment step to perform lookup of the AMI, as part of the deployment pipeline

This can raise questions around what options exist for looking up AMI IDs by name (or tags) from within CloudFormation, similar to how one might do this using Terraform’s data.aws_ami.

In practice, this becomes the responsibility of the deployer, as it is often straightforward to modify the deployment pipeline to perform the lookup at time of deploy. Options around hardcoding of an AMI can be frustrating, as when copying an AMI to another region, the ID will be different between regions.

Populating the Systems Manager Parameters for each of the AWS Accounts & Regions making use of the AMIs is possible, but typically involves active infrastructure that is continuously pushing new AMI ID parameters to accounts, and expiring unused ones.

This is why within some CloudFormation stacks, you’ll see the use of AMI Mappings, which compute the AMI ID and bake them into the CloudFormation Template:

AWSTemplateFormatVersion: '2010-09-09'
Mappings:
  AWSRegionToAMI:
    ap-south-1:
      AMIID: ami-2ed19c41
    eu-west-2:
      AMIID: ami-e3051987
    eu-west-1:
      AMIID: ami-760aaa0f
    ap-northeast-2:
      AMIID: ami-fc862292
    ap-northeast-1:
      AMIID: ami-2803ac4e
    sa-east-1:
      AMIID: ami-1678037a
    ca-central-1:
      AMIID: ami-ef3b838b
    ap-southeast-1:
      AMIID: ami-dd7935be
    ap-southeast-2:
      AMIID: ami-1a668878
    eu-central-1:
      AMIID: ami-e28d098d
    us-east-1:
      AMIID: ami-6057e21a
    us-east-2:
      AMIID: ami-aa1b34cf
    us-west-1:
      AMIID: ami-1a033c7a
    us-west-2:
      AMIID: ami-32d8124a

This creates a bit of a frustration point in that it isn’t communicated what machine image the AMI IDs are referring to. This can require an additional step of looking up one of the AMI IDs to determine the operating system, architecture, or release date. This can be addressed using CloudFormation’s Metadata section which can describe the AMIs. This relies on this section being kept up to date, or the build tooling being responsible for ensuring the Metadata is populated.

Alternatively, you might consider the use of the CloudFormation Macro, (AWS::Include)[https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/create-reusable-transform-function-snippets-and-add-to-your-template-with-aws-include-transform.html].

Use the AWS::Include transform, which is a macro hosted by AWS CloudFormation, to insert boilerplate content into your templates. The AWS::Include transform lets you create a reference to a template snippet in an Amazon S3 bucket.

This is a pre-processor, which allows for a pattern of re-use within CloudFormation Templates. This means that the AWSRegionToAMI mapping can be changed to a descriptive path of an S3 Object which contains the mapping. This S3 Object can be continuously updated as the AMI is mirrored to more regions.

Mappings:
  AWSRegionToAMI:
    Fn::Transform:
      Name: AWS::Include
      Parameters:
        Location: "s3://ami-inventory-20230621021141012900000001/amis/ubuntu/images/ubuntu-xenial-16.04-amd64-server-20210928.json"

This makes it possible to reference the region to AMI ID by a descriptive path, which clearly communicates that we are selecting the Ubuntu Image for AMD64 that was released in 2021. The exact AMI ID in-use can still be made accessible through an Output of the CloudFormation Template.

This allows for removing the hardcoded AMI Mapping, and doesn’t require provisioning of Systems Manager Parameters within all AWS Accounts deploying this CloudFormation Template. The consequence of this though is that we’ve obfuscated the AMI IDs in-use by our CloudFormation Template into an “ami inventory”, which may or may not be desirable.

A note on Coupling

The usage of AWS::Include introduces a coupling for our infrastructure as code to our infrastructure. The macro AWS::Include is a pre-processor, that does not support any mechanism of substitution within its arguments. This means that the parameter Location is hardcoded. This reduces the portability of our CloudFormatio Templates, as they are now tightly coupled to the access permissions of the AWS S3 bucket that contains the boilerplate for our AMI Mappings.

This can be a fine tradeoff, especially if the bucket is expected to be publicly accessible, or the CloudFormation Templates are only intended for internal use. However this coupling means additional cognitive overhead when working with new AWS Accounts or Organizations. As it is now necessary to both update the bucket policy for access, as well as any (AMI sharing configuration)[https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/sharingamis-explicit.html].

One could evaluate the use of AWS CloudFormation template macros by writing a IncludeFromS3 macro that is capable of performing lookup of resources from variable bucket locations, removing the reliance on the specific CloudFormation Bucket. Although this transitions to a capability requirement that any AWS Account Region must have the custom macro registered.