Pullthrough cache for mirrored public ECR images

Mirroring OSS images into the Public ECR Registry for the ECR pullthrough cache

Table Of Contents

Today I Explained

When working with container images from third-party registries (DockerHub, Quay, GitHub Container Registry), it can sometimes result in you failing to pull the container images. One such example of this error is:

CannotPullContainerError: inspect image has been retried 5 time(s): httpReaderSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/manifests/sha256:<....> Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

This kind of error occurs due to the rate limiting restrictions that DockerHub places on anonymous or free tier users. This kind of issue can spring up when working with Terraform modules, Helm Charts or CloudFormation Templates that default (or hardcode) the fully qualified domain name (FQDN) for the container image to DockerHub. Unless an alternative registry is used, DockerHub is used as the source of the image which for anonymous pulls often hit the rate limit.

In practice, addressing this often means a one-off copying of external images into internal registries using docker pull & docker push or crane with a “will automate” to come at a later date. This can be necessary as not all public images have setup alternative mirrors on public registries such as GitHub Container Registry, or Quay.

An alternative approach that can be considered is leveraging the ECR Public Registry for mirroring the artifacts. This registry is subject to its own set of conditions, but does allow for authenticated pulls using AWS IAM. A notable benefit of mirroring open source images into the public registry is that enables the use of a pullthrough cache for the images.

This can work by having the workloads capable of “pushing” images into repositories local to the AWS Account, which are created on-demand when making a pull action to source the container images from the public registry. This provides the benefit of working with a local cache of the images within an AWS Account, as well as working with a public source for the container images itself.

A note on mirroring

A different approach that can be taken with the mirroring of images into the Public Registry is attestation. The workflow that is responsible for mirroring images into the public registry is capable of creating an attestation claim which represents the idea of “We’ve checked this image, and found it to meet our requirements”. This provides an avenue for an organization to contribute to the broader ecosystem of trust around the use of open source software.