Managing an AWS Organization using Terraform

Using Terraform to manage AWS Accounts & Organizations

Table Of Contents

Today I Explained

AWS Organizations & accounts are the de-facto approach for separating workloads based on function, compliance requirements, or facilitating an organization’s logical architecture separations. This is because an AWS account is a hard boundary, for which it requires explicit permissions to cross this boundary. For isolating workloads from other areas of the organization (production from staging), it is an effective solution.

One of the difficulties that arises with this approach of separating workloads by organization units & accounts is silo’ing. At the top level the organization units, controls & metadata associated with AWS Accounts doesn’t permeate the account themselves. This can be a pain point, as rather than having an account describe itself as “production” or “staging”, alternative metadata services arise that are responsible for answering these questions. This works, but is introducing synchronization & publish/response patterns which aren’t always needed.

An alternative approach is taking advantage of Terraform for the management of the organization units, accounts, the assignment of metadata, and the accessibility of the dataset.

Organization units can be defined using single resources, composing them together to create the larger tree structure based on references:

# organizations.tf
resource "aws_organizations_organizational_unit" "lunar_lobster" {
  name      = "Lunar Lobster"
  parent_id = aws_organizations_organization.example.roots[0].id
}

resource "aws_organizations_organizational_unit" "ll_workloads" {
  name      = "Workloads"
  parent_id = aws_organizations_organizational_unit.lunar_lobster.id
}

resource "aws_organizations_organizational_unit" "ll_security" {
  name      = "Security"
  parent_id = aws_organizations_organizational_unit.lunar_lobster.id
}

resource "aws_organizations_organizational_unit" "ll_audit" {
  name      = "Audit"
  parent_id = aws_organizations_organizational_unit.lunar_lobster.id
}

# ...

## Visual:
#
#    ┌─────────┬────Lunar Lobster─────┬────────────┐
#    │         │          │           │            │
#    ▼         ▼          ▼           ▼            ▼
# Security    Audit   Workloads      ...          ...

A tree structure with a top level node named ‘OU Root’ with 5 child nodes named, ‘Security, ‘Audit’, ‘Workloads’, ‘…’ and ‘…’ to mean “and so on”.

Taking advantage of Terraform modules, AWS Accounts can be defined first as a data-only Terraform module that is responsible for collecting all of the metadata, configuration & other properties about an AWS Account. This can be considered the “definition” of an AWS Account. The module responsible for provisioning the AWS Account itself, using a resource like aws_organizations_account will be a separate module. This has the key benefit of allowing relational mapping between AWS Accounts to be defined, such as peering, logging, backups or networking.

# acc_logarchive
module "def_logarchive" {
  source = "./modules/aws-organizations-account-definition"

  name    = "logarchive"
  type    = "production"
  ou      = aws_organizations_organizational_unit.ll_audit.id
  regions = [ "us-east-1" ]
  # ...
}

module "acc_logarchive" {
  source = "./modules/aws-organizations-account"

  metadata = module.def_logarchive.result

  relationship = [
    {
      type    = "peer",
      account = module.def_management.result
    },
    {
      type    = "backup",
      account = module.def_businesscontinuity.result
    }
  ]
}

The wrapper modules for creating the accounts aren’t just responsible for creating the account, but connecting into the account to setup parameter store entries for essential metadata. This makes it easier for workloads running within the AWS Accounts to reference regional variables to have default behaviour that is appropriate for the kind of environment the workloads will be operating within. With the accounts provisioned, the set of all AWS Accounts can be combined into a single module that is responsible for populating an S3 bucket that encodes the metadata of the AWS Organization.

module "metadata_api" {
  source = "./modules/aws-organizations-s3api"

  bucket   = module.metadata_bucket.bucket 
  accounts = [
    module.acc_logarchive.result,
    # ...
  ]
}

By populating an S3 bucket with this data, it enables working around the limitation of list-accounts, which is that it can only be called from an organization’s management account or by a member account that is a delegated administrator. Internal services wishing to make use of this data, are then able to pull from it using the S3 API of the AWS SDK, or through an http apigateway frontend for s3.\

aws s3 cp s3://metadata.myawsorganization.dev/api/v1/accounts/logarchive -
{
  "name": "logarchive",
  "type": "production",
  "organization": {
    "id": "...",
    "name": "Audit"
  }
}