Third party secrets management with dedicated vault AWS Accounts

Using the dedicated vault AWS Accounts as a hard boundary for managing third-party secrets

Table Of Contents

Today I Explained

Organizations don’t typically grant every individual administrator permissions in third party services, or the ability to create new application integrations. The ability to create slack apps, github apps, oauth clients or service accounts is locked down, typically to the IT or CorpOps groups.

When these integrations are created, the credentials used by this application integrations are passed along to the developers, and published in a secrets storage mechanism like AWS SecretsManager. As these integrations evolve, it becomes important that these secrets can be rotated, tested, evaluted in isolation. This requires efforts to be taken to identify all of the existing third party credentials scattered throughout the infrastructure, and keeping them inline with the new expectations. This can be frustrating as it involves delving into arcane knowledge, and automation unfriendly work.

An alternative approach is centralizing these third party secrets into dedicated AWS Accounts known as “vaults” (not to be confused with Hashicorp Vault). The secrets are provisioned within the vault accounts, and using the resource policy for cross-account secretsmanager secrets to share the secrets within the organization. An option for improving the accessibility of the Secrets ARNs is publishing the ARNs as parameter store values within the consumer accounts as accessible names like /vault/third_party/github.com/apps/myapplication.

       ┌────OU Root────┬───────┐
       │               │       │
       ▼               ▼       ▼
   ┌─Vault─┐       Workloads  ...
   ▼       ▼           ▼
┌─AWS─┐ ┌─AWS─┐     ┌─AWS─┐
│ *** │ │ *** │     │ @@@ │
└─────┘ └─────┘     └─────┘

------------------------------

┌─AWS───────┐       ┌───AWS──────┐
│           │       │ GitHub App │
│ @@ @@@ @@ │◄──────┤   ******   │
│           │       └────────────┘
└───────────┘

A least privilege concern that exists with this approach is that the secrets are now accessible at the account-level, in comparison to the option of restricting the access to the service. It is posisble to restrict it to the service, but creates a dependency sequence when deploying new services that requires updating the resource policy to match any new consumer. This makes this more suited for single service per account or closely related services per account account, to meet least privilege concerns.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::ProdAccount001:root",
          "arn:aws:iam::ProdAccount002:root"
        ]
      },
      "Action": "secretsmanager:GetSecretValue",
      "Resource": [
        "arn:aws:secretsmanager:<Region>:<AccountId>:secret:SecretName"
      ]
    }
  ]
}

A note on Least Privilege

Granting an entire AWS Account permissions to read a secret can feel concerning, given that these accounts often have many services operating within them or have many individuals with access into them. The approach of having the secret live alongside the service feels like it can be tailored to meet least privilege & operational needs of a service.

A consideration on this is the benefits of centralizing the management of these secrets. Most likely the way that IT/CorpOps are passing these secrets along is using a password manager (lastpass, 1password, keeper), creating an equivalent to the vault accounts. The notable difference being these are not typically programmatically accessible within the infrastructure. Having them located within the infrastructure removes a copy & paste step from the rotation workflow.

Working with the vault accounts, it is then possible to adopt pull/push patterns for making the secrets available to services within the accounts that the service lives. Similar to pullthrough caches, it is possible to support caches within the accounts that make the secrets accessible.