Home Blog Talks Open Source Cloud Security Resources
← All posts

AWS Access Keys - A Reference

AWS Access Keys are the credentials used to provide programmatic or CLI-based access to the AWS APIs. This post outlines what they are, how to identify the different types of keys, where you’re likely to find them across the different services, and the order of access precedence for the different SDKs and tools.

What are AWS Access Keys?

AWS Access Keys are credentials used to authenticate to the AWS APIs. Any time you execute the aws command line tools, or use any kind of tool or script that interacts with AWS, access keys are what you use to identify yourself to AWS. They’re tied to an IAM principal — either an IAM user or an IAM role. There are, broadly speaking, two primary types of access keys:

  • Access keys tied to an IAM user — these do not expire, and are commonly used to allow systems hosted outside of AWS to authenticate to AWS resources. They’re also often seen used by engineers in organisations that have not implemented single sign-on (SSO) to authenticate to their AWS estate.
  • Temporary keys — these are typically issued to AWS services through instance roles or similar, or as a result of an IAM role being assumed. Unlike keys tied to an IAM user, they are usually only valid for a maximum of 12 hours (though frequently much less).

Access keys are broken down into three primary components:

  • Access Key ID - roughly equivalent to a username, this is the unique identifier for the set of access keys you’re using. This can be found in CloudTrail logs when using long-lived access keys.
  • Secret Access Key - the secret component of any set of credentials, these are used to sign requests to the AWS API. Consider them equivalent to a password, and protect them accordingly.
  • Session Token - only required with temporary keys, these are passed alongside the Secret Access Key in the X-Amz-Security-Token header or query string field.

Identifying access key types

The first four characters of the access key ID will tell you where the access key originated, and knowing the common ones can speed up debugging or incident investigations. The AWS documentation lists all the IAM identifier prefixes, including those used by access keys, but the main ones for access keys are listed below:

Key prefixKey type
ABIAAWS STS service bearer token
AKIAAccess key tied to an IAM user
ASIATemporary (AWS STS) access key IDs

Beyond the details shared above, exactly how the identifiers are generated remains internal implementation details that AWS haven’t seen fit to share, and the same to a degree with the secret access keys and session tokens. Scott Piper and Aidan Steele have done some interesting analysis on the access key IDs, which is well worth a read if you’d like to dig further into this topic.

Where can they be stored?

There are a number of different places that access keys can be found. Some are generic locations that can be used within AWS or externally, and some are specific to the AWS service being used.

A summary of places to check

If you’ve got a shell and you’re not sure what AWS service you’ve landed in, the below is a summary of places to look:

  • Environment variables
  • ~/.aws/credentials and ~/.aws/config
  • http://169.254.169.254/latest/meta-data/iam/security-credentials/ROLE-NAME
  • 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI where $AWS_CONTAINER_CREDENTIALS_RELATIVE_URI is an environment variable defined within the container
  • $AWS_CONTAINER_CREDENTIALS_FULL_URI with $AWS_CONTAINER_AUTHORIZATION_TOKEN sent in the Authorization header, where $AWS_CONTAINER_CREDENTIALS_FULL_URI and $AWS_CONTAINER_AUTHORIZATION_TOKEN are environment variables defined within the container

Generic Locations

There are some standard locations that libraries and tools will look for keys, no matter where they’re running. From an offensive perspective, these are great places to look when you gain a foothold somewhere that you expect to find AWS keys, especially developer workstations.

Environment variables

Access keys can be passed in as environment variables to a given environment. They are always stored in the following environment variables, named much as one would expect:

  • AWS_ACCESS_KEY_ID - access key ID
  • AWS_SECRET_ACCESS_KEY - secret access key
  • AWS_SESSION_TOKEN - session token

Configuration files

AWS provides two on-disk locations where credentials can be stored. These files were originally intended as configuration files for the AWS CLI, but the SDKs and other tools will also pick access keys up from these locations. On a typical developer’s system, these will often contain a number of different profiles, with the necessary information and credentials to authenticate to a range of AWS accounts across an AWS Organization.

  • ~/.aws/credentials — intended as the proper location to store access keys
  • ~/.aws/config — holds the configuration information for each profile, but may also include credentials

Ben Kehoe has published a more detailed explanation of these files and how they operate. This, in turn, prompted an explanation from one of the original engineers, James Saryerwinnie, to provide the backstory as to why the config and credentials files have been implemented the way they have. Ben’s also just published a great guide on how best to use these files, and how to avoid stuffing them full of temporary credentials, which is well worth a read if this is something you or your colleagues commonly do.

Service Specific Locations

There are several different methods used to provide credentials to code executing inside AWS services. While the SDKs and CLI will transparently check all the right places, it’s useful to know where they live under the hood.

EC2

Credentials associated with EC2 instance profiles are provided via the instance metadata service (IMDS), which lives at 169.254.169.254. It contains a range of useful data about the instance and its configuration — hackingthe.cloud has a useful summary posted here. http://169.254.169.254/latest/meta-data/iam/security-credentials/ will list the roles provisioned on the instance, and a request to http://169.254.169.254/latest/meta-data/iam/security-credentials/ROLE-NAME will return the access keys associated with the ROLE-NAME role. Obviously, replace ROLE-NAME with the name of the role you’re interested in.

IMDS version 1

An example curl command to get access keys from an instance running IMDS version 1 is shown below:

curl http://169.254.169.254/latest/meta-data/iam/security-credentials/ROLE-NAME

IMDS version 2

AWS built a hardened version of the IMDS following breaches involving Server-Side Request Forgery attacks being used to steal credentials from an instance’s IMDS. As a result, if IMDS version 2 is enabled, a token is required to access data contained within. To access IMDSv2, the process is:

  • Request a token by submitting a PUT request to http://169.254.169.254/latest/api/token
  • Make a request to other endpoints on the metadata service, passing the aforementioned token in the X-aws-ec2-metadata-token header.

An example curl command is shown below.

TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token"` && curl http://169.254.169.254/latest/meta-data/iam/security-credentials/ROLE-NAME -H "X-aws-ec2-metadata-token: $TOKEN"

GuardDuty evasion

If you steal credentials from an instance’s metadata service in an account with GuardDuty enabled and use them from outside of AWS, the UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration.OutsideAWS GuardDuty finding will trigger. At the time of writing it is possible to bypass this by creating an EC2 instance in an AWS account you control and using the credentials from there.

Lambda

Unlike many other AWS compute services, Lambda provides role credentials as environment variables, as described above. Once you have code execution inside a Lambda function, the printenv command will return all environment variables, including the access key id, secret access key and session token. Equally, the following shell commands will retrieve the access keys and session token from within a Lambda function:

echo $AWS_ACCESS_KEY_ID
echo $AWS_SECRET_ACCESS_KEY
echo $AWS_SESSION_TOKEN

Elastic Container Service (ECS)

ECS uses an instance metadata service, much like EC2, only for ECS this is hosted at 169.254.170.2. Additionally, the ECS team took a different approach to harden ECS tasks against server side request forgery attacks by generating a random UUID for each role in each task execution. As such, credentials are found at http://169.254.170.2/v2/credentials/<random-uuid>. The AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variable is set within the container to inform applications where to access credentials, and the AWS SDKs transparently look up this environment variable when looking for credentials in the instance metadata service. As such, from within the container, the following command will retrieve credentials for the task execution role:

curl 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI

The team at Rhino Security Labs also published a great blog post on reconfiguring ECS task definitions to recover credentials associated with the task.

A number of other services appear to be built on top of ECS, and thus also use the same access method described above:

SageMaker Notebooks

SageMaker notebooks appear to operate inside an ECS container. The following command will retrieve access keys for the assigned role when executed within a SageMaker Notebook shell:

curl 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI
App Runner

App Runner appears to use ECS under the hood, and thus has an ECS-style instance metadata service available. The following command when executed inside App Runner will retrieve access keys for the assigned role:

curl 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI
CodeBuild

CodeBuild containers appear to run on top of ECS. To retrieve access keys associated with the role assigned to a CodeBuild build project, alter the buildspec to execute the following command:

curl 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI
Batch

AWS Batch appears to be built on top of ECS. The following command executed within Batch will retrieve access keys associated with the Batch execution role:

curl 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI

Elastic Kubernetes Service (EKS)

Methods for providing access keys to containers inside EKS vary a lot, depending on who set the cluster up and when. The canonical approach suggested by AWS now is IAM Roles for Service Accounts (IRSA), which is an EKS feature designed to map IAM roles to pods using Kubernetes service accounts. AWS have posted a pretty good description of how it functions in their EKS Best Practices Documentation. The short of it is that credentials are acquired using an OIDC token via the sts:AssumeRoleWithWebIdentity API call. The process looks like this:

  • A pod with a service account tied to an IAM Role will call a public OIDC discovery endpoint for AWS IAM upon startup
  • This endpoint signs the OIDC token issued by the cluster
  • This signed JWT token is transparently used by the CLI or SDKs to call sts:AssumeRoleWithWebIdentity
  • sts:AssumeRoleWithWebIdentity returns temporary access keys associated with the relevant IAM role

When the pod is spun up, the EKS control plane injects the role ARN and the web identity token into the pod. These are defined in environment variables:

AWS_ROLE_ARN=arn:aws:iam::AWS_ACCOUNT_ID:role/IAM_ROLE_NAME
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token

As this is a relatively new feature, you may also find credentials in these places:

  • IMDS for the underlying nodes, if EKS on EC2 is deployed and kiam or similar isn’t deployed
  • Injected as secrets as environment variables inside a container
  • Injected as a mounted file inside a container
  • Stored in application configuration files for the app inside a container
  • The usual AWS configuration files
  • Access keys tied to IAM users stored in the cluster’s secrets store

CloudShell

CloudShell provisions temporary credentials with a matching set of permissions to the user or role instantiating the CloudShell shell. As such, these will often have a fairly useful permission set, if you can get inside a developer’s CloudShell shell.

CloudShell runs on one of the AWS container services, however its access key provisioning appears to operate differently to ECS. The $AWS_CONTAINER_CREDENTIALS_FULL_URI environment variable defines the address and path for the metadata service endpoint that supplies the credentials, however from testing this does not appear to be randomised. From instantiating a few different CloudShell instances, it appears to always resolve to http://localhost:1338/latest/meta-data/container/security-credentials. To provide a layer of protection against SSRF, an Authorization header must also be sent with the request, containing the value of the AWS_CONTAINER_AUTHORIZATION_TOKEN environment variable. An example curl command to retrieve the shell’s access keys is shown below:

curl $AWS_CONTAINER_CREDENTIALS_FULL_URI -H "Authorization: $AWS_CONTAINER_AUTHORIZATION_TOKEN"

Step Functions

Step Functions are used to orchestrate a wide range of other AWS services. The credentials used by a Step Function are not accessible to any compute functionality that it triggers. However, any compute services (such as Lambda functions) that are triggered by a Step Function function as they do elsewhere, and the access keys for any associated IAM roles will be made available in the same way.

Elastic Beanstalk/CodeStar etc

A number of AWS services act as wrappers or abstractions around numerous others in order to make it easier to deploy applications. In these cases, the location of access keys tied to any assigned roles will depend on the underlying services in use.

Lightsail

Lightsail does not allow IAM roles to be assigned to instances. If there are access keys to be found, they’ll be manually configured by the system administrator. As such, they’ll likely be in on-disk configuration files, the environment variables, or hardcoded in application source code.

AWS API calls that return access keys

Kinnaird McQuade posted a detailed run-down of the different AWS API calls that return credentials, and is worth keeping in your back pocket as a reference. I’ve listed the known calls that return AWS access keys below. Given the size of the AWS APIs, it’s probably best not to consider this list exhaustive, but it’s a useful starting point for policy writing, security monitoring and penetration testing.

Access Key Precedence

Each of the main AWS tools and libraries maintain their own order of precedence for loading access keys, and they’re not completely aligned with each other. There’s also several SDK-specific options to make the SDKs work in a manner more familiar to developers working in a particular language. I’ve listed the most common below for reference.

CLI

The CLI documentation gives the following order of precedence:

  1. Environment variables
  2. CLI credentials file — ~/.aws/credentials, C:\Users\USERNAME\.aws\credentials on Windows.
  3. CLI configuration file — ~/.aws/config on Linux or macOS, C:\Users\USERNAME\.aws\config on Windows.
  4. Container credentials
  5. Instance profile credentials

In practice, however, it’s built on top of Botocore, and thus actually follows the same search order as Boto3, including the expanded set of search locations listed under the Boto3 section below. The particular load order is described in the source code here (Thanks to Ben Kehoe for pointing that out!).

Boto3 Python SDK

Per the Boto3 documentation:

  1. Passing credentials as parameters in the boto.client() method
  2. Passing credentials as parameters when creating a Session object
  3. Environment variables
  4. Shared credential file (~/.aws/credentials)
  5. AWS config file (~/.aws/config)
  6. Assume Role provider
  7. Boto2 config file (/etc/boto.cfg and ~/.boto)
  8. Instance metadata service on an Amazon EC2 instance that has an IAM role configured.

Boto3 is built on Botocore, like the CLI, and so uses the same load order. This is defined in the source code here.

.NET SDK

Per the AWS SDK for .NET documentation:

  1. Credentials that are explicitly set on the AWS service client (within the application source code)
  2. A credentials profile with the name specified by a value in AWSConfigs.AWSProfileName.
  3. A credentials profile with the name specified by the AWS_PROFILE environment variable.
  4. The [default] credentials profile.
  5. SessionAWSCredentials that are created from the AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN environment variables, if they’re all non-empty.
  6. BasicAWSCredentials that are created from the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables, if they’re both non-empty.
  7. IAM Roles for Tasks, if running as an ECS task.
  8. Credentials from the EC2 instance metadata service.

Go SDK

Per the Go SDK documentation, assuming you’re using the default credential provider:

  1. Passing credentials as a parameter to the client
  2. Passing credentials when creating a session
  3. Environment variables.
  4. Shared credential file (/.aws/credentials) / AWS config file (/.aws/config) (order not specified)
  5. If your application uses an ECS task definition or RunTask API operation, IAM role for tasks.
  6. If your application is running on an Amazon EC2 instance, IAM role for Amazon EC2.

Javascript SDK

Per the Javascript SDK documentation:

  1. Credentials that are explicitly set through the service-client constructor
  2. Environment variables
  3. The shared credentials file
  4. Credentials loaded from the ECS credentials provider
  5. Credentials that are obtained by using a credential process specified in the shared AWS config file or the shared credentials file
  6. Credentials loaded from AWS IAM using the credentials provider of the Amazon EC2 instance