Kubernetes on AWS - IAM Roles for Service Accounts via Terraform

Introduction Prerequisites IAM Roles for Service Accounts - Manual Setup IAM Roles for Service Accounts - Automated Setup Helpful Links

Introduction

In this blog post, we will walk through a Terraform setup for IAM roles for service accounts. This allows us to provide AWS permissions to Kubernetes workloads. There are several other options to so:

Attach an IAM role to your ec2 instances.

Mount AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY as environment variables.

Mount credentials file to ~/.aws/credentials

However, IAM roles for service accounts provides more granular permission control than attaching a role to a node which probably runs multiple pods. Additionally, it is simpler to configure and more secure because we don’t have to deal with credentials.

Prerequisites

Kubernetes on AWS - From Zero To Production

Kubernetes on AWS - EKS Setup with Terraform

git clone git@github.com:canida-software/k8s-on-aws.git

IAM Roles for Service Accounts - Manual Setup

Let’s walk through a manual setup to understand how to provide AWS permissions to a service account. You don’t need to reproduce this because we will do the same from code afterwards but it would help you to understand all the concepts.

The service account permissions attached to a pod will automatically be used if your pod’s application is built on top of an AWS SDK (basically everything talking to AWS is built on top of one of their SDKs).

The following policy is used to provide access to a subset of secrets in the AWS Secretsmanager.

{
    "Statement": [
        {
            "Action": [
                "secretsmanager:GetResourcePolicy",
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret",
                "secretsmanager:ListSecretVersionIds"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:secretsmanager:eu-central-1:054000737513:secret:k8s-main/*"
            ]
        },
        {
            "Action": "secretsmanager:ListSecrets",
            "Effect": "Allow",
            "Resource": "*"
        }
    ],
    "Version": "2012-10-17"
}

We can attach the above policy to a role and attach the role to an IAM user. Then, we could generate credentials for the IAM user and pass them to our application.

However, what we will do instead is to create the role ExternalSecrets, attach the policy and additionally create the following trust relationship for the role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::054000737513:oidc-provider/oidc.eks.eu-central-1.amazonaws.com/id/81E1F9ED8100DE07F6B96AF57C8191BC"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.eu-central-1.amazonaws.com/id/81E1F9ED8100DE07F6B96AF57C8191BC:sub": "system:serviceaccount:external-secrets:aws-secretsmanager"
                }
            }
        }
    ]
}

The trust relationship says that a principal authenticated by our cluster’s OIDC provider can assume the role. Without any additional conditions, that would mean anybody with cluster access can assume the role. Therefore, we add a condition to restrict the sub (Subject claim). I.e. only the service account aws-secretsmanager in the namespace external-secrets can assume the role.

Next, we can attach the role to a service account as follows:

# aws-secretsmanager-sa.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: aws-secretsmanager
  namespace: external-secrets
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::054000737513:role/k8s-main/ExternalSecrets

Then, we can attach the service account to a pod:

# aws-cli-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: awscli
	namespace: external-secrets
  labels:
    app: awscli
spec:
  serviceAccountName: aws-secretsmanager
  containers:
  - image: amazon/aws-cli
    command:
      - "sleep"
      - "604800"
    imagePullPolicy: IfNotPresent
    name: awscli
  restartPolicy: Always

To check whether the pod really uses our the AWS role ExternalSecrets, we can call the AWS cli from the pod.

k apply -f aws-secretsmanager-sa.yaml aws-cli-pod.yaml
kubectl exec -it awscli -- aws sts get-caller-identity

The result shows that our caller identity for the AWS API is the assumed role ExternalSecrets. I.e. our pod successfully accesses AWS with the role linked to the service account.

{
    "UserId": "AROAQZEVR3TU7U26A2MUZ:botocore-session-1657996279",
    "Account": "054000737513",
    "Arn": "arn:aws:sts::054000737513:assumed-role/ExternalSecrets/botocore-session-1657996279"
}

Given the length of this section, we already motivated the next sections which automates the above steps.

IAM Roles for Service Accounts - Automated Setup

First, you need to adapt your terraform backend in backend.tf . You can use the same bucket that you created for storing the state of your eks cluster.

terraform {
  backend "s3" {
    bucket = "canida-terraform"
    key    = "k8s-main/iam-roles.tfstate"
    region = "eu-central-1"
  }
}

You need to modify external-secrets-policy.json because it limits secrets access to a specific prefix in a specific AWS account. I use one prefix for all the secrets related to my k8s-main cluster. Additionally, you need to modify canida.tfvars . You can retrieve the oidc_url by switching to the k8s-on-aws/eks folder and executing terraform output.

The IAM roles for service accoutns terraform setup uses a custom module iam-roles/modules/iam-role-for-serviceaccount which encapsulates most logic to create an IAM role for a service account. Feel free to explore the module to understand how it automates the manual setup.

cd ./k8s-on-aws/iam-roles

# install Terraform modules
terraform init

# setup the cluster and configure it using the tfvars file
terraform apply -var-file canida.tfvars

The outputs of the terraform file contains the ARN’s of the created roles. We will need them in the following section to attach them to service accounts to provide AWS permissions to our pods.

Helpful Links

https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html