Table of Contents
IntroductionPrerequisitesIAM Roles for Service Accounts - Manual SetupIAM Roles for Service Accounts - Automated SetupHelpful Links
Introduction
In this blog post, we will walk through a Terraform setup for IAM roles for service accounts. This allows us to provide AWS permissions to Kubernetes workloads. There are several other options to so:
- Attach an IAM role to your ec2 instances.
- Mount AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY as environment variables.
- Mount credentials file to ~/.aws/credentials
However, IAM roles for service accounts provides more granular permission control than attaching a role to a node which probably runs multiple pods. Additionally, it is simpler to configure and more secure because we don’t have to deal with credentials.
Prerequisites
Kubernetes on AWS - From Zero To ProductionKubernetes on AWS - EKS Setup with Terraformgit clone git@github.com:canida-software/k8s-on-aws.git
IAM Roles for Service Accounts - Manual Setup
Let’s walk through a manual setup to understand how to provide AWS permissions to a service account. You don’t need to reproduce this because we will do the same from code afterwards but it would help you to understand all the concepts.
The service account permissions attached to a pod will automatically be used if your pod’s application is built on top of an AWS SDK (basically everything talking to AWS is built on top of one of their SDKs).
The following policy is used to provide access to a subset of secrets in the AWS Secretsmanager.
{
"Statement": [
{
"Action": [
"secretsmanager:GetResourcePolicy",
"secretsmanager:GetSecretValue",
"secretsmanager:DescribeSecret",
"secretsmanager:ListSecretVersionIds"
],
"Effect": "Allow",
"Resource": [
"arn:aws:secretsmanager:eu-central-1:054000737513:secret:k8s-main/*"
]
},
{
"Action": "secretsmanager:ListSecrets",
"Effect": "Allow",
"Resource": "*"
}
],
"Version": "2012-10-17"
}
We can attach the above policy to a role and attach the role to an IAM user. Then, we could generate credentials for the IAM user and pass them to our application.
However, what we will do instead is to create the role
ExternalSecrets, attach the policy and additionally create the following trust relationship for the role:{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::054000737513:oidc-provider/oidc.eks.eu-central-1.amazonaws.com/id/81E1F9ED8100DE07F6B96AF57C8191BC"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.eu-central-1.amazonaws.com/id/81E1F9ED8100DE07F6B96AF57C8191BC:sub": "system:serviceaccount:external-secrets:aws-secretsmanager"
}
}
}
]
}The trust relationship says that a principal authenticated by our cluster’s OIDC provider can assume the role. Without any additional conditions, that would mean anybody with cluster access can assume the role. Therefore, we add a condition to restrict the
sub (Subject claim). I.e. only the service account aws-secretsmanager in the namespace external-secrets can assume the role. Next, we can attach the role to a service account as follows:
# aws-secretsmanager-sa.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: aws-secretsmanager
namespace: external-secrets
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::054000737513:role/k8s-main/ExternalSecretsThen, we can attach the service account to a pod:
# aws-cli-pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: awscli
namespace: external-secrets
labels:
app: awscli
spec:
serviceAccountName: aws-secretsmanager
containers:
- image: amazon/aws-cli
command:
- "sleep"
- "604800"
imagePullPolicy: IfNotPresent
name: awscli
restartPolicy: AlwaysTo check whether the pod really uses our the AWS role ExternalSecrets, we can call the AWS cli from the pod.
k apply -f aws-secretsmanager-sa.yaml aws-cli-pod.yaml
kubectl exec -it awscli -- aws sts get-caller-identity
The result shows that our caller identity for the AWS API is the assumed role ExternalSecrets. I.e. our pod successfully accesses AWS with the role linked to the service account.
{
"UserId": "AROAQZEVR3TU7U26A2MUZ:botocore-session-1657996279",
"Account": "054000737513",
"Arn": "arn:aws:sts::054000737513:assumed-role/ExternalSecrets/botocore-session-1657996279"
}Given the length of this section, we already motivated the next sections which automates the above steps.
IAM Roles for Service Accounts - Automated Setup
First, you need to adapt your terraform backend in
backend.tf . You can use the same bucket that you created for storing the state of your eks cluster.terraform {
backend "s3" {
bucket = "canida-terraform"
key = "k8s-main/iam-roles.tfstate"
region = "eu-central-1"
}
}You need to modify
external-secrets-policy.json because it limits secrets access to a specific prefix in a specific AWS account. I use one prefix for all the secrets related to my k8s-main cluster. Additionally, you need to modify canida.tfvars . You can retrieve the oidc_url by switching to the k8s-on-aws/eks folder and executing terraform output.The IAM roles for service accoutns terraform setup uses a custom module
iam-roles/modules/iam-role-for-serviceaccount which encapsulates most logic to create an IAM role for a service account. Feel free to explore the module to understand how it automates the manual setup.cd ./k8s-on-aws/iam-roles
# install Terraform modules
terraform init
# setup the cluster and configure it using the tfvars file
terraform apply -var-file canida.tfvars
The outputs of the terraform file contains the ARN’s of the created roles. We will need them in the following section to attach them to service accounts to provide AWS permissions to our pods.
Helpful Links
