Karpenter on EKS MNG¶
This pattern demonstrates how to provision Karpenter on an EKS managed node group. Deploying onto standard EC2 instances created by an EKS managed node group will allow for daemonsets to run on the nodes created for the Karpenter controller, and therefore better unification of tooling across your data plane. This solution is comprised of the following components:
- An EKS managed node group that applies both a taint as well as a label for the Karpenter controller. We want the Karpenter controller to target these nodes via a
nodeSelectorin order to avoid the controller pods from running on nodes that Karpenter itself creates and manages. In addition, we are applying a taint to keep other pods off of these nodes as they are primarily intended for the controller pods. We apply a toleration to the CoreDNS addon, to allow those pods to run on the controller nodes as well. This is needed so that when a cluster is created, the CoreDNS pods have a place to run in order for the Karpenter controller to be provisioned and start managing the additional compute requirements for the cluster. Without letting CoreDNS run on these nodes, the controllers would fail to deploy and the data plane would be in a "deadlock" waiting for resources to deploy but unable to do so. - The
eks-pod-identity-agentaddon has been provisioned to allow the Karpenter controller to utilize EKS Pod Identity for AWS permissions via an IAM role. - The VPC subnets and node security group have been tagged with
"karpenter.sh/discovery" = local.namefor discoverability by the controller. The controller will discover these resources and use them to provision EC2 resources for the cluster. - An IAM role for the Karpenter controller has been created with a trust policy that trusts the EKS Pod Identity service principal. This allows the EKS Pod Identity service to provide AWS credentials to the Karpenter controller pods in order to call AWS APIs.
- An IAM role for the nodes that Karpenter will create has been created along with a cluster access entry which allows the nodes to acquire permissions to join the cluster. Karpenter will create and manage the instance profile that utilizes this IAM role.
- An SQS queue has been created that is subscribed to certain EC2 CloudWatch events. This queue is used by Karpenter, allowing it to respond to certain EC2 lifecycle events and gracefully migrate pods off the instance before it is terminated.
Code¶
The areas of significance related to this pattern are highlighted in the code provided below:
################################################################################
# Cluster
################################################################################
module "eks" {
source = "terraform-aws-modules/eks/aws"
version = "~> 20.10"
cluster_name = local.name
cluster_version = "1.29"
# Give the Terraform identity admin access to the cluster
# which will allow it to deploy resources into the cluster
enable_cluster_creator_admin_permissions = true
cluster_endpoint_public_access = true
cluster_addons = {
coredns = {
configuration_values = jsonencode({
tolerations = [
# Allow CoreDNS to run on the same nodes as the Karpenter controller
# for use during cluster creation when Karpenter nodes do not yet exist
{
key = "karpenter.sh/controller"
value = "true"
effect = "NoSchedule"
}
]
})
}
eks-pod-identity-agent = {}
kube-proxy = {}
vpc-cni = {}
}
enable_efa_support = true
vpc_id = module.vpc.vpc_id
subnet_ids = module.vpc.private_subnets
eks_managed_node_groups = {
karpenter = {
instance_types = ["m5.large"]
min_size = 2
max_size = 3
desired_size = 2
labels = {
# Used to ensure Karpenter runs on nodes that it does not manage
"karpenter.sh/controller" = "true"
}
taints = {
# The pods that do not tolerate this taint should run on nodes
# created by Karpenter
karpenter = {
key = "karpenter.sh/controller"
value = "true"
effect = "NO_SCHEDULE"
}
}
}
}
tags = merge(local.tags, {
# NOTE - if creating multiple security groups with this module, only tag the
# security group that Karpenter should utilize with the following tag
# (i.e. - at most, only one security group should have this tag in your account)
"karpenter.sh/discovery" = local.name
})
}
output "configure_kubectl" {
description = "Configure kubectl: make sure you're logged in with the correct AWS profile and run the following command to update your kubeconfig"
value = "aws eks --region ${local.region} update-kubeconfig --name ${module.eks.cluster_name}"
}
################################################################################
# Controller & Node IAM roles, SQS Queue, Eventbridge Rules
################################################################################
module "karpenter" {
source = "terraform-aws-modules/eks/aws//modules/karpenter"
version = "~> 20.9"
cluster_name = module.eks.cluster_name
# Name needs to match role name passed to the EC2NodeClass
node_iam_role_use_name_prefix = false
node_iam_role_name = local.name
create_pod_identity_association = true
tags = local.tags
}
################################################################################
# Helm charts
################################################################################
resource "helm_release" "karpenter" {
namespace = "kube-system"
name = "karpenter"
repository = "oci://public.ecr.aws/karpenter"
repository_username = data.aws_ecrpublic_authorization_token.token.user_name
repository_password = data.aws_ecrpublic_authorization_token.token.password
chart = "karpenter"
version = "0.36.1"
wait = false
values = [
<<-EOT
nodeSelector:
karpenter.sh/controller: 'true'
tolerations:
- key: CriticalAddonsOnly
operator: Exists
- key: karpenter.sh/controller
operator: Exists
effect: NoSchedule
settings:
clusterName: ${module.eks.cluster_name}
clusterEndpoint: ${module.eks.cluster_endpoint}
interruptionQueue: ${module.karpenter.queue_name}
EOT
]
lifecycle {
ignore_changes = [
repository_password
]
}
}
VPC¶
module "vpc" {
source = "terraform-aws-modules/vpc/aws"
version = "~> 5.0"
name = local.name
cidr = local.vpc_cidr
azs = local.azs
private_subnets = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 4, k)]
public_subnets = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 8, k + 48)]
enable_nat_gateway = true
single_nat_gateway = true
public_subnet_tags = {
"kubernetes.io/role/elb" = 1
}
private_subnet_tags = {
"kubernetes.io/role/internal-elb" = 1
# Tags subnets for Karpenter auto-discovery
"karpenter.sh/discovery" = local.name
}
tags = local.tags
}
EC2NodeClass and NodePool¶
---
apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
name: default
spec:
amiFamily: AL2
role: ex-karpenter-mng
subnetSelectorTerms:
- tags:
karpenter.sh/discovery: ex-karpenter-mng
securityGroupSelectorTerms:
- tags:
karpenter.sh/discovery: ex-karpenter-mng
tags:
karpenter.sh/discovery: ex-karpenter-mng
---
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
name: default
spec:
template:
spec:
nodeClassRef:
name: default
requirements:
- key: "karpenter.k8s.aws/instance-category"
operator: In
values: ["c", "m", "r"]
- key: "karpenter.k8s.aws/instance-cpu"
operator: In
values: ["4", "8", "16", "32"]
- key: "karpenter.k8s.aws/instance-hypervisor"
operator: In
values: ["nitro"]
- key: "karpenter.k8s.aws/instance-generation"
operator: Gt
values: ["2"]
limits:
cpu: 1000
disruption:
consolidationPolicy: WhenEmpty
consolidateAfter: 30s
Deploy¶
See here for the prerequisites and steps to deploy this pattern.
Validate¶
-
Test by listing the nodes in the cluster. You should see four Fargate nodes in the cluster:
-
Provision the Karpenter
EC2NodeClassandNodePoolresources which provide Karpenter the necessary configurations to provision EC2 resources: -
Once the Karpenter resources are in place, Karpenter will provision the necessary EC2 resources to satisfy any pending pods in the scheduler's queue. You can demonstrate this with the example deployment provided. First deploy the example deployment which has the initial number replicas set to 0:
-
When you scale the example deployment, you should see Karpenter respond by quickly provisioning EC2 resources to satisfy those pending pod requests:
-
Listing the nodes should now show some EC2 compute that Karpenter has created for the example deployment:
kubectl get nodes NAME STATUS ROLES AGE VERSION ip-10-0-38-109.us-west-2.compute.internal Ready <none> 11s v1.29.3-eks-ae9a62a # <== EC2 created by Karpenter ip-10-0-38-5.us-west-2.compute.internal Ready <none> 3m54s v1.29.3-eks-ae9a62a ip-10-0-9-38.us-west-2.compute.internal Ready <none> 3m49s v1.29.3-eks-ae9a62a
Destroy¶
Scale down the deployment to de-provision Karpenter created resources first:
terraform destroy -target="module.eks_blueprints_addons" -auto-approve
terraform destroy -target="module.eks" -auto-approve
terraform destroy -auto-approve
See here for more details on cleaning up the resources created.