Helm Release Add-ons

Starting with EKS Blueprints v5 we have made a decision to only support the provisioning of a certain core set of add-ons. On an going basis, we will evaluate the current list to see if more add-ons need to be supported via this repo. Typically you can expect that any AWS created add-on that is not yet available via the Amazon EKS add-ons will be prioritized to be provisioned through this repository.

In addition to these AWS add-ons, we will also support the provisioning of certain OSS add-ons that we think customers will benefit from. These are selected based on customer demand (e.g. metrics-server) and certain patterns (gitops) that are foundational elements for a complete blueprint of an EKS cluster.

One of the reasons customers pick Kubernetes is because of its strong commercial and open-source software ecosystem and would like to provision add-ons that are not necessarily supported by EKS Blueprints. For such add-ons the options are as following:

With helm_release Terraform Resource

The helm_release resource is the most fundamental way to provision a helm chart via Terraform.

Use this resource, if you need to control the lifecycle add-ons down to level of each add-on resource.

With helm_releases Variable

You can use the helm_releases variable in EKS Blueprints Add-ons to provide a map of add-ons and their respective Helm configuration. Under the hood, we just iterate through the provided map and pass each configuration to the Terraform helm_release resource.

E.g.

module "addons" {
  source  = "aws-ia/eks-blueprints-addons/aws"
  version = "~> 1.0"

  cluster_name      = "<cluster_name>"
  cluster_endpoint  = "<cluster_endpoint>"
  cluster_version   = "<cluster_version>"
  oidc_provider_arn = "<oidc_provider_arn>"

  # EKS add-ons
  eks_addons = {
    coredns = {}
    vpc-cni = {}
    kube-proxy = {}
  }

  # Blueprints add-ons
  enable_aws_efs_csi_driver                    = true
  enable_aws_cloudwatch_metrics                = true
  enable_cert_manager                          = true
  ...

  # Pass in any number of Helm charts to be created for those that are not natively supported
  helm_releases = {
    prometheus-adapter = {
      description      = "A Helm chart for k8s prometheus adapter"
      namespace        = "prometheus-adapter"
      create_namespace = true
      chart            = "prometheus-adapter"
      chart_version    = "4.2.0"
      repository       = "https://prometheus-community.github.io/helm-charts"
      values = [
        <<-EOT
          replicas: 2
          podDisruptionBudget:
            enabled: true
        EOT
      ]
    }
    gpu-operator = {
      description      = "A Helm chart for NVIDIA GPU operator"
      namespace        = "gpu-operator"
      create_namespace = true
      chart            = "gpu-operator"
      chart_version    = "v23.3.2"
      repository       = "https://nvidia.github.io/gpu-operator"
      values = [
        <<-EOT
          operator:
            defaultRuntime: containerd
        EOT
      ]
    }
  }

  tags = local.tags
}

With this pattern, the lifecycle of all your add-ons is tied to that of the addons module. This allows you to easily target the addon module in your Terraform apply and destroy commands. E.g.

terraform apply -target=module.addons

terraform destroy -target=module.addons

Installing Helm releases before the add-ons

Helm installs the CRDs from a chart's crds/ directory only on the first install and never upgrades or deletes them. Projects such as Karpenter therefore ship the CRDs as a separate chart, which has to be installed before the chart that uses them.

Releases passed in pre_helm_releases are installed before every other add-on in this module (including helm_releases). They take the same attributes as helm_releases. E.g.

module "addons" {
  source  = "aws-ia/eks-blueprints-addons/aws"
  version = "~> 1.0"

  ...

  pre_helm_releases = {
    karpenter-crd = {
      namespace     = "kube-system"
      chart         = "karpenter-crd"
      chart_version = "1.6.0"
      repository    = "oci://public.ecr.aws/karpenter"
    }
  }

  enable_karpenter = true
  karpenter = {
    chart_version = "1.6.0"
  }
}

Keep the version of the CRD chart in step with the version of the add-on that uses it (1.6.0 for both above).

Optionally, set skip_crds = true on the add-on so that its chart does not install the CRDs from its own crds/ directory.

Like helm_releases, releases in pre_helm_releases default to wait = false: the add-ons are ordered after the pre releases are submitted, not after their resources are ready. Set wait = true on a pre release when the add-ons need its resources (e.g. a webhook or a controller) to be running first.

When pre_helm_releases is set, create_delay_duration is applied twice: once before the pre releases and once more before the other add-ons.

Releases in pre_helm_releases do not wait for the Amazon EKS add-ons created by this module through eks_addons, because those add-ons are themselves created after pre_helm_releases. Use create_delay_dependencies to wait for the add-ons created outside of this module (e.g. by the terraform-aws-modules/eks/aws module).

With EKS Blueprints Addon Module

If you have an add-on that requires an IAM Role for Service Account (IRSA), we have created a new Terraform module terraform-aws-eks-blueprints-addon that can help provision a Helm chart along with an IAM role and policies with permissions required for the add-on to function properly. We use this module for all of the add-ons that are provisioned by EKS Blueprints Add-ons today.

You can optionally use this module for add-ons that do not need IRSA or even just to create the IAM resources for IRSA and skip the helm release. Detailed usage of how to consume this module can be found in its readme.

This pattern can be used to create a Terraform module with a set of add-ons that are not supported in the EKS Blueprints Add-ons today and wrap them in the same module definition. An example of this is the ACK add-ons repository which is a collection of ACK helm chart deployments with IRSA for each of the ACK controllers.