AWS re:Invent hasn’t officially begun, yet there is a game changing new feature to EKS to make you run Kubernetes like a pro!!!

In this blog we will investigate what EKS Auto Mode is all about and illustrates how to enables EKS Auto Mode on existing cluster and try to migrate to our managed node groups workloads to EKS auto mode of an existing cluster but we will be not able to do that, find out why in the blog!

I have also analyzed the price you will pay for EKS Auto v/s the simplicity of operations it provides to help you make an informed decision whether to enable it or not.

I have created a terraform repository to quickly reproduce this blog with just 2 commands, let’s conquer EKS Auto Mode.

Motivation

December 2, 2024 (JST) – Just ahead of re:Invent 2024, AWS released an exciting update introducing EKS Auto Mode.

What Does This Update Mean for You?

• It automates all kinds of management for your EKS cluster which allow AWS to also set up and manage the infrastructure that enables the smooth operation of your workloads.
• Cluster infrastructure managed by AWS includes many Kubernetes capabilities as core components, as opposed to add-ons, such as compute autoscaling, pod and service networking, application load balancing, cluster DNS, block storage, and GPU support
• You can deploy a new EKS Auto Mode cluster or enable EKS Auto Mode on an existing cluster

Note: Terraform support(still ongoing) not yet there so for this blog we will use AWS console.

Cost considerations of this Update

• According to https://aws.amazon.com/eks/pricing/
• You will stay pay for 2 components :
  • Amazon EKS cluster pricing: *$0.10 per cluster/hour *($0.60/cluster for extended)
  • But if you use auto mode will have to pay extra!!!

You pay for Amazon EKS Auto Mode based on the duration and type of Amazon EC2 instances launched and managed by EKS Auto Mode. The Amazon EKS Auto Mode prices below are in addition to the Amazon EC2 instance price, which covers the EC2 instances themselves.

In simple words EKS Auto Mode charges a management fee that varies based on the EC2 instance type launched, in addition to your regular EC2 instance costs.

What does it actually automate

EKS auto mode is designed to automate data plane components so here are the following components being automated along with the features:

Automated Data Plane Components in EKS Auto Mode

• Compute Management: Automates key aspects of cluster compute, including node provisioning, scaling, upgrades, and load balancing, for hassle-free Kubernetes operations.
  • Nodes:
    • Automatically selects optimized AMIs with essential services.
    • Secures nodes using SELinux, read-only root files, and disallowed SSH/SSM access.
    • Includes GPU support for NVIDIA and Neuron GPUs.
  • Auto Scaling:
    • Uses Karpenter to monitor and deploy new nodes for unschedulable Pods.
    • Dynamically terminates unused nodes to optimize resources.
• Upgrades:
  • Applies OS and component updates with minimal workload disruption.
  • Enforces a 21-day maximum node lifetime for security and stability.
• Load Balancing:
  • Integrates with Elastic Load Balancing (ALB & NLB) for automated provisioning and scaling.
  • Provides production-ready load balancing aligned with AWS best practices.
• Storage Automation
  • Configures ephemeral storage with volume settings, encryption, and deletion policies automatically managed for optimal use.
• Networking Automation
  • Manages Pod and service connectivity with IPv4/IPv6 support and extends IP spaces using secondary CIDR blocks.

Enabling auto mode on Existing Cluster

• In this blog we will see how to enable on an existing cluster as AWS blog already covered EKS auto mode on new cluster
• To mimic existing cluster, you can deploy EKS cluster with EKS managed node groups using Terraform in my repo

Migrations Supported

• At the writing of this blog EKS support migrating from Karpenter to EKS Auto Mode, Migrate from EKS Managed Node Groups to EKS Auto Mode, Migrating from EKS Fargate to EKS Auto Mode Nodes
• To keep this blog simple I will show how to migrate from EKS managed Node Groups to EKS Auto Mode (hopefully :))

Note: Also remember to take a look on how to configure Kubernetes based resource to be owned by either self-managed controllers or EKS Auto Mode.

Prerequisites

• Updating IAM permissions and configuring core EKS Auto Mode settings
• Minimum required version of certain Amazon EKS Add-ons Perspective.

Updating IAM permissions and configuring core EKS Auto Mode settings

• Cluster IAM role of an EKS Cluster cannot be changed after the cluster is created. EKS Auto Mode requires additional permissions on this role. You must attach additional policies to the current role.

• Navigate to Cluster IAM role and adding the following policies
  • AmazonEKSComputePolicy
  • AmazonEKSBlockStoragePolicy
  • AmazonEKSLoadBalancingPolicy
  • AmazonEKSNetworkingPolicy
  • AmazonEKSClusterPolicy ( already exist for me )

• Edit the trust Policy and add sts:TagSession to allow Action.

Minimum required version of certain Amazon EKS Add-ons

Perspective.

• I have set the terraform code to fetch latest versions of add-ons so you don’t have to do anything.

Create a Karpenter Node Pool

• EKS auto uses Karpenter behind the scenes for scaling.

Karpenter, a node provisioning tool that helps optimize cluster scaling and resource utilization. With Karpenter’s NodePool resource, you can define specific requirements for your compute resources, including instance types, availability zones, architectures, and capacity types.

• Select Create recommended roles for Node IAM Role
• I am using preconfigured node pool, but you can configure your own using your own yaml file like this example

• Select default permissions and create role

• Once the role is created and you select next you can see notifications for auto mode being enabled

• Once enabled you can confirm the auto mode option using Overview Tab in the console

• As enabled during enable step EKS created node pools too.

Disable EKS Auto Modę

You can disable EKS Auto Mode on an existing EKS Cluster. This is a destructive operation.

• EKS will terminate all EC2 instances operated by EKS Auto Mode.
• EKS will delete all Load Balancers operated by EKS Auto Mode.

Migrate from EKS Managed Node Groups to EKS Auto Mode

When transitioning your Amazon EKS cluster to use EKS auto mode, you can smoothly migrate your existing workloads from managed node groups using the eksctl CLI tool.

This process ensures continuous application availability while EKS auto mode optimizes your compute resources. The migration can be performed with minimal disruption to your running applications.

Prerequisites

• Cluster with EKS Auto Mode enabled
• eksctl CLI installed and connected to your cluster. For more information, see Set up to use Amazon EKS.
• Karpenter is not installed on the cluster.

Current state of Pods

• 2 pods are running on managed node groups.

jatin.mehrotra@CK0662-001 eks-auto-mode % kubectl get po -o wide
NAME                    READY   STATUS    RESTARTS   AGE     IP           NODE                         NOMINATED NODE   READINESS GATES
test-65b7dbddd4-jdxhw   1/1     Running   0          2m29s   10.0.2.52    ip-10-0-2-85.ec2.internal    <none>           <none>
test-65b7dbddd4-lzd4r   1/1     Running   0          2m29s   10.0.1.118   ip-10-0-1-128.ec2.internal   <none>           <none>

• As per docs, Use the following eksctl CLI command to initiate draining pods from the existing managed node group instances. EKS Auto Mode will create new nodes to back the displaced pods.

eksctl update auto-mode-config --cluster eks-auto-test --drain-nodegroup  --region us-east-1 --profile ck-test

• I thought it’s my eksctl version but looks like my eksctl hasn’t got update for this flag so far. There is open issue too for the official eksctl repository

• my eksctl version output (its the latest version)

eksctl info
eksctl version: 0.194.0

• The other commands for eksctl are running for my cluster so definitely not the connection problem

jatin.mehrotra@CK0662-001 eks-auto-mode % eksctl get addons  --cluster eks-auto-test  --region us-east-1 --profile ck-test
2024-12-02 12:29:41 [ℹ]  Kubernetes version "1.30" in use by cluster "eks-auto-test"
2024-12-02 12:29:41 [ℹ]  getting all addons
2024-12-02 12:29:44 [ℹ]  to see issues for an addon run `eksctl get addon --name <addon-name> --cluster <cluster-name>`
NAME                            VERSION                 STATUS  ISSUES  IAMROLE UPDATE AVAILABLE                                                                     CONFIGURATION VALUES     POD IDENTITY ASSOCIATION ROLES
amazon-cloudwatch-observability v2.5.0-eksbuild.1       ACTIVE  0
aws-ebs-csi-driver              v1.37.0-eksbuild.1      ACTIVE  0
coredns                         v1.11.1-eksbuild.8      ACTIVE  0               v1.11.3-eksbuild.2,v1.11.3-eksbuild.1,v1.11.1-eksbuild.13,v1.11.1-eksbuild.11,v1.11.1-eksbuild.9
kube-proxy                      v1.30.6-eksbuild.3      ACTIVE  0
vpc-cni                         v1.19.0-eksbuild.1      ACTIVE  0

Update: Dec 3, 12:20 AM JST

• Even though AWS team has released a new version of eksctl, issue still persist as well as documentation is also incorrect.

It needs --drain-all-nodegroups as well as cluster-config file which is very difficult to generate for existing cluster created by console or terraform.

eksctl update auto-mode-config --drain-all-nodegroups -f eks-config.yaml --profile ck-test

We need to wait more so that AWS can fix this bug. There is new issue for this bug.

Investigating the Update the Kubernetes Version of an EKS Auto Mode cluster

• The compute capability of Amazon EKS Auto Mode controls the Kubernetes version of nodes. After you upgrade the control plane, EKS Auto Mode will begin incrementally updating managed nodes. EKS Auto Mode respects pod disruption budgets.
• You do not have to manually upgrade the capabilities of Amazon EKS Auto Mode, including the compute autoscaling, block storage, and load balancing capabilities.
• Auto Mode simplifies the version update process by handling the coordination of control plane updates with node replacements, while maintaining workload availability through pod disruption budgets.
• When upgrading an Auto Mode cluster, many components that traditionally required manual updates are now managed as part of the service.
• After you initiate a control plane upgrade, EKS Auto Mode begins replacing nodes in your cluster. The new nodes have the corresponding new Kubernetes version. EKS Auto Mode observes pod disruption budgets when upgrading nodes.
  • CoreDNS
  • KubeProxy
  • AWS Load Balancer Controller
  • Karpenter
  • AWS EBS CSI Driver
• However You are still responsible for updating:
  • Apps and workloads deployed to your cluster
  • Self-managed add-ons and controllers
  • Amazon EKS Add-ons

From Solutions Architect Perspective

• In this blog we saw how EKS Auto mode simplifies Kubernetes operations by offloading infrastructure management to AWS.
• It automatically selects the best EC2 instances, optimizes compute costs, and dynamically scales resources based on demand.
• This feature enhances security, performance, and availability while reducing the need for deep expertise, capacity planning, and manual management.
• But as we saw in the blog its still new feature there might be a need for updates and bug fixes.
• Yes, its true the its simplifies operations and allows users to run Kubernetes like a PRO in secure manner but you have to pay a price for it.

Initially I think this will increase the cost but the real benefits would kick in after a sustained usage to offset the extra price with ease of operations.