Skip to main content

EKS Node Debug Nightmare, How to SSH into EKS Nodes

· 3 min read
Abdulmalik
Abdulmalik
AppSec Engineer

Have you ever faced that dreaded moment when your EKS nodes suddenly go into a NotReady state? If you're managing an Amazon Elastic Kubernetes Service (EKS) cluster, this scenario might be all too familiar. While checking cluster logs gives you some insight, sometimes you need direct access to the nodes themselves.

The Challenge with EKS Node Access​

Unlike regular EC2 instances or Kubernetes pods, accessing EKS nodes isn't as straightforward. However, there are secure and efficient methods to gain SSH access when you need to debug node-level issues.

The most secure approach involves using AWS Systems Manager (SSM) through proper IAM role configuration. Here's how to set it up:

If You're Using Infrastructure as Code (IaC)​

For those using Terraform or OpenTofu, you can easily attach the required IAM policy to your EKS cluster. Here's the configuration:

module "eks" {
  source          = "terraform-aws-modules/eks/aws"
  version         = "20.24.0"
  cluster_name    = var.eks-name
  cluster_version = var.k8s-version

  eks_managed_node_group_defaults = {
    iam_role_additional_policies = {
      AmazonSSMManagedInstanceCore = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
    }
  }
  // ... other configurations
}

The key addition here is the AmazonSSMManagedInstanceCore policy, which enables secure node access without managing SSH keys.

Manual Configuration Through AWS Console​

If you prefer using the AWS Console, follow these steps:

  1. Navigate to the IAM dashboard
  2. Locate the IAM role attached to your EKS node group
  3. Under "Permissions policies", add the AmazonSSMManagedInstanceCore policy
  4. Save your changes

Method 2: Using kubectl Plugins​

For a more Kubernetes-native approach, you can use the kubectl-ssh-node plugin. This method requires cluster-admin privileges but offers a simpler command-line interface.

Quick Setup and Usage​

Once the plugin is installed, accessing nodes is as simple as:

kubectl ssh node YOUR-NODE-NAME

Security Benefits​

Using SSM for node access offers several advantages:

  • No need to manage SSH keys
  • Access is controlled through IAM policies
  • All connections are logged and auditable
  • Secure communication through AWS's internal network

Troubleshooting Tips​

When debugging EKS nodes, remember to:

  1. Check node status first: kubectl get nodes
  2. Review system logs once connected
  3. Monitor resource usage
  4. Check kubelet status and logs

Conclusion​

Having secure access to your EKS nodes is crucial for effective debugging and maintenance. Whether you choose the IAM role approach or kubectl plugins, these methods provide secure and reliable access when you need it most.

Got questions or feedback? Drop a comment below!

Looking to learn more about EKS management? Stay tuned for more articles on Kubernetes troubleshooting and AWS best practices.

Till next time, Peace be on you 🤞🏽


Comments