EKS Node Debug Nightmare, How to SSH into EKS Node

Ever been in that situation where the nodes are going into a NotReady state?

Yeah, that type of situation where you check the cluster logs and everything suggests something is actually causing an error, and now you want to exec into the nodes, just like you have been doing with pods or the way you ssh into those ec2 instance.

its actually possible with the eks nodes too, but i think it doesn't just look straight forward like the normal instance but here is what i have done in the past, that did work for me.

if you have deployed your EKS cluster in a declarative manner with terraform or opentofu, then this will be an easy feat.

All you need to do is attach the AmazonSSMManagedInstanceCore IAM Policy to the cluster as an additional policy, here's what the syntax should look like in your IaC.

module "eks" {
  source                                 = "terraform-aws-modules/eks/aws"
  version                                = "20.24.0"
  cluster_name                           = var.eks-name
  cluster_version                        = var.k8s-version
  .....

  eks_managed_node_group_defaults = {
    iam_role_additional_policies = {
      AmazonSSMManagedInstanceCore = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
    }
  }

 .....
}

the key thing here is the IAM Policy that was added to the managed node group defaults block.

iam_role_additional_policies = {
      AmazonSSMManagedInstanceCore = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
    }

This will allow you exec into your eks worker node with ease and its even more secure than declaring those ssh public and private key thing, eliminating security burden of keeping your private keys safe, haha.

At the same time you can add the this IAM policy manually to your eks node via console, go to the IAM roles attached to your eks node group, under the permissions policies, add the permission and pick the AmazonSSMManagedInstanceCore policy.

Also there is a kubectl plugins that can do this magic and there really no need for managing any SSH keys, luksa/kubectl-plugins project offers this convenient solution. This plugin provides allows you obtain shell access to your eks nodes, assuming you have cluster-admin privileges.

with just kubectl ssh node YOUR NODE NAME you can exec into your eks nodes

Well, that's it, folks! I hope you find this piece insightful and helpful.

Till next time, Peace be on you ✌️

Comments