I used to treat Vault like a snowflake appliance: click init, paste root token, vault write policies by hand, openssl for TLS, Terraform manifests for backups. That series still lives on this blog. Production taught a sharper split.
Ask an engineering or infrastructure team what is happening inside their CI/CD pipeline at runtime. The answer is usually a blank stare. Artifact signing, SBOMs, and pinned Actions SHAs matured. Runtime insight into the job itself lagged behind.
The image signature said yes. Provenance said GitHub Actions on main. The image still shipped with a build that opened the wrong sockets and ran a binary nobody named in the workflow. That is not a signing-tool bug. That is a missing link in the chain of trust.
If you are still using classic GitHub PATs in your environments "just for pulling, read or packaging," you are not being pragmatic. You are accepting forever access as normal. That is a culture problem. Tooling only follows.
Publish returns ack. Consumer never sees the message. Restart a pod and a stuck queue suddenly drains. Realtime join works on one pod; another user never sees it. Websocket adapter events vanish depending on which replica you hit. Same cluster. Same Service DNS. Different JetStream stores. That is the smell.
Do you even know the supply chain risk your Helm charts carry? I bet you don't.
best bet, you are still not rotating your DB passwords and tokens across your infra, but if the software integrations you use, has a stable passwordless option, i think you shouldnt keep a static credentials option then?
kubectl get pods is fine. kubectl describe is fine. Then you hit kubectl logs on a pod that just landed on a Karpenter node and get 401 Unauthorized. Same cluster. Same context. Pods on the system pool still give you logs. That split is the smell.
On EKS, "run Karpenter" is mostly: chart, IRSA, NodePool, EC2NodeClass. On AKS the same sentence hits a wall of prerequisites Microsoft already documented. Then a quieter wall nobody puts in the getting-started: regional vCPU quota.
Your audit logs are quiet. Falco did not fire. The pod filesystem looks clean. And the threat actor is still in the cluster.
Every container security program ends up doing the same three things: scan the image with Trivy in CI, sign it with cosign, put up a dashboard that counts CVEs. Then they call the problem "in progress."
That checklist is the easy 20% of the work. It makes a lot of noise but cuts very little risk.
Your team has pinned every third-party action to an immutable SHA. Runners are GitHub-hosted, ephemeral, isolated. Someone ran zizmor once and filed a ticket. The posture feels done.
On an engagement I worked alongside Olakojo testing a product. Between Ola is one of the best offensive security engineers in this space, and watching him work is a lesson in patience: he does not spray exploits and hope. He reads the app, finds the innocent looking path, and walks in through the front door everyone believed was secure.
A Reddit thread on r/devops stuck with me: someone at a 600-developer org with 2,000 repositories, Okta pushing users via SCIM, was redesigning RBAC and asking whether GitHub teams could realistically be managed with IaC. The replies split fast: Entra groups, access-request tickets, safe-settings, Terraform, and one blunt take: "Don't. Unless you have a full team of Terraform experts."
If you stay in this space long enough, you'll one day face real incident before you retire, pivot, or quietly stop answering pages. Dont let it catch you off guard, compose and face it, haha, its your turn.