Skip to main content
Abdulmalik
AppSec Engineer
View all authors

From securing conventional product CI/CD to securing AI/ML product pipelines

You already secure a normal app end to end. Secrets scan. SAST. SCA. Build. Scan. Sign. Approve. Deploy. That is DevSecOps.

Same gates on train and model promote. New artifacts. Poisoned data or an unsigned checkpoint still lands without them. Extra gate software CI rarely names: pin the training dataset by digest.

Skill transfer: DevSecOps to AI security. No GPU needed. Act A is the app stack. Act B is gates on train → registry promote → API, CPU sklearn stand-in. Kubeflow, MLflow, SageMaker: same gates, different knobs. Signing and SBOM / ML-BOM / AIBOM are adjacent.

MLOps is the ML lifecycle. AIOps is AI for IT/ops. Not the same. This post is AI security on ML and LLM pipelines.

Related:

From software SBOM to ML-BOM and AIBOM

You already generate an SBOM for apps and container images. An AI system still leaves two blind spots. The weight pack on disk is not a PyPI graph. The model string your agent actually calls often never appears in pip freeze either.

I treat that as three inventories, not one mash-up JSON:

  1. Software SBOM: train/serve packages (cyclonedx-py / Syft) → Grype
  2. ML-BOM: CycloneDX machine-learning-model for the pack on disk (weights, hashes, pedigree) → hash-check
  3. AIBOM: what the running system calls: local tags, Modelfiles, cloud model IDs in config

Signing the pack and attesting the train job are separate controls. Those live in From signing containers to signing models. Do not treat this as a second signing tutorial.

Related:

Don't put the token in the agent

At work we kept finding the same miss. Product engineering wired Cursor to GitHub with a PAT in the laptop MCP config. CI ran Claude Code or an internal AppSec agent with GH_TOKEN or AWS keys in the job env. The agent could open a PR or touch S3. Sandbox looked fine. Then a tool dumped env, a log printed the secret, or a prompt-injected step ran printenv. The secret was already inside. The sandbox did not save us.

The fix is not "use OIDC in the deploy job." We already do that. The fix is a visible call path with three named pieces:

  1. Agent. Cursor, Claude Code, Copilot Chat with MCP, or an internal AppSec agent in CI. Decides. Sends tools/call. No durable secret in its env.
  2. Broker. Separate service. Company remote MCP on ECS / Cloud Run / Kubernetes, GitHub's remote MCP (https://api.githubcopilot.com/mcp/), or an egress / tool gateway such as agentgateway. Secrets live on the MCP task role, server env, or IRSA. Not on the laptop. Not in the agent job env.
  3. Upstream. Real api.github.com or AWS API call leaves from the broker. Agent only ever sees tool args and tool results.

Related:

From signing containers to signing models

I already know how to sign containers. Key-based Cosign, keyless OIDC, AWS Signer + Notation without public Rekor. That still applies when the artifact is a model directory. What changed is the shape of the thing I hash.

Signing answers one question: are these the exact bytes a known identity published? That is integrity of the artifact. It does not answer how the model was built, which base it came from, or what packages sit next to it at serve time. Those are separate controls. You do not need an SBOM, ML-BOM, or AIBOM to sign. Inventory is optional and adjacent if you want a bill of materials later. This post owns signing end to end, then shows provenance in CI.

Related:

You cannot deny every tool and still ship

Product wants the patch before standup. Security wants a gateway that can say no. I have sat between them. Both are right until someone makes it one dial.

Deny every tool and the agent is a chatbot. I have watched engineers stop waiting. They open the IDE on the laptop, shell still on, kubeconfig still in the environment, and they ship from there. Your deny rate looks perfect. Nobody complains. That quiet is the part that gets me. The control did not win. The work left.

Diagram: an agent sends read files, search code, and change code through a gate that lets safe work continue, while run commands, secrets, and touch production are stopped by default. A dashed emergency override expires. A dotted quiet bypass goes around the gate where everything is still on.

Same argument without the table. Let the everyday work through. Fail closed on the three that can ruin your week. Any override dies on a clock. And go find the agent that never hits the gate.

Allow the shell because the tests will not run, and you handed a confused deputy a production write. I already wrote that one up: when your bot becomes the payload. This is the deal I will defend when both sides are in the room.

You cannot deny every tool and still ship. Sort them by what they can break. Shell, secrets, and prod write fail closed. Read stays. A write that can only touch the PR branch stays. Take those away and you do not have a PR agent. You have a linter with opinions.

Agent-on-agent: when your bot becomes the payload

You ship an outer agent that plans. It spawns a "helpful" inner agent to dig, summarize, or call tools. Product demo looks clean. Trust model does not: classic confused deputy, just with LLMs.

The inner agent is often the one with broader tools, longer memory, or the system prompt that names secrets and schemas. Untrusted content does not need to jailbreak the outer model. It only needs to steer the inner one into echoing what it already knows, or calling a tool the outer agent would never have been allowed to call.

Your nested helper can become the payload. Prompt leak. Tool/schema leak. Same family of failure as approving an MCP server once and hoping every later tools/call is fine (MCP gateways).

Access requests belong where the team already works

Most access request "systems" are a scavenger hunt. Ticket portal for some tools. Slack DM for others. A spreadsheet for the weird ones. Someone with admin rights eventually clicks Approve in a console you do not have. The requester waits. The approver forgets. Onboarding docs live somewhere else.

The worst option is the DM. Almost no trail of when it was asked, why it was needed, or who said yes. That is a time bomb for offboarding, incident response, compliance, and the 2am question: who has maintain on that repo?

The fix is not a prettier portal. Treat access request as a product lifecycle: catalog → route → grant → onboard. Put it where the team already works. Slack for SaaS catalogs. GitHub Issues for repo and org roles. Same product idea either way: who, what role, why, who approves, audit trail.

What GuardDuty AI Protection actually detects (and what it doesn't)

AWS shipped GuardDuty AI Protection as GA in mid-July 2026. If you already run Bedrock or SageMaker AI and GuardDuty, this is the protection plan that finally looks at model invocation data events instead of only management-plane noise.

It is not "AI security done." It is a specific data-source plan with three finding types. Know what it watches, how to turn it on, and what to do when Low severity findings show up. Then know what it will never see.

MicroVM runners protect the host. Your supply chain still runs inside the job.

Teams buy Firecracker-/Kata-backed runners (or DIY) and talk like the CI supply chain problem is closed. Hardware isolation. Fresh VM per job. Tear it down. The vendor deck lands. The budget conversation moves on.

A microVM protects your host from the job. You still have to protect the job from the supply chain.

MCP this, MCP that. Attack surface, gateways, and cMCP.

Your engineering, infra, or security team is wiring agents to tools inside real environments: GitHub, Stripe, Slack, cloud APIs, filesystem servers, whatever the agent needs to act. Same session. Fine.

The quieter question is MCP gateway security: once those tools are reachable, who is actually allowed to say no before the upstream runs?

An MCP security gateway sits between the agent client and upstream MCP servers. It evaluates each tools/call against policy (allow or deny), keeps an audit trail, and optionally attests that path in a TEE. Raw MCP is the protocol and the catalog. The gateway is the control plane that can refuse a specific call before credentials, files, or APIs get touched.

That is different from a dumb reverse proxy that only terminates TLS or multiplexes routes. Auth at the door is not authz on the tool. Approving an MCP server once is not the same as deciding whether read_file on .env should ever leave the client.

Runtime Trace attestation: signed image, dishonest process

The image signature said yes. Provenance said GitHub Actions on main. The image still shipped with a build that opened the wrong sockets and ran a binary nobody named in the workflow. That is not a signing-tool bug. That is a missing link in the chain of trust.

Eradicate long-lived tokens before they eradicate you

If you are still using classic GitHub PATs in your environments "just for pulling, read or packaging," you are not being pragmatic. You are accepting forever access as normal. That is a culture problem. Tooling only follows.

Your JetStream is fine. You just have two of them.

Publish returns ack. Consumer never sees the message. Restart a pod and a stuck queue suddenly drains. Realtime join works on one pod; another user never sees it. Websocket adapter events vanish depending on which replica you hit. Same cluster. Same Service DNS. Different JetStream stores. That is the smell.

AKS + Karpenter: kubectl logs returns 401 Unauthorized

kubectl get pods is fine. kubectl describe is fine. Then you hit kubectl logs on a pod that just landed on a Karpenter node and get 401 Unauthorized. Same cluster. Same context. Pods on the system pool still give you logs. That split is the smell.

Container Image SBOM Blind Spots: What Scanning Your Image Still Misses

Every container security program ends up doing the same three things: scan the image with Trivy in CI, sign it with cosign, put up a dashboard that counts CVEs. Then they call the problem "in progress."

That checklist is the easy 20% of the work. It makes a lot of noise but cuts very little risk.

Kubernetes Runtime Security: The Silence That Should Keep You Up at Night

On an engagement I worked alongside Olakojo testing a product. Between Ola is one of the best offensive security engineers in this space, and watching him work is a lesson in patience: he does not spray exploits and hope. He reads the app, finds the innocent looking path, and walks in through the front door everyone believed was secure.

Zero Trust Network Access vs VPN: The Art of the Rabona

A rabona is not the obvious pass. You wrap your leg behind the standing one, hit the ball at an awkward angle, and somehow the play opens up anyway. VPN vs Zero Trust feels like that at first, same pitch, different move entirely.

AWS Cloud Costs Nightmare, Cutting It To Elongate That Startup Runway

Cloud costs at early stage startups rarely spiral because of recklessness.

They spiral because the team was moving fast, the architecture made sense at the time, and nobody had the bandwidth to revisit it.

By the time the bill becomes a problem, the decisions are already baked in.

I have been on both sides of this, burning through it personally, and leading teams trying to unwind it before the runway ran out. Here is what has actually worked.

Not enough pod IPs on EKS: secondary CIDR and prefix delegation

Problem: pods stuck Pending because the VPC CNI has no more IPs (not enough addresses, failed ENI/IP assignment). Fix in this post: secondary VPC CIDR + VPC CNI prefix delegation, without rebuilding the cluster for IPv6.

Sooner or later, your Elastic Kubernetes Service (EKS) Cluster will run out of IP allocation for your workloads, pods and all.

2024 In Review

A lot of uncertainty this year fr fr. Like, A LOT lot. But we move.

Tried many things i feared trying in the last two years

EKS Node Debug Nightmare, How to SSH into EKS Nodes

Have you ever faced that dreaded moment when your EKS nodes suddenly go into a NotReady state? If you're managing an Amazon Elastic Kubernetes Service (EKS) cluster, this scenario might be all too familiar. While checking cluster logs gives you some insight, sometimes you need direct access to the nodes themselves.

Sign container images with AWS Signer and Notation (no public Rekor)

This post is AWS Signer + Notation for private ECR signing. It is not Cosign, and it does not use Sigstore's public Rekor.

(URL note: the live slug still has a historical typo, aws-signerion. The corrected path redirects here so old links and Search Console entries keep working.)

Look, I get it. Everyone is using Cosign and Rekor for container signing these days. I've used it myself (check out my previous post if you're curious). But when you're working on private projects, using Sigstore's public Rekor instance isn't really an option.

MongoDB IAM authentication on EKS with IRSA (passwordless)

Short answer: use Atlas / MongoDB IAM authentication (authMechanism=MONGODB-AWS) with an EKS IRSA role on your pod's service account. The driver picks up the web identity token. No static DB password in the env.

Azure twin (Workload Identity + MONGODB-OIDC): MongoDB passwordless on AKS.

best bet, you are not rotating your password and tokens across your infrastructure and deployment but if the software you use has the passwordless authentication option and it's stable why not go for it?

2023 In Review

We try, next year we go again. ✌🏽✌🏽 E go be.

For the longer community story behind the engineering posts, see Leading Communities.

Add Kubernetes clusters to ArgoCD declaratively (multi-cluster GitOps)

You've probably gotten to a point where you need to manage multiple clusters using GitOps, knowing that managing the argocd instance itself can be considered tedious or painful, haha, meaning you sure do not want to install new argocd instances on other new Kubernetes clusters.

Applying Network Security using VPC Flow Logs with Terraform

You have deployed resources on AWS and hardened the application layer, but the network layer is often overlooked. If an EC2 instance or Redis node ends up in a public subnet by mistake, VPC Flow Logs are the fastest way to detect unexpected traffic.

Leading Communities

Been following the tech communities in Ekiti from 100 Level, passionate about every bit of it, the way I swiftly leave classes to the Tech Hub even made my colleagues nickname me "Techub".

But then there was no clear path, no focus, looking around there were no cyber security communities, so what was I doing? I joined the dev communities, going to every event just to take the swag and yes learning too.

Managing Secrets in Infrastructure As Code with Terraform

Okay, you've moved your infrastructure provisioning from visiting the console page and now adopted IaC ( Infrastructure as Code) for provisioning your infrastructure using Terraform.

So along the way, you discovered that you will need some sensitive credentials like GitHub token to use with aws amplify, datadog API and key deployments?

Install Karpenter on EKS with Terraform

What this is: Terraform to put Karpenter on an existing EKS cluster (policy, IRSA, instance profile, Helm), then provision nodes that match your workloads. If you want Cluster Autoscaler instead, this post is not that path.

Struggling to pick the right autoscaler for your Kubernetes cluster? Trust me, I get it. With all the options out there, choosing between Cluster Autoscaler, Karpenter, and others can be overwhelming.

Here's the deal - while both Cluster Autoscaler and Karpenter are backed by AWS, I've found Karpenter to be consistently faster at both scaling up and down. Let me show you how to set it up.

How to install ArgoCD on Kubernetes with Terraform

Install ArgoCD with Terraform: deploy the argo-cd Helm chart to an existing Kubernetes cluster, connect a private Git repo over SSH, create an Application, and optionally wire a GitHub webhook so sync is not stuck on the 3-minute poll.

This is the day-one install path. Already running ArgoCD? Jump to More ArgoCD on this blog for issues, multi-cluster, and Helm secrets.

Run github action locally using act

Yeah, being doing the CI/CD implementations via github workflow lately and I am also trapped in the process of making commits to trigger the workflows or better still making empty commits, haha.

Leveraging git hook for hardcoded secrets scanning in a codebase

Almost everyone knows how to use .gitignore, the git file that helps in keeping sensitive files like .env out of the tracking, commit, and pushing process, and also unwanted folders like node_modules and all.

But do you know secrets, hardcoded credentials, and API aren't easy to deal with using a .gitignore file? you don't want to keep your config.js or config.go file out of the commit process, these are essential files to your project.

Documentation as a gateway to Open Source

Documentation is a vital part of any open source project or software.

It is the entry point or a fall back option for users of any open source project or software to read usage, installation instructions, to fix issues and learn more about the project.

Open Source Internships Program in 2024

Before we start, I will like to say, All these programs listed are not mandatory for you to contribute to open source.

But you can contribute to open source even if you don't get selected for these programs, you can always contribute to open source programs anytime you want.

Stored XSS on Private Bounty Program (My First Bug)

So I started participating in bug bounty not so long and after a lot of read ups and web app practice, I found a stored XSS on quite a big education platform which i was using for learning web development last year.