IR Without a SIEM: Responding to a Supply Chain Event in Under an Hour

Background

In March 2026, a widely-used Python AI library was compromised at the package registry level. Malicious versions were live on PyPI for roughly 40 minutes before quarantine. The payload targeted environment variables, cloud credentials, SSH keys, git configs, and anything else a developer machine tends to accumulate. It also went after /tmp.

I wasn't running the affected versions in production. But I had the package in my orbit — demos, experimental repos, local tooling. When the disclosure started spreading, I had a decision to make: assume I was clean, or treat it as an active incident and work the problem. I chose the latter.

What followed was about 45 minutes of focused triage. No SIEM. No EDR. No IR retainer. Just standard tooling, a methodology, and the discipline to not skip steps just because I was probably fine.

The Attack Surface I Was Defending

My exposure was modest but real: a few Vercel-hosted demos pulling in Python dependencies, GitHub repos with varying levels of dependency pinning, and a dev machine with environment variables that absolutely did not need to leave my home network. AI tooling tends to accumulate credentials fast — LLM provider keys, cloud tokens, API secrets tucked into .env files. That's the target profile the payload was written for.

Triage Methodology

First question: am I in the exposure window? The compromised versions were specific — I needed to confirm what was actually installed, not guess.

# Check installed version across environments
pip show litellm

# Also check uv caches — easy to miss
find ~/.cache/uv -name "litellm_init.pth" 2>/dev/null

The payload in the more aggressive version (1.82.8) dropped a .pth file into site-packages/. Python processes .pth files automatically on every interpreter startup — no import required. If that file was present, the machine was compromised whether or not you ever called import litellm.

# Primary IOC: the malicious .pth file
find / -name "litellm_init.pth" 2>/dev/null

# Persistence mechanism: sysmon dropper
ls -la ~/.config/sysmon/sysmon.py 2>/dev/null
ls -la ~/.config/systemd/user/sysmon.service 2>/dev/null

# The crypto/exfil component — what I was specifically hunting for
find / -name "sysmon.py" 2>/dev/null

The earlier version (1.82.7) injected directly into proxy_server.py — a different vector, same intent. I checked that too.

# Git history inspection — did anything touch the proxy server file?
git log --all --oneline -- '*proxy_server.py'

# Look for unexpected file modifications in the install
pip show -f litellm | grep -E "\.pth$|proxy_server"

The /tmp Angle

The payload staged exfiltration data in /tmp before sending it out. On Linux with AppArmor properly configured, that path is a lot less useful to an attacker than they'd like. I'd set up AppArmor profiles on this machine specifically because transient write targets like /tmp are a classic staging ground — not because I'd anticipated this specific attack, but because defense-in-depth means your controls don't need to know the threat in advance.

# Quick check for any remnants in /tmp
ls -la /tmp/
find /tmp -name "*.py" -o -name "*.sh" -o -name "*.b64" 2>/dev/null

Secret Rotation

I wasn't in the installation window. My checks came back clean. I rotated anyway.

The reasoning: credential rotation costs me 20 minutes. Leaving potentially-compromised secrets in place costs me indefinitely. When the blast radius includes LLM provider keys, cloud tokens, and SSH credentials, there is no risk calculus where "probably fine" beats "definitely rotated."

• All LLM API keys (any provider with an active key on this machine)
• Cloud provider credentials — AWS, anything else in ~/.aws/ or environment
• Any tokens with write access to GitHub repos
• SSH keys associated with active infrastructure

Host Wipedown

The machine got a full wipedown. Not because I found evidence of compromise, but because I'd rather rebuild a dev environment than spend the next six months wondering. This is a legitimate IR decision, not paranoia — the cost of a clean rebuild on a personal dev machine is low; the cost of operating on an uncertain host is high and ongoing.

Wipedown checklist, in rough order:

• Remove affected package versions across all environments and caches
• Purge pip and uv caches: pip cache purge, rm -rf ~/.cache/uv
• Audit and rotate secrets stored in .env files, shell history, and credential stores
• Review ~/.gitconfig, ~/.ssh/, ~/.aws/ for unexpected changes
• Check shell history for unexpected network calls or subprocess invocations
• Verify GitHub repos for unexpected commits, workflow changes, or Dependabot impersonation

What the Broader Incident Shows

This wasn't an exotic attack. It was a credential stealer distributed through a trusted package registry, targeting a dependency class (AI tooling) that has grown faster than most teams' security hygiene around it. The payload specifically went after the secrets profile of an AI developer: LLM keys, cloud tokens, SSH access to repos that might contain more secrets. That's not coincidence.

The supply chain is the perimeter now. Not in a metaphorical sense — in the sense that running pip install on a trusted package is a lateral movement opportunity if someone upstream has a bad day. Every dependency is an implicit trust decision. Most of us are making those decisions implicitly, at scale, continuously.

The memefication of these events on social media is real, and understandable — but when someone actually gets hit, it stops being a meme fast. Credential theft from a dev machine means your cloud infrastructure is the next incident. Your customers' data is the incident after that.

Takeaways

• Pin your dependencies. A lockfile (poetry.lock, uv.lock) would have been immune to this attack — it pins to a known-safe version regardless of what's live on PyPI.
• You don't need a SIEM to respond. You need a methodology, IOCs, and the discipline to actually run the checks even when you think you're probably fine.
• AppArmor and similar controls earn their keep precisely in unplanned scenarios. Defense-in-depth isn't about predicting the attack — it's about ensuring your controls don't need to.
• Rotate on uncertainty. The asymmetry of cost almost always favors rotation.
• Awareness is a control. Knowing what packages your tooling pulls in, following security advisories for the ecosystems you work in, and having a mental model of your own attack surface — that's not optional hygiene anymore.

The tools I used here are available to anyone with a terminal. What matters is knowing what to look for and having the situational awareness to look when it counts.