Securing Agentic CLIs with Deterministic Guardrails

The Breakthrough

PostHog implemented a deterministic security scanner called Warlock that sits outside the agent loop, using YARA rules to enforce security policies and block malicious behavior before an LLM is ever consulted for triage.

What Actually Worked

• Decoupled Enforcement: The security architecture separates detection from judgment. Deterministic YARA rules act as the bouncer, while an LLM layer acts only as an advisor to filter false positives.
• Supply Chain Scanning: Because the agent consumes context from internal documentation and example apps, the team scans content at the source (when skills are built) and at the point of use (when the agent runs) to prevent prompt injection via poisoned context.
• Fail-Closed Triage: The LLM triage layer is designed to reduce noise, but if the model fails or behaves unexpectedly, the system defaults to a fail-closed state, killing the agent session to protect the user.
• Contextual Rule Design: Rather than blocking broad keywords like "ignore," rules require a combination of a verb and an instruction-flavored noun to trigger, reducing false positives from legitimate code comments.

Context

PostHog's Wizard is an agentic CLI that automates SDK installation and dashboard setup. As the tool gained popularity, the team identified that an agent with shell access and a large context window effectively functions as a "malware starter pack." The team moved away from relying on system prompts for security, instead building a multi-layered defense that treats the agent's own context supply chain as a primary attack vector.

Notable Quotes

• "Attacks compose, code review does not."
• "If it isn't enforced deterministically, it is not enforced."
• "Detecting a problem is one job and deciding what to do about that problem is a totally different job."