Automating PR Generation from Observability Signals

Signal Processing and Grouping

To move from raw observability data to actionable code fixes, the pipeline must first filter out noise and group disparate signals into coherent problem reports. A primary challenge is that off-the-shelf embedding models prioritize structural similarity over semantic meaning, causing errors to cluster with other errors rather than with related Slack messages or session replays. To solve this, the system uses an LLM to generate descriptive queries for each signal, then performs embeddings on those queries instead of the raw signal data. This ensures that a Slack message about an onboarding issue and a related error log are correctly grouped together.

Agentic Research and Execution

Once a report exceeds a specific weight threshold, it is passed to a research agent running in a sandboxed environment. This agent utilizes an MCP (Model Context Protocol) server to pull in supplementary data, such as logs or external context from tools like Linear and Notion, to ground its analysis. If the problem is deemed actionable, the system clones the repository into a sandbox and uses the Claude agent SDK to generate a fix. The pipeline supports iterative development by snapshotting the sandbox state; if CI fails or a human reviewer leaves a comment, the agent rehydrates the snapshot to continue refining the code until the PR is green.

Lessons in Pipeline Design

• Specificity is critical: Agents will attempt to fix any problem they are given, so the system must filter for actionable reports. Error tracking data is typically specific enough for immediate code fixes, while session replays and Slack messages often require more human oversight.
• Prioritize experimentation over cost: Early in development, the team avoided using agents due to token costs. They found this to be a mistake, as running an agent against the same problem 100 times reveals patterns that allow developers to replace expensive agentic steps with cheaper, one-shot LLM calls or fine-tuned models.
• Production-grade evals: Testing on local data using vibe checks is insufficient for pipelines handling diverse customer data. The team emphasizes the necessity of evaluating performance on representative production data to avoid fumbling in the dark.