Evals-Driven Development for Mental Health AI

The Breakthrough

SonderMind replaced generic, over-sensitive LLM safety filters with a modular, clinician-in-the-loop evaluation architecture that treats clinical judgment as a testable, version-controlled asset in CI.

What Actually Worked

• Modular Guardrail Architecture: The team decoupled guardrails from the core model, implementing them as separate, dedicated LLM calls. This isolation makes the safety logic harder to jailbreak and easier to iterate on without affecting the primary conversational agent.
• Clinician-Led Annotation Loop: Licensed professionals review flagged conversation traces to define the "expected observation." These annotations are converted into typed evaluation schemas that serve as regression tests for every model or prompt change.
• Clinical Rubric Integration: Annotators use a structured rubric to tag traces with specific metadata, including turn index and category, which allows the engineering team to replay conversations up to the exact point of failure during testing.
• Custom Safety Calibration: The team explicitly disabled the built-in safety filters of frontier models to avoid over-calibration, opting instead for a custom-built system calibrated to distinguish between active crisis scenarios and past-tense reflection.

Context

General-purpose LLMs are often too conservative for mental health applications, leading to "false positive" guardrail triggers that isolate vulnerable users. SonderMind built a system where clinical experts define the ground truth for safety, ensuring that the AI provides support when appropriate and surfaces human resources only when a crisis is detected. The team has open-sourced their evaluation datasets to provide a shared baseline for other developers in the high-stakes mental health space.

Notable Quotes

• "The win isn't that this one box sentence got fixed; it's that the entire self-harm category got lifted."
• "We're not going for more triggers here; we're going for more correct triggers."
• "We can't just promise safety; we need to deliver the most rigorous systems we can, especially in mental health."

Content References

• Tool: SonderMind Mental Health AI Coach (Sonder), developed by SonderMind, recommended as the primary subject of the talk.