Closing Observability Gaps in AI Agents with Microsoft Foundry

Agent Drift and the Observability Gap

Agents degrade over time due to model updates, prompt tweaks, and accumulating edge cases, creating a widening gap between intended behavior (the 'platform') and actual performance (the 'train'). Speakers Amy Boyd and Nitya Narasimhan use the London Underground's 'Mind the Gap' signs as an analogy: platforms evolve but trains vary, requiring constant checks for fit. Safety guardrails warn users of risks, mirroring agent monitoring for customer interactions. Developers need ongoing evaluations to detect drift early, throughout the lifecycle—from build to production and fleet-scale management. Without this, non-determinism (inherent to agents) leads to unreliable production systems passed to customers.

The core problem: Agents extend LLMs with tools for knowledge and execution, but choosing from 2M+ Hugging Face models or 11K+ in Azure's catalog is paralyzing without baselines. For a new travel agent at fictional Contoso, developers lack eval datasets, prompting a need for zero-to-prototype tools that handle model selection, data generation, quality checks, and safety against adversarial prompts.

Tracing with OpenTelemetry for Multi-Agent Workflows

Observability starts at build time with tracing built on OpenTelemetry (OTel) standards. Foundry instruments agents regardless of build tools, aggregating traces in its control plane. This reveals tool calls, messages, and workflow steps, crucial for multi-agent systems where debugging complexity explodes.

For a weather agent example: Evaluate intent resolution (e.g., 'local weather' from 'What's the weather in London?'), tool call correctness (e.g., expected API), and task adherence in the final response. Traces pinpoint failures across the lifecycle, enabling targeted fixes amid non-determinism (expect percentages, not absolutes).

Integration with Azure Monitor pulls in infra/data signals, satisfying IT admins while developers use preferred stacks. This hybrid approach—build anywhere, observe centrally—scales to fleet-wide views of many multi-agent systems.

Built-in and Custom Evaluators Across Dimensions

Foundry embeds evaluators for quality (e.g., response relevance), risk/safety (e.g., hallucinations, toxicity), and agent-specific metrics (intent resolution, tool selection, task completion). Users mix built-ins with customs for scenarios like operational metrics on tool calls.

Continuous evals trigger on code changes; scheduled ones run periodically. This closes the evaluate-debug-optimize loop: Scores alone aren't enough; insights drive prompt tweaks or rollbacks.

"Agents are non-deterministic that's not just a problem for demos that's also a problem for real life when you actually get to production and the reliability and consistency is starting to become when you start passing this out to customers you need to be managing the non-determinism that comes inside those agents." – Amy Boyd, emphasizing production realities over demo illusions.

Observe Skill: Zero-to-Optimized Agent in One Prompt

The demo highlight: 'Observe skill' automates from no dataset. Point it at an agent; it generates eval data, batches evaluations, optimizes prompts, A/B tests versions, and rolls back to the best—showing reasoning at each step.

For the travel agent (hotel/car/flight booking): It surfaces unknown failures like poor tool chaining or edge-case handling. This accelerates prototyping: No manual data curation; AI handles dataset creation, metric computation (quality/safety), and iteration.

Tradeoffs: Relies on Foundry's coding agent for automation; costs ~$10 for workshop runs (free tier viable with Discord credits). Non-determinism means iterative refinement, but transparency in skill reasoning reveals blind spots.

"You point it at an agent with no eval dataset, no baselines, nothing. It generates the dataset, runs batch evaluations, optimizes the prompt, compares versions, and rolls back to the best one... all from a single prompt." – Description framing the skill's power, per video notes.

Red Teaming and Fleet-Scale Safeguards

Safety distinguishes normal-user evals (quality) from adversarial attacks (safeguarding). Red teaming uses a second AI to probe vulnerabilities with malicious prompts, akin to hiring a burglar to test home security.

Microsoft collaborates on open-source red teaming repos (e.g., 'Pirate') and offers one-click Foundry options. Fleet control centralizes observability across agents from any host, monitoring dimensions like security and performance.

Future: Managing 'many multi-agent systems' via unified views, integrating cloud monitoring.

"The difference between the second and the third is that the second assumes that your users are acting normally the third says I have a malicious user who's going to try to prompt attack my solution." – Nitya Narasimhan, clarifying quality vs. safety evals.

Workshop Resources for Hands-On Replication

Fork the GitHub repo (all branches for evolving workshops like AIE-Europe), use GitHub Codespaces with dev containers for instant env setup (VS Code in browser, pre-installed tools for notebooks/skills). Paths: SDK for beginners; coding agent for advanced.

Join Discord's AI Engineer channel for support. Azure account needed (~$10 cost); org repos offset compute.

"This repo is actually almost like a 4-hour workshop that we're trying to compress into this so what I want you to think about is this is a cooking show i'm going to show you the baked goods but I want you to look at the repo." – Nitya Narasimhan, urging repo use over live follow-along.

Key Takeaways

• Instrument agents early with OTel tracing for workflow visibility, aggregating multi-agent traces in Foundry's control plane.
• Evaluate holistically: Intent → tool call → task adherence, using built-ins for quality/safety/agent metrics or customs.
• Use 'Observe skill' to bootstrap evals: Generates data, optimizes prompts, shows reasoning—ideal for zero-baseline prototypes.
• Differentiate quality evals (normal users) from red teaming (adversarial); integrate with Azure Monitor for IT buy-in.
• Prototype fast via Codespaces/dev containers; fork repo for ongoing workshops, engage Discord for tweaks.
• Manage non-determinism via continuous/scheduled evals; expect percentages, focus on optimize loop.
• Scale to fleets: Centralize observability for many agents, regardless of build/host.
• Tradeoff: Automation speeds iteration but incurs compute; transparency reveals hidden failures worth the cost.