How do you handle AI agent observability and debugging in production?

AI agent observability requires tracing every decision and action through the agent pipeline. Implementation: assign a unique trace ID to each request, instrument every node (LLM call, tool execution, memory retrieval) with timing, inputs, outputs, and token usage. Tools: LangSmith (purpose-built for LLM applications), Arize Phoenix (open-source LLM observability), Weights & Biases Prompts, and custom OpenTelemetry instrumentation. Key telemetry: LLM call latency and token usage per step, tool call success/failure rates and latency, reasoning traces (full agent thought process), retrieval quality scores (relevance of retrieved context), and end-to-end task completion metrics. Debugging workflow: when an agent fails, replay the trace to see the exact sequence of thoughts, tool calls, and observations. Identify where the reasoning diverged: wrong tool selected, incorrect parameters, misleading tool response, or flawed reasoning. Common failure patterns: tool parameter hallucination, infinite loops (same tool called repeatedly), context window exhaustion (too much accumulated context), and cascading failures from one bad tool response. Monitoring dashboards: real-time success rate, latency percentiles, cost per request, error categorization, and trending issues. Set alerts for success rate drops, latency spikes, and unusual tool call patterns. Use A/B testing frameworks to compare agent versions before full deployment.