Overview
PandaProbe is an open-source platform purpose-built for AI agent engineering. It captures full agent executions as sessions, traces, and spans, lets teams score those runs with agent-specific metrics, and turns the resulting data into ongoing production monitoring. Where general LLM logging tools stop at single prompt-response pairs, PandaProbe is designed around the messier reality of multi-step agent trajectories — chains of LLM calls, tool invocations, sub-agents, and external APIs.
Under the hood, the platform is licensed under Apache 2.0 and ships with a hosted PandaProbe Cloud as well as a fully self-hostable open-source build. A single instrument() call wires up automatic tracing for supported frameworks like LangGraph, LangChain, CrewAI, Google ADK, the Claude Agent SDK, and the OpenAI Agents SDK, while decorators provide manual instrumentation for in-house orchestration code or raw model APIs.
The project leans into composable evaluations: trace-level checks for individual reasoning steps, session-level checks for whole conversations, and lightweight structural signals so teams aren't forced to run an expensive judge model on every span. Combined with scheduled production evaluations, PandaProbe targets teams that need traces, evaluations, and monitoring in one agent-engineering workflow rather than plain request logs alone.
Key Features
- Full agent tracing — Capture sessions, traces, and spans across LLMs, tools, sub-agents, and custom logic. MCP tool calls are automatically traced for supported frameworks, and a single
instrument()call activates collection. - Trace and session evaluations — Score individual traces or whole sessions using configurable, agent-specific metrics. Async and sampled evaluation modes keep evaluation costs bounded as traffic scales.
- Production monitoring — Schedule recurring evaluations against new traces and sessions to catch regressions, drifting prompts, and tool failures without manual review.
- Performance analytics — Track latency, token cost, error rates, and quality trends over time, with breakdowns by agent, tool, model, and metric.
- Composable metrics and trajectory signals — Combine LLM-as-judge scoring with cheap structural checks (e.g., trajectory shape, tool-call sequence) so most traces are screened without a judge call.
- Manual instrumentation via decorators — Wrap any function, custom tool, or workflow step in a decorator to extend tracing into bespoke agent architectures or internal SDKs.
- Framework-agnostic SDKs — First-class integrations for LangGraph, LangChain, CrewAI, Google ADK, the Claude Agent SDK, and the OpenAI Agents SDK, plus direct support for OpenAI, Anthropic, and Gemini APIs.
How to Get Started
PandaProbe offers two getting-started paths depending on data residency and operational preferences:
- PandaProbe Cloud — Sign up on pandaprobe.com, create a project, and copy the API key. Install the SDK in your agent codebase, call
instrument()once at startup, and traces start flowing within minutes. The Hobby tier is free and requires no credit card, making it suitable for early prototyping. - Self-hosted (Apache 2.0) — Clone the open-source repository at
chirpz-ai/pandaprobeon GitHub and follow the deployment documentation to run the platform on your own infrastructure. All core features — tracing, evaluations, monitoring, analytics — are available in the open-source build, with no feature gating. - Manual instrumentation — For non-supported frameworks or custom orchestration code, decorate functions, tool calls, or workflow steps directly. This is the recommended approach for internal agent SDKs that don't match a standard framework shape.
After traces are flowing, teams typically set up evaluation metrics (e.g., correctness, tool-call accuracy, response coherence) and then promote those metrics into scheduled monitoring jobs that run against production traffic.
Pricing & Plans
PandaProbe uses a freemium model with a permanent free tier and pay-as-you-go overages on paid plans. The Apache 2.0 open-source build is also available at no cost for self-hosting.
| Plan | Price | Base Traces / mo | Trace Evals / mo | Session Evals / mo | Seats | Support |
|---|---|---|---|---|---|---|
| Hobby | $0 / forever | 100 | 100 | 10 | 1 | Community / GitHub |
| Pro (Popular) | $29 / mo | 5,000 (PAYG after) | 5,000 (PAYG after) | 100 (PAYG after) | 2 | |
| Startup | $299 / mo | 50,000 (PAYG after) | 50,000 (PAYG after) | 1,000 (PAYG after) | 10 | Private Slack, retention controls |
| Enterprise | Custom | Custom | Custom | Custom | Unlimited | SSO, dedicated team, SLA, hybrid/self-host |
| Open Source | Free (Apache 2.0) | Self-hosted | Self-hosted | Self-hosted | Unlimited | Community |
All paid tiers include human annotation and stack the previous tier's features (Pro = Hobby + more, Startup = Pro + more, Enterprise = Startup + more). Pay-as-you-go pricing applies once the monthly base is exhausted on Pro and Startup.
How It Compares
The agent observability space is crowded; PandaProbe's positioning is most useful when compared against a few common alternatives:
- vs Langfuse — Langfuse is a broader open-source LLM observability platform with a more mature public ecosystem, while PandaProbe positions itself more narrowly around agent-first tracing, evaluations, and monitoring. PandaProbe is narrower and agent-first: its public docs present trace/session evaluations, scheduled monitoring, and agent-framework tracing as core workflow primitives. Teams already invested in Langfuse usually evaluate PandaProbe when they need richer agent-specific eval workflows.
- vs LangSmith — LangSmith is a hosted LangChain/LangGraph-focused observability and evaluation platform; PandaProbe is Apache 2.0 and self-hostable, while still supporting LangGraph/LangChain-style tracing. PandaProbe is also explicitly framework-agnostic, with native integrations for CrewAI, Google ADK, the Claude Agent SDK, and the OpenAI Agents SDK.
- vs Helicone / Phoenix / Braintrust — Helicone, Phoenix, and Braintrust overlap with PandaProbe across LLM observability, tracing, and evaluation, but their positioning differs by deployment model, target workflow, and evaluation depth. PandaProbe sits at the intersection: tracing, evaluation, and monitoring as one product, optimized for autonomous-agent traffic patterns rather than single-shot prompts.
For teams whose agents already run on a supported framework, the differentiator is usually how cleanly trajectory-level metrics fit production monitoring without bolting on multiple tools.
Best For
- AI engineering teams running agents on LangGraph, CrewAI, Google ADK, the Claude Agent SDK, or the OpenAI Agents SDK who need trajectory-level evaluations rather than single-prompt logging
- Open-source-first organizations that need full self-hosting control under Apache 2.0 with no feature gating
- Teams scaling from prototype to production who want scheduled evaluations to detect quality regressions on real traffic
- Engineers building custom agent orchestration in-house who want decorator-based manual instrumentation alongside framework auto-tracing
- Startups prototyping agents on a free tier and graduating to predictable pay-as-you-go billing as traffic grows
FAQ
Is PandaProbe really open source?
Yes. The PandaProbe core platform is licensed under Apache 2.0 and the source is available on GitHub at chirpz-ai/pandaprobe. The open-source build includes all core features — tracing, evaluations, monitoring, analytics, APIs, and deployment documentation — with no feature gating, similar in spirit to other open-source AI agent infrastructure projects that ship a fully featured self-hostable build.
How does PandaProbe handle frameworks it doesn't natively support?
For non-supported frameworks or custom orchestration code, PandaProbe provides manual instrumentation via decorators. Wrapping any function, tool call, or workflow step exposes it to the tracing pipeline, which means custom or internal agent SDKs can be observed without a framework switch.
What is the difference between trace evaluations and session evaluations?
Trace evaluations score a single trace — typically one user request and the spans it produced (LLM calls, tool calls, sub-agent invocations). Session evaluations score whole multi-turn sessions, which is closer to how conversational or long-running agents actually need to be judged. Pricing tiers count them separately.
How are evaluation costs controlled?
PandaProbe supports async and sampled evaluations, plus lightweight structural trajectory signals (e.g., tool-call sequence checks) that don't require a judge model. Combining cheap structural checks with selective LLM-as-judge runs keeps total evaluation cost bounded even as trace volume grows.
Does PandaProbe replace tools like Langfuse or LangSmith?
It depends on the workload. PandaProbe is agent-first: trace/session evaluations and scheduled production monitoring are core primitives. Langfuse covers a broader LLM-observability surface, and LangSmith is tightly integrated with the LangChain stack. Teams running multi-step autonomous agents tend to evaluate PandaProbe specifically for trajectory-level metrics; teams with simple single-prompt workloads may find a general-purpose tool sufficient.
Is there a free tier I can use without paying?
Yes. The Hobby plan is $0/forever and includes 100 base traces, 100 trace evals, and 10 session evals per month, plus 1 seat and community support. The self-hosted Apache 2.0 build is also free with no usage limits beyond your own infrastructure.
What happens when I exceed my plan's monthly trace allowance?
On the Pro and Startup plans, usage above the monthly base allowance is billed pay-as-you-go for traces, trace evals, and session evals. The Hobby tier does not include overage billing; teams expecting to exceed 100 traces per month should plan to upgrade to Pro.
