Evaluating LLM and Agentic Workflows

Guided end-to-end walkthrough: bring a model endpoint or a deployed Agent Studio workflow, choose a bundled benchmark (Spider, τ-bench, safety and truthfulness sets) or upload your own dataset, select metrics, and run the evaluation. The run produces a shareable Phoenix experiment with per-example scores, traces, and model-to-model comparisons.

The Reprise interactive tour (linked from the page banner) and the written Quickstart guide cover the same flow. Watch the demo

• LLM endpoint & safety evaluation: text2sql (Spider, TPC-H), agent tasks, and jailbreak/refusal/truthfulness benchmarks (AdvBench, Do-Not-Answer, JailbreakBench, TruthfulQA) against any OpenAI-compatible API, including self-hosted vLLM.

• Agent Studio workflow testing: black-box evaluation of deployed Cloudera Agent Studio workflows through the kickoff/events API.

• Ragas, LLM-judge, and custom metrics: AgentGoalAccuracy, ToolCallAccuracy, ToolCallF1, SQL execution accuracy, and your own Python metrics registered from the UI or API.

• Phoenix tracing and experiments: OTEL spans per example and one Phoenix project per dataset and model for clean, comparable, versioned results.

• One-click, in-tenant deployment: ships as a Cloudera AMP and single CAI Application (also Docker or CAII Applications); a local Python runner pushes results from CrewAI or custom scripts.

AI teams moving LLMs and agents into production repeatedly face the question “has the team tested this new model, does it suit our use case, and how do we know it is better than the last one?” CAI Eval Platform turns that into data. Teams run the same benchmark suite against every new model release and track drift over time, vet open-source or self-hosted models on Spider, τ-bench, safety, or their own domain dataset before adoption, black-box test Agent Studio workflows the same way they test a model, and give stakeholders a shareable Phoenix experiment link in minutes instead of days.

The platform ships as a single CAI Application where an nginx reverse proxy routes web requests to Arize Phoenix and the FastAPI eval engine. The engine loads datasets, invokes the target LLM endpoint or Agent Studio workflow, computes metrics, and records traces and experiments directly into Phoenix.

Evaluations generate spans per example and store runs in isolated Phoenix projects for seamless side-by-side comparison. The Application supports flexible deployment options, including CAI AMP, CI/CD pipelines, Docker, and CAII Applications.