Obsidian Metadata

type	concept
sources	Source - AI Evals FAQ, Source - Selecting the Right AI Evals Tool, Source - Demystifying Evals for AI Agents

Eval Tooling and Vendors

”What’s your favorite eval vendor?” None.

• The space is intensely competitive; any feature comparison “would be invalidated in a week”.
• The vendors Hamel meets most often: LangSmith, Arize (Phoenix / AX), Braintrust. Also mentioned: Langfuse (self-hosted OSS), Harbor (containerized agent eval runs, Terminal-Bench 2.0), HoneyHive, OpenLLMetry, Humanloop.
• Selection depends mostly on support quality and the human factor (“dare I say, vibes”), plus client size and use case.
• Core features are very similar. Hamel uses vendors as a backend data store and builds notebooks plus custom annotation interfaces on top.
• Anthropic: frameworks are “only as good as the eval tasks you run through them”. Pick one quickly and invest in test cases and graders.

Criteria for assessing a tool (panel with Shreya Shankar & Bryan Bischof)

1. Workflow and developer experience. Time from observing a failure to iterating on a fix (e.g., trace → playground in one click). For notebook-centric teams, look at SDK ergonomics, docs, and integration with existing data tools.
2. Human-in-the-loop support. First-class manual annotation and error analysis. Axial coding is missing from many tools.
3. Transparency and control vs “magic”. Be skeptical of automation without human validation, e.g., an agent that writes a rubric and then immediately scores outputs (“stacking of abstractions”).
4. Ecosystem integration vs walled gardens. Fits your stack; no proprietary DSLs; export to common formats is a must.

Mid-2025 snapshots (take with salt)

Tool	Liked	Critiques
LangSmith	Smooth trace → playground; Prompt Canvas; dataset schema detection; Annotation Queue	Weak side-by-side prompt comparison; cluttered UI; AI-generated examples can be homogeneous
Braintrust	Clean UI; structured, SME-first process; strong HITL; the “Money Table” of failure modes	”Loop” auto-scorer writes the rubric and scores (false security); proprietary BTQL; clunky synthetic-data round trips
Arize Phoenix	Notebook-centric, export to pandas; open source and local-first, “hackable”; clear prompt UI	Output readability (markdown rendering); point stats rather than histograms; monolithic system-prompt editor (wants ablatable components)

Recorded panel commentary with more lessons (stacked abstractions, homogeneous one-click synthetic data, the “money table”, ablatable prompt components): Video - LLM Eval Tools Compared (LangSmith, Braintrust, Phoenix).

Gaps you’ll likely fill yourself

Table stakes most tools do well: trace logging, metrics tracking, prompt playgrounds, annotation queues. Expect to build:

1. Error analysis and pattern discovery: clustering similar issues (e.g., recognizing “persona–tone mismatch” across “wrong tone for investor” and “too casual for luxury buyer”), semantic search, rewriting notes into clean taxonomies.
2. AI assistance throughout the workflow: categorizing observations, proposing fixes (e.g., after 20 cases of omitted pet policies, suggest prompt edits; notice missing SQL WHERE clauses), and data analysis in AI notebooks (Julius, Hex) such as “location ambiguity errors spike 3× when users mention neighborhoods”.
3. Custom evaluators over generic metrics (“proposing unavailable showing times”, “omitting budget constraints”).
4. APIs for custom annotation apps: true bulk export (no paginating thousands of requests through timeout-prone endpoints) and efficient annotation write-back.

Vendors addressing these gaps are a strong signal they understand practitioners.

Related

Annotation Tools and Review Interfaces · Internal Eval Platforms · What LLMs Can and Can't Automate in Evals