Obsidian Metadata
| type | source |
| subtype | video |
| urls | https://www.youtube.com/watch?v=y0vm_fjkejo, https://www.youtube.com/watch?v=97iykOemOn4, https://www.youtube.com/watch?v=wcYnzHJlUR0 |
| speakers | Hamel Husain, Shreya Shankar, Bryan Bischof, Harrison Chase, Wade Gilliam, Arize product lead |
| durations | 46:14, 41:42, 32:40 |
Videos: LLM Eval Tools Compared (LangSmith · Braintrust · Arize Phoenix)
Three “Mystery Science Theater”-style panels. Each vendor completed course homeworks 1–2 (write a system prompt for a recipe bot, build a query set, generate synthetic data, open- and axial-code traces), and Hamel, Shreya, and Bryan commented live. Written summary: Source - Selecting the Right AI Evals Tool and Eval Tooling and Vendors. Extra color from the recordings:
Recurring panel reactions (tool-agnostic lessons)
- “Stacked abstractions” are a red flag. An AI generates the rubric, scores against it, then optimizes the prompt against that score (99% average!). As one panelist put it, you can’t optimize toward an objective you don’t know yet. Automation is fine for exploration (e.g., finding examples to inspect), not for business decisions at this stage.
- Beware the headline number. A demo showing 99% will be read by someone as “done”, or by an executive who then wants it higher.
- One-click synthetic data from a few seed examples is homogeneous. Generating from lowercase seeds produced lowercase queries. Vendors should ask the user for dimensions (a small form) before generating. Bryan: a small, high-quality set beats a big, messy one. At Hex they multiplied a few great evals 5–10× with tightly controlled prompting and still kept humans reviewing.
- Involving the real users as SMEs (Wade’s family wrote the queries) got strong praise, since that’s real data.
- Markdown rendering and readable fonts in trace views matter. Point statistics per row (tokens, latency) are eye candy. Show distributions or histograms if you show anything at all.
- The “money table” (traces × failure modes) is exactly what you want after axial coding. Beyond that, the panel preferred a notebook over proprietary query languages.
- Axial coding with an LLM: give it context. It’s fine to use the terms “open codes / axial coding”, since LLMs know grounded theory. Once a failure mode repeats, turn it into a dropdown tag. You need a spreadsheet-like view of each example next to its axial code to debug the grouping. (Harrison’s LLM grouping produced more “failure modes” than distinct notes.)
- UI design matters: jargon, padding, rounded corners, and Fitts’s law (big, close buttons speed up labeling). A “Tinder-style” rapid review UI for SMEs, maybe on a separate subdomain that hides the internals.
- Prompt components: Bryan wants system prompts built from ablatable components so you can test the effect of removing each section.
- Final verdicts: no landslide. LangSmith has a strong annotation queue and trace→playground flow. Braintrust has the most intuitive UI and a structured SME process. Phoenix is notebook-centric, open source, and hackable. Bryan: none beat the tool he built himself for his own task, so invest in a vendor once you hit real production requirements.
→ Eval Tooling and Vendors · Synthetic Data · What LLMs Can and Can't Automate in Evals