Obsidian Metadata
| type | source |
| subtype | video |
| url | https://www.youtube.com/watch?v=B_DMMlDuJB0 |
| speakers | Hugo Bowne-Anderson, Hamel Husain, Emil Sedgh (Rechat CTO) |
| duration | 1:04:40 |
Video: Data and DevOps Tools for Evaluating and Productionizing LLMs (Vanishing Gradients)
This 2024 conversation became the blog post Source - Your AI Product Needs Evals. It adds operational detail to Rechat Lucy.
Details not in the blog
- Where the data started: an internal QA team held daily conversations with Lucy in a debug mode (shortcut menu of capabilities, a link straight to the LangSmith trace) and reported issues. Humans didn’t scale, which motivated synthetic generation.
- LangChain/LangSmith rationale: abstractions plus an ecosystem. Example: a self-healing option that re-prompts when function-call JSON doesn’t match the schema cut a whole class of errors to near zero. Traces were logged to LangSmith (OpenLLMetry was mentioned as an open alternative).
- Assertions are really integration tests. They call the real LLM and run on every commit that touches AI code in GitLab CI. Results go to Postgres → Metabase, including side-by-side comparisons of a base model vs a fine-tuned one on the same dataset. Fixing one failure raised another, which shows why systematic tests matter.
- They stayed in TypeScript, matching the existing stack and CI, rather than splitting off a Python service. Hamel notes that a lot of LLM app code in the wild is TypeScript.
- Custom review app (Shiny for Python, open-sourced pieces as “langfree”): pulls traces out of LangSmith into Airtable, shows the multi-turn conversation, function calls and results, filters by feature/scenario, and supports accept/reject with an editable final output (they only edit the final message, since they don’t trust themselves to edit mid-trace).
- Synthetic pipeline: GPT-4 writes ~50 requests per tool (e.g., “ask your assistant to send an email”), Lucy runs them, the CI assertions auto-filter bad outputs, and survivors go to the DB for human review. Humans only see pre-vetted cases.
- Fine-tuned GPT-3.5 on the curated data. It eliminated an “invalid placeholder” failure and empty listing-search responses. Prompt changes and fine-tuning data must be versioned together. If you delete an instruction from the prompt, scrub it from the training data too.
- Future work: track the correlation between a reward model or oracle LLM and human judgments, and reduce human review once correlation is high enough. A/B tests with user feedback come later.
- Advice: keep it simple and optimize iteration speed. Emil’s “golden chain” metaphor: start with a wooden chain (Airtable and other hacks) and replace links with gold as needed. Not everything needs to scale.