Obsidian Metadata
| type | source |
| subtype | video |
| url | https://www.youtube.com/watch?v=e2i6JbU2R-s |
| speakers | Hamel Husain, Jacob Carter (Nurture Boss) |
| duration | 12:08 |
Video: Error Analysis: The Highest ROI Technique in AI Engineering
Jacob Carter (founder of Nurture Boss, AI leasing assistants for the apartment industry over voice, SMS and email, built on OpenAI) describes adopting error analysis.
Takeaways
- At launch the product was a black box. Jacob was initially frustrated by “look at your data” advice because it sounded like there had to be a secret, but after reading thousands of conversations he found it hugely valuable.
- What reading data gives you:
- A product roadmap: you see what users ask that the product can’t do.
- How users assume the product works, which never matches your assumptions. This surfaces bugs and adjustments.
- An intuition for how the model will respond, which informs tools, RAG, and prompts.
- Their custom viewer: a text conversation with human/AI turns, tool calls and tool responses, good/bad plus a free-text note. An AI helper then categorizes the negative notes. The top buckets were tour scheduling/rescheduling and handoffs not triggered (the AI should have brought in a human).
- Those notes became bug and feature tickets right away.
- Dates: users say “two weeks from now” or “March 1st” and the model picked wrong dates or years. They built date test cases in Braintrust’s playground and went from 33% to 100% (the Field Guide says 95%).
- Generic evals didn’t fit their setup. Conversations are “evergreen”: leads become residents and renew leases, so there’s no clean start and end. You only learn quirks like this by looking at the data.
Hamel’s summary
Don’t use off-the-shelf metrics. Build your own viewer to remove friction. The simplest form of error analysis is notes → categories → counts, and LLMs can help with the categorizing.