Obsidian Metadata
Case Study: Nurture Boss (Apartment-Leasing AI)
Product: an AI assistant for apartment management. Prospects text or call, and the agent answers questions, books tours, and hands certain conversations to humans. Founder: Jacob.
Error analysis
- Built a simple data viewer: search and filter sessions (by channel: voice/text/chat, by property), annotate with open-ended notes, then aggregate and count errors.
- Bottom-up: spreadsheet rows = conversations → open notes → LLM-assisted taxonomy → map rows to labels → pivot table counts.
- Three issues accounted for > 60% of all problems:
- Conversation flow (missing context, awkward responses)
- Handoff failures (not recognizing when to transfer to humans)
- Rescheduling / date handling
- Date handling was failing 66% of the time (“tour two weeks from now”). After categorizing failures, building targeted tests, and measuring, success went from 33% → 95%.
- The team found so many actionable insights that implementing the fixes took several weeks.
Themes become metric groups
These themes are a natural way to group evals into per-theme pass-all rates (see Combining Evals into a Single Metric).
Automated error-analysis experiment (2026)
100 of their production traces, labeled by a domain expert (39 failures), were used to test Braintrust Loop, Arize Alyx, LangSmith, Codex, Factory Droid, and Claude Code. Best recall was 87%. All missed sales-objection handling, Markdown in SMS, voice interruptions, and missed handoffs. These failures look fine in the trace but violate product goals (see What LLMs Can and Can't Automate in Evals).
Related
Error Analysis · Human Handoff Evaluation
Video
Jacob explains the process himself in Video - Error Analysis Highest ROI Technique (Nurture Boss), including “evergreen” conversations with no clean start or end, which break generic eval tooling.