Obsidian Metadata
| type | source |
| subtype | video |
| url | https://youtu.be/R_HnI9oTv3c |
| speakers | Bryan Bischof (Theory Ventures, ex-Hex), Hamel Husain |
| duration | 56:13 |
Video: Stop Managing AI Projects Like Traditional Software (“Burn your roadmap”)
Bryan explains how to operationalize evals in planning. This expands Capability Funnel and Experiment Roadmaps and Transition Failure Matrix.
Why feature roadmaps fail for AI
Roadmaps act like a ledger of dated, interchangeable tickets you can plan marketing and hiring around. For genuinely new AI work you’re exploring territory that isn’t on the map, so date-committed features are fiction. They’re fine for well-trodden builds (e.g., implementing a known paper).
Experiment-driven roadmap recipe
- Start with a question (hypothesis), not a feature. “Can we find relevant tables for NL→SQL by row values?” instead of “build a keyword index”. “Should we rerank or interleave RAG results?” instead of “seven vendor meetings about rerankers”.
- Attach a metric: table recall, per-attribute extraction accuracy, binary classification accuracy, the effect of the relevant doc’s rank.
- Estimate the effect size using small pre-experiments and data analysis (e.g., “we retrieve the right table 73% of the time; we expect single-digit gains”).
- Plot ROI vs effort to prioritize. Now PMs and engineers can have honest priority conversations.
Evals should show which assumptions are false
Capability funnel for NL→SQL: syntactically valid → executes → returns relevant results → matches intent → solves the user’s actual problem (e.g., ARR over time or by segment, not a single number).
- Evaluate components, not just final pass/fail. “Did it work?” gives you nowhere to go. (Hamel’s variation: do end-to-end error analysis first, then zoom into components.)
- Steps are dependent, and failing early fails everything downstream. Failures near the top of the funnel are worth more, and fixing them can reveal new failures lower down.
- For agentic “spaghetti”, linearize: pick the paths that matter and order the dependencies. If you can’t define start → steps → end, you may not know what you’re trying to achieve.
- Human-in-the-loop products (e.g., Stitch Fix stylists) are evaluated on whether the candidates handed to the human were good enough.
- Building the funnel needs human-labeled answers, then back-propagating what each step must have produced.
Analyze like a data scientist
- Hunt for data stories. A number going up isn’t enough.
- Compare experiments with before/after transition matrices over funnel states. Two treatments may improve different stages, and the funnel tells you which matters more.
- Multiple-comparisons worry is lower when you commit to a hypothesis about a specific stage up front.
- You don’t need a credentialed data scientist, but you do need data acumen and truth-seeking.
Getting started in an org stuck on sprints
Take one new capability, write out its assumptions and funnel, and propose experiments per stage. Or build the funnel for your most important existing capability. Showing leaders where it fails (e.g., “12% fail at step one”) makes the roadmap prescriptive, which is what roadmaps were always supposed to be.
→ Capability Funnel and Experiment Roadmaps · Transition Failure Matrix · Multi-Step Workflow Evaluation