Obsidian Metadata
| type | concept |
| sources | Source - A Field Guide to Rapidly Improving AI Products |
Capability Funnel and Experiment-Based Roadmaps
Your AI roadmap should count experiments, not features
Traditional roadmaps (“Launch sentiment analysis by Q2”) assume we know what’s possible. With AI you’re testing the boundaries of feasibility, so teams either ship something subpar to hit a date or miss the date entirely. Either way, trust erodes.
The key metric for AI roadmaps isn’t features shipped, it’s experiments run.
Capability funnel (Bryan Bischof, Hex)
Break AI performance into progressive levels of utility. Example for a query assistant:
- Generates syntactically valid queries
- Queries execute without errors
- Queries return relevant results
- Queries match user intent
- Optimal queries that solve the user’s problem
This makes progress visible before the final goal is reached, and shows executives where things break. It maps naturally to binary checks per stage and to a Transition Failure Matrix.
Timeboxed feasibility (Eugene Yan)
“First, I take two weeks to do a data feasibility analysis (‘do I have the right data?’)… Then an additional month for technical feasibility (‘can AI solve this?’). After that, if it still works I’ll spend six weeks building a prototype we can A/B test.”
“I try to reassure leadership with timeboxes. At the end of three months, if it works out, then we move it to production. At any step of the way, if it doesn’t work out, we pivot.”
Content-moderation example: an experimentation roadmap with 2-week updates. Nothing worked for 2–3 months, then a new paradigm solved ~80% within a month. A feature roadmap would have killed the project.
Experiment recipe (Bryan Bischof)
- Frame work as a question/hypothesis (“Can we find relevant tables by row values?”), not a feature (“build a keyword index”).
- Attach a metric (e.g., table recall).
- Estimate the effect size with small pre-experiments (“73% today; expect single-digit gains”).
- Plot ROI vs effort and prioritize.
- Use the failure funnel to generate hypotheses, and commit to the stage you expect to move (which limits multiple-comparison worries). Start with one new capability, or build the funnel for your most important existing one. Showing where it fails makes the roadmap prescriptive. See Video - Stop Managing AI Projects Like Traditional Software.
Culture of sharing failures
Eugene’s “fifteen-five”: a weekly update that takes 15 minutes to write and 5 to read, documenting failures and successes, plus no-prep sharing sessions. Normalize failure as the main source of learning.
Evaluation infrastructure is the foundation
Without evals you’re guessing whether experiments work. GitHub Copilot example: heavy upfront investment in offline eval infrastructure (cloning repos at scale, running their existing unit tests to verify completions) let the team run thousands of experiments and say “this improved quality by X%“.
Related
Making the Case for Evals · GitHub Copilot · Source - A Field Guide to Rapidly Improving AI Products