Obsidian Metadata
| type | source |
| url | https://ai-execs.com/2_intro.html |
| authors | Hamel Husain |
Source: 99% of Executives Are Misled by AI Advice
URL: https://ai-execs.com/2_intro.html#a-case-study-in-misleading-ai-advice
Case: Casetext’s CEO claimed that after passing 100 tests, the odds of 100% accuracy on 100k real inputs are “very high”. YC partners amplified it, and founders started chasing 100% pass rates.
Why it’s wrong: in AI, a perfect score is a red flag. It suggests leakage (tests too similar to prompts or training data) or tests too weak to measure what matters. Perfect scores also stop you from differentiating models and finding improvements. “The goal of evaluations isn’t to pat yourself on the back for a perfect score.”