Obsidian Metadata
| type | reading-path |
| source | https://hamel.dev/notes/llm/evals/start/reduce-eval-cost/index.html |
Reading Path: Evals Take Too Much Time or Money
“We’re spending too much effort reviewing outputs, maintaining tests, or running evaluators.”
- Source - Its Hard to Eval Is a Product Smell: product changes can cut review effort
- Source - Selecting the Right AI Evals Tool · Error Discovery Skill Workflow
- Source - Do Automated Evals Work: what automated error analysis catches and misses
- Videos: Shreya’s assisted error analysis; Isaac’s Building Custom Eval Tools With Coding Agents
Review interfaces and tools: Internal Eval Platforms · Annotation Tools and Review Interfaces · Eval Tooling and Vendors
Maintenance and workload: When to Build an Automated Evaluator · Eval Maintenance and Cadence · Trace Sampling Strategies · How Many Examples Do You Need · Benevolent Dictator
Automation, outsourcing, models: LLM-as-Judge · Judge Validation (cheap classifiers like Jev) · What LLMs Can and Can't Automate in Evals · Annotation Outsourcing · Prompt Optimization · Model Selection