Obsidian Metadata

type	concept
sources	Source - AI Evals FAQ, Source - Demystifying Evals for AI Agents

Model Selection

Many developers treat switching models as the main lever for improvement. Start with Error Analysis first.

“I suggest not thinking of switching model as the main axes of how to improve your system off the bat without evidence. Does error analysis suggest that your model is the problem?” — Hamel (office hours)

• Public model benchmarks help pick a starting model. Product evals tell you whether it works for you.
• Once you have evals, upgrading models is fast. Run the suite, see strengths, tune prompts, and upgrade in days, while teams without evals spend weeks testing (Anthropic).
• Judge models: start with the most capable, then optimize cost after alignment (see LLM-as-Judge).
• Re-run error analysis after any model switch (see Eval Maintenance and Cadence).