Obsidian Metadata
| type | concept |
| sources | Source - Your AI Product Needs Evals |
Three Levels of Evaluation (Rechat framework)
From Your AI Product Needs Evals (2024). Cost rises from Level 1 to Level 3, which sets the cadence:
| Level | What | Cadence |
|---|---|---|
| 1. Unit tests | Fast, cheap assertions (Code-Based Evals) scoped by feature × scenario, plus generic tests | Every code change |
| 2. Human & model eval | Log Traces, review them in a custom viewer, label good/bad, align an LLM critic with a human (LLM-as-Judge); includes debugging | Set cadence |
| 3. A/B testing | Confirm the product drives the desired user behavior and outcomes | After significant product changes, for mature products only |
- Get through a good portion of Level 1 before moving to model-based tests.
- There’s no strict formula for when to introduce each level. Balance fast user feedback, user perception, and product goals.
- A/B testing LLM products isn’t very different from other products (see the Eppo blog). It’s fine to defer it until you’re ready to show real users.
The virtuous cycle
Iteration speed depends on three things together: (1) evaluating quality, (2) debugging (logging and inspecting data), (3) changing behavior (prompting, fine-tuning, code). “Many people focus exclusively on #3, which prevents them from improving their LLM products beyond a demo.”
Eval infrastructure gives you more for free
- Fine-tuning: 99% of fine-tuning work is assembling good data, and the eval system already is a data generation and curation engine (generate synthetic inputs, filter with L1/L2 tests, curate with the viewer). Fine-tuning is best for syntax, style, rules; RAG supplies context and facts.
- Debugging: a searchable trace DB, error-flagging mechanisms, log navigation, and fast re-testing of fixes. Eval and debugging infrastructure overlap heavily.
Key takeaways (from the post)
- Remove ALL friction from looking at data.
- Keep it simple. Don’t buy fancy LLM tools; use what you have first.
- You’re doing it wrong if you aren’t looking at lots of data.
- Don’t rely on generic eval frameworks. Build an eval system specific to your problem.
- Write lots of tests and update them often.
- Use LLMs to unblock building the eval system: generate test cases, write assertions, synthesize data, critique and label.
- Reuse eval infrastructure for debugging and fine-tuning.
Related
Rechat Lucy · CI Evals vs Production Monitoring · Source - Your AI Product Needs Evals