Honeycomb Query Assistant

Obsidian Metadata

type	case-study
sources	Source - Using LLM-as-a-Judge, Source - Your AI Product Needs Evals

Case Study: Honeycomb Query Assistant

Product: a natural-language → Honeycomb query generator. Domain expert: Phillip Carter.

Process

• Hamel sent Phillip a spreadsheet every few days with the NLQ, the generated query, the LLM critique, and the LLM outcome (good/bad). Phillip filled in his own critique, outcome, and desired response for 25–50 examples at a time.
• Hamel iterated on the critique model’s prompt until it matched Phillip. The judge prompt included the query-language docs, guidelines, and few-shot examples of Phillip’s critiques (see the full prompt in LLM-as-Judge).
• > 90% agreement in three iterations. Raw agreement was acceptable here only because the data was ~50% failures. With imbalance, use TPR/TNR (Judge Validation).
• Tools were low-tech: Excel/Sheets.

Human side

“Seeing how the LLM breaks down its reasoning made me realize I wasn’t being consistent about how I judged certain edge cases.” — Phillip Carter

The process standardized the expert’s own criteria (Criteria Drift) and surfaced insights about the product, AI capabilities, and user needs that were “often more valuable than creating a LLM judge”.

Example judge critiques

• Good: correct IP filter, COUNT grouped by trace.trace_id.
• Bad: “slowest trace” query that doesn’t group by trace.trace_id, so it shows MAX(duration) over time rather than the slowest trace.
• Bad: a “per hour” analysis with time_range 3600s is insufficient; it needs ≥ 36000s for meaningful hourly patterns.

Related

Critique Shadowing · LLM-as-Judge