Obsidian Metadata

type	template
sources	Source - Evals Skills for Coding Agents, Source - Using LLM-as-a-Judge

Synthetic Data Prompts

Method: Synthetic Data. Generate inputs only, run them through the real system, then do Error Analysis.

0. Dimensions worksheet

Dimension 1: [Name] — [What it captures / which failure hypothesis it targets]
  Values: [a, b, c]
Dimension 2: ...
Dimension 3: ...

Hand-write 20 tuples, and review them with the domain expert.

1. Tuple generation prompt

Generate 10 random combinations of ({dim1}, {dim2}, {dim3})
for a {your application description}.

The dimensions are:
{dim1}: {description}. Possible values: {values}
{dim2}: {description}. Possible values: {values}
{dim3}: {description}. Possible values: {values}

Output each tuple in the format: ({dim1}, {dim2}, {dim3})
Avoid duplicates. Vary values across dimensions.

2. Tuple → query prompt (separate call!)

We are generating synthetic user queries for a {your application}.
{Brief description of what it does.}

Given:
{dim1}: {value}
{dim2}: {value}
{dim3}: {value}

Write a realistic query that a user might enter. The query should
reflect the specified persona and scenario characteristics.

Example: "{one of your hand-written examples}"

Now generate a new query.

3. Persona / scenario style prompts (LLM-judge guide)

Feature	Scenario	Persona	Prompt	Assumption (not in prompt)
Order Tracking	Invalid data	Frustrated customer	”Generate a user input from someone who is clearly irritated and impatient, using short, terse language to demand information about their order status for order number #1234567890. Include hints of previous negative experiences.”	Order #1234567890 does not exist
Contact Search	Multiple matches	New user	”Create a user input from someone who seems unfamiliar with the system, using hesitant language and asking for help to find contact information for a person named ‘Alex’…”	Multiple contacts named Alex exist
Meeting Scheduler	Ambiguous	Busy professional	”Simulate a user input from someone who is clearly in a hurry, using abbreviated language and minimal details to request scheduling a meeting…”	–
Content Recommendation	No matches	Expert user	”Produce a user input from someone who demonstrates in-depth knowledge of their industry… Use the information in this article… {{article}}“	No such articles exist

4. Rechat-style bulk generation

Imagine if Zillow was able to parse natural language. Come up with 50 different ways users would
be able to search listings there. Use real names for cities and neighborhoods.
You can use the following parameters: <...>
Output should be a JSON code block array. Example: ["Homes under $500k in New York"]

5. RAG synthetic QA

Given a chunk of text, extract a specific, self-contained fact from it.
Then write a question that is directly and unambiguously answered by that fact alone.
Return output in JSON format: { "fact": "...", "question": "..." }
Chunk: "{text_chunk}"

6. Quality filter (optional)

Rate each generated query’s realism 1–5 with few-shot examples. Discard anything below 3 (or keep only 4–5 for RAG queries).