Obsidian Metadata

type	concept
sources	Source - AI Evals FAQ, Source - A Field Guide to Rapidly Improving AI Products

Prompt Management

Versioning: store prompts in Git (Hamel’s preference)

There’s an unavoidable tension between keeping prompts close to the code and keeping them in a place non-technical stakeholders can reach.

Git treats prompts as software artifacts: versioned, reviewed, and deployed atomically with application code.
Non-technical folks can use the GitHub web UI or GitHub Desktop. At GitHub, Hamel saw lawyers and accountants do this effectively. (Ben Balter’s GitHub for non-technical roles is a primer.)
Vendor prompt-management tools (Arize, Braintrust, LangSmith) allow quick iteration but add indirection. They can’t easily execute your application’s code (tools, RAG, agents), so it’s hard to test prompts with your real system.

Jupyter notebooks (when your codebase has Python entry points) let you iterate on the actual agent with full tool and RAG capabilities, and widgets give you mini UIs. (Teresa Torres, as a PM, used notebooks for the whole eval and experimentation lifecycle.)
If notebooks don’t fit, build an integrated prompt environment: an admin mode of your real UI that exposes prompt editing and debugging. It goes beyond standalone prompt playgrounds (Phoenix, LangSmith, Braintrust), which don’t include your RAG, orchestration, and business logic.
Either way, version in Git.

Nothing beats experimentation. Models and providers treat them differently, so move instructions between them and measure with evals.

General guidelines:

System prompt = the model’s constitution: static rules for all requests. Identity, behavioral constraints, output format, standing instructions. “You are a medical assistant. Never provide diagnoses. Always recommend consulting a healthcare provider.”
User prompt: the actual task, relevant context, few-shot examples, data to process (documents, query-specific variations).
When unclear, prefer the user prompt. It’s more portable across models and easier to debug.