Obsidian Metadata
| type | source |
| url | https://hamel.dev/blog/posts/prompt/ |
| authors | Hamel Husain |
| published | 2024-02-14 |
Source: “F*** You, Show Me The Prompt.”
URL: https://hamel.dev/blog/posts/prompt/ · Hamel Husain
Many libraries rewrite or construct prompts for you (guardrails, guidance, instructor, LangChain, DSPy). The final prompt is “a natural-language description of what these tools are doing” and the fastest way to understand them. So intercept the API calls.
How: run mitmproxy (mitmweb), install its CA cert, and scope it to Python by setting HTTPS_PROXY=http://127.0.0.1:8080, REQUESTS_CA_BUNDLE, and SSL_CERT_FILE.
Findings:
- Guardrails: 2 calls with an XML-schema prompt plus a “fix this JSON” retry. “A whole lot of ceremony.”
- Guidance: 7 calls, 5 of them near-redundant “idea” generations. One-shot generation of ideas would be more diverse.
- LangChain SmartLLMChain: 4 calls (ideas → critique → resolve); the prompt has a typo, so it probably wasn’t tested.
- Instructor: function-schema extraction is a “zero-cost abstraction”, but its validation flow uses 3 calls.
- DSPy: the “minimal” example took 30+ minutes and hundreds of calls (bootstrapping few-shot demos with random search).
Questions to ask once you see the prompt: Is this framework necessary? Should I just take the final prompt and drop the framework? Could I write a better prompt? Is the number of API calls reasonable?
Why it matters for evals: never adopt off-the-shelf LLM guardrails or judges blindly. Look at the prompt (see Guardrails vs Evaluators, Prompt Optimization). Minimize accidental complexity.