Thought UmweltThought Umwelt

thought umwelt

From Noob to Automated Evals In A Week (as a PM) w/Teresa Torres

4 min read

Obsidian Metadata

channelHamel Husain
urlhttps://www.youtube.com/watch?v=N-qAOv_PNPc
published2025-08-15

Summary

Product management expert Teresa Torres recounts her journey of building and evaluating an \“AI Interview Coach\” from the ground up. The video details her transition from subjective \“vibe checks\” to establishing a robust, automated evaluation framework. She shares how she leveraged various tools, including Airtable for traces, Jupyter Notebooks for eval design, Python with ChatGPT for analysis, VS Code for investigations, and Claude for custom annotation tools. Torres also discusses the collaboration dynamics between PMs and engineers on AI products and answers questions regarding user feedback, the necessity of a technical background for AI development, and the intricate micro-decisions involved in building an AI application.

Key Takeaways

  • Iterative AI Evaluation: The critical importance of moving beyond subjective assessments to build automated, systematic evaluation frameworks for AI products.
  • Hands-on PM Engagement: Product Managers can, and benefit greatly from, getting hands-on with AI development and evaluation, even starting with limited technical knowledge.
  • Leveraging Modern Tools: Practical application of tools like Airtable for data tracing and annotation, Jupyter Notebooks for eval design, and LLMs (ChatGPT, Claude) for learning Python, analyzing results, and building custom tools.
  • Diverse Evaluation Methodologies: The value of combining different evaluation approaches, such as LLM-as-Judge models and traditional code-based assertions, for comprehensive AI quality assessment.
  • Collaborative AI Development: Strategies for effective collaboration between Product Managers and engineers in the context of AI product building, emphasizing shared understanding and tools.
  • Continuous Improvement through Feedback: The necessity of capturing and annotating end-user feedback to continuously refine and improve AI product performance.
  • Empowerment for Non-Technical Roles: An encouraging perspective that a deep technical background is not a prerequisite to start building and evaluating AI solutions.

Mindmap

graph TD
    A[From Noob to Automated Evals: Teresa Torres' Journey] --> B(The Product: AI Interview Coach)
    A --> C(The Problem: Evaluating AI Quality)
    C --> C1(Moving Beyond Vibe Checks)
    A --> D(Evaluation Journey & Tools)
    D --> D1(Airtable: Traces & Annotation)
    D --> D2(Jupyter Notebooks: First Evals Design)
    D --> D3(Eval Examples: LLM-as-Judge vs. Code-Based)
    D --> D4(Learning Python w/ChatGPT for Analysis)
    D --> D5(VS Code & Custom Tools for Investigation)
    D --> D6(Claude: Building Custom Annotation Tool)
    A --> E(From Personal Project to Production App)
    A --> F(PM-Engineer Collaboration on AI)
    A --> G(Q&A)
    G --> G1(Capturing End-User Feedback)
    G --> G2(Technical Background for AI?)
    G --> G3(What's Next for Teresa?)
    G --> G4(Micro-Decisions of Building an AI App)

Notable Quotes

Note: The actual transcript content was not provided, so these are potential moments for insightful quotes based on the video’s chapter markers.

Graph View