Thought UmweltThought Umwelt

thought umwelt

Rechat Lucy

3 min read

Obsidian Metadata

typecase-study
sourcesSource - Your AI Product Needs Evals, Source - A Field Guide to Rapidly Improving AI Products

Case Study: Rechat’s Lucy (Real-Estate AI Assistant)

Product: Rechat is a SaaS app for real-estate agents (contracts, listings, creative assets, appointments). Lucy is its conversational assistant.

The plateau

Prompt engineering got fast early progress. As Lucy’s surface area grew, performance plateaued:

  1. Fixing one failure mode created others (whack-a-mole).
  2. No visibility into effectiveness beyond vibe checks.
  3. Prompts grew long and unwieldy trying to cover edge cases.

What they built

  • Level 1: hundreds of assertions, scoped by feature × scenario (e.g., Listing Finder: one/multiple/no matches → len(listing_array) checks) plus generic tests (the no-exposed-UUID regex). Run in CI. Results tracked over time in Metabase, which showed error prevalence dropping after fixes.
  • Synthetic test inputs from an LLM, e.g., 50 “create a contact” instructions, each paired with a lookup query; the test asserts the CRM returns exactly one match. 
    Write 50 different instructions that a real estate agent can give to his assistant to create
contacts on his CRM. The contact details can include name, phone, email, partner name,
birthday, tags, company, address and job.
For each of the instructions, you need to generate a second instruction which can be used
to look up the created contact.
. The results should be a JSON code block with only one string as the instruction like the following:
[["Create a contact for John (johndoe@apple.com)", "What's the email address of John Smith?"]]
  • Level 2: traces logged in LangSmith. A custom Shiny-for-Python viewer (built in under a day) showed the tool and scenario, synthetic vs real input, filters, and links to the CRM and trace system. Outputs were editable to curate fine-tuning data. Labels were good/bad.
  • Test DB of listings designed to trigger edge cases. Synthetic queries grounded in real IDs, schedules, and business rules (see Synthetic Data).
  • Fine-tuning fixed failure modes prompting couldn’t, using data from the eval pipeline (generate → filter with L1/L2 → curate).

Lessons

  • Evaluation sits at the center of the improvement loop (see Three Levels of Evaluation).
  • Eval infrastructure doubles as debugging and fine-tuning data infrastructure.
  • Failure examples from Rechat work in the FAQ: suggesting unavailable showings (assertion), confusing client personas / wrong tone for luxury vs investor clients (judge), omitting budget constraints or pet policies.

Three Levels of Evaluation · Code-Based Evals · Synthetic Data

More from the podcast

See Video - Data and DevOps Tools for LLMs (Rechat, Vanishing Gradients): a QA team using debug mode as the first data source, LangChain self-healing for malformed function calls, assertions as integration tests on every AI-related commit, Airtable behind the Shiny review app, CI assertions auto-filtering synthetic data before human review, Metabase comparisons of base vs fine-tuned models, and versioning prompts and fine-tuning data together. The advice: start with a “wooden chain” and upgrade links over time.

Graph View