NurtureBoss is the canonical short case study for what a working error-analysis loop looks like in production. The team built an AI assistant for the apartment industry, read their own conversations, and one failure category dropped from 66 percent to roughly 5 percent. The vignette is from Hamel's field guide and is summarized here as the worked example the rest of the error analysis section refers back to [1].
The product and the problem
NurtureBoss's assistant handles apartment-leasing conversations, including tour scheduling and rescheduling, across voice, text, and chat. When the founder, Jacob, needed to improve it, his team built a simple viewer for reading conversations, with a space beside each one for open-ended notes about failure modes. They started from the conversations, not from a list of metrics [1].
The cycle
The team annotated conversations in a spreadsheet, one row per conversation, writing open-ended notes on any undesired behavior. An LLM then helped build a taxonomy of common failure modes, each row was mapped to failure-mode labels, and the labels were counted [1]. A pivot table was enough for the counting.
Three categories accounted for over 60 percent of all problems: conversation flow issues (missing context, awkward responses), handoff failures (not recognizing when to transfer to a human), and rescheduling problems (struggling with date handling). The date category was already visible after annotating dozens of conversations: the assistant failed 66 percent of the time when users asked for a tour a relative amount of time out, such as two weeks from now [1].
The fix and the result
Instead of reaching for new tools, the team looked at actual conversation logs, categorized the types of date-handling failures, built specific tests to catch them, and measured improvement against those tests [1]. The date-handling success rate went from 33 percent to 95 percent, so the failure rate fell from roughly 66 percent to roughly 5 percent. The write-up does not describe the change to the system itself, and this page does not guess at one. The loop is the point: read, name the failure mode specifically, test for it, re-measure.
Figure: Error analysis result at NurtureBoss: the date-handling failure rate fell from 66% to about 5% (success rate 33% to 95%). The team annotated dozens of conversations, categorized the date failures, built specific tests, and re-measured, with no new tools.
Why the cycle worked
Three properties of the loop generalize.
Specificity. The label was not "hallucination" or "bad scheduling". It was date handling, with the failure types inside it categorized separately. A test is easy to write against a label that precise.
People close to the product did the reading. The team built a viewer fitted to their own conversations and wrote the notes next to each one, which made reading cheap enough to actually happen [1].
The labeled conversations became a substrate. The notes and counts seeded the tests for the date failures, and the same move on the other two categories turns a one-time analysis into regression coverage. Future regressions get caught at offline eval before merge, not in a user complaint three weeks later [2].
What the vignette does not say
The case is a clean win because the failure had a concrete, testable shape. Not every cycle produces an order-of-magnitude drop. The write-up also says the analysis found more than the team could fix at once: they needed several weeks just to implement fixes for the problems already found [1]. What it shows is the lower bound of effort that can produce a real result: dozens of annotated conversations, three counted categories, tests for the worst one. Teams without an eval program are not failing because the work is hard; they are failing because it has not been put on a calendar.
One cycle is not the end. The top failure mode shifts; cycle one's second-largest becomes cycle two's largest. The discipline is the cadence, not the heroics.
How to run your version
Pull one hundred diverse traces, weighted toward the segment where complaints are loudest, and keep a few random ones in the batch so failure modes your current signals miss still show up [3]. Annotate at least the first thirty yourself before letting an LLM or agent suggest anything, because early suggestions bias what you notice [3]. Have a domain expert write one-line notes. Cluster into four to seven candidate failure modes. Pick the top three by frequency times severity. Ship one targeted fix per mode. Re-pull, re-count. Promote the labeled set into a stage-2 offline eval so the next regression blocks at CI rather than reaching a user [4]. Start with code-based checks where a rule can catch the failure (a relative date resolved to the wrong calendar day is one) and keep an LLM judge for failures that need judgment [3]. First cycle: one person-week. After that: ten to twenty traces a week, plus a fresh hundred-trace pass every two to four weeks while the product is changing, and monthly once it is stable [3].
Where a failure needs judgment, the labeled traces start paying off at scale in a stage-2 judge, and building one that agrees with your domain expert is the subject of LLM-as-Judge.