Step 7 of 7: From Idea to Impact
Offline eval quality feeds in-product AI quality (accept/edit/retry rates), which feeds user behavior (adoption, retention), which feeds business impact (revenue, cost saved). Most teams only report the business number to leadership and only watch offline eval day to day, and never prove the two are connected. That gap is why AI investment gets cut.
Layer 1 to Layer 2: eval pass rate on adversarial cases should predict edit rate in production. If eval says 91% pass but production edit rate is 60%, your eval set doesn't match production distribution. Fix the eval, don't just tune the model.
Layer 2 to Layer 3: edit rate under 30% should correlate with agents actually adopting the tool rather than routing around it. Track "drafts requested" as a percentage of eligible cases. A low request rate despite good edit rates means a UX or trust problem, not a model problem.
Layer 3 to Layer 4: adoption plus low edit rate should translate to time-per-resolution dropping. This is the number finance and leadership actually care about, but it only holds up if Layers 1-3 are all healthy.
2. PostHog Event Structure for AI Features: the exact event schema built around AI-specific decision points, not just clicks.
3. The Metrics Tree: full dashboard spec across all 4 layers, ready to build from.
4. Worked Example: the week the metrics chain nearly broke on the dispute tool, and the Design-layer fix that closed the gap.
5. Common Pitfalls: why reporting the business-impact number with no supporting story from the other 3 layers is the fastest way to lose funding when someone asks a hard question.
6. Nuance Most Guides Skip: novelty effect vs habit formation, and why a metrics stack that can't tell them apart will lie to you about a failing feature.
You've completed the free journey: Research, Think, Define, Design, Build, Review, Measure. The paid playbook gives you the full depth at every step, plus one running example product taken through all 7 stages end to end.