Step 2 of 7: From Idea to Impact
Every AI feature idea rests on a stack of assumptions: about the user, the data, the model, and the business. Most PMs only test the top one ("will users like this?") and skip the ones underneath that actually kill projects: does the data exist, is the model good enough at this specific task, will this be cheap enough at scale.
Climb down the ladder before you climb up the roadmap.
Desirability: do users actually want it? Test with a fake-door prompt in the UI ("resolve this dispute with AI assistance") and measure click-through before building anything. If under 15% of eligible users click, the desirability assumption fails regardless of what you build.
Data: does the ground truth exist? For the dispute tool: does a labeled set of "correctly resolved disputes with the reasoning that resolved them" exist anywhere, case notes, past agent decisions, regulatory rulings? If not, you need a data-collection phase before a build phase, not instead of one.
Capability: can current models actually do this task at acceptable quality? Don't ask "is GPT-4 good at writing." Run 20 real historical disputes through the model cold and have a senior support agent grade the outputs blind against the human-written originals. This is a half-day test that kills more bad ideas than any amount of planning.
Viability: does the unit economics work at scale? At dispute volume (a few hundred/week) with frontier model pricing, is the cost per resolution meaningfully below the cost of a human agent's time on the same ticket, after you add in review overhead?
2. Claude Prompt Chain for Assumption Challenge: the exact prompt that gets Claude to argue against your own idea instead of validating it.
3. Assumption-to-Experiment Map: full template mapping each assumption to its cheapest test, time-to-answer, and kill threshold.
4. Worked Example: how a Data-layer test on the dispute feature nearly killed the project, and the 2-day pivot that saved it.
5. Common Pitfalls: why testing Desirability first (because it's easiest) is the mistake that costs teams a full sprint.
6. Nuance Most Guides Skip: why AI features fail unevenly, and what that means for how you sample your assumption tests.