Q21. How do you collaborate with a data science team when scoping an AI feature?
Type: Behavioral | Difficulty: Medium
What they're testing: Working with DS beyond "give me a model." Bringing problem + constraint, not solution.
Framework hint:
Bring problem + business metric
Define acceptance criteria jointly
Agree eval set upfront
Iterate weekly on outputs
Clear PM vs DS responsibility split
Q22. Walk me through how you'd handle model drift in production — you shipped 3 months ago and users are complaining more now.
Type: Case | Difficulty: Hard
What they're testing: Post-launch AI ops. Most PMs think shipping is done; drift is where real AI PM work lives.
Framework hint:
Detect: monitor + complaint tracking
Diagnose: input shift vs output degradation
Root cause: new segments, new uses, vendor updates
Fix: retrain, adjust prompts, update index
Prevent: continuous eval + alerting
Q28. What's the difference between training data, fine-tuning data, and eval data — and how would you source each for a resume-scoring product?
Type: Case | Difficulty: Medium
What they're testing: Basic ML fluency. Can you talk to engineers without cringing.
Framework hint:
Definitions of each
Resume-scoring: labeled pairs for fine-tune, diverse held-out for eval
Sourcing with user consent + synthetic edge cases
Bias audit
Labeling cost and inter-rater agreement
Q30. Your model is 92% accurate in eval but users hate it. What went wrong?
Type: Case | Difficulty: Medium
What they're testing: Accuracy vs user value. Distribution mismatch, cost of errors, latency, presentation.
Framework hint:
Eval vs production distribution gap
Error asymmetry — which 8% and cost
Non-accuracy factors: latency, tone, presentation
UI overselling mismatch
Rebuild eval + cost-weighted metrics
🔓 16 more questions in this segment alone — including grounding architecture, cold-start data strategy, and third-party data evaluation. Get the full 100 on Gumroad →