After launch: confirming the win holds
When you roll out a winner, 5% of visitors keep seeing the original for 28 days (a holdout). The rest see the winner.
- If the winner stops beating the original, you get an "effect faded" alert suggesting a rollback.
- After 28 days the result is confirmed and the holdout is released, so no test code keeps running for nothing.
- Holdout visitors are re-randomised separately from the original test, so nobody's past assignment biases the check.