# How do you know personalization caused the lift? | PersonalizeIQ

Source: https://personalizeiq.com/blog/measure-personalization-incrementality/

PersonalizeIQ answers


Run a holdout: a comparable group that never sees personalization. If the personalized group does not beat it on profit or conversion, the lift is not proven.

## Why before-and-after lies

Conversion went up after you launched personalization. It also went up after the sale started, the email went out, and the season changed. Before-and-after comparisons credit personalization with every improvement that happened at the same time, which is why teams that measure this way always see lift and never learn anything.

The problem is selection as much as timing. Personalized experiences go to the visitors most likely to engage, because the targeting is working. Comparing engaged visitors to everyone else measures the targeting, not the treatment.

## The holdout design

Hold back a random, comparable slice of traffic from all personalization and serve them the default experience. Randomize at the visitor level, not the session level, so the same person does not flip between groups. Keep the holdout stable across the test window; a holdout that leaks treated visitors is worse than no holdout.

Size the holdout for the decision, not for comfort. Ten percent is the common default, and it is usually enough to detect meaningful lift within a few weeks on a healthy store. Smaller holdouts take longer to read; that is the tradeoff, not a reason to skip the control.

## Reading the result honestly

Compare the groups on profit or conversion, not clicks. Personalization that increases engagement while decreasing margin is a failure wearing success metrics. Include the cost of the tool and the team in the math; incrementality has to clear the full cost, not just beat zero.

When the holdout wins, believe it. The most valuable outcome of incrementality testing is killing personalization that does not earn its keep, because that budget and attention go to the variants that do.

## How long should a holdout run?

Until the result is stable, usually two to four full business cycles. A week is not enough; seasonality inside the week will fool you.

## Should the holdout see a generic experience or the old personalized one?

The generic default. Comparing new personalization against old personalization measures the upgrade, which is a different and also useful question, but it is not incrementality.

## What if leadership will not allow a holdout?

Run it on a small slice and frame it as insurance: a five percent holdout costs little and protects every lift claim the team makes.

## Reviewed

Published Oct 1, 2026.

## Reviewed

Published Oct 1, 2026.