Test, measure, implement, repeat. All to answer the most important question: Did our campaign result in conversions? You may track and know your catalog response rate, but do you know exactly how much more effective it is than your other marketing efforts? This nugget of important information is called incremental lift. Your catalog’s performance compared to other media campaigns, be it social, email or website ads, is the lift. So, do you know how much heavy lifting your catalog campaigns are doing? There are a lot of ways to measure the success of a marketing endeavor, but holdout testing is one method that holds true and comes directly from the scientist’s mouth. 

If you’ve ever participated in a research study, you’ll recognize holdout testing as a randomized control trial. When you have two groups of people with similar behavior and demographics, and you give one group a pill and the other a placebo, you can measure the pill’s effectiveness. Holdout testing in marketing is conducted in a similar manner.

First, you identify a population of your customers with similar behavior and demographics, and randomly assign them to two equally sized groups: One that receives your catalog and one that doesn’t receive it. The group that will not receive the catalog is the control group or the holdout group. After the campaign is complete, you measure the response rate and the sales of each group. For the holdout group, you’ll likely measure the sales and response rates of other your other direct advertising strategies such as email. The holdout group’s results act as the baseline against which you can measure your catalog campaign’s incremental lift.

You can keep this test simple, measuring only the incremental lift, or you could set parameters based on catalog design or feature product and use those insights to optimize for future direct mail campaigns. For example, if you conduct a holdout test of a catalog with a featured product in a particular region and it doesn’t produce the lift you need, you know that you may need to swap it out with another. Or perhaps, you see significant lift in a certain region, so you could make a greater investment on it in future mailings. Who knew that the customers who receive nothing could tell you a lot about your strategies.