If you are a regular reader of IWCO Direct’s blog, you’ll know that we are active proponents of direct mail testing as part of a comprehensive direct marketing strategy. When we test direct mail, we learn what creative messaging and formats (and data) achieve the best results for our clients. At the same time, testing is never about resting on one’s laurels. We can always do better, and response will degrade over time without fresh creative tested and rolled out on a regular basis. One of our mottos is “always be testing.”
But direct mail testing can be time consuming, expensive and carries some risk. New creative must be developed, printed and mailed in statistically significant quantities in order to achieve valid results. This can take months or even years, but the business may not want to wait that long or make the necessary investment. There’s usually a trade-off involved—a more cautious, iterative approach to testing reduces risk of poor performance, but rarely leads to significant gains in response.
Testing new direct mail formats may carry more risk but usually leads to better results over time. Then we have the costs of creative, production, and postage, which can be substantial. As opposed to the more common A/B split testing, multivariate testing (MVT) enables us to test for the optimal combination of creative and messaging components across a package, leading to faster learning. But multivariate testing can also be costly, time consuming and require sufficient mail volume to analyze each element within the test.
A Solution to Direct Mail Testing Constraints
Is there a potential way around these testing challenges? There is—it’s online direct mail testing, also known as conjoint analysis. Conjoint analysis is a multivariate statistical technique to determine how people value various attributes, such as the benefits, features, and functions of a product or service. In direct mail, it is used to determine the optimal messaging and/or creative product attributes that appeal to the target audience.
In a conjoint analysis, we present competing creative versions online, often built by our creative team, to a carefully selected research panel audience that matches the client target universe. The audience selects the creative that is most appealing to them. A conjoint will focus either on creative components or messaging. In testing creative components, we will typically put two versions of an outer envelope, letter, and inserts on the screen to be viewed and “voted on” by the panel member. In testing messaging, we test headlines, teasers, benefits, calls to action, and addressing barriers to response. Result numbers become part of a multivariate analysis that provides a total score for each combination of test components. The top scoring combination wins.
Faster Insights, Faster Results
In direct mail online testing, a conjoint analysis addresses cost and timing challenges. We can test many more ideas much faster, at a lower cost than a live multivariate analysis. We greatly reduce our clients’ risks and costs as there are no production and postage costs. A statistically significant audience is used, but requires a much smaller audience than a live MVT, as every person is reviewing two concepts/offers per screen and completing roughly eight screens, the equivalent of mailing a person 16 live mailings. We gain faster results and learning, a timeframe of approximately eight weeks. The best part of conjoint analysis? It works, and is typically then validated through live testing.
For more on using online direct mail testing to optimize direct mail performance, reach me at
Subscribe via email to our Stevie® and Feedspot award-winning blog and get a fresh post delivered weekly to your inbox. We promise to keep it interesting, but you can easily unsubscribe if we don’t.