Direct Marketing Testing

Testing direct mail campaigns or any marketing campaign is critical to understanding the campaign success or need for adjustments.

Why should you test your DM (direct marketing?)  First of all, because it pays!  It also limits your losses, and maximizes profitability.  In today’s competitive markets and with an increased demand on marketing professionals to be accountable, testing is more important than ever. Although it costs money and time, effective testing will provide information that you can use in future campaigns or business strategies.

Possible Testing Objectives – Improving the Bottom Line

First determine your testing objective. As with other marketing concepts, it should be simple, clear, easy to follow and implement. Most importantly, it has to allow for data gathering and analysis.

Determining the viability of a product or service is a fundamental reason to test. Many companies take a sequence of steps before their products are ready for market.  The figure below shows the seven stages of the new product process with market testing being stage six.

Strategy Development

Idea Generation

Screening & Evaluation

Business Analysis

Development

Market Testing

Commercialization

 

Other testing objectives might be to increase sales, test response rate, increase average order volume, or the frequency of purchases. Various test models determine how you can change your business practices, principles, or marketing channels to increase the bottom line.

Other testing objectives might be to decrease promotion costs, credit loss and collection costs, or cost of goods sold.

Let’s review the specific objective of increasing the average order value.  What might you test? First of all, try featuring a proven higher priced product over products that are less proven to sell and are lower priced. Try offering volume discounts or a “bundled” set of products.

Many companies test using lower priced add-ons, trying to move higher priced products. For example, computer retailers are notorious for moving high priced laptops by offering a free printer, software packages or other peripheral devices.

Try testing the effectiveness of group related products that work as cross sell. For example, buy a cake and get a cake cover for half off, buy a car and get free oil changes for as long as you own the car. Grocery chains have test coupons at checkout for future dollars off on items similar to that day’s shopping experience.  If you buy Pampers^® baby wipes, they print out a coupon for Huggies^®.

If your company sells products in a catalog or over the Internet, test offering free shipping. Amazon.com offers free shipping on certain products that total more than $25. They do the math for you as you add items to your cart. This incentive keeps consumers shopping to avoid shipping charges. They test different products over time to see which outperform others and increase the overall sales volume or value of the order at that time.

Retailers use “Free-Gift-with-Purchase” test. Sometimes they do this to move products that are slow, but they also do it to test increasing the order value. Finally, test discounting future purchase based on current purchases over a certain dollar amount. This forces the consumer to buy…buy…buy up to a certain predetermined point. This is called the “spending over” amount strategy. For example, try “spend $50 today, get $5, $10, $25 off next time” or “spend $100 today, get $15 off your next purchase.”  The key is to test different amounts of money.

Test important issues

In DM testing there are five major elements or issues that can be tested

• List
• Offer
• Package
• Copy
• Timing

Each one of these issues has the potential for unlimited types of testing. And compounded with each other –you get a lot of testing potential.

List – Rather than mail to everyone on a single list, rent the minimum number of names on a few lists from different sources, and test an identical package, offer and copy with each one. Once you've selected the winning list source, you can roll out in force. Other ways to test lists are by doing an Nth name selection or random selection. N is the number divided by the interval of random selection. For example, N would equal all the names on the list divided by size of the desired test sample. Or, it could equal a specific number within the total count – such as, 1 of every 100^th name will be selected or 1 of every 150^th name.

Offer – Effective offers are important to the bottom line. How will you know which offer to make or which one will work?  You don’t! The consumer will tell you through purchases and feedback analyzed through testing. The decision for a consumer to take one offer over another is steeped in emotion, as is the decision-making process to buy. Testing within offers must have a control within the copy, the packaging, the list and the timing. Only change the offer itself.

Try offers such as 90-Day Free Trial, Money Back Guarantee, or an Extended Guarantee/Warranty. Other offers could revolve around how they pay you, such as, bill me, installment terms, revolving credit, or X number of days same as cash. You can even test the X number of days by changing the number itself.

Try using the word “Free.” It has tremendous power in today’s maze of offers. Which of these do you think out-pulled increasing lift response?  Note: they all mean exactly the same thing.

• ½ Off
• 50% Off
• 2 For the Price of 1
• Buy 1 & Get 1 Free

According to a study done by AT&T, “Buy 1 and Get 1 Free” had the highest response.  Fifty percent off came in a close second.  The word “free” has power.

Package Formats – Try testing envelopes over self-mailers, flat sizes against letter size, tubes and boxes. You can test oversized letter/card mailers against smaller size postcard mailers and even the weight of paper. When testing package formats, always test at least two different types or even three. As with all testing, keep your copy, list, offer and timing constant within the package test. This ensures a true testing environment without other variables tainting your test.

Timing of Drops – Testing the timing of mail drops is easy and probably the least expensive of all testing techniques. Time your marketing material arrival for a Monday versus a Friday or even a Saturday. Try timing around the beginning, middle or end of the month. Different times of the year around holidays or other significant dates can be tested.

Basic Testing Terminology

• Control – is the most successful previously used effort, which also applies to creative. If you aren’t tracking your marketing results, you won’t know which is your control.
• Roll out – continuation mailing of a list or creative effort
• Test Cell or Panel – A segment receiving a different creative approach
• Statistically Valid Sample – Adequate quantity to ensure predictability of a test

Basic Testing Rules

Testing costs should be in perspective to the expected results. Don’t spend $10,000 on testing and only $1,000 on the whole campaign. Interestingly enough, companies that don't want to test something important, like price, often test low-impact variables such as a live stamp vs. metered or indicia postage. They may even test one color over another. Test the major elements first such as copy, graphics, packaging, list and timing.

Test consumer options to an audience. For example, test the medium of response such as phone, fax, Internet, e-mail, snail mail, or in-store capture all at once.  These are options that you wouldn’t want to test on an individual basis. Why, because it costs too much to test these minor impact elements over major impact elements. Also, the very nature of the response medium yields its own raw data of results. You will be able to count how many responses came in via e-mail, phone and/or fax.

Test a single element to isolate the effect of that element, or test an entirely new concept or package, but do not test multiple major elements at the same time. Yankee Magazine did a package test where the test yielded a 31 percent lift in response over the control. The problem was that more than one element was changed, which made it impossible to tell what made the difference in lift (response over the last piece). The chart below outlines the control elements and the test elements. Notice the letter went from four pages to two pages and they eliminated the four-color brochure in the test. It is impossible to tell which element change effected the response lift.

Later they tested using a four-color brochure and a two-page letter resulting in a 2 percent lift. Then they tested a four-page letter with no brochure resulting in a 2 percent lift. Now, after these two other tests were completed, they know that the two elements changed in the original test, together yielded a higher 31% lift overall.

31% Lift

Finally, don’t test individual component elements until that effort is a proven winner. See the figure below. Plan how you will track the results before you test. If you have no tracking plan in place, you are wasting your time and money in testing.

How big should your test be?

The larger the test sample, the more you will regress to the mean. The larger the sample, the better the statistic sample. If you mail to the entire universe (your entire list), your results will equal the true responsiveness of the universe. If you mail an Nth name selection, your results will equal the performance of the test cell or panel, not the true performance of the universe. You must consider how closely the test panel represents the universe, at what confidence level and within what range of response. That is, if your test cell/unit is only 100 records out of a million in your universe (entire file), your test panel does not adequately represent the total universe.

How then do you get a statistically valid sample? There are three factors that determine test sample quantity.

First, consider the anticipated response rate. If it’s 2 percent, then sample 2 percent of the file. If your response rate is expected to be 15 percent, then sample 15 percent of the test file. This is based on historic breakeven results from the Direct Marketing Association library.

Secondly, review the confidence level or interval you need to roll out. Typically, this is 90-95 percent for direct marketers. For example, 90 percent of the time, the true response results, from rollout to the entire universe, will be between the test sample estimated response ± the sampling variance.

Lastly, calculate sample variance and range of response. Your response range will equal the estimated response rate ± the sampling variance / error. You can control the sample variance / error. The chart below outlines this example.

The use of statistic probability tables will help to better and more easily determine sample sizes for specific desired response range with error factors built in.

Testing through price barriers

Test significant price differences, not pennies and nickels. These are called major barriers such as $10, $20, $50, $100, $500. You will more than likely see a 20-30 percent drop in response if you go through an acceptable major price barrier. You can back off a nickel or penny on these major barriers such as $9.95, $19.95…$49.99, etc, but once you pass through $300-400 it’s usually OK to round off, but you should test it.

Testing minor barriers is effective in reducing your overall drop in response rates. Try testing minor barriers of $5 increments such as $15, $25, $45, and so on. More than likely, you will see a 5-15 percent drop in response versus the higher 20-30 percent in the major price barriers. These minor price barriers typically affect impulse and lower priced items, such as magazines. You can repackage these lower priced items to add perceived value.

Additionally, you can test using the concepts of under-the-full-dollar-amount, which conveys a bargain. Use variety when you have multiple items on a page, such as a catalog or web page. This concept works at all price levels, even expensive items when positioned as a bargain. Avoid doing this on luxury items that are one-of-a-kind and those that use exclusivity in luxury positioning.

Do the math for them

Below is an example of a mailer sent out from an American Airlines test where they did the math to show the value in the mileage plan they were offering.

Avoiding disasters in testing

Using the 10 times rule pyramid testing system will help to keep the testing manageable and reduce testing waste. Rule of 2 x 10 is two mailings of 10 times the first mailing. Mail to your hotline buyers first. These are the 20 percent of your customers/clients who do 80 percent of your business or buyers who just bought from you. They have a higher probability of buying from you again. Secondly, mail to your test sample, then the second test mailing should be 10 times the amount of your first test mailing. After these test results are analyzed, then mail to the entire universe.

Testing isn’t always easy or inexpensive. Executed with some thought and planning, it will yield a wealth of information about your direct marketing. It will also prepare you for future mailings. Ultimately, testing will provide information for you to be accountable to your boss and show what works and what doesn’t. It prevents those around you from using their own bias and opinions to detract from what works for your product or service and what sells to your customers. People do things and buy products for their reasons, not yours or your boss’s.

Although some of the principles outlined in this article are complex in nature, I have provided you with immediate and easy-to-apply concepts, such as testing of the list, offer, packaging and timing. These major concept elements are fairly inexpensive to test on your next campaign.