Actionable Insights From APT's Retail Practice

8 Tips for Improving Your Testing Process

February 13th, 2015 | Posted by Haley Jackson in Retail

Most retailers use in-market tests to help inform decisions. But just running tests isn’t enough. Based on our work with dozens of retail brands, here are eight tips for improving your testing process.

  1. Grow your innovation funnel. You should never be left with just one idea to try – if it doesn’t work, what’s left? Ideas for testing should span marketing, merchandising, operations, pricing, and capital investments. Many retailers generate these ideas from a broad range of sources, including talking to customers and managers in the field, competitor actions, and importantly, analyzing their own transaction and customer data more deeply. Often ideas are filtered through taste panels, focus groups, surveys, and the like, which can help weed out a fraction of the ideas that may not work. Ultimately, however, all of the resulting ideas are unproven until they are tested in the real world. For example, new promotions could give away money to customers who would have paid full price for the promoted item, targeting Millennials could alienate other customer segments, etc. It’s easy to make any idea sound good and support it with customer feedback, management anecdotes, executive intuition, or survey data, but the reality is that many innovative ideas don’t work. Organizations need to have enough ideas in the funnel so that they can throw away the ones that don’t work and refine and target the ones that do.

  1. Test in the right number of locations. Testing in the right number of stores is a critical component of a useful and efficient Test & Learn process. Tests run in too few stores are wasted efforts. Executives then cannot be confident in the results and often complain that “testing isn’t predictive.” Testing is predictive, if done right. Make sure you run tests in enough stores to measure a breakeven lift with statistical confidence. On the other hand, testing in too many stores is unnecessarily costly and risky.
  1. Choose the right stores. Testing in just the stores in the market surrounding your headquarters is probably not the best way to go (sound familiar?) – the results simply won’t be applicable to the rest of your system. For example, increasing price only in your highest-volume locations will not help you understand the effectiveness of a price increase in a lower-volume location. The selected test group should be similar to the rest of your network based on such factors as local market demographics, the competitive environment, store format, and volume. While there are often real world constraints, leading Test & Learn retailers have all overcome these challenges.
  1. Have a well-matched control group. It isn’t a test if there isn’t a control group, as without a control you have no way of knowing what would have happened otherwise. Further, the selected group of control stores must be as similar to the test stores as possible, besides the action taken. Having the right control group ensures that you can isolate the true cause-and-effect relationship between the tested action and key performance measures.
  1. Don’t stop at “did this work?” Many programs work well in some situations but not others. You should move beyond just measuring overall program effectiveness to identifying the types of stores and customers that respond best to a program. With this information in hand, you can target your programs to only the customers and store locations that will generate a positive response.
  1. Prioritize high-value, risky tests. It’s easy to use testing to validate ideas that you already know will work. While many changes should be validated through Test & Learn, organizations should prioritize tests that are higher-risk and higher-reward. It’s okay for most new ideas to fail (that’s why you tested it!) – if you’re able to identify the big winners, and quickly improve or discard the losers, the whole testing process will pay off. In fact, the only sustainable way to generate a competitive advantage is to test ideas that sound risky, are not being pursued by your competitors, and can move the needle on your profits.
  1. Use existing differences across your network as a natural test vs. control opportunity. Often there is intentional or inadvertent variation in pricing, merchandising, operational, and marketing strategies across different markets and stores. For instance, some stores may charge a higher price for a particular item or staff fewer employees during peak hours. Comparing these stores to the rest of the network could reveal profitable strategies to implement more broadly. Organizations should identify these “natural experiments” and use a test vs. control approach to isolate their impact.
  1. Use tests to generate hypotheses for future tests. The best Test & Learn organizations are always refining their businesses, and frequently use learnings from previous tests to fuel their next one. For example, if a 5% increase in marketing spend worked, how do you know a 10% increase wouldn’t work better? That’s a great test.

Testing and learning the right way is worth millions of dollars in added profits.


Be Sociable, Share!

You can follow any responses to this entry through the RSS 2.0 Both comments and pings are currently closed.