Breakthrough Performance: Why Test and Learn Isn’t Always Enough

‘Data-driven’ and ‘test and learn’ are part of the default lexicon of modern marketing. Arguably, rendered meaningless by their overuse yet still worn like a badge of honour as we try to convince ourselves that every campaign decision is rationalised and fully supported by data in a spreadsheet.

We run endless split and multivariate tests, obsessing over minuscule conversion lift and call it optimisation. What if the nuggets of insight we’re looking for are a cypher? What if the performance breakthroughs we all hope for are hidden, more than we can know, existing only as contingent knowledge?

Contingent knowledge, or more simply an unknown unknown, is a truth that isn’t currently known but becomes knowable only by taking a specific, often unconventional, path. It’s the answer to a question you didn’t even know to ask. Instead of simply confirming a hypothesis, sometimes we need to push past the edge of the known to discover something entirely new.

Often, the biggest challenge during testing isn’t the logistics of ideating and deploying 160 new ads but the mindset that informs the approach. Testing budgets are usually constrained by the need to deliver immediate results, begging the question, is a test even a test if it cannot fail? The pressure to deliver results during testing often forces us into a cycle of marginal, incremental improvement, where the test is only considered successful if it yields an immediate performance lift. This approach, while valid and safe, can stifle genuine learning by limiting creativity and precluding original thinking.

The law of unintended consequences comes into play, too, when thinking about what constitutes a valuable test. What if a ‘failed test,’ one that doesn’t produce an immediate positive return and as such is dismissed as a waste of money, reveals something far more valuable? A creative that tanks when shown to one audience might fly when shown to another audience. An ad that wouldn’t get past the concept stage according to received marketing wisdom might resonate deeply with a niche audience segment. These seemingly nonsensical moments are when contingent knowledge is revealed. We should place a value on the failure to deliver a direct, intended outcome if that failure uncovers a new, completely unintended truth. We can only get there by being willing to push past the inertia of immediate gratification and allow ourselves to analyse the full spectrum of reactions, not just the ones we were hoping for.

This isn’t a call to throw any madcap concept at the wall and hope something sticks but instead for a reframing of how we approach testing and a move away from immediate incremental gratification in search of more meaningful performance uplifts. A great place to start is by carving out a dedicated budget for true experimentation. Of course, this isn’t always easy, but by untethering a small pool of capital from the demands of direct ROI we free ourselves to potentially unlock game changing insights. Think of it as a research and development fund for your marketing efforts. This allows for genuine, scientific testing where you can isolate key variables and run experiments with clarity.

Crucially, this approach requires pushing past assumptions until you have statistically significant data to uphold them and remaining open to stress testing your strategy. A test suggesting that a specific creative format, landing page or offer doesn’t perform is not necessarily eternally true. Shifting user behaviour, evolving market dynamics and constant platform changes mean data-driven insights date very quickly. Be ready to be proven wrong.

What constitutes success or failure during testing is another component of this more open approach to experimentation. Maybe a specific ad variation didn’t move the needle in terms of conversion but, on closer inspection, it drove a significant click-through rate uplift from a specific age demographic. The ad didn’t ‘work’ but this seemingly insignificant piece of data may be a breadcrumb that could lead to a larger breakthrough when we refine creative and double down on the responsive audience.

The rapid development of AI and the introduction of technologies like META Andromeda means flexible, open approaches to testing have never been more important. Automation and personalisation mean we are daily ceding control to a burgeoning intelligence vastly more powerful than us. Manual, interest-based targeting will soon be obsolete, and our role will be to strategically feed ad delivery systems with a rich, considered palette of creative options. The machine then takes on the role of uncovering contingent knowledge – matching creative to the right person and making a connection we as human beings would never have known existed.

In the end, the most valuable insights often exist just out of our field of vision, rarely do they sit in the well-lit spaces we assume. Untapped performance awaits and can be unlocked by our willingness to embrace risk, to accept a failure as a valuable investment and to see the world not just as a collection of known facts but as a reserve of unknown unknowns. Our challenge as marketers is simple: be braver, embrace the risk to find the rewards.

The author: Rob Lewis is paid social director at WPR. He’s a paid social performance marketing specialist, with a focus on direct response revenue and lead generation, and extensive experience buying media across all platforms for B2C and B2B brands.