Brief

The goal of the August multivariant test was to create fresh summer designs to beat the control.

The Challenge

The control had been running for a month or two without being beaten. It was a relatively straightforward design, but it had come close to losing. The nearest result was "winner non-significant" – technically beaten, but not statistically conclusive, so the control held.

The Approach

Every test starts with a brainstorm. Around five designers and non-designers each pitch at least five ideas – quality doesn't matter at that stage, the point is to think differently. From there, I selected four directions: Colorbomb Expolding Out of Phone, Waffle Gameboard, a Colorbomb Gameboard, and 3D Rainbow Stripes Candies. I roughed out each layout on paper before building the finals in Photoshop.

Outcome

The 3D Rainbow Stripes Candies creative beat the control – and did so with a statistically significant win, which is the meaningful threshold. A non-significant win leaves the result in doubt; a significant one confirms the new creative genuinely outperforms. The three losing variants – Colorbomb Expolding Out of Phone, Waffle Gameboard, a Colorbomb Gameboard – each explored a different visual territory, which is the point of multivariant testing: casting wide enough to find what actually lands.