How to get results with A/B testing

A/B testing is a powerful tool to use when you want to test different versions of a page or section safely. When you have KPI’s to hit and you can’t afford to have a dip in results whilst you test new elements it’s a very powerful method. On any typical week I will have a round 10+ running on different sections of our main funnel at whocanfixmycar.com.

A/B testing is a powerful tool to use when you want to test different versions of a page or section safely.

How to decide what to test

This is the easy bit, I test everything that I go live with. This helps to keep releases smaller, makes you think about the impact and also makes releasing new things a lot safer and less stressful.

I am going to use one example throughout, this is what we call the “Comparison tile”.

Step one

Using feedback from drivers and the development of job specific funnels I made some additions to the comparison tile and introduced some new styles. This was the MOT funnel.

Step two

Decide to do a 50/50 split or a 30/80 split depending on the impact that bad results couple have on the full core funnel.

Step three

Lets do this, The developers to set up the tests or we use google’s optimizely.

How to monitor the test

I find the best way to monitor results is to have a live dashboard, which if you have a data team you can create. Or you can run the results every few days. It depends on the complexity of the test to decide on the best way to monitor.

For example you may be working out the conversion on a form which may be easy to monitor with Google analytics funnel with users in the top and drop off. Or you may need to look in the database to find the actual conversion of a process (Purchase or choosing a job type.)

How to decide when to release

So it’s important to set targets to make sure you are improving on results. For example we have KPIs set for each of our funnels. To improve on these funnels we need to be constantly improving the flows and conversions. To decide on if you are going to release it or not, look for consistent improvements on conversion 2 weeks is ideal as i find sometimes results can look good or bad within the first few days and starts to level our over a longer period of time.

What if the results are tied? If its a draw, lets go for it.

What if its a draw

If it’s a draw I would normally extend the test or if I am feeling brave push it live as its not having a negative effect. This also means that the styles / layouts are consistent throughout the apps as opposed to some having the old and some the new.