The importance of A/B split testing your website

For seasoned digital marketers A/B testing or split testing your web pages is nothing new. It’s been around for years and it’s a really simple addition to your planning, testing or even as part of your digital strategy. Why are we writing a blog about then I hear you ask? Good question, one that has a simple answer. I’ve been working in Auckland for 9 months now and I’m being exposed to new brands and digital agencies every week and there is a common theme; split testing seems to almost be an alien concept.

A colleague of mine actually described split testing as my ‘secret weapon’. It’s a nice way of putting it, although the word secret does rather back-up my point. In many ways split testing is a weapon, it has countless benefits both from a digital marketing standpoint and as an incredibly powerful way to get your point across to clients or stakeholders.

Below I’ve outlined what split testing is and some of the key benefits of testing your website or webpage’s.

What is A/B or split testing?

For those that aren’t seasoned digital marketers we will start at the beginning. A/B testing, split testing (or even bucket testing) is the act of comparing two versions of a web page to see which performs best.

A software or analytics package will show the two versions of the page automatically to visitors at a typical rate of 50%. This in theory means that one visitor could visit your website at 3pm and see version A of your page. A second visitor arrives at 3:01pm and sees version B.

Using the split testing software platform website owners can determine by what measure to determine a winner. This could be a conversion (lead capture, sale etc), an event (e.g. a PDF download) or even how long a person spends on the page.

When a winning page is found you then have the choice to continue with your existing one or switch to the new version.

Why should I split test my web pages?

Split testing is used to help you understand your website and your visitors with the end goal typically to improve your conversions or engagement. Whilst there is a huge amount that can be written about a/b testing and the potential benefits I’ve decided at some personal and practical benefits that extend beyond your website and into working with a stakeholder, boss or client.

#1 - Make website changes with low risk, high reward

Let’s say you work generating leads for private healthcare customers. Your landing page results are down but the page has worked perfectly for 6 months. The easy solution is to try to do some conversion rate optimisation on the page and find out what is wrong and then fix it. That makes sense but you don’t want to throw the baby out with the bath water, it could be your problem isn’t webpage related.

Instead, take your optimisation recommendations and build a duplicate page and split test it. This way you can accurately measure the changes against the old page and see if it makes a difference. If it does, fantastic, if it doesn’t then make some changes to your new page and test again.

Another example on a smaller scale. I own a small Auckland based social media agency. I’ve just been through a cool expensive re-brand which includes our office mascot Dave the cat. We are finding that since introducing him to the page we are getting much higher bounce rates. It’s just a theory but if we take him off will that make a difference? Rather than sorting the branding, do a simple split test by removing him and see what happens.

Whilst these may not be the world’s greatest example what it does is show the various benefits of split testing. It can allow you to make and test changes to your site and pages without overhauling, without re-designing or getting an agency to do an entire audit. You remove the potential risk of making a snap decision or decision based on anecdotal feedback. You can try it even if you think it maybe not quite right, if it isn’t you’ll know pretty quickly.

#2 - Make a great case for change

Not entirely dissimilar to proving a point split testing can be the backbone of making a case. Here’s another example of how split testing has helped me over the years.

I was working at a business that used online forms as their key lead generation means. I’d ran some form testing and found that we had a high percentage of users drop off on the field where we asked the user ‘where did you hear about us’. This meant we were losing out on conversions because of this field.

Google Analytics experiments example screenshot

I took this information to a senior but was told it’s a necessary field to help with marketing spend. Ok, fair enough. What if I create a split test without it? Fast forward two weeks and I have the results of my test, the form without the HDYHAU field has won by over 105%. I can now put a case together that has tangible numbers attached to them. I can accurately say that by removing the field we could generate double the amount of enquiries in a year. It’s a compelling argument and one that I won. Two crowns is just greedy.

#3 - Prove a point

This seems a rather petty way to describe this particular point but I’ve worked as a digital consultant and client-side long enough to know that just telling someone you’re right about something isn’t enough.

A few years ago I was working with a particularly difficult stakeholder who despite having no digital (or even marketing) background insisted she was right based entirely on seniority. She had heard that long pages with lots of content was a bad thing and so wanted a landing page to have minimal copy and be image heavy. I suggested that whilst she was broadly correct we had no evidence in our industry and for our market that this was the right approach. The trouble was I had no solid evidence either way, so she ‘won’.

Well, she would have won had I not suggested creating an alternative version of the page with more text, less imagery and a different layout. We then split test this and ran it for two weeks. After two weeks the page with more content had more engagement and generated more actions (a PDF download).

That’s the thing, whilst her basic digital knowledge was sound and based on ‘best practice’ it didn’t mean she was right and remember best practice isn’t a blanket rule. In the end she backed down and from then on was far more open to my suggestions, I’d proven I was the expert and I had the means to prove rather than speculate. I was presented with a crown and I wore it until I left.

What software can I use to split test?

So you agree that split testing is the way forward? That’s great, now you just need to know what platform to use and how much will it cost. Well the three paid ‘big hitters’ in this space are:

Each of these platforms have their own pros and cons and come at different costs. For more information on what the best platforms are you can check out Conversion Sciences 20 best A/B testing tools.

Our recommended A/B tool

Our recommendation however is to use Google Analytics Experiments. Whilst Experiments isn’t as powerful as some of the other tools we’ve mentioned it has two benefits. One it’s free. Two you should already have Google Analytics on your site so you could be ready to use it instantly.

I’ve used experiments consistently for probably the past five years for day-to-day use mainly due to ease of access and it’s an easy handover once I move on from a client.

Experiments in its current guise will soon cease to exist and will become Google Optimize. Here at NZ Digital Marketing we’ve had a play around with Optimize and whilst it’s not as easy to pick up and use for new users it’s far more powerful in almost every respect. For those looking to get started easily I’d recommend getting experiments running and check back to learn more about Optimize.

If you want to jump straight to Optimize now here’s a video that gives you more detail and a tutorial on how to get started with Google Optimize.

Using Hotjar to generate leads banner

Optimising your website user experience with Hotjar

At NZ Digital Marketing we love analytics and conversion tools and we particularly love Hotjar. We’ve had some fantastic results using it in some very different organisations over the past 12 months. No matter whether you’re B2B, B2C a government organisation or you’re selling hipster antiques online you should be using Hotjar to compliment your existing analytics.

What is Hotjar?

Hotjar has been a player in the market for 12 months or so now although none of the key features are particularly unique. The reason we love it is the combination of features coupled with the price and a market leading interface.

In short; Hotjar uses analytics and feedback to really help you understand your web and mobile visitors.

Let’s take a look at the four key features and how these features can make a difference to your site and how it could (and possibly should) influence your digital marketing strategy.

Four key features:

#1 Heatmaps

Heatmaps? They are as old as time! We did say it wasn’t unique but Hotjar’s heat mapping tool more than does the job. Hotjar includes all your standard mapping options:

  • Click maps - Showing where users click their mouse on any given page
  • Movement map – Where users move their mouse on the page, this can provide insight on how a user is absorbing the copy or content on your page
  • Scroll maps – This shows how far down the page users are willing to scroll on average.

Hotjar webpage heatmap example

If we had a dollar for every time a marketer or communications professional has said “everything has to appear above the fold or people won’t see it” over the years we’d be slightly richer. It’s like this one piece of info was handed out to everyone on their first day at work and they’ve held onto it ever since. Whilst there is nothing wrong in essence with the statement it definitely doesn’t hurt to be able to back this up, every industry and website is different and the beauty of using the scroll map is that if you get this retort you can simply point to the scroll map and say “yes you’re right but 75% of people are seeing everything below the fold”

Heatmapping is an essential tool in the digital marketing cannon. It allows you to see just how your users are interacting with your pages, whether you’re testing a new landing page or thinking about making changes to your homepage you want to be running some heatmapping before making any big design decisions. It can also be a great compliment to an A/B split test as you can track both variants in Hotjar and see how the interaction varies on each layout or design. You can also split the maps into desktop, mobile and tablet traffic.

#2 Visitor Recordings

Visitor recordings are a sort of half-way house solution for those that don’t have the time to do some face-to-face user testing sessions or the capacity/budget to run something like The visitor recordings do exactly what they say on the tin, they record visitor sessions as video clips that play for the duration of a user’s session.

Visitor recording screenshot from Hotjar

As Hotjar only takes a proportion of your sessions you can’t watch every single user journey but you can near as damn it. The beauty of the recording sessions is the filter option, you can filter by the landing page, the exit page or any page in between. Couple this with the heatmapping tool and suddenly making an informed decision on a new page or journey becomes a lot clearer. When starting a role with a new client or organisation we’ve found that using visitor recordings really helps get your head around the user behaviour in that industry and more specifically on that website. Even if you don’t plan to use that information directly at the time it’s a quick and interesting way to get orientated and a brilliant base for future decisions.

#3 Web Form Analytics

The web form testing feature is probably our favourite and without doubt the one feature that has brought us the most recent success. The benefits of this data are too many to go into detail here but you can read more about the benefits of web form testing and the potential huge uplift on our conversion rates on our web forms article.

Form analytics provide detailed information on your web forms including:

  • Overall form interaction (Sessions) - This tells you how many users have visited your form page. How many have then gone on to interact (click at least one field) and how many have left the page without interacting. This gives you insight into the immediate impact your form page is having on a user.
  • Field-by-field analytics - Field analytics are where you’ll find your core field information. Each field has three measurements: Time spent on field - the length of time it takes the user to fill in the field. % of people who had to go back and re-fill the field in. A % of those that left it blank. You also get the number of interactions on that field and....
  • Drop-off - The field in which the user gave up on your form and left the page without submitting. The drop-off is possibly the most important factor, if you can identify the reasons for the drop-offs your conversions can benefit hugely.
  • Successful submits - How many users successfully submitted the form out of the number of users that reach the end of the form. This essentially tells you if there is a problem at the very end of the form e.g. your button text or CTA is off-putting for some reason.

Web form analysis screenshot in Hotjar

#4 Feedback

A few bigger New Zealand brands are making use of the Hotjar feedback polls on their websites, it’s possible you’ve maybe even used one and not known.

The feedback poll is a little popup widget that allows you to ask your important questions directly to users on your website in real-time. Understand what they want, what’s preventing them from achieving it and how you can help. Hotjar then provides the data in a raw form, in a word cloud or if you’ve asked users to rate their experience you’ll get an overall net promoter score to help gauge your users satisfaction.

But popups? Pah, aren’t Google punishing mobile sites with popups? Yes, you’re right they do but this isn’t that sort of popup. It doesn’t look like an advert and I’m probably doing it a disservice by calling it a popup, but hey it helped me write that neat little opening line and I can’t be bothered to go back and re-write it now.

Anyway, the key thing here is that the widget is discreet and highly customizable, it can appear anywhere on your website at whatever point in the users journey you choose. For example; it can appear as soon as the page loads, when a user is about to abandon a desktop page or after a predetermined number of seconds. You can also edit the appearance of the widget to blend into your site and brand colours.

Example Hotjar user feedback poll

Feedback in action

We used the feedback feature whilst working with an NGO and at the time I was dismayed by the amount of people who were willing to put their time into providing answers to questions with little to no incentive to do so. In hindsight a service first website user's goals are significantly different to that of a commercial website user. You can generally expect an uplift in response on service first websites.

This organisation had a lot of anecdotal feedback about how poor the website was and this was backed up with a simple eye test. However we lacked tangible evidence from real users, the feedback tool gave us everything we needed to start the process of rebuilding the whole journey.

A minus net promoter score coupled with an average experience of 4/10 and hundreds of ‘constructive’ responses made the ensuing discovery and pitch stage a synch. You may not need to use the feedback tool for such dramatic reasons, you could just want feedback on a single page, or you want to understand why your ornamental lamp isn’t selling.

Like with the visitor recordings this shouldn’t be a replacement for actual user testing or gathering offline insights with real life users. It’s a very good secondary option as the volume of data you can gather can far exceed what most organisations can afford to put into user testing or focus groups.


We've not gone into detail about other features such as conversion funnels, surveys and user testing recruiting partly because we’ve not used all of them significantly enough and partly because we’ve typed the word Hotjar so much over the last 1500 words that we’re concerned Google will penalize us for keyword stuffing.

The summary is of course that pretty much no matter what type of website you have you should be using these tools, if you’re a conversion led business then you should 100% be using them and either way you should probably be using H****r.