Background
A/B Testing is a method of testing in which you experiment with variations of your website design or content and measure your website activity to see how well the variations performed.
We have previously done an analysis to see how many clicks each area-of-interest tile on the Academics page receives. Click here to see the analysis of the academics page tile click-through rate.
This analysis seemed to indicate that the further down the page a particular tile was, the less likely it was to get clicked on.
We then developed a plan to A/B test the academics page to see if we can change the click-through rate by changing the tile order.
Example
Experiment
Our plan for this A/B test is to change the order of the tiles on the page such that the last tile in the list – Sustainability – will be displayed in first position. Everything else would stay the same.
Every visitor to the website during the experiment period will be divided equally into one of two groups. 50% of visitors will see the original design (control group) and 50% of visitors will see the variation. (Because the group assignment is done via cookie, once a user has been assigned to a experiment group that will continue to see that particular version until they clear their browser cookies.)
Measurement/Objective
For this test, we will be measuring success by seeing how many people in each group go on to view the /academics/sustainability page during their visit. (Note that the only link to the /academics/sustainability page is from the academics page.)
Hypothesis
Moving the “Sustainability” tile into the first position should result in more page views of the /academics/sustainability page.
Outcome
The experiment has not yet reached a definitive answer, but the data so far seems to indicate that moving the sustainability tile to the first position results in more clicks to this page.
However the boost in clicks was not as high as I would have thought.
Questions
Why was there only a minimal boost in clicks to the sustainability page? I suspect that this may be a case where the last tile listed alphabetically – sustainability – also happened to have a lower level-of-interest. Therefore moving it to the top caused it to outperform somewhat compared to its original position, but since the level of interest may not have been high in the first place, there was only so much improvement to be had.
We could test for this effect by trying the same test on a tile which outperformed what you would expect based on its position on the page (e.g. “Health and Well-being” or “Science and Biomedicine”). These tiles have done quite well in terms of click-throughs, indicating a high amount on ‘natural’ interest. Perhaps moving one of these tiles to the front would have a larger effect on click-throughs.
Suggestions for Further Testing
The academics page is a very important page for us. As we have seen, it is regularly the second-most viewed page across our entire web presence. Therefore, anything we can do to make even small improvements to the usability and value of this page is likely to have a beneficial effect.
There are a number of options for additional testing to improve this page. A couple we are considering are:
- Removing the slider on the top-left of the page. This slider takes-up some valuable ‘real estate’, which could perhaps be better-used by providing space for more tiles.
- Changing the titles of one or more tiles. For example, the “Global” tile seems to do surprisingly poorly – perhaps due to its title. It is the only tile for which the title does not match the others. Suggestions for better names?
Improvements
- It would be nice to be able to limit the experiment targeting rules to exclude Mason users, but there is no ready-made way to do that. (We could filter-out city=”Fairfax” traffic.)
- It would be nice to implement the anti-flicker ‘page-hiding’ snippet, so that users would not possibly see a brief ‘flicker’ on their screen as the experiment loads. This is a bit more technically complicated that you might think.
Other Notes
You can identify website users seeing a particular variation in Google Analytics.