Reflections from the River

Meta-analysis of CTR from natural search results (SERPS)

Key message: The click-through rate from different positions within the top 10 search results can be predicted using the equation CTR%=0.59 x Position^-1.94.

New data on the click-through rates (CTR) from different positions within natural search results (SERPs) is always accompanied by a flurry of blog posts and comments from the SEO community. When AOL released data on the click-through rates from SERPs by their customers it was instantly seized upon by several bloggers (e.g. 1, 2). More recently, when Wunderman presented click-through data at an Ad-Tech event (pdf) a blog post presenting the data attracted 26 comments. This, perhaps, isn’t surprising since such data can form the basis for estimating the commercial value to a client of moving up to the top ten, top three or even top result on the search engines.  What is perhaps more surprising is that there is no definitive and accepted data on CTR from different positions on search results pages. Of the 26 comments on the Wunderman data, mentioned above, 15 of them questioned or challenged the data, its source, its validity and/or its value.

Meta-analysis

One approach to deriving definitive data on a topic is meta-analysis. The principle behind meta-analysis is that ‘truth usually lies amidst many approximations’. Re-analysing data from several sources enables trends to be identified with greater accuracy (the effect of outliers in any single data set is minimised) and with greater confidence (it is based on several sets of independently collected data).

Five data sets on click-through rates from natural search were found through web-based research.

1. Andrew Gerhart (Google Analytics data; click-throughs=6759; date=2009; source data)
2. AOL (Transaction log data; click-throughs=4,926,623; date=2006; source data)
3. Granka et al (Experimental study; click-throughs=275: date=2004; source data pdf Fig 1)
4. Pan et al (Experimental study; click-throughs=200: date=2006; source data Fig 3, ‘normal condition’)
5. Wunderman (transaction log data?; click-throughs=?; date=2007; source data pdf page 6)

The data from these sources was collated, normalised (the percentages were re-calculated for only the top 10 natural search results) and then analysed using the SPSS statistics application.

An advantage of these data sets (fortuitously!) is their diversity: two are from transaction logs, two are research data and one is from Google Analytics. They also range in sample size from a couple of hundred to several million. Consistencies found in such diverse data are likely to have wide applicability.

Results

The graph below shows the percentage of click-throughs from the top 10 natural search results, from the five data sets, and also the best-fit regression line through the data.

Effect of search results position on click-through rate

Regression analysis gives a best-fit equation of CTR=0.59 x Position^-1.94 (R²=0.96 p<0.001).

Interpretation and use

Given the diversity of data sources, there is a remarkable level of consistency within the data. It is not possible to give meaningful measures of variance because each data point is itself an average with unknown variance. Even without this, however, we are able to predict that moving a search result from position 10 to position 3 will increase click-through volumes by and average of 10-fold and moving from position 3 to position 1 will increase click-through volumes by 8-fold. Combining this model with data on search volumes allows us to estimate actual visit numbers. So, for example, given that there were 450,000 searches for ‘digital camera’ on Google in the UK during June, we can estimate that a site currently appearing at position 5 on the search results page could attract an extra 57,500 visits per month if they moved up to position 2 (search volume data from Google Keyword Tool).

Until additional data emerges, it is hoped that this model can form a basis for industry-wide agreement on the general relationship between position in search results and click-through rates.