Measuring impact on the web

Like many development research organizations, IFPRI staff often debate the question of how to best measure the impact of our work. As opposed to some of our partner CG Centers, IFPRI’s work focuses on policy-level interventions, which are rather different from producing new crop varieties for the purpose of increasing agricultural yields, household incomes, etc. Thus, it could be claimed (and in fact has been claimed by some) that IFPRI primarily is in the business of producing “intellectual varieties.” Defining what we produce, however, is quite different from knowing how to measure impact, which effectively underscores the importance of choosing which indicators it’s worth paying attention to. Below I list a few potential indicators for measuring impact on the web while trying to provide my own impressions on their overall value and utility for measuring impact.

  • Page views/visitors: It seems like many webmasters and other mangers of web statistics reports have been questioning the value of these for years, but they continue to be reported year after year. Criticisms include the lack of proper contextualization (from what constitutes a ‘hit,’ ‘visitor session,’ etc. to how do we compare our statistics with others) as well as the lack of any kind of clear correlation between these easily-measurable indicators and others designed to measure impact.
  • Downloads: It seems to follow that the more a particular publication is downloaded, the more people are reading it, thereby expanding its sphere of influence. In practice, however, the same person might be downloading a particular publication multiple times and even more important than the how many question is the by whom question. As with page views/visitor sessions, Google Analytics‘ mapping features can be highly useful in obtaining this type of data to make sure that research outputs are reaching audiences in the countries they’re intended to benefit.
  • Citations: Once again, context is everything. Many research organizations only report on their peer-review publications, and it follows that they are only interested in citations in PR journals, books, etc. Several online databases offer citation tracking information, including Google Scholar, ISI Web of Science, CrossRef’s Forward Linking Program, and CiteSeer, among others. One might be tempted to add these citation figures up in order to obtain a clearer picture of overall impact, but this would not yield such a picture since several of these services count the same citations. Thus, benchmarking your organization’s citations vis-a-vis your counterparts would probably prove more effective in monitoring the impact of your research publications.
  • Mentions in the media/blogs: It also seems like many research organizations have been monitoring this for years, but the addition of services like RSS feeds and automated alerts from Google News has made it much easier than in the past. The same also applies to monitoring the blogs via advanced search operators for Google and Google Blog searches as well as Technorati.
  • RSS feeds: Once again, many are questioning the value of measuring ‘hits’ as an indicator for impact. Instead, they argue, we’d be much better served in paying attention to the number of feed subscribers out there, though this can also fluctuate heavily over time. For our purposes, Feedburner fits the bill rather nicely, even while sticking only to their free, basic service.
  • Search engine rankings: Obviously, there’s one search engine that need not be named (at least, not for the umpteenth time in this post!) that most organizations pay attention to above all others, but few fail to take into account differences across different country platforms. My own advice is to pay attention to your ranking among search results for several keywords your organization may be targeting and look into purchasing ad words for your institution via the Google Grants program (free for non-profits).

5 Responses

  1. […] easily extend them to the rest of the web. Pete Shelton makes some helpful suggestions of how to Measure the impact on the web or here is one for […]

  2. There is always the caveat for traditional scientific citation analysis and the use impact factors for scientific journals that one should never compare across different subject domains, as publishing and citation behaviour is so different in different domains. I would think that the same is true for web impact measurements

    Hugo Besemer

  3. Two important points I failed to mention earlier and am glad Christian and Hugo brought them up: ratios and research areas. Obviously, when benchmarking the number of citations vis-a-vis the World Bank, for example, it makes much more sense to compare the number of citations per article than the number of overall citations since the Bank produces so much more material than IFPRI. Also, many website analysis reports compare links per page and the ratio of PDF documents to web content (as indicators of site quality and value, respectively) across peer organizations.

    Also, Hugo’s point about differing publishing and citation behavior across disciplines is well documented, which is why ISI et al. generally breaks down their citation reports thusly. He also hints that the same should apply to the other indicators I mentioned. I think that they would be in any comprehensive website analysis, which should compare the organization’s web presence with other organizational websites in their research area.

  4. […] and since weblogs are new social media, the standard impact measures, according to Peter Shelton of Web Tastings, are the […]

  5. […] new social media, there are standard impact measures reflected in blog statistics. Peter Shelton of Web Tastings, describes web impact measures to […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: