ReCal: reliability calculation for the masses

ReCal (”Reliability Calculator”) is an online utility that computes intercoder reliability coefficients for content analysis data. It is compatible with Excel, SPSS, Stata, OpenOffice, Google Docs, and any other database, spreadsheet, or statistical application that can export CSV files. Select one of the following two links to get started:

I need to compute reliability for two coders only. (ReCal2)

I need to compute reliability for three or more coders. (ReCal3)

To date, ReCal (2 and 3 combined) has been successfully executed a total of times by persons other than the author.

• For those of you visiting this site in search of Excel formulae for calculating intercoder reliability: Use ReCal; it’s better than a spreadsheet. I actually wrote a couple formulae for calculating ICRCs (intercoder reliability coefficients) in Excel, but I abandoned that path when I realized PHP (the programming language in which ReCal is written) would be much more flexible. Here are several reasons why ReCal is better than Excel for computing ICRCs:
  1. From a programming standpoint, Excel’s basic function language is not very sophisticated in its handling of arrays, which are essential for calculating reliability. This means that VBA (Visual Basic for Applications, a Microsoft proprietary programming language that works inside Excel spreadsheets) would need to be used. Unfortunately, MS has removed VBA from the current Mac version of Office, meaning that an ICRC macro written in VBA would be useless for non-Windows users.
  2. Even if VBA hadn’t been removed from Office for Mac, a VBA macro would still restrict ReCal usage to Excel owners. Coding in PHP and using CSV as an input format creates a “lowest common denominator” in terms of system requirements (since both are non-proprietary) that maximizes the breadth of ReCal’s user base.
  3. PHP generally runs faster than VBA, which you’ll notice if you compare ReCal to PRAM (although technically PRAM is written in VB rather than VBA, the languages are very closely related as their names indicate).
• So hopefully this demonstrates why ReCal is a superior alternative to an ICRC macro written for Excel. If you disagree, see any major disadvantages to using PHP, or have a question/comment, please feel free to respond in comments below.

And here are some notes on of the differences between the two versions of ReCal linked above:

• ReCal for 2 Coders (ReCal2) can calculate reliabilities for multiple variables at once, whereas ReCal for 3+ Coders (ReCal3) can only calculate reliability for one variable at a time. If you have several variables all coded by two coders, the former edition might save you time.
• Although the two utilities share a formally identical data format, they make very different assumptions about what that data represents. ReCal2 assumes that data columns come in pairs, i.e. that columns 1 and 2 represent two coders’ codes for a single variable, cols 3 and 4 represent two coders’ codes for a different variable, etc. By contrast, ReCal3 assumes that each column in the input file represents a different coder’s work on a single variable. Therefore, the same 6-column CSV file would represent 3 different variables coded by 2 coders each to ReCal2, while ReCal3 would interpret it as one variable coded by 6 different coders. For this reason, the only files for which both ReCal2 and ReCal3 will give accurate results are those containing only 2 columns (coders). Submitting data intended for one edition to the other will generate incorrect results!

Questions? Comments? Suggestions? Leave a comment.