ReCal3: Reliability for 3+ Coders

ReCal3 (“Reliability Calculator for 3 or more coders”) is an online utility that computes intercoder/interrater reliability coefficients for nominal data coded by three or more coders. (Versions for 2 coders working on nominal data and for any number of coders working on ordinal, interval, and ratio data are also available.) Here is a brief feature list:

  • Calculates four of the most popular reliability coefficients for multiple coders assessing nominal data: average pairwise percent agreement, Fleiss’ Kappa, average pairwise Cohen’s Kappa, and Krippendorff’s Alpha.
  • Displays all possible pairwise percent agreements and Cohen’s Kappa coefficients, making it easier to root out rogue coders
  • Accepts any range of possible variable values
  • Results should be valid for nominal data coded by three or more coders (other uses are not endorsed, and accurate results are not guaranteed in any case — trust but verify!)

If you have used ReCal3 before, you may submit your data file for calculation via the form below. If you are a first-time user, please read the documentation first. (Note: failure to format data files properly may produce incorrect results!) You should also read ReCal’s very short license agreement before use.

Documentation

ReCal3 operates under the following requirements and assumptions:

  • Data should be nominal
  • Each file should represent multiple coders working on a single variable (note that ReCal3 can only analyze one variable at a time whereas ReCal2 can handle multiple variables simultaneously)
  • Each column should represent a single coder’s work on one variable
  • Each row should represent a single unit of analysis
  • All codes must be represented numerically
  • Input file must be formatted properly
  • All columns must contain the same number of units of analysis with no missing data (see below)

To format your data for ReCal3 analysis, follow these instructions:

  1. Make sure that each of your content analysis code values is represented by a unique number. E.g. 0 = absent, 1 = present, 99 = N/A. Your file must contain no characters other than numeric digits—no letters, no dashes, no decimal points, only digits.
  2. In Excel, SPSS, or another spreadsheet-like program, create a new file.
  3. Enter a different coder’s data into each column of your new file, one unit of analysis per row, ensuring that each row represents the same unit of analysis. Continue until each coder is represented by a single column. The screenshot below shows a file containing six coders’ output on one variable. The differences between their codes indicate that much more coder training is needed!
    • Do not include any header information–the first cell of each column should be each coder’s first code.
    • There should be no missing data; ReCal3 will generate an error otherwise.
    • All data columns must contain the same number of units of analysis (i.e. all columns must end on the same row in your spreadsheet).
  4. Save this file in comma-Separated values (.csv), semicolon-separated values (also .csv), or tab-separated values format (.tsv)—simply changing the file extension to “.csv” or “.tsv” will not work; the file needs to be “saved as” CSV in whatever spreadsheet or stat program you’re using—and choose “comma” as the column or field delimiter (if applicable). Click through any warning messages that may pop up. The file should have a “.csv” or “.tsv” extension. Your file is now ready for analysis; use the file selection box above to locate it on your hard drive.
  5. Here is the full example file from which the screenshot above was taken. It contains one variable, six coders (columns) and 20 units of analysis. In a spreadsheet program it will look like a normal spreadsheet, but a web browser or text editor will display it as a series of comma-separated numbers.

If you’re having trouble getting ReCal3 to work with your data, first check the FAQ/troubleshooting page, and if you don’t find the answer to your question there, send me an email. Feel free also to leave any general questions or comments regarding ReCal3 below in comments.

13 Responses to ReCal3: Reliability for 3+ Coders

  1. John Collins

    Is there a version of ReCal3 anticipated that will perform similar analyses on ordinal and interval data, AND that will accept (the social reality of) missing data. By experience, we all know that judges can never be made to be as compliant as can computer coding.

    EDIT: I have answered this question in the FAQ. ~DEEN

  2. Julie

    Does ReCal3 allow for more than one variable, the way that ReCal2 does? I.e., for 3 coders, can the first three columns represent one variable and the next three columns represent a second variable?

  3. dfreelon

    Julie,

    It does not, and I suppose here is as good a place as any to explain why. The main reason is that I couldn’t think of an intuitive way to do it. ReCal2 simply assumes that every adjacent column pair is a new variable, but ReCal3 can’t do that since it has to be able to handle any number of coders starting with two. I briefly considered requiring users to enter the number of coders for ReCal3, which could then serve as a columnar multiple to produce that ReCal2 effect, but rejected this idea as too complicated.

    ReCal’s error rate is fairly high (first-time users almost always make at least one error before completing a successful run) and I didn’t want to make the system any more complicated than it had to be. Anyone who’s ever designed a user interface will tell you that every extra hoop you force the user through generates more errors and inevitably increases attrition. I realize that my decision privileges new users at the expense of more savvy ones, but I would rather make ReCal more accessible than less, even if that means a bit more clicking for ReCal3 users. And isn’t that what undergrad coders and research assistants are for anyway?

  4. Fawzy

    wonderful, handy and very helpful

  5. Brian Louie

    Is there a probability p value that is associated with Fleiss’ Kappa? If so, it would be great to see this integrated as a feature in future versions.

  6. Venise Wagner

    Thanks so much for this tool. It helped immensely. The p value would be great to include.

  7. nathalie

    AWESOME!! This is so helpful!

  8. Guillermo

    Professor,
    I don’t understand when you say “multiple coders working on a single variable”. I looks to me that each row represents a variable and columns coders. in my case i have 10 survey questions (variables) and approximately 100 respondents. Following your screenshot, I would have thought that columns represent respondents (100 columns in total) and each row a survey question (10 rows in total). Each cell then represents the answer to that respondent/question. does this make sense?
    Thanks again

    Guillermo

    • dfreelon

      Normally intercoder reliability isn’t used to analyze survey results, unless you’re attempting to assess the extent to which your participants agree in their answers. It doesn’t make much sense to me to try to assess reliability with that many participants–you may want to check the established practice for whatever it is you’re trying to do.

  9. D.Ham

    I think the tool is useful. Thanks

  10. Jesse

    Wonderful tool.

    If the same group of raters rate two different data sets is there a way to tell if the differences in alpha scores from one data set from another is significant?

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>