ReCal3: Reliability for 3+ Coders
ReCal3 (”Reliability Calculator for 3 or more coders”) is an online utility that computes intercoder/interrater reliability coefficients for nominal content analysis data coded by three or more coders. (A version for 2 coders is also available.) Here is a brief feature list:
- Calculates four of the most popular reliability coefficients for multiple coders: average pairwise percent agreement, Fleiss’ Kappa, average pairwise Cohen’s Kappa, and Krippendorff’s Alpha.
- Displays all possible pairwise percent agreements and Cohen’s Kappa coefficients, making it easier to root out rogue coders
- Accepts any range of possible variable values
- Results should be valid for nominal data coded by three or more coders (other uses are not endorsed, and accurate results are not guaranteed in any case — trust but verify!)
If you have used ReCal3 before, you may submit your data file for calculation via the form below. If you are a first-time user, please read the documentation first. (Note: failure to format data files properly may produce incorrect results!) You should also read ReCal’s very short license agreement before use.
- Data should be nominal
- Each file should represent multiple coders working on a single variable (note that ReCal3 can only analyze one variable at a time whereas ReCal2 can handle multiple variables simultaneously)
- Each column should represent a single coder’s work on one variable
- Each row should represent a single unit of analysis
- All codes must be represented numerically
- Input file must be formatted properly
- All columns must contain the same number of units of analysis with no missing data (see below)
To format your data for ReCal3 analysis, follow these instructions:
- Make sure that each of your content analysis code values is represented by a unique number. E.g. 0 = absent, 1 = present, 99 = N/A. Your file must contain no characters other than numeric digits—no letters, no dashes, no decimal points, only digits.
- In Excel, SPSS, or another spreadsheet-like program, create a new file.
- Enter a different coder’s data into each column of your new file, one unit of analysis per row, ensuring that each row represents the same unit of analysis. Continue until each coder is represented by a single column. The screenshot below shows a file containing six coders’ output on one variable. The differences between their codes indicate that much more coder training is needed!
- Do not include any header information–the first cell of each column should be each coder’s first code.
- There should be no missing data; ReCal3 will generate an error otherwise.
- All data columns must contain the same number of units of analysis (i.e. all columns must end on the same row in your spreadsheet).
- Save this file in comma-Separated values (.csv), semicolon-separated values (also .csv), or tab-separated values format (.tsv)—simply changing the file extension to “.csv” or “.tsv” will not work; the file needs to be “saved as” CSV in whatever spreadsheet or stat program you’re using—and choose “comma” as the column or field delimiter (if applicable). Click through any warning messages that may pop up. The file should have a “.csv” or “.tsv” extension. Your file is now ready for analysis; use the file selection box above to locate it on your hard drive.
- Here is the full example file from which the screenshot above was taken. It contains one variable, six coders (columns) and 20 units of analysis. In a spreadsheet program it will look like a normal spreadsheet, but a web browser or text editor will display it as a series of comma-separated numbers.
If you’re having trouble getting ReCal3 to work with your data, first check the FAQ/troubleshooting page, and if you don’t find the answer to your question there, send me an email. Feel free also to leave any general questions or comments regarding ReCal3 below in comments.
Is there a version of ReCal3 anticipated that will perform similar analyses on ordinal and interval data, AND that will accept (the social reality of) missing data. By experience, we all know that judges can never be made to be as compliant as can computer coding.
EDIT: I have answered this question in the FAQ. ~DEEN