ReCal for Ordinal, Interval, and Ratio Data (OIR)
ReCal OIR (“Reliability Calculator for Ordinal, Interval, and Ratio data”) is an online utility that computes intercoder/interrater reliability coefficients for ordinal, interval, and ratio data judged by two or more coders. (If you need to calculate reliability for nominal data judged by two coders only, use ReCal2; for nominal data coded by three or more coders, use ReCal3.) Here is a brief feature list:
- Calculates three reliability coefficients:
- Krippendorff’s alpha for ordinal data
- Krippendorff’s alpha for interval data
- Krippendorff’s alpha for ratio data
- Accepts any range of possible variable values, including decimal values and negative numbers
- Results should be valid for ordinal, interval, or ratio data sets coded by two or more coders (other uses are not endorsed, and accurate results are not guaranteed in any case — trust but verify!)
If you have used ReCal OIR before, you may submit your data file for calculation via the form below. If you are a first-time user, please read the documentation first. (Note: failure to format data files properly may produce incorrect results!) You should also read ReCal’s very short license agreement before use.
- Data should be ordinal, interval, or ratio
- Each file should represent two or more coders working on a single variable (similar to ReCal3)
- Each column should represent a single coder’s work on one variable
- Each row should represent a single unit of analysis
- All code values must be represented numerically
- Input file must be formatted properly
- All columns must contain the same number of units of analysis with no missing data (see below)
To format your data for ReCal OIR analysis, follow these instructions (which are identical to those for ReCal3 except that OIR allows decimals and negative numbers):
- Make sure that each of your code values is represented by a unique number. E.g. 0 = absent, 1 = present, 99 = N/A. The complete list of allowed characters in your file is as follows: numeric digits, decimal points, minus signs (for negative numbers).
- In Excel, SPSS, or another spreadsheet-like program, create a new file.
- Enter a different coder’s data into each column of your new file, one unit of analysis per row, ensuring that each row represents the same unit of analysis. Continue until each coder is represented by a single column. The screenshot below shows a file containing six coders’ output on one variable. The differences between their codes indicate that much more coder training is needed!
- Do not include any header information–the first cell of each column should be each coder’s first code.
- There should be no missing data; ReCal will generate an error otherwise.
- All data columns must contain the same number of units of analysis (i.e. all columns must end on the same row in your spreadsheet).
- Save this file in comma-Separated values (.csv), semicolon-separated values (also .csv), or tab-separated values format (.tsv). Simply changing the file extension to “.csv” or “.tsv” will not work; the file needs to be “saved” or “exported” as CSV in whatever spreadsheet or stat program you’re using. Choose “comma” as the column or field delimiter (if applicable). Click through any warning messages that may pop up. The file should have a “.csv” or “.tsv” extension. Your file is now ready for analysis; use the file selection box above to locate it on your hard drive. Before executing, be sure to check off which coefficient(s) you would like to calculate.
- Here is the full example file from which the screenshot above was taken. It contains one variable, six coders (columns) and 20 units of analysis. In a spreadsheet program it will look like a normal spreadsheet, but a web browser or text editor will display it as a series of comma-separated numbers.
If you’re having trouble getting ReCal OIR to work with your data, first check the FAQ/troubleshooting page, and if you don’t find the answer to your question there, send me an email. Feel free also to leave any general questions or comments regarding ReCal OIR below in comments.
Leave a comment