Ok, I am stumped. How can I have a percent agreement of .97 and a Scott’s Pi of-.015? I have two coders coding either Yes (1) or No (0) for the presence of a variable. What am I doing wrong. I find when calculating by hand I get similar results (off by a decimal or so). When using RECAL or calculating Scotts Pi with more than two categories, I don’t get negative Scotts Pi when the percent agreement is high.
Thanks so much for sharing your program and answering my question if you have the time.
Excellent question, Sonya. As with the last question I answered, I’ll provide your raw data (with a new filename) so that others can follow along; hope you don’t mind.
Looking at the data, you’ll immediately notice an interesting characteristic: only the second coder uses the “1″ code. That is, the two coders only ever agree on “0″ codes and never once on a “1″ code. Scott’s pi, Cohen’s kappa, and Krippendorff’s alpha punish this phenomenon severely, the rationale being that coders must show at least some covariation in their agreements to merit high coefficient values. Krippendorff himself addressed this very situation in a recent article:
In the calculation of reliability, large numbers of absences should not overwhelm the small number of occurrences that authors care enough about to report. Without a single concurrence and three mismatches [Krippendorff here is referring to a specific dataset, which just so happens to have the same number of mismatches as Sonya's], the report of finding 2 out of 137 cases [3 out of 99 for Sonya's data] is about as close to chance as one can get—and this is born out by the near zero values of all the chance-corrected agreement coefficients. (2004, p. 425)
Thus, when one coder only uses one of two coding categories, and the other uses both, chance-corrected reliability will always be near or well below zero (but percent agreement can still be near 100% as it is not chance-corrected). The only solutions here seem to be either better coder training or a revised coding scheme that allows coders more latitude to agree with one another on different categories.
Reference
Krippendorff, K. (2004). Reliability in content analysis: Some common misconceptions and recommendations. Human Communication Research, 30(3), 411-433.
]]>Thank you so much for a great tool. But, I hope you can help me clear up a discrepancy I’ve noticed in my results for variables that have the same number of agreements/disagreements. For example, variable 1 has 26 agreements and 1 disagreement. So does variable 3. So does variable 5. Yet, the results for variable 1 are: 96.3% agreement and Scott’s pi of 0.924. The results for variable 3 are: 96.3% agreement and Scott’s pi of 0.914. The results for variable 5 are: 96.3% agreement and Scott’s pi of 0.886. Can you please tell me why the Scott’s pi is different for each variable when all the raw data for them is the same (ie same number of agreements and disagreements)? This scenario has occurred on three separate occasions when I’ve submitted my .csv files for analysis.
Great question, Dianne. The answer lies in how Scott’s pi (and Cohen’s kappa and Krippendorff’s alpha) compute reliability as compared to percent agreement. With the latter, equal numbers of agreements and disagreements will always give you the same result, as you noticed. This is not always the case with coefficients that correct for chance agreement, as do Scott’s, Cohen’s, and Krippendorff’s. Their formulae give additional “credit” to data sets with greater variation in agreeing values: in other words, the more different coding categories your coders agree upon, the higher your reliability coefficients will be (the logic being that it is harder to attain stronger levels of agreement on data that is highly distributed across many coding categories).
But I don’t expect you to just take my word for it, so I’ll actually work through your numbers and show the math here. I hope you don’t mind if I provide your raw data here so that other interested parties can follow along—I’ve changed the headers and the filename.
Recall that the formula for Scott’s pi is
(Po - Pe) / (1 - Pe)
where Po is observed agreement and Pe is expected agreement. Observed agreement is simply percent agreement as a fraction of one; thus for all three of your variables it is thus 0.963. Expected agreement is a bit more complex, but essentially what you have to do is:
- Note the number of possible coding categories your coders used (each of your variables used 2 categories, represented by the numbers 0 and 1)
- Start by adding the number of times coder A selected category 1 to the number of times coder B selected category 1 and then divide that sum by the total number of coding decisions (which in your case is 54, or 2x the number of cases). This value is known as the joint marginal proportion for variable 1, category 1. This is equal to (12 + 11) / 54 = 0.426. In this example we need to do this for all category values in all variables, so in total we will need to calculate 6 JMPs so that each variable has 2 (the number of coding categories noted above). The JMP for var 1, category 2 is (15 + 16) / 54 = 0.574. The rest are as follows: var 3 cat 1 = 0.315; var 3 cat 2 = 0.685; var 5 cat 1 = 0.204; var 5 cat 2 = 0.796.
- Now that we have all our JMPs, we need to square them and then sum them within variables to get our expected agreements. So for var 1, we have 0.4262 + 0.5742 = 0.511. The expected agreement values for vars 3 and 5 are 0.569 and 0.676, respectively.
Now we have all the values we need to plug in to our main Scott’s pi formula above.
- For var 1 Scott’s pi is (0.963 - 0.511) / (1 - 0.511) = 0.924;
- for var 3 it is (0.963 - 0.569) / (1 - 0.569) = 0.914;
- for var 5 it is (0.963 - 0.676) / (1 - 0.676) = 0.886.
By now you’ve probably figured out what’s going on. Looking at the data, variable 5 has the most uneven category distribution; you can tell at a glance that it is mostly zeros. This pushes its expected agreement value higher—it is easier to achieve high levels of agreement when a data set mostly falls into one category, so the Scott’s pi formula raises the bar accordingly. Vars 1 and 3 are more balanced in their category distributions, so their expected agreements are lower, making their coefficients higher despite the fact that all three vars have the same number of agreements and disagreements.
I hope this answers your question, Dianne. If not, let me know!
]]>Using this new function is easy. Simply submit your data file(s) to ReCal as usual, and near the bottom of the usual HTML results page you’ll see a button labeled “Export Results to CSV.” Click this button, and your output CSV file should pop up as a download. Give this file an appropriate name (this is important, as the default filename is “output.csv”) and save it to your hard drive. You should now be able to open your results file in any spreadsheet application. (Though it is technically possible, I highly recommend not trying to open these output files in statistical applications such as SPSS or STATA, as they most likely will not display the data properly.) When you export after having accumulated the results of multiple data files via the Save Results History option, the resulting output file will contain all of that data.
If you experience any problems or have any questions about exporting your results, please leave a comment below.
]]>This post is password protected. To view it please enter your password below:
After you run your initial data file through ReCal, you’ll notice another ReCal file form at the bottom of the results page. Below this form is a checkbox captioned “Save results history.” When checked, on the next execution ReCal transfers the currently displayed data output to the new results page and appends the new results to the bottom. This emulates the functionality of the output windows found in SPSS/Stata etc. by creating a history of results in order from oldest to most recent on a single page. Previously, ReCal created a separate results page for each file you wanted to analyze. Now you can collect all your related results in one output page for easy reference. And of course you can always leave the box unchecked to save only one set of results per page.
If you encounter any difficulties or incorrect results using this feature, please let me know ASAP.
]]>Forbes reports today that Yahoo will be shutting down Geocities at some point this year. Geocities is one of the oldest free web hosting services and a mainstay of the pre-Web 2.0 online publishing world along with Tripod and AOL. I note this primarily because the demise of Geocities means that PRAM, ReCal’s most robust competitor in the reliability calculation business, will no longer be available for web download. PRAM is (as of a Google search conducted just minutes ago) hosted exclusively on Geocities, so get it while you can. If you’re reading this after PRAM’s Geocities site has gone offline and would like to use it, let me know and I’ll email you a copy. (see update above)
While I’m on the subject, PRAM was last updated in 2004. Can anyone confirm whether or not it works in Windows Vista? If so, I’d appreciate it if you could let me know via email or a comment below.
]]>Have questions about ReCal, the online intercoder/interrater reliability calculator? Hopefully this page contains the answer you’re looking for. If not, feel free to submit new questions in comments. Back to ReCal main page
ReCal isn’t working with my data. It keeps giving me the following error code:
What applications is ReCal compatible with?
How can I be sure ReCal’s results are accurate?
I need help understanding/interpreting/improving my results; what resources are available to me?
What are CSV files and why does ReCal require them?
Why doesn’t ReCal work with Excel/SPSS/Word/[insert other proprietary software package here] files?
What happens to my data when I submit it to ReCal?
What are the functional differences between ReCal2 and ReCal3?
Error 1—You should never see this error, at all. Ever. If you do, please let me know ASAP because it indicates a dire system error.
Error 2—This error occurs in two cases: first, when your file runs above 100,000 bytes in size; and second, when your file is 0 bytes in size. It helps prevent incorrect and corrupt file types from being processed (CSV files are rarely that large and never that small). Double-check your file and make sure it is indeed a non-corrupt CSV file.
Error 3—This error occurs when your data file contains characters other than numeric digits (with the exception of alphabetic letters on the first row). ReCal’s requirements in this regard are generally quite strict—the digit “1″ by itself would pass muster, whereas “1.00″ would not due to the decimal point. Similarly, negative numbers won’t work in ReCal due to the minus sign—you’ll need to convert them to positive numbers. The only exception is that header text for each column may be included on the first row (a la SPSS); in this case, the entire first row will be ignored and calculation will begin on the second row. Make sure to scour your entire file for any characters other than numeric digits except on the first row.
Some users might see this error even when they are absolutely certain that their file contains only numbers. The problem in these cases may be that the “CSV” file is delimited by a character other than commas or semicolons. (See What are CSV files and why does ReCal use them? if you don’t know what this means.) To determine whether this is the problem, open your file in a basic text editor (not MS Word) such as Wordpad in Windows or TextEdit in Mac. If you see a series of numbers separated by anything other than commas or semicolons, you will need to run a Find/Replace command to convert whatever the separating character is into commas.
Error 4—This error is caused by missing data in your file, commonly seen in spreadsheet software as blank cells surrounded by data. Missing data violates the assumptions of all the coefficients ReCal computes; therefore it is not accepted. Fill in the missing data on the line indicated or delete the line entirely and try again.
Error 5—This error occurs for ReCal2 only and indicates an odd number of columns, whereas ReCal2 requires an even number of columns. Recall that ReCal2 assumes that each pair of columns constitutes two coder’s judgments on a single variable. If the number of columns in the data file is odd, the final column has no corresponding column with which reliability can be calculated. Double-check the number of columns in your file and try again.
Error 7—ReCal requires that each file submitted to it feature a “.csv” extension at the end of the filename. It is critical to understand that a file cannot be converted to CSV format simply by changing its extension! See How do I create CSV files? for more details on this point.
Error 8—This error occurs when all the rows in your file do not contain the same number of codes. For example, if rows 1-10 of a hypothetical 20-row file contain three columns of data, and rows 11-20 contain four, the file will trigger error 8. ReCal would return incorrect results if it attempted to analyze it. There are two ways to solve this problem: the first would be to delete the 4th column from rows 11-20, leaving 20 rows with 3 columns each; this solution only works for ReCal3. The second is to add the missing data to rows 1-10, creating 20 rows of 4 columns each which could be analyzed in ReCal2 or ReCal3 depending on the nature of the data.
What applications is ReCal compatible with? ReCal can read data from any software application that has the ability to “Save As” or “Export” files in CSV format. (More on CSV and why ReCal uses it here.) This includes SPSS, Stata, S-PLUS, SAS, Excel, Google Docs, MS Access, OpenOffice/NeoOffice, Minitab (as of version 15), and more.
How can I be sure ReCal’s results are accurate? Unfortunately, there is no way I can show a priori that ReCal will furnish accurate results for all possible datasets. 100% certainty is only possible in certain branches of pure mathematics; in the real world all sorts of things can go wrong. With ReCal, for example, very rarely people will format their data incorrectly yet manage to see results anyway, which of course will be incorrect. Barring this scenario, I would certainly encourage you to test ReCal against other reliability calculators, especially if its results appear flawed. A list of alternative reliability calculators can be found here.
I would prefer to use an Excel spreadsheet to calculate intercoder reliability. Why should I use ReCal? Here are several reasons why ReCal is superior to spreadsheets for reliability calculation:
- From a programming standpoint, Excel’s basic function language is not very sophisticated in its handling of arrays, which are essential for calculating reliability. This means that VBA (Visual Basic for Applications, a Microsoft proprietary programming language that works inside Excel spreadsheets) would need to be used. Unfortunately, MS has removed VBA from the current Mac version of Office, meaning that an ICRC macro written in VBA would be useless for non-Windows users.
- Even if VBA hadn’t been removed from Office for Mac, a VBA macro would still restrict ReCal usage to Excel users. ReCal doesn’t require anything other than a web browser and an internet connection.
- PHP generally runs faster than VBA, which you’ll notice if you compare ReCal to PRAM (although technically PRAM is written in VB rather than VBA, the languages are very closely related as their names indicate).
- I don’t know of any publicly available spreadsheets that calculate Scott’s Pi, Cohen’s Kappa, or Krippendorff’s Alpha. (In fact, this was the main reason I created ReCal in the first place.) If you know of any, please let me know and I’ll link to them.
I need help understanding/interpreting/improving my results; what resources are available to me? Probably the best intercoder reliability resource on the web is Matthew Lombard’s site, which presents the basics of how to calculate, use, and interpret reliability statistics. Beyond that, you may be interested in the extended discussions found in Content Analysis: an Introduction to Its Methodology by Klaus Krippendorff and/or The Content Analysis Guidebook by Kim Neuendorf. Finally, if you have a question of general interest that isn’t already answered on either this site or Lombard’s, you can ask me and I’ll answer it publicly if I can.
How do I create CSV files? The specific instructions on how to do this differ depending on which application you are using, but in Excel and SPSS I believe you use either the “Export” or “Save As” command and select “CSV” or “Comma-Separated Values” as your file format. It is important to remember that merely changing a file’s extension manually to “.csv” does not convert the file format itself to CSV; you must use your application’s Export or Save As function.
What are CSV files and why does ReCal require them? CSV stands for “Comma-Separated Values” and is a non-proprietary method of representing tabular (spreadsheet) data that can be read and exported by a wide range of applications (wikipedia entry here). ReCal requires CSV files because doing so maximizes compatibility across software applications and operating systems.
Why doesn’t ReCal work with Excel/SPSS/Word/[insert other proprietary software package here] files? See the answer to the question above.
My data cannot easily be reformatted to conform to ReCal’s specifications. What alternatives are available? Click here to view a list of alternative reliability calculators.
What happens to my data when I submit it to ReCal? Your data file is uploaded to a private folder on my web hosting account for troubleshooting purposes. In lieu of actual user feedback, reviewing user data directly is the only way I can identify and fix bugs. See the fine print for more info.
Is there a version of ReCal3 forthcoming that will perform similar analyses on ordinal and interval data and that will accept missing data? The short answer to this is not immediately, as I am (as of fall 2009) reading for my exams and won’t have time to think about implementing major new features until after January at the earliest. I address the two components of this question in greater detail immediately below:
- Ordinal/interval-level data: I personally know far less about these kinds of coefficients than I know about nominal coefficients. I know that Spearman’s rho can be used to calculate reliability for ordinal-level data, while Lin’s concordance can be used for interval or ratio data, but I couldn’t say whether these are the most widely accepted coefficients for these data levels. If any experts in this area are interested in consulting with me on the development of online calculators for non-nominal data, that would greatly expedite the process.
- Missing data: Of all the coefficients ReCal computes, only Krippendorff’s alpha is equipped to handle missing data (though it is not currently configured to do so in ReCal). Thus, one solution here would be to add new code to ReCal that would integrate missing data compatibility into its implementation of Krippendorff’s alpha. Another solution would be to implement a strategy of casewise deletion—that is, deleting all cases which were not evaluated by all coders. This strategy can be implemented either on the server side (through code that has yet to be written) or on the client side (wherein the researcher manually removes all cases with missing data). Casewise deletion is probably best used when the number of incomplete cases is small, but then, content analysis data sets with large amounts of missing reliability data are problematic from a broader validity standpoint.
Generally, if you’re interested in certain new features, please let me know via comment or email—knowing that users actively want a particular function makes me more likely to develop it.
What are the functional differences between ReCal2 and ReCal3? There are two main differences between the two:
- ReCal for 2 Coders (ReCal2) can calculate reliabilities for multiple variables at once, whereas ReCal for 3+ Coders (ReCal3) can only calculate reliability for one variable at a time. If you have several variables all coded by two coders, the former edition might save you some time.
- Although the two utilities share a formally identical data format, they make very different assumptions about what that data represents. ReCal2 assumes that data columns come in pairs, i.e. that columns 1 and 2 represent two coders’ codes for a single variable, cols 3 and 4 represent two coders’ codes for a different variable, etc. By contrast, ReCal3 assumes that each column in the input file represents a different coder’s work on a single variable. Therefore, the same 6-column CSV file would represent 3 different variables coded by 2 coders each to ReCal2, while ReCal3 would interpret it as one variable coded by 6 different coders. For this reason, the only files for which both ReCal2 and ReCal3 will give accurate results are those containing only 2 columns/coders. Submitting data intended for one edition to the other will generate incorrect results!
This post is password protected. To view it please enter your password below: