ReCal error log entry #1: invariant values (updated 5/12/09)

***UPDATED 5/12/09; see the bottom of this post.***

The purpose of this post is to explain the following ReCal error:

*Scott’s pi/Cohen’s kappa/Fleiss’ kappa could not be calculated for this variable due to invariant values.

You should only see this error when two conditions apply simultaneously to your data: 1) all of your coders have attained 100% agreement and 2) they have all selected the same variable value for every unit of analysis. (If you see it under any other circumstances, please let me know, as it means the code is flawed and needs to be fixed.) For example, assume a five-unit of analysis reliability sample of a binary variable with possible values 1 and 0. If both coders decide that all five units should be rated 0 or that all five units should be rated 1, the “invariant values” scenario, or IVS (I’m sure someone’s come up with a better name for it) occurs. Scott’s pi, Cohen’s kappa, and Fleiss’ Kappa are all undefined when this happens (Fleiss’ kappa is slightly more robust in that the more coders in the reliability pool, the less likely they all are to choose the same value for every unit).

The reason for this is that when the two IVS conditions obtain, the mathematical definition of expected agreement for these coefficients is 1. Let’s take a look at the example specified in the previous paragraph:

As you can see, the IVS is in effect because all values for this variable are equal to 1. Percent agreement for this variable is obviously 100%; observed agreement is 1. The number of 1s for coders 1 and 2 is 5 for both, for a total of 10 decisions. The first, and only, joint marginal proportion for Scott’s pi is equal to (5 + 5) / 10 = 1. Expected agreement then becomes 1² = 1. The Scott’s pi equation would thus be:
(observed - expected) / (1 - expected) = (1 - 1) / (1 - 1)

But this leads to division by zero, which basic arithmetic tells us is undefined. Thus, Scott’s pi (and Cohen’s kappa, which behaves similarly) are undefined under the IVS. Fleiss’ kappa is similarly nonexistent when all coders assign the same value to all units.

Krippendorff’s alpha, on the other hand, is immune to this problem. Recall its basic form:
a = 1 - Do/De
When observed disagreement (D_o) is 0, D_o/D_e simplifies to 0, and a equals 1. This is one instance in which Krippendorff’s alpha improves upon its predecessors.

Update 5/12/09: A colleague recently pointed out to me that the above reasoning regarding the nature of Krippendorff’s alpha under invariant values is incorrect. It will in fact be undefined just like Scott’s pi and Cohen’s kappa due to the fact that D_e = 0 when all coders select the same value for all units, thus invalidating the entire expression. ReCal’s code has been amended to address this flaw.