***UPDATED 5/12/09; see the bottom of this post.***

The purpose of this post is to explain the following ReCal error:

*Scott’s pi/Cohen’s kappa/Fleiss’ kappa could not be calculated for this variable due to invariant values.

You should only see this error when two conditions apply simultaneously to your data: 1) all of your coders have attained 100% agreement and 2) they have all selected the same variable value for every unit of analysis. (If you see it under any other circumstances, please let me know, as it means the code is flawed and needs to be fixed.) For example, assume a five-unit of analysis reliability sample of a binary variable with possible values 1 and 0. If both coders decide that all five units should be rated 0 or that all five units should be rated 1, the “invariant values” scenario, or IVS (I’m sure someone’s come up with a better name for it) occurs. Scott’s pi, Cohen’s kappa, and Fleiss’ Kappa are all undefined when this happens (Fleiss’ kappa is slightly more robust in that the more coders in the reliability pool, the less likely they all are to choose the same value for every unit).

The reason for this is that when the two IVS conditions obtain, the mathematical definition of expected agreement for these coefficients is 1. Let’s take a look at the example specified in the previous paragraph:

As you can see, the IVS is in effect because all values for this variable are equal to 1. Percent agreement for this variable is obviously 100%; observed agreement is 1. The number of 1s for coders 1 and 2 is 5 for both, for a total of 10 decisions. The first, and only, joint marginal proportion for Scott’s pi is equal to (5 + 5) / 10 = 1. Expected agreement then becomes 1² = 1. The Scott’s pi equation would thus be:
(observed - expected) / (1 - expected) = (1 - 1) / (1 - 1)

But this leads to division by zero, which basic arithmetic tells us is undefined. Thus, Scott’s pi (and Cohen’s kappa, which behaves similarly) are undefined under the IVS. Fleiss’ kappa is similarly nonexistent when all coders assign the same value to all units.

Krippendorff’s alpha, on the other hand, is immune to this problem. Recall its basic form:
a = 1 - Do/De
When observed disagreement (D_o) is 0, D_o/D_e simplifies to 0, and a equals 1. This is one instance in which Krippendorff’s alpha improves upon its predecessors.

Update 5/12/09: A colleague recently pointed out to me that the above reasoning regarding the nature of Krippendorff’s alpha under invariant values is incorrect. It will in fact be undefined just like Scott’s pi and Cohen’s kappa due to the fact that D_e = 0 when all coders select the same value for all units, thus invalidating the entire expression. ReCal’s code has been amended to address this flaw.

Whew . . . just made it through my AOIR presentation relatively unscathed. I think the crowd was a bit cool to my subject area—I only got one comment at the end, a compliment, and from the president of AOIR no less. Still, I was hoping folks would grill me a bit more, but then I suppose there’s always a chance someone might have something to say after all the session presentations are done. The panel is good so far, but not thematically unified—maybe the audience wasn’t familiar with the general research area. I really should try and get on a proposed panel for my next conference; they seem to be draw folks with some level of topic expertise.