Comments for dfreelon.org

Comment on ReCal: The Fine Print by dfreelon

dfreelon — Tue, 09 Mar 2010 20:15:28 +0000

Dolores,

Thank you for your question. The math used in ReCal can be found in my Worked Examples document here.

This document contains all of the math that is implemented in the program itself. Its underlying code has not yet been released (you’d need to know PHP to understand it anyway), but you can confirm that its output is correct by using these examples to calculate your own reliability coefficients by hand if you wish. If you use this document, please make sure to cite it.

Please let me know if you have further questions. Best, ~DEEN

Comment on ReCal: The Fine Print by Dolores Gonzalez

Dolores Gonzalez — Tue, 09 Mar 2010 19:53:44 +0000

I need to know the calculus that are developed in order to obtain the intercoder reliability since I am writing a paper and need to specify how I obtain this reliability.
Thanks,
Dolores

Comment on ReCal error log entry #1: invariant values (updated 5/12/09) by dfreelon

dfreelon — Tue, 16 Feb 2010 17:56:30 +0000

Brian,

Thank you for your comment. First let me assure you that ReCal did function as intended, reporting invariant values exceptions only for those variables for which the two conditions you mention obtain. Because the invariant values exception is calculated on a per-variable basis, it is not intended to refer to the situation in which all the cases in a multi-variable file are set to the same variable value.

I believe the confusion here stems from differing uses of the term “case”—you seem to be using it as a synonym for “variable” (i.e. column pair), whereas I (prior to editing for clarity) used it in two conflicting senses: yours as well as as a synonym for “unit of analysis” (i.e. corresponding value pair in the same row). ReCal’s output also uses this latter sense; the “N cases” is always the total number of rows in the file, since all variables in a multi-variable file must contain the same number of cases.

I can see how this would be confusing, so I have amended this post to refer to “units of analysis” and “variables,” removing all uses of the term “case.” I hope this clarifies my meaning.

Comment on ReCal error log entry #1: invariant values (updated 5/12/09) by Brian

Brian — Mon, 15 Feb 2010 17:15:32 +0000

Dear Deen, I’m getting this error when I’m running Recal on my coding results, but I’m not sure if the conditions are exactly as you pointed out above:

1) all of your coders have attained 100% agreement and
2) they have all selected the same variable value for every case.

In my results “undefined” is displayed when indeed the coders have attained 100% agreement but not when they selected the same variables for “every case”. It only shows undefined for the cases where the coders have 100% agreement and selected the same variable value.

To clarify, I have coding results for 2 coders which evaluated 16 cases on the presence of 54 variables (0 = absent, 1 = present). And only the cases where there is a 100% agreement the undefined error is shown. Not when all the cases have a 100% agreement.

Maybe this is exactly what you meant but your example and the corresponding text seemed to indicate that the error should only occur when there’s 100& agreement in all the cases.

Yours truly,

Brian

Comment on ReCal: reliability calculation for the masses by Ostkirchen Gabriele; Tolic Ivana

Ostkirchen Gabriele; Tolic Ivana — Wed, 27 Jan 2010 10:47:35 +0000

Dear Mr. Freelon,
you helped us a lot. We have a huge amount of data
out of different studies in which we analysed children’s
stories about their pain experiences concerning an
accident by bicycle, headaches, abdominal pain and
getting a vaccination by the doctor. These stories
were all analysed by 2 independent coders and we
already knew, that only calculating the percentage
agreement is not enough. Then we read the paper of
Prof. Lombard and got to know, that you developped
a programm, doing all necessary calculations to
present the intercoder - reliability as requested.

We can’t thank you enough for this great work.
It helps us and atleast about 20 students
who are working in our project.

Thank’s a lot. Others will write you, as soon as
they use your programm.

Best regard
Gaby Ostkirchen
Ivana Tolic

Comment on ReCal: reliability calculation for the masses by Jonas

Jonas — Sat, 16 Jan 2010 17:16:46 +0000

Very usefull. Thank you for your effort on making content analysis an easier job. I will tell everyone about this tool. Keep up the good work!

Comment on ReCal: reliability calculation for the masses by Eric Poldner

Eric Poldner — Wed, 06 Jan 2010 14:47:03 +0000

I will use this site for testing my content analysis results during my phd research! The topic of my research is quantitative content analysis of student’s reflective writings in Teacher Education. Eric

Comment on ReCal: reliability calculation for the masses by Eric Poldner

Eric Poldner — Wed, 06 Jan 2010 14:39:36 +0000

Thanks for this great tool , before I visited this website I used PRAM and the macro of Krippendorf in SPSS. But this tool is indeed faster and very handy! My ompliments to you! I shall recommend this website to toher researchers. Eric

Comment on ReCal: reliability calculation for the masses by Bob

Bob — Fri, 18 Dec 2009 18:51:27 +0000

easy and convenient to use. It is faster than SAS MACRO that I am using.

Great job!!!

Bob

Comment on ReCal: reliability calculation for the masses by Sonya

Sonya — Tue, 15 Dec 2009 01:38:12 +0000

Ok, I am stumped. How can I have a percent agreement of .97 and a Scott’s Pi of-.015? I have two coders coding either Yes (1) or No (0) for the presence of a variable. What am I doing wrong. I find when calculating by hand I get similar results (off by a decimal or so). When using RECAL or calculating Scotts Pi with more than two categories, I don’t get negative Scotts Pi when the percent agreement is high.

Thanks so much for sharing your program and answering my question if you have the time.

Happy Holidays!

Sonya

EDIT: I have answered this question.