Comments for dfreelon.org http://dfreelon.org Fri, 30 Jul 2010 00:59:45 +0000 hourly 1 http://wordpress.org/?v=3.0 Comment on ReCal3: Reliability for 3+ Coders by Brian Louie http://dfreelon.org/utils/recalfront/recal3/#comment-749 Brian Louie Fri, 30 Jul 2010 00:59:45 +0000 http://dfreelon.org/?page_id=18#comment-749 Is there a probability p value that is associated with Fleiss' Kappa? If so, it would be great to see this integrated as a feature in future versions. Is there a probability p value that is associated with Fleiss’ Kappa? If so, it would be great to see this integrated as a feature in future versions.

]]> Comment on ReCal2: Reliability for 2 Coders by John http://dfreelon.org/utils/recalfront/recal2/#comment-747 John Sun, 25 Jul 2010 21:39:45 +0000 http://dfreelon.org/?page_id=17#comment-747 One word. Awesome! This program gets the job done. Good work and keep bring it. One word. Awesome! This program gets the job done. Good work and keep bring it.

]]> Comment on ReCal3: Reliability for 3+ Coders by Fawzy http://dfreelon.org/utils/recalfront/recal3/#comment-742 Fawzy Mon, 28 Jun 2010 10:15:04 +0000 http://dfreelon.org/?page_id=18#comment-742 wonderful, handy and very helpful wonderful, handy and very helpful

]]> Comment on Intercoder Reliability Worked Examples by Toi http://dfreelon.org/2009/10/30/intercoder-reliability-worked-examples/#comment-728 Toi Mon, 17 May 2010 14:51:28 +0000 http://dfreelon.org/?p=56#comment-728 Hi Deen, I'm a qualitative researcher for a large non-profit. We're coding some video from kids' video diaries. Everything in ReCal seems to be working well EXCEPT when we're in near complete agreement, and there is only one divergent rating. So for example, with three coders, we can have 10 different cases for the same variable. So out of the 30 judgments, we can have 29 1s and one 2, and the output on the Krippendorff is a fat 0. I must admit that I am not well-versed in stats, so, this is a little out of my realm. All of the other statisticians we have on staff have never heard of Krippendorff's alpha, so they can't really help explain why this is the case. Can you help? Thanks, Toi Hi Deen,

I’m a qualitative researcher for a large non-profit. We’re coding some video from kids’ video diaries.

Everything in ReCal seems to be working well EXCEPT when we’re in near complete agreement, and there is only one divergent rating. So for example, with three coders, we can have 10 different cases for the same variable. So out of the 30 judgments, we can have 29 1s and one 2, and the output on the Krippendorff is a fat 0.

I must admit that I am not well-versed in stats, so, this is a little out of my realm. All of the other statisticians we have on staff have never heard of Krippendorff’s alpha, so they can’t really help explain why this is the case.

Can you help?

Thanks,

Toi

]]> Comment on ReCal: reliability calculation for the masses by Ostkirchen Gabriele; Lüdecke Hildegard http://dfreelon.org/utils/recalfront/#comment-727 Ostkirchen Gabriele; Lüdecke Hildegard Tue, 11 May 2010 15:13:56 +0000 http://dfreelon.org/?page_id=16#comment-727 Dear Mr. Freelon, would it be possible, you send us your opinion on our problem? Since we calculate intercoder-reliability for different sub-studies of our project with your programme we easily get the reliabilty-results, including the amount of coder-differences. We learnt that our coding-system ameliorated over the time, and we started to use your programme to help us to sharpen up our category-system by adding examples and by reformulating the rules. Hildegard did a complete analysis of the mistakes (disaggrements) found by ReCal2 and up to now 5 mistakes are remaining. In her thesis she wants to present the first calculation with about 60 disagreement, than a table with all commented disaggrements and then she executes a new reliability analysis and of course nearly all categories show an aggrement of 100%. Can we do it like this? Or do you propose another way? We find it very necessary to sharpen up our system through the process we explained above. An expert of methods like you, has he any arguments against this procedure? Thanking you in anticipation for you soon reply we remain with best wishes Gaby and Hildegard Dear Mr. Freelon, would it be possible, you send us your opinion on our problem? Since we calculate intercoder-reliability for different sub-studies of our project with your programme we easily get the reliabilty-results, including the amount of coder-differences. We learnt that our coding-system ameliorated over the time, and we started to use your programme to help us to sharpen up our category-system by adding examples and by reformulating the rules. Hildegard did a complete analysis of the mistakes (disaggrements) found by ReCal2 and up to now 5 mistakes are remaining. In her thesis she wants to present the first calculation with about 60 disagreement, than a table with all commented disaggrements and then she executes a new reliability analysis and of course nearly all categories show an aggrement of 100%. Can we do it like this? Or do you propose another way? We find it very necessary to sharpen up our system through the process we explained above. An expert of methods like you, has he any arguments against this procedure? Thanking you in anticipation for you soon reply we remain with best wishes Gaby and Hildegard

]]> Comment on ReCal3: Reliability for 3+ Coders by dfreelon http://dfreelon.org/utils/recalfront/recal3/#comment-725 dfreelon Mon, 03 May 2010 16:11:08 +0000 http://dfreelon.org/?page_id=18#comment-725 Julie, It does not, and I suppose here is as good a place as any to explain why. The main reason is that I couldn't think of an intuitive way to do it. ReCal2 simply assumes that every adjacent column pair is a new variable, but ReCal3 can't do that since it has to be able to handle any number of coders starting with two. I briefly considered requiring users to enter the number of coders for ReCal3, which could then serve as a columnar multiple to produce that ReCal2 effect, but rejected this idea as too complicated. ReCal's error rate is fairly high (first-time users almost always make at least one error before completing a successful run) and I didn't want to make the system any more complicated than it had to be. Anyone who's ever designed a user interface will tell you that every extra hoop you force the user through generates more errors and inevitably increases attrition. I realize that my decision privileges new users at the expense of more savvy ones, but I would rather make ReCal more accessible than less, even if that means a bit more clicking for ReCal3 users. And isn't that what undergrad coders and research assistants are for anyway? Julie,

It does not, and I suppose here is as good a place as any to explain why. The main reason is that I couldn’t think of an intuitive way to do it. ReCal2 simply assumes that every adjacent column pair is a new variable, but ReCal3 can’t do that since it has to be able to handle any number of coders starting with two. I briefly considered requiring users to enter the number of coders for ReCal3, which could then serve as a columnar multiple to produce that ReCal2 effect, but rejected this idea as too complicated.

ReCal’s error rate is fairly high (first-time users almost always make at least one error before completing a successful run) and I didn’t want to make the system any more complicated than it had to be. Anyone who’s ever designed a user interface will tell you that every extra hoop you force the user through generates more errors and inevitably increases attrition. I realize that my decision privileges new users at the expense of more savvy ones, but I would rather make ReCal more accessible than less, even if that means a bit more clicking for ReCal3 users. And isn’t that what undergrad coders and research assistants are for anyway?

]]> Comment on ReCal3: Reliability for 3+ Coders by Julie http://dfreelon.org/utils/recalfront/recal3/#comment-724 Julie Mon, 03 May 2010 15:31:17 +0000 http://dfreelon.org/?page_id=18#comment-724 Does ReCal3 allow for more than one variable, the way that ReCal2 does? I.e., for 3 coders, can the first three columns represent one variable and the next three columns represent a second variable? Does ReCal3 allow for more than one variable, the way that ReCal2 does? I.e., for 3 coders, can the first three columns represent one variable and the next three columns represent a second variable?

]]> Comment on ReCal FAQ and Troubleshooting Page by dfreelon http://dfreelon.org/2009/02/20/frequently-asked-questions-about-recal/#comment-695 dfreelon Wed, 14 Apr 2010 16:32:36 +0000 http://dfreelon.org/?p=48#comment-695 Andrew, Thanks for letting me know; I'll take your suggestion under advisement next time I have occasion to add features to ReCal. If anyone else is interested in weighted kappa or other features, please leave a comment so that I can gauge demand. Andrew,

Thanks for letting me know; I’ll take your suggestion under advisement next time I have occasion to add features to ReCal. If anyone else is interested in weighted kappa or other features, please leave a comment so that I can gauge demand.

]]> Comment on ReCal FAQ and Troubleshooting Page by Andrew Maxwell http://dfreelon.org/2009/02/20/frequently-asked-questions-about-recal/#comment-694 Andrew Maxwell Wed, 14 Apr 2010 12:48:24 +0000 http://dfreelon.org/?p=48#comment-694 I am interested in the difference between analyzing the data for simple "matches" versus "no-match" and the weighted Kappa, that considers how far away are the misses. SPSS does not calculate weighted kappa, do you? Thanks Andrew I am interested in the difference between analyzing the data for simple “matches” versus “no-match” and the weighted Kappa, that considers how far away are the misses.

SPSS does not calculate weighted kappa, do you?

Thanks

Andrew

]]> Comment on ReCal: reliability calculation for the masses by Paul Lambert http://dfreelon.org/utils/recalfront/#comment-690 Paul Lambert Mon, 12 Apr 2010 03:54:38 +0000 http://dfreelon.org/?page_id=16#comment-690 Hi There, Thanks for making this tool available as it provides a quick and easy way to work out reliability. Well done! Paul, South Australia Hi There, Thanks for making this tool available as it provides a quick and easy way to work out reliability.

Well done!

Paul, South Australia

]]>