Comments on: ReCal: reliability calculation for the masses http://dfreelon.org Fri, 30 Jul 2010 00:59:45 +0000 hourly 1 http://wordpress.org/?v=3.0 By: Ostkirchen Gabriele; Lüdecke Hildegard http://dfreelon.org/utils/recalfront/#comment-727 Ostkirchen Gabriele; Lüdecke Hildegard Tue, 11 May 2010 15:13:56 +0000 http://dfreelon.org/?page_id=16#comment-727 Dear Mr. Freelon, would it be possible, you send us your opinion on our problem? Since we calculate intercoder-reliability for different sub-studies of our project with your programme we easily get the reliabilty-results, including the amount of coder-differences. We learnt that our coding-system ameliorated over the time, and we started to use your programme to help us to sharpen up our category-system by adding examples and by reformulating the rules. Hildegard did a complete analysis of the mistakes (disaggrements) found by ReCal2 and up to now 5 mistakes are remaining. In her thesis she wants to present the first calculation with about 60 disagreement, than a table with all commented disaggrements and then she executes a new reliability analysis and of course nearly all categories show an aggrement of 100%. Can we do it like this? Or do you propose another way? We find it very necessary to sharpen up our system through the process we explained above. An expert of methods like you, has he any arguments against this procedure? Thanking you in anticipation for you soon reply we remain with best wishes Gaby and Hildegard Dear Mr. Freelon, would it be possible, you send us your opinion on our problem? Since we calculate intercoder-reliability for different sub-studies of our project with your programme we easily get the reliabilty-results, including the amount of coder-differences. We learnt that our coding-system ameliorated over the time, and we started to use your programme to help us to sharpen up our category-system by adding examples and by reformulating the rules. Hildegard did a complete analysis of the mistakes (disaggrements) found by ReCal2 and up to now 5 mistakes are remaining. In her thesis she wants to present the first calculation with about 60 disagreement, than a table with all commented disaggrements and then she executes a new reliability analysis and of course nearly all categories show an aggrement of 100%. Can we do it like this? Or do you propose another way? We find it very necessary to sharpen up our system through the process we explained above. An expert of methods like you, has he any arguments against this procedure? Thanking you in anticipation for you soon reply we remain with best wishes Gaby and Hildegard

]]> By: Paul Lambert http://dfreelon.org/utils/recalfront/#comment-690 Paul Lambert Mon, 12 Apr 2010 03:54:38 +0000 http://dfreelon.org/?page_id=16#comment-690 Hi There, Thanks for making this tool available as it provides a quick and easy way to work out reliability. Well done! Paul, South Australia Hi There, Thanks for making this tool available as it provides a quick and easy way to work out reliability.

Well done!

Paul, South Australia

]]> By: Matt J. Duffy http://dfreelon.org/utils/recalfront/#comment-635 Matt J. Duffy Tue, 23 Mar 2010 14:54:06 +0000 http://dfreelon.org/?page_id=16#comment-635 Thank you, Deen! What a fantastic program -- made my intercoder reliability calculation easy. Thanks as well for answering my questions via email. You're the best! Thank you, Deen! What a fantastic program — made my intercoder reliability calculation easy. Thanks as well for answering my questions via email. You’re the best!

]]> By: Ostkirchen Gabriele; Tolic Ivana http://dfreelon.org/utils/recalfront/#comment-615 Ostkirchen Gabriele; Tolic Ivana Thu, 18 Mar 2010 14:16:10 +0000 http://dfreelon.org/?page_id=16#comment-615 Dear Mr. Freelon, thank you for your helpful tool. I am member of the study-group of Mrs. Dr. Ostkirchen. After calculating the reliabilities we hat categories with bad results. That's why we startet an analysis of the mistakes. This helped us to improve our category system. Best regard Hildegard Lüdecke Dear Mr. Freelon, thank you for your helpful tool. I am member of the study-group of Mrs. Dr. Ostkirchen.

After calculating the reliabilities we hat categories with bad results. That’s why we startet an analysis of the mistakes. This helped us to improve our category system. Best regard Hildegard Lüdecke

]]> By: Ostkirchen Gabriele; Tolic Ivana http://dfreelon.org/utils/recalfront/#comment-559 Ostkirchen Gabriele; Tolic Ivana Wed, 27 Jan 2010 10:47:35 +0000 http://dfreelon.org/?page_id=16#comment-559 Dear Mr. Freelon, you helped us a lot. We have a huge amount of data out of different studies in which we analysed children's stories about their pain experiences concerning an accident by bicycle, headaches, abdominal pain and getting a vaccination by the doctor. These stories were all analysed by 2 independent coders and we already knew, that only calculating the percentage agreement is not enough. Then we read the paper of Prof. Lombard and got to know, that you developped a programm, doing all necessary calculations to present the intercoder - reliability as requested. We can't thank you enough for this great work. It helps us and atleast about 20 students who are working in our project. Thank's a lot. Others will write you, as soon as they use your programm. Best regard Gaby Ostkirchen Ivana Tolic Dear Mr. Freelon, you helped us a lot. We have a huge amount of data out of different studies in which we analysed children’s stories about their pain experiences concerning an accident by bicycle, headaches, abdominal pain and getting a vaccination by the doctor. These stories were all analysed by 2 independent coders and we already knew, that only calculating the percentage agreement is not enough. Then we read the paper of Prof. Lombard and got to know, that you developped a programm, doing all necessary calculations to present the intercoder – reliability as requested.

We can’t thank you enough for this great work. It helps us and atleast about 20 students who are working in our project.

Thank’s a lot. Others will write you, as soon as they use your programm.

Best regard Gaby Ostkirchen Ivana Tolic

]]> By: Jonas http://dfreelon.org/utils/recalfront/#comment-553 Jonas Sat, 16 Jan 2010 17:16:46 +0000 http://dfreelon.org/?page_id=16#comment-553 Very usefull. Thank you for your effort on making content analysis an easier job. I will tell everyone about this tool. Keep up the good work! Very usefull. Thank you for your effort on making content analysis an easier job. I will tell everyone about this tool. Keep up the good work!

]]> By: Eric Poldner http://dfreelon.org/utils/recalfront/#comment-551 Eric Poldner Wed, 06 Jan 2010 14:47:03 +0000 http://dfreelon.org/?page_id=16#comment-551 I will use this site for testing my content analysis results during my phd research! The topic of my research is quantitative content analysis of student's reflective writings in Teacher Education. Eric I will use this site for testing my content analysis results during my phd research! The topic of my research is quantitative content analysis of student’s reflective writings in Teacher Education. Eric

]]> By: Eric Poldner http://dfreelon.org/utils/recalfront/#comment-550 Eric Poldner Wed, 06 Jan 2010 14:39:36 +0000 http://dfreelon.org/?page_id=16#comment-550 Thanks for this great tool , before I visited this website I used PRAM and the macro of Krippendorf in SPSS. But this tool is indeed faster and very handy! My ompliments to you! I shall recommend this website to toher researchers. Eric Thanks for this great tool , before I visited this website I used PRAM and the macro of Krippendorf in SPSS. But this tool is indeed faster and very handy! My ompliments to you! I shall recommend this website to toher researchers. Eric

]]> By: Bob http://dfreelon.org/utils/recalfront/#comment-516 Bob Fri, 18 Dec 2009 18:51:27 +0000 http://dfreelon.org/?page_id=16#comment-516 easy and convenient to use. It is faster than SAS MACRO that I am using. Great job!!! Bob easy and convenient to use. It is faster than SAS MACRO that I am using.

Great job!!!

Bob

]]> By: Sonya http://dfreelon.org/utils/recalfront/#comment-508 Sonya Tue, 15 Dec 2009 01:38:12 +0000 http://dfreelon.org/?page_id=16#comment-508 Ok, I am stumped. How can I have a percent agreement of .97 and a Scott's Pi of-.015? I have two coders coding either Yes (1) or No (0) for the presence of a variable. What am I doing wrong. I find when calculating by hand I get similar results (off by a decimal or so). When using RECAL or calculating Scotts Pi with more than two categories, I don't get negative Scotts Pi when the percent agreement is high. Thanks so much for sharing your program and answering my question if you have the time. Happy Holidays! Sonya <em>EDIT: I have <a href="http://dfreelon.org/2009/12/14/from-the-mailbag-121409/" rel="nofollow">answered this question.</a></em> Ok, I am stumped. How can I have a percent agreement of .97 and a Scott’s Pi of-.015? I have two coders coding either Yes (1) or No (0) for the presence of a variable. What am I doing wrong. I find when calculating by hand I get similar results (off by a decimal or so). When using RECAL or calculating Scotts Pi with more than two categories, I don’t get negative Scotts Pi when the percent agreement is high.

Thanks so much for sharing your program and answering my question if you have the time.

Happy Holidays!

Sonya

EDIT: I have answered this question.

]]>