ReCal: reliability calculation for the masses

posted September 22nd, 2008 by dfreelon

***UPDATE 06/21/2010*** ReCal OIR (for ordinal, interval, and ratio data) is now available for public testing. Please report all bugs and issues and double-check all coefficients before presenting or publishing.

ReCal (”Reliability Calculator”) is an online utility that computes intercoder/interrater reliability coefficients for nominal, ordinal, interval, or ratio-level data. It is compatible with Excel, SPSS, STATA, OpenOffice, Google Docs, and any other database, spreadsheet, or statistical application that can export comma-separated (CSV), tab-separated (TSV), or semicolon-delimited data files. Additional technical details, worked examples, and evidence of ReCal’s accuracy are available in the peer-reviewed article “ReCal: Intercoder Reliability Calculation as a Web Service.”

ReCal consists of three autonomous modules each specialized for different types of data. The following table will help you select the module that best fits your data. (If you do not know whether your data are considered nominal, ordinal, interval, or ratio, please consult this Wikipedia article to find out more about these levels of measurement.) For:

Level of measurement N of coders Use
Nominal 2 coders only ReCal2
Nominal 3 or more coders ReCal3
Ordinal, interval, or ratio Any N of coders ReCal OIR

Please visit the ReCal FAQ/troubleshooting page if you have questions or are experiencing difficulty getting ReCal to work with your data. If you still have questions please contact me directly.

ReCal’s source code was last updated on 06/21/2010. To date, ReCal (2, 3, and OIR combined) has been successfully executed a total of times by persons other than the developer.
32 comments.

32 comments for “ReCal: reliability calculation for the masses”

  1. Wow! This simple tool instantly makes content analysis a more desirable and easier method! Thank you!

    2. posted December 20th, 2008 at 4:26 pm PST by Mohamad Elmasry

  2. This was very simple to use, and (I think) it worked beautifully. Thanks for building it and making it available.

    3. posted December 28th, 2008 at 7:56 pm PST by Geoff Archer

  3. Wonderful tool, and I’ll recommended it to others. ReCal is especially helpful for data in an Excel spreadsheet, because Excel has no easy way for calculating intercoder reliability.

    4. posted February 14th, 2009 at 2:56 pm PST by Christopher Martin

  4. I’m using this to create some examples for the research methods class I’m teaching. Thanks for providing such a helpful tool!

    5. posted February 20th, 2009 at 2:31 pm PST by Dave Voelker

  5. It’s quite useful. Thanks a lot!

    6. posted March 18th, 2009 at 11:31 pm PST by Liao

  6. This is so useful. I was looking everywhere for a decent app, and to have it web-based is just great!

    Thanks for helping me to beat deadline on a big (for me, anyway) conference paper.

    7. posted March 31st, 2009 at 9:00 pm PST by Mark P.

  7. A very useful product, but I would strongly encourage you to give users a viable option for exporting the results. I have tried every way know to man and I just can get the data into a useful (reportable) format.

    8. posted April 28th, 2009 at 8:35 am PST by Eric

  8. Thanks. This was extremely helpful. So far I have only used the tool with sample data but will return when the second coder has finished.

    9. posted June 5th, 2009 at 11:02 am PST by Carole Bell

  9. very good job!

    thank you so much! be sure that i will cite the link in my Phd thesis.

    keep on your useful work!

    regards,

    Chrysi Rapanta

    10. posted June 8th, 2009 at 6:10 am PST by Chrysi Rapanta

  10. this was so easy to use- thank you! It saved me alot of work!

    11. posted June 19th, 2009 at 12:04 pm PST by patrice

  11. This was absolutely amazing (and absolutely free); so quick and simple and the guidelines were excellent and easy to follow. Thank you SO much!

    12. posted July 2nd, 2009 at 10:08 am PST by Vania

  12. A really useful tool. Many thanks

    13. posted July 10th, 2009 at 9:41 am PST by David Deacon

  13. Many thanks for providing this service. Infact, this is great service for those who intensely need it. Once again thank you very much and please try more to help others in the same way.

    WISAL ASS Professor, KUST

    14. posted July 18th, 2009 at 6:17 am PST by wisal

  14. Absolutely brilliant!

    15. posted July 30th, 2009 at 3:02 am PST by Scott Kinder

  15. Outstanding service. Easy to use program with clear and concise output. Many thanks.

    16. posted September 4th, 2009 at 3:27 am PST by James Sauer

  16. Very handy! Thanks for making this available.

    17. posted September 15th, 2009 at 11:19 am PST by Annie Pettit

  17. It certainly saves me lots of sleepless nights looking for the solution. Timely for my final touches on the thesis. Certainly an excellent invention.And thanks for earlier reply to my email.

    18. posted October 18th, 2009 at 12:49 am PST by KinGan

  18. Great tool…this is first time I am using ReCal and I have only words of admiration for it! So simple yet so great. I will definitely reference it in paper I am ready to publish. Thanks a lot, o

    19. posted October 24th, 2009 at 10:22 am PST by Ognjen Brborovic

  19. This is very easy and quick reliability test.

    20. posted October 29th, 2009 at 10:20 pm PST by Chansuk Kang

  20. Thanks for your tool!!

    21. posted December 6th, 2009 at 7:03 am PST by snowfield

  21. Thank you so much for making this available to frantic students! Wonderfully helpful and easy to use!! Appreciatively, Michele

    22. posted December 7th, 2009 at 5:53 pm PST by Michele

  22. Thank you so much for a great tool. But, I hope you can help me clear up a discrepancy I’ve noticed in my results for variables that have the same number of agreements/disagreements. For example, variable 1 has 26 agreements and 1 disagreement. So does variable 3. So does variable 5. Yet, the results for variable 1 are: 96.3% agreement and Scott’s pi of 0.924. The results for variable 3 are: 96.3% agreement and Scott’s pi of 0.914. The results for variable 5 are: 96.3% agreement and Scott’s pi of 0.886. Can you please tell me why the Scott’s pi is different for each variable when all the raw data for them is the same (ie same number of agreements and disagreements)? This scenario has occurred on three separate occasions when I’ve submitted my .csv files for analysis. Best wishes Dianne

    EDIT: I have answered this question.

    23. posted December 9th, 2009 at 6:27 pm PST by dianne

  23. Ok, I am stumped. How can I have a percent agreement of .97 and a Scott’s Pi of-.015? I have two coders coding either Yes (1) or No (0) for the presence of a variable. What am I doing wrong. I find when calculating by hand I get similar results (off by a decimal or so). When using RECAL or calculating Scotts Pi with more than two categories, I don’t get negative Scotts Pi when the percent agreement is high.

    Thanks so much for sharing your program and answering my question if you have the time.

    Happy Holidays!

    Sonya

    EDIT: I have answered this question.

    24. posted December 14th, 2009 at 5:38 pm PST by Sonya

  24. easy and convenient to use. It is faster than SAS MACRO that I am using.

    Great job!!!

    Bob

    25. posted December 18th, 2009 at 10:51 am PST by Bob

  25. Thanks for this great tool , before I visited this website I used PRAM and the macro of Krippendorf in SPSS. But this tool is indeed faster and very handy! My ompliments to you! I shall recommend this website to toher researchers. Eric

    26. posted January 6th, 2010 at 6:39 am PST by Eric Poldner

  26. I will use this site for testing my content analysis results during my phd research! The topic of my research is quantitative content analysis of student’s reflective writings in Teacher Education. Eric

    27. posted January 6th, 2010 at 6:47 am PST by Eric Poldner

  27. Very usefull. Thank you for your effort on making content analysis an easier job. I will tell everyone about this tool. Keep up the good work!

    28. posted January 16th, 2010 at 9:16 am PST by Jonas

  28. Dear Mr. Freelon, you helped us a lot. We have a huge amount of data out of different studies in which we analysed children’s stories about their pain experiences concerning an accident by bicycle, headaches, abdominal pain and getting a vaccination by the doctor. These stories were all analysed by 2 independent coders and we already knew, that only calculating the percentage agreement is not enough. Then we read the paper of Prof. Lombard and got to know, that you developped a programm, doing all necessary calculations to present the intercoder – reliability as requested.

    We can’t thank you enough for this great work. It helps us and atleast about 20 students who are working in our project.

    Thank’s a lot. Others will write you, as soon as they use your programm.

    Best regard Gaby Ostkirchen Ivana Tolic

    29. posted January 27th, 2010 at 2:47 am PST by Ostkirchen Gabriele; Tolic Ivana

  29. Dear Mr. Freelon, thank you for your helpful tool. I am member of the study-group of Mrs. Dr. Ostkirchen.

    After calculating the reliabilities we hat categories with bad results. That’s why we startet an analysis of the mistakes. This helped us to improve our category system. Best regard Hildegard Lüdecke

    30. posted March 18th, 2010 at 6:16 am PST by Ostkirchen Gabriele; Tolic Ivana

  30. Thank you, Deen! What a fantastic program — made my intercoder reliability calculation easy. Thanks as well for answering my questions via email. You’re the best!

    31. posted March 23rd, 2010 at 6:54 am PST by Matt J. Duffy

  31. Hi There, Thanks for making this tool available as it provides a quick and easy way to work out reliability.

    Well done!

    Paul, South Australia

    32. posted April 11th, 2010 at 7:54 pm PST by Paul Lambert

  32. Dear Mr. Freelon, would it be possible, you send us your opinion on our problem? Since we calculate intercoder-reliability for different sub-studies of our project with your programme we easily get the reliabilty-results, including the amount of coder-differences. We learnt that our coding-system ameliorated over the time, and we started to use your programme to help us to sharpen up our category-system by adding examples and by reformulating the rules. Hildegard did a complete analysis of the mistakes (disaggrements) found by ReCal2 and up to now 5 mistakes are remaining. In her thesis she wants to present the first calculation with about 60 disagreement, than a table with all commented disaggrements and then she executes a new reliability analysis and of course nearly all categories show an aggrement of 100%. Can we do it like this? Or do you propose another way? We find it very necessary to sharpen up our system through the process we explained above. An expert of methods like you, has he any arguments against this procedure? Thanking you in anticipation for you soon reply we remain with best wishes Gaby and Hildegard

    33. posted May 11th, 2010 at 7:13 am PST by Ostkirchen Gabriele; Lüdecke Hildegard

Leave a comment

dfreelon.org is the personal web site of Deen Freelon. Any opinions expressed herein are his alone and do not necessarily reflect the views of his employers or sponsoring organizations.
all content © dfreelon.org 2008. Design credits: Arcsin (template), aNieto2k (Wordpress theme), dfreelon (additional theme modifications)