Despite the existence of functions to calculate percentage agreement and Kappa score on coding done by two or more coders in various software programs, there remains few published resources discussing this approach to qualitative data analysis. For the actual STEPS in coding and generating the percentages and scores, please refer to the references at the end. Here, I share some pointers after adopting this approach and interpreting the results in a full year project with in-depth interview data with three coders.
Important considerations:
Pros of this approach:
Cons of this approach:
Resources*
ATLAS.ti 8 Online HowTo Documents – Inter-Coder Agreement Analysis with ATLAS.ti 8 Windows. (2019). https://atlasti.com/manuals-docs/
MAXQDA 2018 Online Manual: Chapter 24 - Teamwork - Intercoder Agreement. (2019). https://www.maxqda.com/help-max18/teamwork/problem-intercoder-agreement-qualitative-research
NVivo 12 for Windows Online Help - Coding Comparison Query. (2019). https://help-nv.qsrinternational.com/12/win/v12.1.82-d3ea61/Content/queries/coding-comparison-query.htm
*I am not satisfied with the few work detailing this approach in the current literature and therefore have excluded them from this Resources list.
Important considerations:
- Ensure the coding framework for different coders is the same or largely the same if you merge projects. Otherwise, percentage agreement or Kappa score will not be generated/will be meaningless for codes/nodes that exist only in one project.
- Set rules on how much text to code, for example, whole paragraphs, sentences or just the key words. The percentages and scores will always be low if different coders select different amount of text.
- Do not rely solely on the generated numbers to decide on the degree of agreement. Always click on the results to see the text in the original transcripts and discuss the codes/nodes that are divergent.
- Do not get stuck on arriving at a high or perfect percentage agreement or Kappa. It is easy to get caught up in the process while losing sight of the purpose of this approach. If in doubt, refresh your memory on “Differences between Qualitative and Quantitative Research 101”.
- Remember, calculating percentage agreement and Kappa score with qualitative coding is a controversial topic. Some argue that it is imposing a quantitative mindset on qualitative data. Be extra cautious!
Pros of this approach:
- For the quantitative mind, the numbers provide reassurance that the resulting themes or theories are objective and rigorous.
- It provides a systematic way to eliminate codes/nodes that fall below a threshold percentage or score.
- Scientists love this! Some believe that any results not backed up by numbers and statistics are subjective and anecdotal.
Cons of this approach:
- It could take away the power of interpretation and data immersion, which are the fundamentals in qualitative analysis, if all the results need to be quantified.
- It is more time-consuming than other analysis methods, especially in setting up precise coding rules, merging projects, generating numbers and discussing the results.
- There is little published work on this topic. Researchers are often left on their own to figure out how to apply it in their projects.
Resources*
ATLAS.ti 8 Online HowTo Documents – Inter-Coder Agreement Analysis with ATLAS.ti 8 Windows. (2019). https://atlasti.com/manuals-docs/
MAXQDA 2018 Online Manual: Chapter 24 - Teamwork - Intercoder Agreement. (2019). https://www.maxqda.com/help-max18/teamwork/problem-intercoder-agreement-qualitative-research
NVivo 12 for Windows Online Help - Coding Comparison Query. (2019). https://help-nv.qsrinternational.com/12/win/v12.1.82-d3ea61/Content/queries/coding-comparison-query.htm
*I am not satisfied with the few work detailing this approach in the current literature and therefore have excluded them from this Resources list.