Categories &

Functions List

Function Reference: fleissKappa

statistics: kappa = fleissKappa (X)

statistics: kappa = fleissKappa (X, name, value)

statistics: [kappa, stats] = fleissKappa (…)

Fleiss’ kappa, the agreement among several raters beyond chance.

kappa = fleissKappa (X) returns Fleiss’ kappa for m raters who each assigned the same n subjects to categories, X holding one row per subject and one column per rater. The ratings may be numeric or logical, categorical, or text in a cell array of character vectors or a string array. A subject with a missing rating, NaN, an undefined category, an empty text or a missing string, is left out, every subject having to be rated by all m raters.

With N(i,j) the number of raters who put subject i in category j, p(j) the proportion of all ratings in category j, Pbar the mean over the subjects of (sum (N(i,:) .^ 2) - m) / (m * (m - 1)), the proportion of pairs of raters who agree, and Pe = sum (p .^ 2), $$ \kappa = \frac{\bar P - P_e}{1 - P_e}, $$ 1 for perfect agreement and 0 for agreement no better than chance. With two raters it is Scott’s pi, not Cohen’s kappa, chance being estimated from the ratings of all raters together; cohenKappa measures the agreement of two raters from their own frequencies.

The following Name-Value pairs are supported:

NameValue
'Categories'The categories: numeric for numeric ratings, text for text or categorical ratings. The default is the sorted values of numeric ratings, the categories of categorical ratings, unused ones included, and the sorted texts of text ratings. A category no rater used has a kappa of NaN in stats.
'Exact'false (default) for Fleiss’ kappa, or true for the variant of Conger (1980), which takes each rater’s own frequencies into account and equals Cohen’s kappa for two raters. It has no test, and stats then holds NaN for its statistic and p-value.

[kappa, stats] = fleissKappa (…) also returns a structure with the fields

pothe observed agreement Pbar
pethe agreement expected by chance
sethe standard error of kappa when there is no agreement beyond chance (Fleiss, Nee and Landis, 1979)
zvalthe statistic of the test that kappa is 0, kappa / se
pthe two-sided p-value of that test
nthe number of subjects
ratersthe number of raters m
countsthe n-by-c matrix N
categoriesthe categories, a numeric column or a cell array of character vectors
categorykappathe kappa of each category, the agreement on whether a subject belongs to it
categoryzvalthe statistic of the test that each category’s kappa is 0
categorypthe two-sided p-value of that test

No confidence interval is given, there being no standard large-sample standard error for kappa away from 0. Kappa is undefined when chance alone predicts complete agreement, as when every rating falls in one category, and such ratings are refused. MATLAB has no Fleiss’ kappa, so fleissKappa is specific to Octave.

References:

  1. J. L. Fleiss. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378–382, 1971.
  2. J. L. Fleiss, J. C. M. Nee and J. R. Landis. Large sample variance of kappa in the case of different sets of raters. Psychological Bulletin, 86(5):974–977, 1979.
  3. A. J. Conger. Integration and generalization of kappas for multiple raters. Psychological Bulletin, 88(2):322–328, 1980.

See also: cohenKappa, friedman

Source Code: fleissKappa

Six psychiatrists diagnosed 30 patients (R's irr::diagnoses)

 D = [4 4 4 4 4 4; 2 2 2 5 5 5; 2 3 3 3 3 5; 5 5 5 5 5 5; 2 2 2 4 4 4;
      1 1 3 3 3 3; 3 3 3 3 5 5; 1 1 3 3 3 4; 1 1 4 4 4 4; 5 5 5 5 5 5;
      1 4 4 4 4 4; 1 2 4 4 4 4; 2 2 2 3 3 3; 1 4 4 4 4 4; 2 2 4 4 4 5;
      3 3 3 3 3 5; 1 1 1 4 5 5; 1 1 1 1 1 2; 2 2 4 4 4 4; 1 3 3 5 5 5;
      5 5 5 5 5 5; 2 4 4 4 4 4; 2 2 4 5 5 5; 1 1 4 4 4 4; 1 4 4 4 4 5;
      2 2 2 2 2 4; 1 1 1 1 5 5; 2 2 4 4 4 4; 1 3 3 3 3 3; 5 5 5 5 5 5];
 [kappa, stats] = fleissKappa (D)
kappa = 0.4302
stats =

  scalar structure containing the fields:

    po = 0.5556
    pe = 0.2199
    se = 0.024374
    zval = 17.652
    p = 9.8511e-70
    n = 30
    raters = 6
    counts =

       0   0   0   6   0
       0   3   0   0   3
       0   1   4   0   1
       0   0   0   0   6
       0   3   0   3   0
       2   0   4   0   0
       0   0   4   0   2
       2   0   3   1   0
       2   0   0   4   0
       0   0   0   0   6
       1   0   0   5   0
       1   1   0   4   0
       0   3   3   0   0
       1   0   0   5   0
       0   2   0   3   1
       0   0   5   0   1
       3   0   0   1   2
       5   1   0   0   0
       0   2   0   4   0
       1   0   2   0   3
       0   0   0   0   6
       0   1   0   5   0
       0   2   0   1   3
       2   0   0   4   0
       1   0   0   4   1
       0   5   0   1   0
       4   0   0   0   2
       0   2   0   4   0
       1   0   5   0   0
       0   0   0   0   6

    categories =

       1
       2
       3
       4
       5

    categorykappa =

       0.2448
       0.2448
       0.5200
       0.4711
       0.5661

    categoryzval =

        5.1920
        5.1920
       11.0309
        9.9941
       12.0092

    categoryp =

       2.0800e-07
       2.0800e-07
       2.7124e-28
       1.6172e-23
       3.1801e-33