cohenKappa
statistics: kappa = cohenKappa (x, y)
statistics: kappa = cohenKappa (T)
statistics: kappa = cohenKappa (…, name, value)
statistics: [kappa, ci] = cohenKappa (…)
statistics: [kappa, ci, stats] = cohenKappa (…)
Cohen’s kappa, the agreement between two raters beyond chance.
kappa = cohenKappa (x, y) returns Cohen’s kappa
for two raters who each assigned the same subjects to categories, the
ratings of the first in the vector x and of the second in y.
The ratings may be numeric or logical, categorical, or text: a character
matrix with one rating per row, a cell array of character vectors, or a
string array, the two raters’ of the same kind. A subject whose rating by
either rater is missing, NaN, an undefined category, an empty text
or a missing string, is left out.
kappa = cohenKappa (T) takes the agreement table
instead, a square matrix of counts whose element T(i,j) is
the number of subjects the first rater assigned to category i and
the second to category j.
With po the proportion of subjects on whom the raters agree and
pe the proportion expected by chance from each rater’s own
frequencies of the categories,
$$ \kappa = \frac{p_o - p_e}{1 - p_e}, $$
1 for perfect agreement, 0 for agreement no better than chance, and
negative for less. Weighted kappa counts partial agreement between
ordered categories, weighting each pair of categories i and
j by w(i,j), 1 when they are the same, in both po
and pe.
The following Name-Value pairs are supported:
| Name | Value |
|---|---|
'Weights' | 'none' (default), unweighted kappa;
'linear', w(i,j) = 1 - abs (i - j) / (c - 1);
'quadratic', w(i,j) = 1 - (i - j)^2 / (c - 1)^2, for
c categories in their order; or a c-by-c matrix of
agreement weights between 0 and 1, with ones on its diagonal. |
'Categories' | The categories, in their order: numeric for numeric ratings, text for text or categorical ratings. The default is the sorted values of numeric ratings, the categories of categorical ratings in their own order, unused ones included, and the sorted texts of text ratings. A category neither rater used counts towards c and spaces the others: placed between used categories it changes linear and quadratic kappa, placed after them it does not. Not accepted with T, whose rows and columns are the categories. |
'Alpha' | The significance level of the confidence interval, a scalar in the range (0,1). The default is 0.05. |
[kappa, ci] = cohenKappa (…) also returns the
100 (1 - alpha) percent confidence interval of kappa, from its
large-sample standard error (Fleiss, Cohen and Everitt, 1969), clipped to
[-1, 1].
[kappa, ci, stats] = cohenKappa (…) also
returns a structure with the fields
po | the observed agreement, weighted if weights are given |
pe | the agreement expected by chance |
se | the standard error of kappa that ci is based on |
zval | the statistic of the test that kappa is 0, kappa over its standard error when there is no agreement beyond chance |
p | the two-sided p-value of that test |
n | the number of subjects |
table | the agreement table |
categories | the categories, a numeric column or a cell array of character vectors |
weights | the matrix of weights used |
Kappa is undefined when chance alone predicts complete agreement, as when
both raters use a single category, and such ratings are refused.
Agreement among more than two raters is measured by fleissKappa.
MATLAB has no Cohen’s kappa, so cohenKappa is specific to Octave.
References:
See also: fleissKappa, crosstab, limitsOfAgreement
Source Code: cohenKappa
Two raters who sorted 20 subjects into three ordered grades
x = [1, 2, 3, 1, 2, 3, 2, 2, 1, 3, 3, 1, 2, 1, 3, 2, 2, 3, 1, 1]; y = [1, 2, 2, 1, 3, 3, 2, 1, 1, 3, 2, 1, 2, 2, 3, 2, 3, 3, 1, 2]; [kappa, ci] = cohenKappa (x, y)
kappa = 0.4737 ci = 0.1566 0.7907
Partial agreement between neighbouring grades counted
[kappa, ci] = cohenKappa (x, y, 'Weights', 'quadratic')
kappa = 0.7200 ci = 0.5218 0.9182