Cohen’s kappa takes into account disagreement between the two raters, but not the degree of disagreement. This is especially relevant when the ratings are ordered (as they are in Example 2 of Cohen’s Kappa).
To address this issue, there is a modification to Cohen’s kappa called weighted Cohen’s kappa. The weighted kappa is calculated using a predefined table of weights which measure the degree of disagreement between the two raters, the higher the disagreement the higher the weight. The table of weights should be a symmetric matrix with zeros in the main diagonal (i.e. where there is agreement between the two judges) and positive values off the main diagonal. The farther apart are the judgments the higher the weights assigned.
We show how this is done for Example 2 of Cohen’s Kappa where we have reordered the rating categories from highest to lowest to make things a little clearer. We will use a linear weighting although higher penalties can be assigned for example to the Never × Often assessments.
Example 2: Repeat Example 2 of Cohen’s Kappa using the weights in range G6:J9 of Figure 1, where the weight of disagreement of Never × Often is twice the weights of the other disagreements.
We first calculate the table of expected values (assuming that outcomes are by chance) in range A14:E19. This is done exactly as for the chi-square test of independence. E.g. cell B16 contains the formula =B$10*$E7/$E$10.
The weighted value of kappa is calculated by first summing the products of all the elements in the observation table by the corresponding weights and dividing by the sum of the products of all the elements in the expectation table by the corresponding weights. Since the weights measure disagreement, weighted kappa is then equal to 1 minus this quotient.
For Example 1, the weighted kappa (cell H15) is given by the formula
Note that if we assign all the weights on the main diagonal to be 0 and all the weights off the main diagonal to be 1, we have another way to calculate the unweighted kappa, as shown in Figure 2.
Observation: Using the notation from Cohen’s Kappa where pij are the observed probabilities, eij = piqj are the expected probabilities and wij are the weights (with wji = wij) then
The standard error is given by the following formula:
Note too that the weighted kappa can be expressed as
From these formulas, hypothesis testing can be done and confidence intervals calculated, as described in Cohen’s Kappa.
Real Statistics Function: The Real Statistics Resource Pack contains the following function:
WKAPPA(R1, R2, lab, alpha) = returns a 4 × 1 range with values kappa, the standard error and left and right endpoints of the 1 – alpha confidence interval (alpha defaults to .05) where R1 contains the observed data (formatted as in range M7:O9 of Figure 2) and R2 contains the weights (formatted as in range S7:U9 of the same figure).
If range R2 is omitted it defaults to the unweighted situation where the weights on the main diagonal are all zeros and the other weights are ones. Range R2 can also be replaced by a number r. A value of r = 1 means the weights are linear (as in Figure 1), a value of 2 means the weights are quadratic. In general this means that the equivalent weights range would contain zeros on the main diagonal and values (|i−j|)r in the ith row and jth column when i ≠ j.
If lab = TRUE then WKAPPA returns a 4 × 2 range where the first column contains labels which correspond to the values in the second column. The default is lab = FALSE.
Observation: Referring to Figure 1 and 2, we have WKAPPA(B7:D9,G6:J9) = WKAPPA(B7:D9,1) = .500951 and WKAPPA(M7:O9) = .495904. We If we highlight a 4 × 2 range and enter WKAPPA(B7:D9, G6:J9,TRUE,.05) we obtain the output in range Y7:Y10 of Figure 3. For WKAPPA(M7:O9,,TRUE,.05) we obtain the output in range AA8:AB11 of Figure 7 of Cohen’s Kappa.
Real Statistics Data Analysis Tool: The Reliability data analysis tool supplied in the Real Statistics Resource Pack can also be used to calculate Cohen’s weighted kappa.
To calculate Cohen’s weighted kappa for Example 1 press Ctrl-m and choose the Reliability option from the menu that appears. Fill in the dialog box that appears (see Figure 7 of Cronbach’s Alpha) by inserting B7:D9 in the Input Range and G7:J9 in the Weights Range, making sure that Column headings included with data is not selected and choosing the Weighted kappa option. The output is shown on the left side of Figure 25.5.3.
Alternatively you can simply place the number 1 in the Weights Range field. If instead you place 2 in the Weights Range field (quadratic weights) you get the results on the right side of Figure 3.
Figure 3 – Weighted kappa with linear and quadratic weights