Kendall’s coefficient of concordance (aka Kendall’s W) is a measure of agreement among raters defined as follows.
Definition 1: Assume there are m raters rating k subjects in rank order from 1 to k. Let rij = the rating rater j gives to subject i. For each subject i, let Ri = . let be the mean of the Ri and let R be the squared deviation, i.e.
Now define Kendall’s W by
Observation: By algebra, an alternative formulation for W is
(see proof of Property 2 of Wilcoxon Rank Sum Test), and so
If all the Ri are the same (i.e. the raters are in complete disagreement), then by Definition 1, W = 0. In fact, it is always the case that 0 ≤ W ≤ 1.
Property 1: When k ≥ 5 or m > 15, m(k–1)W ~ χ2 (k–1).
Observation: We can use this property to test the hypothesis that W = 0 (i.e. there is no agreement among the raters).
Example 1: Seven judges rank order the same eight movies with the results shown in Figure 1. The average rank is used in cases of ties. Calculate Kendall’s W for this data and test whether there is no agreement among the judges.
Figure 1 – Kendall’s W
We see that W = .635 (cell C16), which indicates some level of agreement between the judges. We also see that (cell C18) and that the p-value = 5.9E-05 < .05 = α, thereby allowing us to reject the null hypothesis that there is no agreement among the judges.
Note too that we calculated the sums of the values in each row of data to make sure that the data range contained ranked data. Since there are 8 subjects the sum of rankings on each row should be 1 + 2 + ∙∙∙ + 7 + 8 = 8 ∙ 9 / 2 = 36, which it does.
Observation: W is not a correlation coefficient and so we can’t use our usual judgments about correlation coefficients. It turns out, however, that there is a linear transformation of W that is a correlation coefficient, namely
In fact it can be shown that r is the average (Spearman) correlation coefficient computed on the ranks of all pairs of raters.
For Example 1, r = .574 (cell C19).
Observation: In cell C22, we show how to compute W based on the alternative formulation for W given above. What is quite interesting is that the χ2 value for W given above is equal to the χ2 value used for Friedman’s test. Since we can calculate that value using the supplemental formula FRIEDMAN(R1), by Property 1, it follows that
For Example 1, this calculation is shown in cell C23.
Real Statistics Function: The Real Statistics Resource Pack contains the following array function:
KENDALLW(R1, lab, ties): returns a column vector consisting of and p-value where R1 is formatted as in range B5:I11 of Figure 1. If lab = TRUE, then instead of a 5 × 1 range the output is a 5 × 2 range where the first column consists of labels; default: lab = FALSE. If ties = TRUE then the ties correction as described below is applied (default = FALSE).
For Example 1, KENDALLW(B5:I11, TRUE) returns the output shown in Figure 2.
Figure 2 – KENDALLW output
Real Statistics Data Analysis Tool: The Reliability data analysis tool supplied in the Real Statistics Resource Pack can also be used to calculate Cohen’s weighted kappa.
To calculate Kendall’s W for Example 1 press Ctrl-m and choose the Reliability option from the menu that appears. Fill in the dialog box that appears (see Figure 7 of Cronbach’s Alpha) by inserting B4:I11 in the Input Range and choosing the Kendall’s W option.
Observation: The definition of W is fine unless there are a lot of ties in the rankings. When there are a lot of ties, the following revised definition of W can be used.
Definition 2: For each rater j, define
where the g are all the groups of tied ranks for rater j and tg = the number of tied ranks. E.g. for judge 1 in Example 1, there are no ties and so T1 = 0. For judge 2 there is one group of tied ranks (for 4 and 5) and so T2 = 23 – 2 = 6. Similarly T3 = T4 = T5 = 6. For judge 6 there are two such groups and so T6 = 6 + 6 = 12 and for judge 7 there is one group with three ties (3, 4, 5) and so T7 = 33 – 3 = 24. Thus T = 0 + 6 + 6 + 6 + 6 + 12 + 24 = 60.
Now define W as follows.
Example 2: Repeat Example 1 taking ties into account.
The calculations are shown in Figure 3.
Figure 3 – Kendall’s W with ties
Here we handle the ties using the same approach as in Example 3 of Kendall’s Tau. In particular, the non-zero cells in each row of the range L5:S11 will correspond to the first element in a group of ties. The value of each such cell will be one less than the number of ties in that group. E.g. cell L5 contains the formula
If you highlight the range L5:S11 and press Ctrl-R and Ctrl-D you will fill in the whole range with the appropriate formulas. This works provided the cells in A5:A11 and J5:J11 are blank (or at least non-numeric). Cell C16 will contain the formula to calculate T.
We see that the value of W hasn’t changed much even though we have quite a few ties.
Observation: The Real Statistics Reliability data analysis tool described above also contains a Kendall’s W with ties option. When this option is selected the ties correction described above is applied.