In Intraclass Correlation we reviewed the most common form of the intraclass correlation coefficient (ICC). We now review other approaches to ICC as described in the classic paper on the subject (Shrout and Fleiss). In that paper the following three classes are described:
Class 1: For each of the n subjects a set of k raters is chosen at random from a population of raters and each of these raters rate that subject, but each subject is potentially rated by different raters.
Class 2: k raters are chosen at random from a population of raters and these k raters rate all n subjects.
Class 3: Each of the n subjects are rated by the same k raters and the results address only these k raters.
The ICC values for these classes are respectively called ICC(1, 1), ICC(2, 1) and ICC(3, 1). Each of these measures the reliability of a single rater. We can also consider the reliability of the mean rating. The intraclass correlation for these are designated ICC(1, k), ICC(2, k) and ICC(3, k).
Real Statistics Function: The Real Statistics Resource Pack contains the following supplemental function:
ICC(R1, class, type, lab, alpha): outputs a column range consisting of the intraclass correlation coefficient ICC(class, type) of R1 where R1 is formatted as in the data range of Figure 1 of Intraclass Correlation, plus the lower and upper bound of the 1 – alpha confidence interval of ICC. If lab = TRUE then an extra column of labels is added to the output (default FALSE). class takes the values 1, 2 or 3 (default 2) and type takes the values 1 (default) or k where k = the number of raters. The default for alpha is .05.
For example, the output from the formula =ICC(B5:E12,2,1,TRUE,05) for Figure 1 of Intraclass Correlation is shown in Figure 1 below.
Figure 1 – Output from ICC function
Real Statistics Data Analysis Tool: The Reliability data analysis tool supplied in the Real Statistics Resource Pack can also be used to calculate the ICC.
To calculate ICC for Example 1 press Ctrl-m and choose the Reliability option from the menu that appears. Fill in the dialog box that appears (see Figure 3 of Cronbach’s Alpha) by inserting B5:E12 in the Input Range and choosing the ICC option. The output is shown in Figure 2.
Figure 2 – Output from ICC data analysis tool
We next show how to calculate the various versions of ICC.
Class 1 model
For class 1 the model used is
where μ is the population mean of the ratings for all the subjects, μ + βj is the population mean for the jth subject and εij is the residual, where we assume that the βj are normally distributed with mean 0 and that the εij are independently and normally distributed with mean 0 (and the same variance). This is a one-way ANOVA model with random effects.
As we saw in One-way ANOVA Basic Concepts
The subjects are the groups/treatments in the ANOVA model. In this case, the intraclass correlation, called ICC(1,1), is
The unbiased estimate for var(β) is (MSB – MSW)/k and the unbiased estimate for var(ε) is MSW. A consistent (although biased) estimate for ICC is
For Example 1 of Intraclass Correlation, we can calculate the ICC as shown in Figure 3.
Figure 3 – Calculation of ICC(1, 1)
First we use Excel’s Anova: Single Factor data analysis tool, selecting the data in Figure 1 of Intraclass Correlation and grouping the data by Rows (instead of the default Columns). Alternatively we can first transpose the data in Figure 1 of Intraclass Correlation (so that the wines become the columns and the judges become the rows) and use the Real Statistics Single Factor Anova data analysis tool.
The value of ICC(1, 1) is shown in cell I22 of Figure 1, using the formula shown in the figure.
The confidence interval is calculated using the following formulas:
For Example 1 of Intraclass Correlation, the 95% confidence interval of ICC(1, 1) is (.434, .927) as described in Figure 4.
Figure 4 – 95% confidence interval for ICC(1,1)
ICC(1, 1) measures the reliability of a single rater. We can also consider the reliability of the mean rating. The intraclass correlation in this case is designated ICC(1, k) and is calculated by the formulas
ICC(1, 4) for Example 1 of Intraclass Correlation is therefore .914 with a 95% confidence interval of (.754, .981).
Class 2 model
This is the model that is described in Intraclass Correlation. For Example 1 of Intraclass Correlation, we determined that ICC(2, 1) = .728 with a 95% confidence interval of (.408, .950). These are the results for a single rater. The corresponding formulas for the mean rating are as follows:
ICC(2, 4) for Example 1 of Intraclass Correlation is therefore .914 with a 95% confidence interval of (.734, .987).
Class 3 model
The class 3 model is similar to class 2 model, except that var(α) is not used. The intraclass correlation, called ICC(3, 1), is given by the formula
Using the terminology of Two Factor ANOVA without Replication (as for case 2), we see that (MSRow–MSE)/k is an estimate for var(β) and MSE is an estimate for var(ε). A consistent (although biased) estimate for ICC is
Figure 5 – Calculation of ICC(3,1) and 95% confidence interval
ICC(3, 4) for Example 1 is therefore .915 with a 95% confidence interval of (.748, .981).
Observation: Class 3 is not so commonly used since by definition it doesn’t allow generalization to other raters.
Observation: ICC(3, k) = Cronbach’s alpha. For Example 1 of Intraclass Correlation, we see that =CRONALPHA(B5:E12) has value .915, just as we saw above for ICC(3, 4).