There are two common measures of effect size used for ANOVA and contrasts: one based on Cohen’s d (see Effect Size for Samples) and the other based on the correlation coefficient r (see Basic Concepts of Correlation). We will cover the first type here and the second type in Other Measures of Effect Size for ANOVA.
For pairwise contrasts we can use Cohen’s measure of effect size, namely
This can be extended to the comparison of more complicated contrasts. E.g. for the null hypothesis H0: we can use the following value for g
The interpretation of g is the same as for a comparison of two means.
This measure of effect size can be extended to the omnibus ANOVA. The following measure is called the root-mean-square standardized effect (RMSSE).
In Excel, this can be calculated as
=SQRT(DEVSQ(R1) / ((k–1)*R2)
where R1 is the array of group means and R2 is a cell that contains MSE. The group means differ on average by d standard deviations from the grand mean. E.g. if d = .8, then the group means differ by 80% of a standard deviation from the grand mean, which is a sizable difference.
Observation: When no better information is available, a rule of thumb is that d = .10 is a small effect, .25 is a medium effect and .40 or more is a large effect. To calculate power you can employ G*Power (available for free on the Internet) using the above values of d. You can also use the capabilities described in Power for One-way ANOVA.
Example 1: Calculate the effect size d (RMSSE) for the ANOVA in Example 2 of Basic Concepts for ANOVA.
Using the Excel formula given above, d = SQRT(DEVSQ(I7:I10)/(H15*I16)) = .618 (referring to Figure 2 of Basic Concepts for ANOVA), which is quite a high value.
Example 2: Calculate the effect size d (RMSSE) for the ANOVA in Example 3 of Basic Concepts for ANOVA using the One Factor ANOVA supplemental analysis tool.
Figure 1 from Confidence Interval for ANOVA displays the output from the Real Statistics One Factor ANOVA analysis tool used to perform this analysis. This figure is replicated as follows:
Figure 1 – Effect size from Real Statistics ANOVA data analysis tool
We can see from Figure 1 that the RMSSE effect size is 0.597509 (cell M14). The figure also shows the omega square effect size measurement (cell N14) which is explained in Other Measures of Effect Size for ANOVA).
Observation: Another related measure of effect size is
where is as described above. Thus, when all the groups are equal in size m, we have
We will use this measure of effect size when we discuss power and sample size requirements (see Power for One-way ANOVA).
Example 3: Calculate the effect size d for the contrast in Example 4 of Planned Comparisons for ANOVA.
Figure 2 replicates Figure 7 from Planned Comparisons for ANOVA and shows the output from the Real Statistics Contrast data analysis tool. In particular, Cohen’s d (cell V39) = ABS(T36)/N39 = 0.39.
Figure 2 – Effect size from Real Statistics Contrast data analysis tool
When you select the Contrast tool you will be presented with the dialog box shown in Figure 3. You need to select the type of alpha correction that you want, namely no experiment-wise correction, the Bonferroni correction or the Dunn/Sidák correction (as explained in Planned Comparisons). In any case you set the alpha value to be the experiment-wise value (defaulting as usual to .05). You also need to make sure the Independent type of contrast is selected (the default).
Note too that the contrast output that results from the tool will not contain any contrasts. You need to fill in the desired contrasts directly in the output (e.g. for Example 3 you need to fill in the range O32:O35 in Figure 2 with the contrasts you desire).
Figure 3 – Dialog box for Contrasts in data analysis tool