Power of Chi-square Tests

The power of the goodness of fit or chi-square independence test is given by

image7322where F is the cumulative distribution function (cdf) for the noncentral chi-square distribution χ2(df), xcrit is the χ2(df) critical value for the given value of α and λ = w2n is the noncentrality parameter where w is the φ effect size (see Chi-square Effect Size), even for larger than 2 × 2 contingency tables.

Example 1: Calculate the power for Example 3 of Goodness of Fit.

The power of this test is 23%, as shown in Figure 1.

power-goodness-of-fit

Figure 1 – Power of goodness of fit test

We can use the CHISQ_POWER function to achieve the same result, namely CHISQ_POWER(B9,B4,B5) = .230126.

Real Statistics Functions: The following function is provided in the Real Statistics Resource Pack:

CHISQ_POWER(w, n, df, α, m, prec) = the power of a chi-square goodness of fit or independence test where w = Cohen’s effect size, n = the sample size, df = degrees of freedom and α = alpha (default = .05).

CHISQ_SIZE(w, df, 1−β, α, m, prec) = the minimum sample size required to obtain power of at least 1−β (default .80) in a chi-square goodness of fit or independence test where w = Cohen’s effect size and α = alpha (default = .05).

Here m = the maximum number of terms in the infinite sum that will be calculated (default 1000) and prec = desired level of accuracy of the power calculation (default 0.000000001).

Example 2: How big a sample is required to achieve power of 80% for a chi-square test of independence for a 3 × 3 contingency table with medium effect size (i.e. w = .3)?

We can use Excel’s Goal Seek capability as shown in Figure 2.

Sample size independence test

Figure 2 – Using Goal Seek to find the sample size

Upon pressing the OK button, the value in cell G10 changes to .80 and the value in cell G9 changes to 132.6031. Thus a sample size of 133 is required.

The sample size requirement can also be obtained using the Real Statistics formula

=CHISQ_SIZE(.3,4,.8,.05)

37 Responses to Power of Chi-square Tests

  1. Reza says:

    Hi Charles,
    Thanks for putting these tools together.
    I have two sets of data. First set is generated by a machine (Oi) and the second is manually calculated (Mj). I know log(Oi/Mj) ~N(0, c^2/Mj). I need to measure the c^2. Can I use goodness of fit to calculate the sample size required for that?
    Any comment on that is greatly appreciated.
    Reza

    • Charles says:

      Reza,
      You should be able to estimate c from the sample using a goodness of fit approach, but I don’t understand the part about sample size. Not sure what statistical power would mean in this case.
      Charles

  2. Glyn says:

    Charles
    Hope you are well.
    I’ve been looking at your site recently to solve a problem regarding the calculation of the power of a chi-square test. I have reproduced your results above in Excel 2016 but I am having problems understanding the implementation for the calculation of the BETA term in cell B11 of example 1. I do understand that this may be complicated but would you be able to point me in the right direction.
    Regards
    Glyn

  3. Martine says:

    Hi Charles,

    This looks so useful! However, I can’t seem to install it. I’m using Excel for Mac 2011 (14.6.8).

    The only Add-On available to me is ‘Solver.xlam’ and there is no option to browse for other Add-Ons. Cmd+M minimises the screen and Ctrl+M seems to do nothing at all.

    Are you able to advise at all?

    Many thanks!
    Martine

    • Martine says:

      Problem solved! Thanks very much for this useful tool.

      May I ask: if I’ve calculated sample size based on a chi-square test of independence (2×2), is it safe to then analyse the data using binary logistic regression? Or is a different sample size calculation required for the latter?

      Thank you again!
      M

      • Charles says:

        Hi Martine,
        What did you do to resolve the problem?
        You would need to provide additional information before I am able to answer your question about whether to use binary logistic regression.
        It is very likely that the sample size requirements for the chi-square test is different from that for binary logistic regression.
        Charles

        • Martine says:

          Hi Charles,

          I just posted a reply, but I don’t think it worked, so I’ll try again. Apologies if you receive two!

          I can’t recall how I solved the installation problem in the end. I do recall that it was something very silly on my part.

          I asked whether it was safe to analyse my data using binary logistic regression, having calculated sample size based on a chi-square test for independence (2×2). You asked for further details. Here they are:

          My design is experimental: participants are randomly assigned to one of two groups. My outcome variable is binary: responses are coded 0 or 1. I’d like to know whether a response of ‘1’ is more frequent (or more likely) in one group than in the other. That is, I’d like to know the effect of group (IV) on response (DV: 0 or 1). So I’d like to use binary logistic regression to assess this.

          And my question is: if I’ve calculated sample size based on a chi-square test for independence (2×2), is it safe to analyse my data using binary logistic regression? Or should I rather calculate sample size based on binary logistic regression?

          Many thanks again!
          M

    • Charles says:

      Martine,
      You need to install the software before trying to use it. When you type the formula =VER() into any cell in a blank Excel spreadsheet what do you see? If you don’t see a version number then the software has not been installed. To install the software you need to do more than simply download it. You need to follow the instructions on the webpage from where you downloaded the software.
      Charles

  4. James T. Lee MD PhD says:

    Something bothers me and bothers me a lot: When we consider computing the “power” of some Chi Squared test is it not a fact that the calculation (even if concealed inside the software of the best stats program) of “power” for any Chi Squared test is always obligated to use OBSERVED DATA (i.e. after the experiment is already over with) to generate what is perceived as the “power value”? But, is this not a species of Post Hoc Power Value ???

    • Charles says:

      Yes, you are rights about this, not just for the chi-square test, but many other tests as well.
      Charles

      • James T. Lee, MD PhD says:

        Then I do not see how the “power calculation” for any chi-square test could possibly be useful for DESIGNING an experiment (i.e. picking optimum “sample size”), since computing the value of such “power” would require ipso facto that the investigator knows what his/her outcome data will be BEFORE an experiment is ever run. In other words, an investigator would need to be clairvoyant.

        What you are talking about in these posts is the entity called “observed power”. Since it can only (and obviously) be computed AFTER an experiment is completed, to talk about such “power values” in the context of planning experiments sure does seem to be absolutely pure gibberish to me.

        Please clear the fog up here. Language is powerful.

        Thanks.

        • Charles says:

          James,
          Yes, when you compute power, it is the observed power.

          The situation is different when you try to estimate the sample size, this is before the experiment. It works like this. Before the experiment, (for any given power and alpha goals that you have) you need to decide on how small an effect you need to detect. If your expectation is that the alternative hypothesis is really correct and by a large amount, then you base the effect size you use on this amount to estimate the sample size that you will need. If it turns out that after running the experiment, the effect size is much smaller than this value, then you will see that the power is smaller than you would have liked (because the sample size you used was too small).

          If you decide in advance that you need to detect a small effect (or just to be on the safe side you want to be sure you can detect a small effect), then you use this value for the effect size to calculate the sample size required.

          Charles

  5. Maximilian says:

    Hello Charles,

    I have problems to calculate the beta respectively the power of the X²-Test. In the statistic tools (statistical Power and Sample Size; Chi-square test) I have the message “Compiler Error in Modul: frmPower …..”
    And with the functions CHISQ_POWER and NCHISQ_DIST i have the Error: Error in Compiler: Projecr or libary not found.

    Can you help me ?

    • Charles says:

      Maximilian,
      A possible cause is that Excel’s Solver is not active. Please press Alt-TI and see whether Solver is on the list of addins with a check mark next to it. Solver is now required to use the Real Stats software.
      Charles

  6. Jonathan Bechtel says:

    Hi Charles,

    In the CHISQ_POWER function you say w should represent Cohen’s D, but the use of this statistic is not used anywhere in the section on the Chi-Square distribution.

    Do you mean we should use the Pearson’s V statistic in its place?

    Thank you.

  7. B. McCook says:

    I am doing something incorrect. trying to determine minimum sample sizes for two chi square tests of independence

    Both will have power equal to 0.80, alpha equal to 0.05, 3 columns, and w equal to 0.3.
    One has 4 rows the other 10.

    Regards,
    Bill

    • Charles says:

      Bill,
      What answer are you getting? Are you using the Statistical Power and Sample Size data analysis tool?
      Charles

  8. Kelvin says:

    Hi Charles,

    Would this sample size calculation be valid even if it implies that the expected number of events in some cells will be less than 5, or even zero, due to the low frequency of each variable in the sample?

    How to account for this?

    • Charles says:

      Hi Kelvin,
      This sample size calculation should be valid if the chi-square test itself is valid. If a relatively small percentage of the cells are 2, 3 or 4, there shouldn’t be a big problem, but I wouldn’t have too much faith in the test if there are cells with 1 or 0. With such samples I would use the Fisher Exact Test if available, although currently I don’t provide an estimate for power. I currently support for the Fisher Exact Test for 2×2, 2×3, 2×4, 2×5, 3×3 and 3×4 contingency tables, depending on how big the sample is.
      Charles

      • Kelvin says:

        Thanks, Charles. Unfortunately the frequencies are too low for the ChiSq test (below 5), while Fisher test only applies to 2 variables, so still not sure how to test low frequencies between multiple variables, or if this is even possible.

  9. Yclept Nemo says:

    You say “Cohen’s φ”, but that doesn’t exist. Do you mean the phi coefficient (https://en.wikipedia.org/wiki/Phi_coefficient) or, Cramer’s Phi (https://en.wikipedia.org/wiki/Cram%C3%A9r's_V_%28statistics%29), or Cohen’s w (https://en.wikipedia.org/wiki/Effect_size#Cohen.27s_w)? The phi coefficient is only defined for 2×2 tables and isn’t suitable for goodness-of-fit 1xk tables. Cramer’s phi is, but requires tables to determine the significance of the effect size. I’m not familiar with Cohen’s w, but it looks like the square root of x^2.

    • Charles says:

      Yclept

      w = square root of the sum of the values (pi – ri)^2 / pi, where pi = the proportion of “cell” i adhering to the null hypothesis and ri = the proportion adhering to the alternative hypothesis. It then turns out that w = square root of the chi-square statistic divided by the sample size.

      This is true for m x n contingency tables as well as 1 x k goodness of fit tables.

      As you have correctly suggested, I have replaced the referenced to “Cohen’s φ” by “the phi coefficient”

      Charles

  10. Ahlam soliman says:

    Thank you very much.
    But I have a question is the sample size calculated is per group or the whole sample?

  11. Andrew Vezey says:

    Hi Charles!

    Thank you very much for this resource. I was wondering, is there an alternate function other than NCHISQ_DIST that I can use to calculate beta? I do not have the Resource Pack so I am trying to find an alternative method. I appreciate the consideration!

    Best,
    -Andrew Vezey

    • Charles says:

      There is no standard Excel function that computes power (i.e. 1 minus beta) for the chi-square tests. You can try to duplicate all the steps that I have described on the website, but this will take a fair amount of work.
      Charles

  12. Andrew says:

    Hi Charles,

    Thanks for putting these tools together. They are of obvious value.

    I did have a quick question regarding power analysis for chi square. I wasn’t able to easily map what is described here with other formula for calculating power that I have seen talked about (e.g. https://www.amstat.org/sections/SRMS/Proceedings/y2006/Files/JSM2006-000389.pdf). Is there a particular formula or approach that what you describe above corresponds to?

    Cheers
    Andrew

    • Charles says:

      Andrew,
      If I understand correctly, the referenced article describes approximate methods for calculating the power of certain tests without using iterative techniques. I have adopted instead the calculations using noncentral distributions. I believe the methods are quite different, but I plan to investigate further to see whether the results are similar.
      Charles

  13. Jason says:

    Hello Sir,
    Thank you for your website info about chi-square power analysis.
    I was wondering if you can help me calculate- How many samples is required to achieve power of 80% for chi square test for independence for a 2×2 contigency table with medium effect size (w-.3)?
    I appreciate,
    thank you,
    Jason

    • Charles says:

      Jason,
      You can use the Statistical Power and Sample Size data analysis tool. Click on the Chi-square and Sample Size options. On the resulting dialog box insert .3 for Effect Size, 1 for df and use the defaults for the other fields. When you press the OK button, you will see that the sample size is 88.
      Charles

      • Jason says:

        Hi Sir Charles,
        I see the mistake I did.
        Thanks for your help.
        Your website is an excellent resource especially for a novice like me.
        Very much appreciated.
        Jason

Leave a Reply

Your email address will not be published. Required fields are marked *