Two Sample Kolmogorov-Smirnov Test

The two sample Kolmogorov-Smirnov test is used to test whether two samples come from the same distribution. The procedure is very similar to the One Kolmogorov-Smirnov Test (see also Kolmogorov-Smirnov Test for Normality).

Suppose that the first sample has size m with an observed cumulative distribution function of F(x) and that the second sample has size n with an observed cumulative distribution function of G(x). Define

image7318

The null hypothesis is H0: both samples come from a population with the same distribution. As for the Kolmogorov-Smirnov test for normality, we reject the null hypothesis (at significance level α) if Dm,n > Dm,n,α where Dm,n,α is the critical value.

For m and n sufficiently large

image7319

where c(α) = the inverse of the Kolmorogov distribution at α, which can be calculated in Excel as

Dm,n,α = KINV(α)*SQRT((m+n)/(m*n))

where KINV is defined in Kolmorogov Distribution. The values of c(α) are also the numerators of the last entries in the Kolmogorov-Smirnov Table.

Example 1: Determine whether the two samples on the left side of Figure 1 come from the same distribution. The values in columns B and C are the frequencies of the values in column A.

Two Sample Kolmogorov-Smirnov

Figure 1 – Two-sample Kolmogorov-Smirnov test

We carry out the analysis on the right side of Figure 1. Column E contains the cumulative distribution for Men (based on column B), column F contains the cumulative distribution for Women and column G contains the absolute value of the differences. E.g. cell E4 contains the formula =B4/B14, cell E5 contains the formula =B5/B14+E4 and cell G4 contains the formula =ABS(E4-F4).

Cell G14 contains the formula =MAX(G4:G13) for the test statistic and cell G15 contains the formula =KSINV(G1,B14,C14) for the critical value. Since D-stat =.229032 > .224317 = D-crit, we conclude there is a significant difference between the distributions for the samples.

We can also use the following functions to carry out the analysis:

Real Statistics Function: The following functions are provided in the Real Statistics Resource Pack:

KSDIST(x, n1, n2, b, m) = the p-value of the two-sample Kolmogorov-Smirnov test at x (i.e. D-stat) for samples of size n1 and n2.

KSINV(p, n1, n2, b, iter, m) = the critical value for significance level p of the two-sample Kolmogorov-Smirnov test for samples of size n1 and n2.

As usual, m = the # of iterations used in calculating an infinite sum (default = 10) in KDIST and KINV and iter (default = 40) = the # of iterations used to calculate KINV.

When the argument b = TRUE (default) then an approximate value is used which works better for small values of n1 and n2. If b = FALSE then it is assumed that n1 and n2 are sufficiently large so that the approximation described previously can be used.

For Example 1, we have the following:

D-crit = KSINV(G1,B14,C14) = .224526

p-value = KSDIST(G14,B14,C14) = .043055

Alternatively we can use the Two Sample Kolmogorov-Smirnov Table of critical values  to find the critical values, or the following functions which are based on this table:

KS2CRIT(n1, n2, α, tails, h) = the critical value of the two sample Kolmogorov-Smirnov test for sample of size n1 and n2 for the given value of alpha (default .05) and tails = 1 (one tail) or 2 (two tails, default) based on the table of critical values. If h = TRUE (default) then harmonic interpolation is used; otherwise linear interpolation is used.

KS2PROB(x, n1, n2, tails, h ) = an approximate p-value for the two sample KS test for the Dn1,n2 value equal to x for samples of size n1 and n2, and tails = 1 (one tail) or 2 (two tails, default) based on a linear interpolation (if h = FALSE) or harmonic interpolation (if h = TRUE, default) of the values in the table of critical values, using iter number of iterations (default = 40).

Note that the values for α in the table of critical values  range from .01 to .2 (for tails = 2) and .005 to .1 for tails = 1. If the p-value is less than .01 (tails = 2) or .005 (tails = 1) then the p-value is given as 0 and if the p-value is greater than .2 (tails = 2) or .1 (tails = 1) then the p-value is given as 1.

For Example 1, we have the following:

D-crit = KS2CRIT(B14,C14, G1) = .229792

p-value = KS2PROB(G14,B14,C14) = .051232

Finally, we can use the following array function to perform the test:

Real Statistics Function: The following function is provided in the Real Statistics Resource Pack:

KS2TEST(R1, R2, lab, alpha, b, iter, m) is an array function which outputs a column vector with the values D-stat, p-value, D-crit, n1, n2 from the two sample KS test for the samples in ranges R1 and R2, where alpha is the significance level (default = .05) and b, iter and m are as in KSINV.

If R2 is omitted (the default) then R1 is treated as a frequency table (e.g. range B4:C13 in Figure 1).

If lab = TRUE then an extra column of labels is included in the output; thus the output is a 5 × 2 range instead of a 1 × 5 range if lab = FALSE (default).

For Example 1, the formula =KS2TEST(B4:C13,,TRUE) inserted in range F21:G25 generates the output shown in Figure 2.

KS test Excel

Figure 2 – Output from KS2TEST function

Example 2: Determine whether the samples for Italy and France in Figure 3 come from the same distribution.

Samples KS test

Figure 3 – Two data samples

We first show how to perform the KS test manually and then we will use the KS2TEST function.

Two sample KS test

Figure 4 – Two sample KS test

The approach is to create a frequency table (range M3:O11 of Figure 4) similar to that found in range A3:C14 of Figure 1, and then use the same approach as was used in Example 1. This is done by using the Real Statistics array formula =SortUnique(J4:K11) in range M4:M10 and then inserting the formula =COUNTIF(J$4:J$11,$M4) in cell N4 and highlighting the range N4:O10 followed by Ctrl-R and Ctrl-D. Finally the formulas =SUM(N4:N10) and =SUM(O4:O10) are inserted in cells N11 and O11.

We can also calculate the p-value using the formula =KSDIST(S11,N11,O11), getting the result of .62169.

We see from Figure 4 (or from p-value > .05), that the null hypothesis is not rejected, showing that there is no significant difference between the distribution for the two samples. The same result can be achieved using the array formula

=KS2TEST(J4:J11,K4:K10,TRUE)

which produces the output in Figure 5.

Two sample KS test

Figure 5 – Output from KS2TEST function

Finally, note that if we use the table lookup, then we get KS2CRIT(8,7,.05) = .714 and KS2PROB(.357143,8,7) = 1 (i.e. > .2).

105 Responses to Two Sample Kolmogorov-Smirnov Test

  1. Chen says:

    Hi, can you point me to the reference of the formula of the critical value? It seems that every website (including Wiki) is talking about the formula, but no one has a real reference.

  2. Chris says:

    I have a short question: What would be the correct interpretation when I have D = 0.011579 and p100.000). I’m not sure if even with this marginal difference in D statistic it’s correct to reject the null hypothesis.
    Thanks in advance

    • Charles says:

      Chris,
      Sorry, but I don’t know what p100.000) means.
      Charles

      • Chris says:

        Yup, me neither. Something went wrong with my keyboard. My question was: What would be the correct interpretation if I have a D = 0,01 and p = 2,2E-12? Because if I just consider the D value I’d say both distributions are equal. On the other hand, if I take into consideration just the p-value I’d say both are different. Both samples are greater than 100.000 records.

      • Chris says:

        Hi Charles, I’d really appreciate if you can provide any insight. Thanks!

        • Charles says:

          Chris,
          I don’t have any further insight based on the information available to me.
          If you send me an Excel file with your calculations, I will try to figure out what is happening.
          Charles

  3. Blyth Hughes says:

    My previous post was incorrect. KSINV(p,n1,n2) gives wrong results for the example of n1 and n2 both = 50 and p outside the interval 0.2 to 0.46. If for a given D, p=KSDIST(D,n1,n2) is calculated for n1 = n2 = 50, then KSINV(p,n1,n2) only returns the original D-values for 0.2 ≤ D ≤ 0.46. Outside this range it produces a constant value for D 0.46.

    The same behaviour pertains for n1 and n2 not equal to 50.

    Has this been addressed in later versions of KSINV for the Mac? (Mine was uploaded to your website in Jan 2014). If not, is there an algorithm that allows for calculation of the limiting values of p in KSINV for given n1 and n2?

    • Charles says:

      Blyth,
      I will look into the accuracy of the KSINV function shortly.
      The latest MAC release is Rel 3.4.3, which was issued on 3 February 2015.
      I don’t believe that I have made any further changes to the KSINV function since then.
      Charles

    • Charles says:

      Blyth,
      The next release of the Real Statistics software will correct this problem. KSINV and KINV should work properly for values of p between .00001986 and .27. This release should be available this week.
      As mentioned elsewhere, I am also working on a new Mac release. This will likely take a little longer since certain Excel support that I use is not available for the Mac and so I needed to create these capabilities myself.
      Charles

  4. Blyth Hughes says:

    In the version of RealStats-2011 that I was able to download for the Mac, there are limitations on KSINV that are not apparent or are not described in the table giving all the Excel functions that are available in REALSTATS-2011.

    For n1 and n2 = 50, KSINV only gives correct answers for 0.2 ≤ p ≤ 0.46 (or thereabouts). For p outside these limits KSINV gives fixed results independent of p. For n1 and n2 different from 50 the range of p also limited (but different).

    Is there a table or some algorithm that would allow these limits to be estimated for given n1 and n2?

    My version was uploaded to your website in January 2014. Perhaps there is a later version that I can download.

  5. Minzi says:

    Hi, I have a questions regarding the sample size. Let’s said I have n=4 for male sample and n=4 for female sample for different age group. How would you calculate the D value and critical value? Would you sum up all 4 sample or get the average, then only compare between two distribution?

    • Charles says:

      Minzi,
      It depends on what you are comparing. If you want to compare females with males for a specific age group, then you follow the instructions described on the referenced webpage. If you want to do something else, then this would depend on the specific comparison you want to make. Don’t expect much, however, if you sample size is so small as n = 4.
      Charles

      • Minzi says:

        Hi Charles,

        Thanks for the reply, I think I may describe it wrongly in my previous question, in each sample I have about 1000 and above raw data for the control and treatment group and I want to compare their specific position. I can easily perform the KS test as described above. However, I repeated the similar experiment for 4 times, and I am having difficulties whether I should sum up all the data or should I take the average for four times for each specific position to run the test.
        Thank you.

  6. Pingback: ABBA BABA test with R | exquisitanaturae

  7. Ruta says:

    Hi Charles,

    If I calculate c(alpha) as KINV(alpha), I get slightly different c(alpha) results than using this formula SQRT(-1/2*LN(alpha/2)) which is recommended in Wikipedia.
    Why does the difference occur?

    • Ruta says:

      Is it because of iterations? If yes, then why are the iterations necessary here and what is the iterations’ meaning here (in practical way)?
      On what does it depend the chosen number of iterations?

      Appreciate your answers!

      • Ruta says:

        In this formula KS2TEST(R1, R2, lab, alpha, b, iter m), when R1 and R2 sample sizes are different:

        1. Does m is equal to bigger sample size or exactly to second (R2) sample size?
        2. is the comma missing between ‘iter’ and ‘m’ in this formula?

        Thanks!

        • Charles says:

          Ruta,
          Yes, the comma is missing from the formula. I have just corrected this webpage. Thanks for identifying this typo.
          KS2TEST can be used even when the sample sizes are different. The sample sizes are calculated from R1 and R2.
          m = the # of iterations used in calculating an infinite sum (default = 10).
          Charles

      • Charles says:

        Ruta,
        If you are referring to the KDIST(x, m) function, then the number of iterations m refers to how many of the terms in an infinite sum are used. Generally m = 10 is sufficient, but you could try m = 20 to see whether you get a different result (the bigger the m the more accurate the result, although at a certain point the won’t be any further improvement).
        Charles

    • Charles says:

      Ruta,
      Both are approximations. For alpha = .05, the Wikipedia value is 1.358102. The KINV value is 1.358099. I don’t know which is more accurate, but usually it won’t make much of a difference.
      Charles

  8. Nishi Karunasinghe says:

    Many thanks for Charles for this explanation and comments sent in by others. These really helped me with my cumulative data comparison.

  9. Clisby says:

    I now understand how to manually calculate the D-crit value by reading your explanations in the comments. However, I think the equation is slightly wrong:
    D-crit = c(alpha)*root((m+n)/(m*n))
    when m and n in the explanation are the sizes of the arrays, not the sums.
    Because of this, I was originally calculating
    D-crit = 1.36*root(20/100)
    and didn’t realize you meant the sums until reading the comments.

    Thanks!

    • Charles says:

      Clisby,
      I am not sure what you mean by the “sums”. In any case the webpage states “Suppose that the first sample has size m … and that the second sample has size n …” Is this incorrect?
      Charles

  10. amir says:

    how to get critical value from the K-S table

    • Charles says:

      Amir,
      The values of c(α) are the numerators of the last entries in the Kolmogorov-Smirnov Table. The other entries in the table are not used for the two sample test.
      Charles

      • Sam Salous says:

        I tried to use the values of c(α) are the numerators of the last entries in the Kolmogorov-Smirnov Table to replicate example 1 above but got:
        D critical = 1.36 * SQRT((80+62)/(80*62)) = 0.2301 instead of your calculated value of 0.224317 and that gives the opposite conclusion.
        Any help in understanding how to calculate D critical is appreciated.
        Thanks

        • Charles says:

          Sam,
          These are two ways to calculate the same thing. The values are not that different. If one answer yields a significant result and the other doesn’t, then the reality is probably borderline and should be reported that way. If the p-value is .0501 then commonly you accept the null hypothesis, while if the p-value is .0499 you reject it. The reality is that the result is pretty much the same, namely an approximately 1 in 20 chance of a type I error.
          Charles

  11. amir says:

    how to find the critical value manually through k-s table test

  12. David says:

    I have sets of Histogram data which I wish to use the KS2TEST on.
    If I have use {=KS2TEST(KS2TEST(A3:A19,C3:C19,TRUE)} I get,
    D-stat 0.352941176
    p-value 0.190007633
    D-crit 0.441915967
    size 1 17
    size 2 17

    But If i put the two columns of data next to each other and use {=KS2TEST(A3:B19,,TRUE)} , I get
    D-stat 0.800345968
    p-value 0.005280863
    D-crit 0.096198483
    size 1 197
    size 2 29999

    In the first example, the sizes represent the number of bins of the histogram data but not the cumulative number of points. Which Should I be using?
    Thanks for the great site.

    • David says:

      Sorry, I pasted into the comment wrong. The first function in excel is correctly (I think) {=KS2TEST(A3:A19,C3:C19,TRUE)}

  13. Mark says:

    Hi Charles,
    I downloaded the most recent Real Statistics Resource Pack, but the “KS2TEST” function doesn’t seem to show up on Excel. I have previously used it on Excel. Is there anything different about the most recent Resource Pack? I have the 2010 32-bit version of Excel.

    Thanks,
    Mark

    • Charles says:

      Mark,
      Nothing has changed. You should see the KS2TEST function on the latest version of the Resource Pack. In any case, you can access the same capability from the Non-parametric Test data analysis tool.
      Charles

      • David says:

        I have a similar problem. The check box is ticked for the real stats pack however it will not work until I reload it from the add-ins menu.

  14. shkey says:

    Hi
    I would like to test which distribution fits my data better, weibull or pareto. could you please help me how I can use Real Statistics pack to do this test.

    Thanks

    Shkey

    • Charles says:

      Sorry Shkey, but the Real Statistics Resource Pack doesn’t currently do this. You can, however, use the approach described on the following webpage to see whether either of these distributions is a good fit.
      Goodness of Fit
      See especially Example 4.
      Charles

      • shkey says:

        Hi Charles,
        Thank you for your reply. Perfect, I’ll follow example 4 but I need your help. e.g. I have 70 datapoint and would like to check lognormal distribution. I’ll find the parameters, mu and sigma. To find out the expected value, should I use :
        E=LOGNORM.DIST(x,mu,sigma,TRUE)?
        then compute column F as suggested in Example 4.
        the df is 70-2-1=67?
        and then do the rest!

        Thanks in advance,

        Shkey

  15. Damir says:

    Hello Charles,

    Thank you for the excellent post and the resource pack. They are both very useful.

    I have a question regarding the KS test. Do you know how to perform a KS test on weighted samples? Calculating cumulative distributions is straightforward, but I am not sure how to calculate the critical value. Does the same formula apply? If yes, should n and m be the unweighted sample sizes?

    Best regards,

    Damir

    • Charles says:

      Hello Damir,
      I am sorry, but I don’t yet support a weighted version of the KS test. I will add this to the list of future enhancements.
      Charles

  16. Mark says:

    Hi Charles,

    Should the Kolmogorv-Smirnov test work now for non-integer values? It doesn’t work for decimal values, but I’m wondering if I’m using an older version.

    Thanks,
    Mark

    • Charles says:

      Mark,
      In the latest version, the values can be decimal numbers. E.g. in Example 2 if you change cell J10 to 6.1 the output does not change, but if you now also change cell K10 to 6.1 the output changes.
      Charles

  17. Shahram Abdolmalaki says:

    Dear Sir

    I hope you are well and have a very happy time .I am working in fisheries
    research center of Guilan province in North of Iran .I have a project on
    population dynamics of invader Freshwater shrimp Macrobrachium nipponens in
    Anzali wetland . I want to know the proceedure for How to test similarity
    among the length frequency distribution between sexes by Kolmogorov-Smirnov
    test whit SPSS

    I wonder if you kindly furnish me by guiding me the procedure for how to
    test this length frequency by this test . As you know due to misbehavior of
    our government ,Iran is under sanction of UN and we hope one day Iran can
    have a good relation with free world .I trust that you will give me your
    kind consideration to my request .I never forget your help.I never forget
    your help
    My postal address is :
    International Sturgeon Research Institute , P.O.Box 41635-3464 ,Rasht ,IRAN

    Thank you in advance and I am looking forward to hearing from you .I am
    sorry to put you in trouble.
    Yours very sincerely ,
    Shahram Abdolmalaki

  18. James says:

    Hi Charles,

    I have a problem here ..
    I have two samples and I did the test. I found D-stat > D-crit, and I reject the null hypothesis. But when I look at p-value, I found a big value (72%) indicating to not reject the null hypothesis. When I use the KS2TEST function, I found the same results. How can I conclude my test? Reject or not reject ?

    Thanks a lot

    • Charles says:

      James,
      If you send me an Excel file with your data and test results I will try to figure out what is happening. You can find my email address at Contact Us.
      Charles

  19. Stacey says:

    Thank you Charles,

    I’ve learned a lot from your nice website and also from your kind help before. Thank you. I have a question though. Now I have a scatter plot (it seems plots are not allowed to put in this comment) of two continuous variables (x, y) for two separate groups (a and b). Group a, for instance, includes points x1(1,2), x2(2,3), x3(1.5,2), etc. Group b includes y1(2,5), y2(3,4), etc. Two groups have the same number of variables. I am wondering whether these two groups are significantly different from each other.
    I looked up in the Internet and some say that a two-sample Kolmogorov–Smirnov test could be done. Is this OK? If not, what kind of measure is preferable?

    Thank you!
    Best,
    Stacey

    • Charles says:

      Stacey,
      Yes, this test can be used. If you only want to test whether the means/medians of the two groups are significantly different then there are other choices (t test, Mann-Whitney). None of these tests require that the two groups have the same number of elements.
      Charles

      • Stacey says:

        Thank you Charles.

        Sorry I did not make my question clear. Each element in two groups has two variables. Part of them are displayed below:

        Group a Group b
        a1(1.48, 2.99) b1 (2.37, 12.8609)
        a2(1.68, 4.98) b2 (1.48, 3.5667)
        a3(2.27, 11.27) b3(1.76, 6.3466)
        a4(1.71, 4.78) b4(1.98, 9.6587)
        a5(1.97, 7.95) b5(1.27, 1.1167)

        Thus now we have four sets of data (each group has two sets of variables). Two groups can be drawn as two scatter plots. So I want to know whether these two distributions are different or similar to each other. Because I saw that two-sample KS was used for two samples. I don’t know whether it will work for my current situation. Thank you.

        • Charles says:

          Stacey,
          I don’t know away of doing this. The only thing I can think of is to create a transformation f of (x,y) into z such that f(x,y) = f(u,v) if and only if x = u and y = v. In this case you can use the two-sample KS on the mapped values.
          The specific transformation f probably depends on the nature of your data.
          Charles

          • Stacey says:

            Charles,
            I think I understood your meaning. I’ll find a proper transformation f for my data. Thank you very much. 🙂
            Stacey

          • Stacey says:

            The first thing popping into my head is euclidean distances of these coordinates. I’ll try it.

    • zachary huang says:

      Are the correct Ns really 80 and 62 for the first sample? or N=10 for each gender?

      I tried the same data in a different program and got a different P value and I think it is due to the differences in N.

  20. Nicole says:

    Hi Charles,

    I would like to know, from where you got the formula you are using in your KSINV function to calculate the critical value. I couldn’t find anything about this calculation.
    Why are you thinking this is more accurate?

    Kind regards,
    Nicole

    • Charles says:

      Hi Nicole,

      The KSINV function is based on the Kolmogorov distribution, which is described on the webpage
      Kolmogorov Distribution

      There is no reason to think that the values from the KSINV function are more accurate for values that are found in the KS Table. For values not in the table the values from KSINV may be more accurate than an interpolation from the KS Table.

      Charles

  21. Andy says:

    Hi Charles

    Thank you for your explanation! I have a question though. I do understand your procedure, but when I copied the cum% values from Fig1 to my stats software (GraphPad Prism) and run a ks test there I get the following results:
    Kolmogorov-Smirnov test
    P value 0,7591
    Exact or approximate P value? Approximate
    P value summary ns
    Significantly different? (P < 0.05) No
    Kolmogorov-Smirnov D 0,3000

    The same happens when I put those values on the online ks test software:
    http://www.physics.csbsju.edu/stats/KS-test.n.plot_form.html
    Here the result is "The maximum difference between the cumulative distributions, D, is: 0.3000 with a corresponding P of: 0.675"

    So in neither of those cases the result happens to be significant. Why is there such a difference?

    Thank you for your help!

    Andy

    PS: The numbers I entered into the programs were:
    Items in Data Set 1:
    5.000E-02 0.188 0.250 0.338 0.338 0.400 0.512 0.675 0.925 1.00

    Items in Data Set 2:
    0.113 0.177 0.194 0.371 0.565 0.629 0.661 0.726 0.855 1.00

    • Andy says:

      I just found the answer to my own question. What I need to use is the “raw” data, not the frequencies. The online test gives the right values (D, is: 0.2290 with a corresponding P of: 0.043), but GraphPad Prism gives me a different P:
      P value 0,0512
      Exact or approximate P value? Approximate
      P value summary ns
      Significantly different? (P < 0.05) No
      Kolmogorov-Smirnov D 0,2290

      I suppose this example shows this can be problematic, as Prism would have considered these two distributions are similar.

    • Charles says:

      Andy,

      When you enter the specified data into GraphPad or the online tool, these software programs expect that you are entering raw data values for the two samples and not cumulative distribution values. In fact, if I enter the same data in the range A1:B10, then the Real Statistics array formula =KS2TEST(A1:A10,B1:B10,TRUE) will output results similar to that given by the online KS tool, as shown below.

      0.05 0.113
      0.188 0.177
      0.25 0.194
      0.338 0.371
      0.338 0.565
      0.4 0.629
      0.512 0.661
      0.675 0.726
      0.925 0.855
      1 1

      Output from =KS2TEST(A1:A10,B1:B10,TRUE)

      D-stat 0.3
      p-value 0.675078154
      D-crit 0.564636591
      size 1 10
      size 2 10

      If you want to use GraphPad or the online tool to get the result for the original data, you must enter the following input data into these tools (unless they provide an option for frequency data input like the Real Statistics software does):

      21 21
      21 21
      21 21
      21 21
      23 21
      23 21
      23 21
      23 23
      23 23
      23 23
      23 23
      23 25
      23 27
      23 27
      23 27
      25 27
      25 27
      25 27
      25 27
      25 27
      27 27
      27 27
      27 27
      27 29
      27 29
      27 29
      27 29
      31 29
      31 29
      31 29
      31 29
      31 29
      33 29
      33 29
      33 29
      33 31
      33 31
      33 31
      33 31
      33 33
      33 33
      35 35
      35 35
      35 35
      35 35
      35 37
      35 37
      35 37
      35 37
      35 37
      35 37
      35 37
      35 37
      35 39
      37 39
      37 39
      37 39
      37 39
      37 39
      37 39
      37 39
      37 39
      37
      37
      37
      37
      37
      37
      37
      37
      37
      37
      37
      37
      39
      39
      39
      39
      39
      39

      Charles

  22. Milad says:

    hello sir
    i try to use two-sample kolmogorov-s test for negative data , but i can not, can you explainfor me what do i do?

    • Charles says:

      I have used this test with negative data without any problems. What you can’t make negative are the frequency values.

      I you send me an Excel file with your data, I can try to figure out where the problem is. See Contact Us for the email address.

      Charles

  23. Mohammad says:

    Hello,
    Thank you for such a nice information.
    the question is that, the maximum difference between cumulative frequencies between two data-sets maybe less than zero (negative) is. and if we deny the negative sign of this value, it would be more maximum than positive deference.
    To clear the question, I would give you an example:
    after calculate the difference between cumulative of two data-sets, the maximum and minimum values are 0.00086 (maximum positive) and -0.009196 (maximum negative) with alpha=0.05.
    Could you please give us information in this case?
    Thank you in advanced

    • Charles says:

      The test uses the maximum of the absolute values of the differences, and so in your example 0.009196 would be used (note that I need to use the absolute value of the maximum negative in this case).
      Charles

  24. Elizabeth says:

    Hi Charles

    I have tried to calculate the critic d from example 1, but I get a different result.
    Can you please write on the exact calculation that you did ?

    I did : 1.36* sqrt{(62+80)/(62*80)}= 0.230114
    and you get 0.224317
    Regards, Elizabeth

    • Charles says:

      Elizabeth,
      Your estimate is close enough. The value I used is a refinement which is supposed to be a bit more accurate.

      Let a = 1.36 and b = sqrt{(62+80)/(62*80)}.

      Instead of a I used A = KINV(.05) = 1.358099…

      Instead of b I used B = 1/(.11*b+1/b+.12)

      The value A*B = 0.224317.

      Charles

  25. Alexandra says:

    Hello! I have a problem using KS2TEST function – I have decimal values, and I get #VALUE instead of the answer. You said in the previous comments above that you revised the formula and that there is a new version that works with decimals. Where can I get it. If there isn’t one, how else can I solve the problem?

    • Charles says:

      Alexandra,
      I just tried an example with decimal numbers and it works fine.
      If you send me an Excel spreadsheet with your data I will try to figure out what has gone wrong.
      Charles

  26. Christoffer says:

    Thank you for an insightful post!

    Out of curiosity. Consider you have three independent samples instead of two, and you wish to investigate whether they come from the same distribution. Is there a K-S test for more than two samples?

    Thank you,

    Christoffer

  27. Mathieu says:

    Hello,
    I’ve reproduce the calculation of two sample K-S test with another sample. I get acceptance of H0 when I reduce alpha. I would have exected the opposite. Am I wrong or is there something wrong with my calculations?
    Thanks!

    ps: I’m trying to use this test to check if two rainfall gauges are in the same homogeneous region. (the sample is total rainfall yr-1 for stations 1 and 2, total rainfall yr-2 for stations 1 and 2, etc.) . Is this a good choice of test? Is there a better suited test for my task?

    • Charles says:

      Mathieu,

      The null hypothesis for this test is that both samples come from a population with the same distribution. The smaller the value of alpha the less likely to reject the null hypothesis (i.e. the critical region is smaller), and so it makes sense that you would reject the null hypothesis at some value of alpha and not reject it at a smaller value. E.g. in a two-tailed test if alpha = .5 then you would always reject the null hypothesis and with alpha = 0 you would never reject the null hypothesis. Also, since hypothesis testing is probabilistic in nature, researcher avoid the phrase “accept the null hypothesis” and use phrases such as “can’t reject the null hypothesis”.

      From the information you have provided, I can’t tell for sure whether two sample KS test is the right test to use. In particular, I don’t know what “the same homogeneous region” means, but if it means that the two regions might as well be considered to be parts of one larger region with the same characteristics, then it seems that the KS test is an appropriate test.

      Charles

      • Mathieu says:

        Thanks for this most appreciated reply. I had a look at the alpha-C(alpha) table and understood I was wrong in the interpretation of alpha.

        However, I’m still struggling to understand what does alpha represent. Let say I can’t reject the null hypothesis for alpha = 0.001. Can I associate this with a certain level of condidence? Or I can only say that the results of the KS test is “weak”, or…?

        Many thanks.

  28. Kate says:

    Hi,

    Does the two-sample Kolmogorov-Smirnov test require the class intervals to be equal across the whole range of the data? I have some very strongly skewed data where using equal classes loses much detail for low values, and gives many classes with zero observations for higher values. Will the K-s test still be valid if I use more classes at lower values and fewer at high values?

    Thank you

    • Charles says:

      Kate,
      If I remember correctly, the two samples must be random samples and so there is no equal interval requirement. See, for example, Example 2 on the referenced webpage.
      Charles

  29. Maria says:

    Hi,
    is a two-tailed test?
    Thank you for your website!
    Best regards

    Maria

    • Charles says:

      Maria,
      This is a two-tailed test. A one-tailed test can also be defined (i.e. F(x) >= G(x)), but that is not what I have done on the referenced webpage.
      Charles

  30. William Agurto says:

    Charles:

    In Real Statistics 3.5 a problem is still present for decimal values when using Kolmogorov-Smirnov test:
    1. With Kolmogorov-Smirnov (raw) option in “Additional Non-parametric Tests” in Data Analysis Tool; and
    2. With the function =KS2TEST
    As Donato, I get “#VALUE!” in both cases when decimal values are used in the columns to be compared.
    I send my data with the purpose of you can analyze it:

    Pretest(R1, D5:D18) Posttest(R2, E5:E18)
    0.997673113 0.264002119
    0.577424025 0.860549053
    0.000204027 0.923131895
    0.912676161 0.996300517
    0.923920025 0.001646899
    0.153323325 0.024959612
    0.309162014 0.964058172
    0.998794874 0.945081912
    0.999989659 0.997708808
    0.963914334 0.087593341
    0.047632851 0.998809053
    0.932418329 0.954355755
    0.999954638 0.38811265
    0.999686141 0.990518966

    KS function was written as follows:
    =KS2TEST(D5:D18,E5:E18,TRUE,I6)
    Data Analysis Tool used the same formula. Results were the following (equal in both cases):

    Two Sample Kolmogorov-Smirnov Test

    Alpha 0.05

    #VALUE! #VALUE!
    #VALUE! #VALUE!
    #VALUE! #VALUE!
    #VALUE! #VALUE!
    #VALUE! #VALUE!

    Thank you.

    William Agurto.

    • Charles says:

      William,
      I corrected this bug and tested the correction, but for some reason the correction was not included in the final version of the Rel 3.5 software. I have just issued a new release, Rel 3.5.1, which I hope now reinstitutes the correction. Sorry for the inconvenience and thanks for bringing this problem to my attention.
      Charles

      • William Agurto says:

        Charles:

        Real Statistics 3.5.1 now is OK for Kolmogorov-Smirnov test. There’s only a little bug: when using the Real Statistics Data Analysis Tool without changing the default value of alfa (.05), results are incorrect, because that tool uses KS2TEST formula with alfa=5 instead of alfa=0.05. But the results are correct if the user types the correct value of alfa in the interface window of “Additional Non-parametric Tests” (0.05). I supposed the mistake origin is this: in the window, alfa= 0,05 (with comma). I supposed it has to be 0.05 (with a dot).

        Thank you.

        William Agurto.

        • Charles says:

          William,
          There are two ways that I know of for changing the decimal delimiter: from the Control Panel or from Excel Options > Advanced. When I Use the Control Panel to make the change the value of alpha performs as expected. When I change the decimal delimiter using both Excel and the Control Panel everything works properly as well. When I only use Excel to change the delimiter the dialog box displays the default value of alpha as 0.05, but output from the data analysis tool is correct (and alpha is displayed as 0,05). In this latter case if instead I try to change the value of alpha in the dialog box (even from 0.05 to 0,05) then the output will display alpha as 5 (as you have said). You can of course change the value of alpha in the output to 0,05 and then the KS2TEST formula will give the correct answer. I will need to look into how to get this latter case to work correctly.
          Charles

          change the value of alpha from Excel

  31. Donato says:

    Sir,

    I’ using KS2TEST with the two samples below in column L and M, rows 2 to 14:

    L M
    2 -0.4470 0.1320
    3 -0.2720 -0.3060
    4 0.5120 -1.3670
    5 -0.2640 -0.6470
    6 0.2280 -0.1230
    7 1.0490 0.5440
    8 0.0980 0.6760
    9 -0.0420 1.8000
    10 0.0960 0.7110
    11 -0.5390 0.6600
    12 -0.3810 0.0910
    13 0.2620 0.7270
    14 0.1680 0.5640

    If I use: {=KS2TEST(L2:M14,,FALSE)} it works and I get the values below:

    1.1005
    0.074847002
    1.166116577
    1
    6

    If I use: {=KS2TEST(L2:L14,M2:M14,FALSE)} it does not work and I get this:

    #VALUE!
    #VALUE!
    #VALUE!
    #VALUE!
    #VALUE!

    • Charles says:

      Donato,

      You are correct that the {=KS2TEST(L2:L14,M2:M14,FALSE)} does not work. The formula works for integer data but not decimal values. I have now fixed the problem and the revised version of the function will be available in the next release of the software, which will be issued this week.

      The correct values of the formula are:

      D-stat 0.461538462
      p-value 0.086993934
      D-crit 0.500652861
      size 1 13
      size 2 13

      The formula {=KS2TEST(L2:M14,,FALSE)} uses data in frequency format, which is different from what you wrote in your comment. With your data, the input should be in the following format:

      K L M
      1.8 0 1
      1.049 1 0
      0.727 0 1
      0.711 0 1
      0.676 0 1
      0.66 0 1
      0.564 0 1
      0.544 0 1
      0.512 1 0
      0.262 1 0
      0.228 1 0
      0.168 1 0
      0.132 0 1
      0.098 1 0
      0.096 1 0
      0.091 0 1
      -0.042 1 0
      -0.123 0 1
      -0.264 1 0
      -0.272 1 0
      -0.306 0 1
      -0.381 1 0
      -0.447 1 0
      -0.539 1 0
      -0.647 0 1
      -1.367 0 1

      Note that the first column (column K) is not used in the formula, but is listed so that you can see the order of the data elements. Obviously this format is more useful with data that has lots of repetitions.

      The output is the same as listed above.

      Charles

  32. William Agurto says:

    Charles:

    I downloaded Real Statistics 3.4 and I used the Data Analysis Tool option “Other non parametric tests”. But the window that appeared allowed entering only one input array. I supposed then I had to enter the data as a unique frequency table, but the answer was different from what I got when used the function “KS2TEST” (when KS2TEST was used with two arrays instead of one frequency table; the results are the same when I used KS2TEST with the frequency table).
    My data sets are:
    Pretest (R1, G11:G30) Posttest (R2, H11:H30)
    72 95
    30 69
    38 61
    52 32
    62 8
    29 46
    17 83
    98 23
    59 16
    79 74
    22 2
    76 16
    22 53
    36 29
    69 45
    27 5
    46 67
    59 76
    55 92
    63 48
    I formed the following frequency table (using the “frequency” Excel function):

    Interval Pretest (M11:M20) Posttest (N11:N20)
    0-10 0 3
    10-20 1 2
    20-30 5 2
    30-40 2 1
    40-50 1 3
    50-60 4 1
    60-70 3 3
    70-80 3 2
    80-90 0 1
    90-100 1 2

    Real Statistics Data Analysis Tool provided the following results (using as array input the frequency table, both columns in R1 array, omitting R2 array, as in your example 1):
    D-stat 0.2
    p-value 0.770952945
    D-crit 0.409428086
    size 1 20
    size 2 20

    That answer is different from the results obtained when using Real Statistics Function KS2TEST with the two arrays (R1 and R2):
    =KS2TEST(G11:G30,H11:H30,TRUE,0.05)

    D-stat 0.25
    p-value 0.497342335
    D-crit 0.409428086
    size 1 20
    size 2 20

    But the answer is the same when using the frequency table:
    =KS2TEST(M11:N20,,TRUE,0.05)

    D-stat 0.2
    p-value 0.770952945
    D-crit 0.409428086
    size 1 20
    size 2 20

    Does it occurs because using the originals arrays R1 (Pre-test) and R2 (Post test) is more accurate than using frequency tables? If so, why doesn’t Real Statistics Analysis Tool allow to choose the two input arrays rather than only the frequency table?
    Are similar differences with other nonparametric tests in Real Statistics 3.4 Data Analysis Tool?
    Thank you very much.

    William Agurto.

    • Charles says:

      William,

      Thanks, as usual, for your very insightful and instructive comment. First of all, when data is organized in a typical frequency table then usually some accuracy is lost. However, if the frequency table doesn’t contain intervals but single values instead then no accuracy is lost. For the example that you have given such a “frequency table” takes the following form:

      Val freq-Pre freq-Post
      2 0 1
      5 0 1
      8 0 1
      16 0 2
      17 1 0
      22 2 0
      23 0 1
      27 1 0
      29 1 1
      30 1 0
      32 0 1
      36 1 0
      38 1 0
      45 0 1
      46 1 1
      48 0 1
      52 1 0
      53 0 1
      55 1 0
      59 2 0
      61 0 1
      62 1 0
      63 1 0
      67 0 1
      69 1 1
      72 1 0
      74 0 1
      76 1 1
      79 1 0
      83 0 1
      92 0 1
      95 0 1
      98 1 0

      This table was created as described in Example 2 of the referenced webpage. If you use the Other Non parametric Tests data analysis tool choosing the second and third columns of the above as your input you will get the same answer as =KS2TEST(G11:G30,H11:H30,TRUE,0.05)

      By the way, I give an example of how to use the Other Non parametric Tests data analysis tool for the two sample KS test on the webpage
      http://www.real-statistics.com/non-parametric-tests/data-analysis-tools-non-parametric-tests/

      As you have mentioned the data analysis tool only supports the frequency table format and not the raw data format. This was done only out of expediency so as not to delay the release. I will add the raw data format shortly.

      Charles

      Charles

  33. Mark says:

    Hi,

    Is there a way to do a large number of Kolmogorov-Smirnov tests at once? I have data separated in columns by a space, but because the output is an array, I can’t simply copy the function along the bottom row, the way I could for your Mann-Whitney function. Thanks!

    • Charles says:

      Mark,

      If you space the columns evenly you should be able to do what you want. You can copy array functions just like you do non-array function by using Ctrl-C and Ctrl-V. You just need to be sure that you copy the entire array. You don’t need to highlight the entire range that you are copying into: you just need to highlight the upper left cell of that range.

      Any reformatting needs to be done after the copying. You can then remove unwanted blank columns. While you can’t delete any cells in a range that contains an array formula, if you don’t need all the cells in the range, then after you finish the copying, you can use formulas such as =D4 to reference just the cells you want. Alternatively you can copy and then paste “values”.

      If none of this works for you, send me a spreadsheet describing what you want to accomplish and I will see whether I can figure out how to do it.

      Charles

  34. Junaid says:

    Hi, nice tutorial, very informative. I had a doubt. Can we use it for matching shapes?

    thanks

    • Charles says:

      It depends on what you mean by the same shape. The two sample KS test determines whether two samples have the same distribution. If the goal is to determine whether the samples come from the same distribution with some change in parameters (e.g both are normal but with different means and/or variances), then the answer is no.
      Charles

  35. Adam says:

    I am trying to use the KS2 function (after downloading/installing tyhe real statistics resource pack) using for example ks2test(c30:p30,c34:p34,TRUE) and I get an error. the c30:p30 is one sample and c34:p34 the other sample of data.

    • Adam says:

      Lets say I recreate your data from Figure 2. I then do the KS2test function and select the first column and second column like you do in your example. the output then just lists “D-stat” in the selected cell and no numbers or other output. Am I missing a step?

      • Charles says:

        Adam,
        The K2TEST function is an array function. You therefore need to highlight a 5 x 2 range and press Ctrl-Shift-Enter. This is explained on the referenced webpage.
        Charles

    • Charles says:

      Adam,
      If the error is the one you referenced in your subsequent comment, then I have addressed this in my response to that comment. If you are getting an error message, then I suggest that you send me the worksheet where this occurs, since the formula should work provided the data is appropriate.
      Charles

      • Alie says:

        Hi.. i dont get same value for the critical value in your example above using the manual formula. Sample Size (m) 80 and sample size2 (n) is 62. Using the formula Dmna = 1.36 Sqrt of (m+n)/(mn), I’ve got the critical value equivalent to 0.23011 while in your sample 0.22437. This could lead to another conclusion to accept the null hypothesis instead. Kindly show me the correct computation. I will be reporting this in our class. Thank you.

        • Charles says:

          Alie,
          Dmna = 1.36 Sqrt of (m+n)/(mn) is the commonly used estimate of the critical value, although I believe that the estimate using the KSINV function gives a more accurate answer. Thus I would use the .22437 figure.
          Charles

  36. fatemeh says:

    hello sir

    i try to use two-sample kolmogorov-s test for two distributions , but i can not find KSDIST and KSINV in excel, how can i find them?

Leave a Reply

Your email address will not be published. Required fields are marked *