One- and two-sample statistics¶

One- and two-sample statistics are statistics where we compare one group to some predifined mean or distribution or compare whether two gropus are different. There are parametric and non-parametric tests.

The t-test¶

The t-test is perhaps one of the simplest statistical tests you can run. Usually in electrophysiology we are comparing two different (independent) groups, comparing two different timepoints or drug treatment within a group (paired, not independent) or comparing whether a single group is different from a predefined mean (one-sample t-test) which can be used for a paired t-test. You can run a t-test using Scipy or Statsmodels in Python. I always recommend getting the confidence intervals of the difference between the two groups. Usually when the confidence intervals of the difference do not include 0 a statistically significant effect. The Further from 0 that your interval is the larger the effect size. Since the p-value is dependent on sample size this helps interpret the p-value. If you are running a t-test between two different groups there are several things to consider.

1. Are your data normally distributed? Yes, go to 2.

2. Are your two groups independent? Yes, use z-test, t-test or Welch’s t-test. No, use paired t-test.

3. Is the variance of your two groups the same? Yes, use standard t-test. No, use Welch’s t-test.

4. Are you sample sizes unequal? Use Welch’s t-test

5. How large are your sample sizes? Larger than ~30 use the z-test

The output¶

You will get the test statistic, a p-value and the degrees of freedom. You can also get confidence intervals. For simple tests like t-tests I prefer to get bootstrapped confidence intervals.

Scipy vs Statmodels¶

I prefer Scipy for the t-tests but, I will include the code for Statsmodels as well.

Two-sample t-test with equal variances¶

Scipy¶

Statsmodels¶

Comparing two-sample t-test with unqual variances with and without Welch’s correction¶

You will see that without Welch’s correction that p-value and the degrees of freedom (df) is inflated. One thing to note is that Welch’s correct will converge towards a standard t-test if the variances and sample sizes are equal.

Scipy¶

Statsmodels¶

One tailed t-test¶

One tailed t-tests are used when you have a specific hypothesis about the direction of your effect. This needs to be choosen before you collect your data. An example of when this would be used is if you have a treatment that “needs” to be more effective than a previous treatment or if you are replicating data and aleady have a hypothesis about the direction of your effect. The direction of you effect is determine by why group you have as one and two. If you expect one to be larger than two (positive differece between the groups) you want larger/greater. If you expect one to be smaller than two (negative difference between the groups) the you want less/smaller.

Paired t-test¶

The paired t-test is used when you have a subject that has two measures and you want to compare if the two measures are different. Paired data (aka within-subject data) violates the independence assumption of a traditional t-test. Also paired t-test will increase your statistical power by reducing between subject variance since each subject essentially acts as its own control.`

Scipy¶

Statsmodels¶

Statsmodels does not have a one sample t-test. Since the one-sample t-test is pretty simple to run we will manually compute it here. If you want to determine whether you difference is different from some other value than zero then you can replace 0 with that value.

Rank-based two-sample statistics¶

There are a variety of non-parametric two-sample rank-based statistics such as Mann-Whitney and Wilcoxon signed-rank test. Rank based tests can be used with continuous or ordinal data. Some things you should be aware of before using rank-based tests. If your data is continuous, try transforming your data before running a rank-based test. I often see people fall back to Mann-Whitney for non-normal data. However, in most cases a simple log-transform would have been enough to make the data fit the assumptions of the t-test and its variants. Another is that for rank-based tests in biological sciences we generally want to test if the location or scale is different but not both because that makes interpreting our data hard. To do so we are most interested in the median. If your two samples each have a distribution (same gamma) with different shapes AND dispersion you may not have a difference in medians (due to location or scale) which is what we really want to test with rank-based tests. The test is still valid you just cannot say where the difference in your samples are. If you do not know the terms location, shape, scale or dispersion check out the Distributions chapter. Non-parametric rank-based statistics really have almost as many assumptions as parametric-based statistics. Rank-based statistics can also be sensitive the number of ties or the number of same values between your two group. Rank-based statistics are also not a substitute for parametric tests when your sample size is small (n=3 for each group). The last case where rank-based tests may be useful is when you have outliers

Mann-Whitney U¶

Mann-Whitney is a rank-based statistical test for two indepedent samples. Rather than compare the mean difference, Mann-Whitney essentially scores how many values are larger than in the other group and creates a U score for each group which is used to generate a p-value. One thing to note that the U value that Scipy outputs changes depending on what group is one and what group is two but, the p value will be the same since the p value is calculated from the smaller U value. The second U value can be calculated $U2 = n1*n2-U1$. Another thing to note is that you have to use a different method to calculate the p value if your sample size is small. The Mann-Whitney tests has a table of precomputed critical values for smaller sample sizes. For the Mann-Whitney test to be valid your data must come from the same distribution type otherwise interpreting your what difference your significant p-value is showing is next to impossible. When reporting the Mann-Whitney p-values you should also report the difference between the medians (or the median of each group), the Hodges-Lehmann estimator which is like an effects size measures for the Mann-Whitney test and the probability of superiority which gives you the probability that values in one group will be large than another when randomly choosing two points.

Some other parameters in the Scipy version are alternative. This is if you are expecting group one to be more or less than group two. The method parameter method is used to define whether to use the exact, asymptotic or permutation p value calculation. “Auto” is probably the best for most things but Scipy recommends passing a Permutation instance if there are less than 10 samples in one of the groups and there are ties.

Brunner-Munzel¶

Brunner-Munzel is a rank-based tests with very few assumptions that tests whether the probability of getting large values in both groups is the same. It is robust to outliers, it can be used when the shape or dispersion of a distributions are different or if the distributions are different. However, there are some assumptions which means it will not be useful in most cases. One is that Scipy recommends that you have a sample size greater than 10 for both groups so most biological experiments will not hit this threshold. The other is that responses are ordinal or can be assumed to be ordinal (i.e. number of votes)

Bootstrap and permutation tests¶

Bootstrap and permutation tests are resampling tests. They are really more a method than an actual test since you can apply them to parametric statistical tests. Some people insist that these tests are better for small sample sizes however, the core problem remains that the small sample size is too small.

Permutation test¶

The permutation test tests the null hypothesis that all samples come from the same distribution. The permutation test takes the follow steps:

1. Get the difference between your two groups

2. Pool your groups

3. Randomly assign values to each group. The number of assigned values should be the same size as the original groups. Ideally you want to choose all unique permutations without repeats. However, sometimes that is not possible with large sample sizes so you get a subset.

4. Find the difference between the groups.

5. Repeat steps 3 and 4 n times.

6. Find whether your difference is significant. The permutation test can also be run on the other tests like the t-test and Mann-Whitney U test.

Bootstrap test¶

The bootstrap test takes a different approach by repeatedly sampling with replacement from each group and comparing the observed difference to the bootstrap difference. Scipy’s bootstrap takes some extra work to get a p-value. We need to get the percentile rank of the observed difference.