approx-rand-test-0.0.3: Approximate randomization test

Stability experimental Daniël de Kok Safe-Infered

Statistics.Test.ApproxRand

Description

This module provides functionality to perform approximate randomization tests (Noreen, 1989).

Synopsis

Description

Approximate randomization tests rely on a simple premise: given a test statistic, if the null-hypothesis (the samples do not differ) is true, we can randomly swap values between samples without an (extreme) impact on the test statistic. Otherwise, the null-hypothesis must be rejected.

The test works by generating a given number of sample shuffles and computing the test statistic for each shuffle. If r is the number of shuffled samples where the test statistic is at least as high as the test statistic applied on the original samples; and N the number of shuffles, then the null-hypothesis is rejected iff (r + 1):(N + 1) < p-value (for one-sided tests).

Two kinds of test are supported:

• Paired sample (`approxRandPairTest`): values from samples are shuffled pair-wise. This requires the samples to have an equal length.
• Unpaired sample (`approxRandTest`): values from samples are shuffled among both samples. Consequently the i-th element of one sample does not bear a relationship with the i-th element of the other sample. The shuffled samples retain the sizes of the original samples.

Both tests can be performed as a one-tailed or two-tailed test.

Examples

Both unpaired and paired sample tests use the `Rand` monad to obtain random numbers. We can obtain a pseudo-random number generator that is seeded using the system clock using the `newPureMT` function (please refer to the documentation of `Pure64` for more information):

``` prng <- newPureMT
```

Suppose that we have the samples `s1` and `s2`. We could now perform a Two-Tailed randomization test with 10,000 shuffles and the mean difference as the test statistic, by running `approxRandTest` in the `Rand` monad (at the p = 0.01 level):

``` evalRandom (approxRandTest TwoSided meanDifference 10000 0.01 s1 s2) prng
```

It is also possible to obtain the test scores of the shuffled samples directly (e.g. to inspect the distribution of test scores) using the 'approxRandScores'/'approxRandPiarScores' functions:

``` evalRandom (approxRandScores meanDifference 10000 0.01 s1 s2) prng
```

Data types

data TestResult Source

The result of hypothesis testing.

Constructors

 Significant Double The null hypothesis should be rejected NotSignificant Double Data is compatible with the null hypothesis

Instances

 Eq TestResult Ord TestResult Show TestResult

Computations with random numbers that can fail.

Approximate randomization tests

Arguments

 :: TestType Type of test (`OneTailed` or `TwoTailed`) -> TestStatistic Test statistic -> Int Number of shuffled sample to create -> Double The p-value at which to test (e.g. 0.05) -> Sample First sample -> Sample Second sample -> Rand TestResult The test result

Apply an approximate randomization test.

In approximate randomization tests, the values of two samples are shuffled among those samples. A test statistic is calculated for the original samples and the shuffled samples, to detect whether the difference of the samples is extreme or not.

Arguments

 :: TestStatistic Test statistic -> Int Number of shuffled samples to create -> Sample First sample -> Sample Second sample -> Rand [Double] The scores of each shuffle

Generate a given number of shuffled samples, and calculate the test score for each shuffle.

This function does not require the samples to have an equal length.

Arguments

 :: TestType Type of test (`OneTailed` or `TwoTailed`) -> TestStatistic Test statistic -> Int Number of shuffled samples to create -> Double The p-value at which to test (e.g. 0.05) -> Sample First sample -> Sample Second sample -> RandWithError TestResult The test result

Apply a pair-wise approximate randomization test.

In pair-wise approximate randomization tests the scores at a given index are swapped between samples with a probability of 0.5. Since swapping is pairwise, the samples should have the same length.

Arguments

 :: TestStatistic Test statistic -> Int Number of shuffled samples to create -> Sample First sample -> Sample Second sample -> RandWithError [Double] The scores of each shuffle

Generate a given number of pairwise shuffled samples, and calculate the test score for each shuffle.

Since the scores at a given index are swapped (with a probability of 0.5), the samples should have the same length.

Test statistics

A test stastic calculates the difference between two samples. See `meanDifference` and `varianceRatio` for examples.

Calculates the difference mean of two samples (mean(s1 - s2)). When the two samples do not have an equal length, the trailing elements of the longer vector are ignored.

Calculates the mean difference of two samples (mean(s1) - mean(s2)).

Calculate the ratio of sample variances (var(s1) : var(s2)).