【Statistics】HYPOTHESIS TEST(SIGNIFICANCE TEST)

本文着重梳理 假设检验 HYPOTHESIS TEST(SIGNIFICANCE TEST)，通过 逻辑性知识 和 概念性知识 两部分厘清该重点内容。

1.Logic of Hypothesis Test

0) What’s hypothesis test？

If we have some doubt in the origin hypothesis or assumption, then we can raise a hypothesis test to prove our doubt or said reject the origin hypothesis.

For example:

Then, we introduce some new concepts on hypothesis:

H0H_0H0 (Null hypothesis): Something we doubt.
HaH_\text{a}Ha (Alternative hypothesis): Our guess, or said the new hypothesis
Single-sided and Double-sided Hypothesis :
- If our new hypothesis is in the form of newhypothesis≠somenumbernew\ hypothesis \neq some \ numbernew hypothesis̸=some number, this is a double-sided hypothesis ;
- If our new hypothesis is in the form of newhypothesis≤somenumbernew\ hypothesis \leq some \ numbernew hypothesis≤some number or newhypothesis≥somenumbernew\ hypothesis \geq some \ numbernew hypothesis≥some number, this is a double-sided hypothesis.

PAY ATTENTION: All the hypothesis is aimed at testing the population parameter.

1) Set up Hypothesis

To set up a Null Hypothesis, we can just need to figure out what is the concerned problem. Or said the feature of null hpothesis is that there is no news if the null hypothesis is actually true.

To set up a lnternative Hypothesis, we use the number of null hypothesis and then choose single-sided or double-sided hypothesis to set up our internative hypothesis.

2) Set up Significance level

PAY ATTENTION: Before we carry out the calculation, we need to set up a significance level. It is an ethical problem if we set up a significance level to suit our calculation result in order to generate a attracting conclusion.

3) Take Sample and Calculate

4) Make Conclusion

2.Concepts of Hypothesis Test

1) What’s p-value and significance level？

p−valuep-valuep−value: It is a probability that current sample statistic occur.

P(observecurrentsamplestatistic∣H0isTrue)P(observe\ current\ sample\ statistic |H_0 \ is \ True) P(observe current sample statistic∣H0 is True)

α\alphaα: We call it significance level. It is a thresold that quantify the word “extreme”. In ohter words, it’s a relatively small probability that indicates whether the p−valuep-valuep−value is small enough that shake our belief on H0H_0H0.
$power $: It is a probability that not making Type II error

P(rejecetH0∣H0isfalse)=1−P(notrejecetH0∣H0isfalse)=P(notmakingTypeIIerror)P(rejecet\ H_0|H_0 \ is \ false) = 1 - P(not \ rejecet\ H_0|H_0 \ is \ false) = P(not\ making \ Type \ II \ error) P(rejecet H0∣H0 is false)=1−P(not rejecet H0∣H0 is false)=P(not making Type II error)

2) Type I Error & Type II Error

i) Understanding the concepts

Meaning of Type I Error ：The origin hypothesis (H0H_0H0) is actually True, but due to some extreme event happens (p−value<αp-value < \alphap−value<α), we consider the hypothesis might be wrong and therefore reject it. It is obivous that if the α\alphaα is too large, we can easily get a Type I Error.

Meaning of Type II Error：The origin hypothesis (H0H_0H0) is actually False, but nothing seems to happen (p−value>αp-value > \alphap−value>α), then we consider the hypothesis should be true and therefore accpet it. It is obivous that if the α\alphaα is too small, we can easily get a Type II Error. Another way to think of Type II Error is that if we can start from using the concept of powerpowerpower instead of p−valuep-valuep−value.

Trade-off problems : There exists a trade-off between Type I Error and Type II Error, which means that we need to set an appropriate α\alphaα to “balance” the error probability of these two type of error. Here is an example on trade-off problem:

Employees at a health club do a daily water quality test in the club’s swimming pool. If the level of contaminants are too high, then they temporarily close the pool to perform a water treatment.

We can state the hypotheses for their test as H0H_0H0 : The water quality is acceptable vs. HaH_\text{a}Ha : The water quality is not acceptable. Consider the following two questions:

In terms of safety, which error has the more dangerous consequences in this setting?

What significance level should they use to reduce the probability of the more dangerous error?

FROM Khan Academy

What will affect the error probability :

Significance level α\alphaα: If α↑\alpha \uparrowα↑, then power↑power \uparrowpower↑ and P(typeIerror)↑P(type \ I \ error)\uparrowP(type I error)↑
Sample size n: If n↑n \uparrown↑, then power↑power \uparrowpower↑. But it doesn’t impact the likelihood of a Type I error.Larger samples are still preferred since they produce less variable results, but we’ll still reject a true H0H_0H0 at a rate equal to the significance level ααα.
Statistic variability : If Statisticvariability↓Statistic\ variability \downarrowStatistic variability↓ , then power↑power \uparrowpower↑. It suits the intuition as when the statistic variablity is low, the outliner should be easier to figured out, and therefore we can hava higher probability to reject it. But we could not control this variable.
Distance from true parameter to H0H_0H0 : If distance↑distance \uparrowdistance↑ , then power↑power \uparrowpower↑. It suits the intuition as when the true value far from H0H_0H0, we can hava higher probability to reject it. But we could not control this variable.

ii) Some associations

看论文经常会接触到以下的表格，出处正是来自statistics

Table of error types	H0H_0H0 is True(in reality)	H0H_0H0 is False(in reality)
Fail to reject	Correct inference	Type II error(False Negative)
Reject	Type I error(False Positive)	Correct inference

ROC曲线

3) Z-test and T-test

3.Others

1.Some common ideas between confidence interval and significance test