21. Hypothesis Testing

This document provides a comprehensive guide to hypothesis testing, covering its fundamental concepts, common pitfalls, and key metrics.

21.1 Hypothesis Testing Guide

Hypothesis testing is a statistical method used to make decisions or draw conclusions about a population based on sample data. It involves formulating a hypothesis about a population parameter and then using sample data to determine whether there is enough evidence to reject that hypothesis.

The general process of hypothesis testing involves the following steps:

1. Formulate Hypotheses: Define a null hypothesis ($H_0$) and an alternative hypothesis ($H_a$).
2. Choose Significance Level ($\alpha$): Determine the probability of rejecting the null hypothesis when it is actually true.
3. Select Test Statistic: Choose an appropriate statistical test based on the data type and research question.
4. Collect Data: Gather sample data relevant to the hypothesis.
5. Calculate Test Statistic: Compute the value of the chosen test statistic from the sample data.
6. Determine P-value: Calculate the probability of observing the sample data (or more extreme data) if the null hypothesis were true.
7. Make a Decision: Compare the p-value to the significance level ($\alpha$).
   • If p-value $\le \alpha$, reject the null hypothesis.
   • If p-value $ > \alpha$, fail to reject the null hypothesis.
8. Interpret Results: State the conclusion in the context of the original research question.

21.2 Null and Alternative Hypotheses

Null Hypothesis ($H_0$)

The null hypothesis represents the default assumption or the status quo. It typically states that there is no effect, no difference, or no relationship between variables. It is the hypothesis that we aim to find evidence against.

Example:

• A new drug has no effect on blood pressure.
• There is no difference in average scores between two teaching methods.

Alternative Hypothesis ($H_a$ or $H_1$)

The alternative hypothesis is what we are trying to find evidence for. It contradicts the null hypothesis and suggests that there is an effect, a difference, or a relationship.

Types of Alternative Hypotheses:

• Two-tailed: States that there is a difference or effect, but does not specify the direction.
  • Example: The average blood pressure is different after taking the new drug. ($H_a: \mu \ne \mu_0$)
• One-tailed (Right-tailed): States that there is a difference or effect in a specific positive direction.
  • Example: The new drug lowers blood pressure. ($H_a: \mu < \mu_0$)
• One-tailed (Left-tailed): States that there is a difference or effect in a specific negative direction.
  • Example: The new drug increases blood pressure. ($H_a: \mu > \mu_0$)

21.3 Statistical Significance

Statistical significance indicates that the observed results are unlikely to have occurred by random chance alone, assuming the null hypothesis is true.

• Significance Level ($\alpha$): This is a pre-determined threshold (commonly set at 0.05, 0.01, or 0.10) that represents the maximum probability of rejecting the null hypothesis when it is actually true (Type I error).
• Significant Result: If the p-value is less than or equal to the significance level ($\alpha$), the result is considered statistically significant. This means we have enough evidence to reject the null hypothesis.
• Non-significant Result: If the p-value is greater than the significance level ($\alpha$), the result is not statistically significant. This means we do not have enough evidence to reject the null hypothesis.

21.4 P-value

The p-value (probability value) is the probability of obtaining test results at least as extreme as the results actually observed, assuming that the null hypothesis is true.

• Low P-value: Indicates that the observed data are unlikely under the null hypothesis, providing evidence to reject $H_0$.
• High P-value: Indicates that the observed data are likely under the null hypothesis, suggesting that we should not reject $H_0$.

Decision Rule:

• If $p \le \alpha$, reject $H_0$.
• If $p > \alpha$, fail to reject $H_0$.

Example: If we are testing a new drug and obtain a p-value of 0.03 with a significance level of 0.05, we reject the null hypothesis. This suggests that the observed effect of the drug on blood pressure is statistically significant.

21.5 Type I and Type II Errors

When making a decision in hypothesis testing, there are four possible outcomes, two of which involve errors:

Type I Error (False Positive)

• Definition: Rejecting the null hypothesis when it is actually true.
• Probability: Denoted by $\alpha$ (the significance level).
• Example: Concluding that a new drug is effective when it actually has no effect.

Type II Error (False Negative)

• Definition: Failing to reject the null hypothesis when it is actually false.
• Probability: Denoted by $\beta$.
• Example: Concluding that a new drug is not effective when it actually is effective.

21.6 Statistical Power

Statistical power is the probability of correctly rejecting a false null hypothesis. It is the probability of detecting an effect when one truly exists.

• Power = 1 - $\beta$

Factors Affecting Power:

• Sample Size: Larger sample sizes generally lead to higher power.
• Significance Level ($\alpha$): A larger $\alpha$ (e.g., 0.10 vs. 0.05) increases power but also increases the risk of a Type I error.
• Effect Size: The magnitude of the difference or relationship being studied. Larger effect sizes are easier to detect, leading to higher power.
• Variability in Data: Lower variability in the data increases power.

Importance of Power: Researchers aim for studies with adequate statistical power to ensure that if an effect truly exists, it is likely to be detected. A study with low power might fail to find a real effect, leading to a Type II error.