What Is Post Hoc Testing

What is Post Hoc Testing? Unveiling the Secrets of Statistical Significance

Post hoc tests are a crucial part of statistical analysis, especially when you're dealing with ANOVA (Analysis of Variance) results. Understanding what they are and when to use them is essential for drawing accurate and meaningful conclusions from your data. This comprehensive guide will delve into the intricacies of post hoc testing, explaining its purpose, various methods, and crucial considerations. By the end, you'll be equipped to confidently apply post hoc tests to your own research and interpret the results with greater precision.

Introduction: The Need for Post Hoc Testing

Imagine conducting an experiment comparing the effectiveness of three different teaching methods on student exam scores. You run an ANOVA and find a statistically significant difference between the groups. Great! But the ANOVA only tells you that a difference exists somewhere among the three methods; it doesn't pinpoint where those differences lie. This is where post hoc tests come in. They are multiple comparison procedures used after an ANOVA (or other similar tests) to determine which specific groups differ significantly from one another. Without post hoc tests, a significant ANOVA result leaves you with a general, inconclusive finding.

When to Use Post Hoc Tests

Post hoc tests are specifically designed for situations where:

• You have a significant ANOVA (or Kruskal-Wallis) result: This indicates that there's at least one significant difference among your groups, but it doesn't reveal which groups are different.
• You have three or more groups: Post hoc tests are necessary when comparing more than two groups because conducting multiple t-tests would inflate the Type I error rate (the probability of falsely rejecting the null hypothesis).

Why Not Just Use Multiple t-Tests?

It's tempting to simply run multiple independent samples t-tests to compare all possible pairs of groups after a significant ANOVA. However, this approach is statistically flawed. Each t-test carries a risk of a Type I error (false positive). When you perform many t-tests simultaneously, the overall probability of making at least one Type I error increases dramatically. This phenomenon is known as the familywise error rate. Post hoc tests address this issue by controlling the familywise error rate, ensuring that your conclusions are more reliable.

Types of Post Hoc Tests: A Diverse Toolkit

Several post hoc tests exist, each with its own strengths and weaknesses, making the choice dependent on your data characteristics and research questions. Here are some of the most commonly used methods:

1. Tukey's Honestly Significant Difference (HSD):

• Purpose: Tukey's HSD is a powerful and widely used post hoc test that controls the familywise error rate effectively. It's particularly robust when the sample sizes are equal across groups.
• Assumptions: Assumes homogeneity of variance (the variances of the groups are equal) and normality of data.
• Strengths: Provides a simple and reliable method for comparing all possible pairs of groups.
• Weaknesses: Can be less powerful than other tests if the sample sizes are unequal.

2. Bonferroni Correction:

• Purpose: A relatively simple method that adjusts the significance level (alpha) for each individual comparison. It divides the original alpha level (usually 0.05) by the number of comparisons being made.
• Assumptions: Minimal assumptions regarding data distribution.
• Strengths: Easy to understand and implement; applicable to a wide range of situations.
• Weaknesses: Can be overly conservative (less likely to detect true differences), especially when conducting many comparisons. This leads to reduced statistical power.

3. Scheffe's Method:

• Purpose: A very conservative post hoc test that controls the familywise error rate for all possible contrasts (comparisons), including complex comparisons involving more than two groups.
• Assumptions: Minimal assumptions regarding data distribution.
• Strengths: Offers strong control over the familywise error rate and is suitable for complex comparisons.
• Weaknesses: Highly conservative, leading to lower power compared to other methods. Less likely to detect significant differences.

4. Newman-Keuls Test:

• Purpose: A step-down procedure that compares groups based on their mean differences. It's less conservative than Tukey's HSD but still controls the familywise error rate.
• Assumptions: Assumes homogeneity of variance and normality of data.
• Strengths: More powerful than Tukey's HSD when sample sizes are unequal.
• Weaknesses: Can be less reliable than Tukey's HSD if assumptions are violated.

5. Dunnett's Test:

• Purpose: Specifically designed for comparing multiple treatment groups to a single control group.
• Assumptions: Assumes homogeneity of variance and normality of data.
• Strengths: More powerful than other methods when comparing to a control group.
• Weaknesses: Not suitable for comparing all possible pairs of groups.

6. Games-Howell:

• Purpose: A post hoc test that is robust to violations of the assumption of homogeneity of variance. It is particularly useful when your groups have unequal variances.
• Assumptions: Does not assume homogeneity of variance.
• Strengths: Powerful and reliable even when variances are unequal.
• Weaknesses: May be less powerful than other tests if homogeneity of variance is met.

Choosing the Right Post Hoc Test:

The selection of the appropriate post hoc test is a crucial step. The ideal choice depends on several factors:

• Sample Size: For equal sample sizes, Tukey's HSD is often preferred. For unequal sample sizes, Games-Howell (if homogeneity of variance is violated) or Newman-Keuls may be more suitable.
• Homogeneity of Variance: If the assumption of homogeneity of variance is violated, Games-Howell is a robust option.
• Type of Comparison: Dunnett's test is best for comparisons against a control group. Scheffe's test is suitable for complex comparisons.
• Desired Level of Conservatism: Bonferroni is highly conservative, while Newman-Keuls is less so.

Interpreting Post Hoc Test Results

Post hoc test results are typically presented in a table showing the pairwise comparisons between groups, along with the p-values. A p-value less than your chosen significance level (e.g., 0.05) indicates a statistically significant difference between the two groups being compared. The table might also include confidence intervals, providing a range of plausible values for the true difference between the group means.

Practical Example:

Let’s say we conducted an ANOVA comparing the effectiveness of three different fertilizers (A, B, and C) on plant growth. The ANOVA showed a significant result (p < 0.05), indicating that at least one fertilizer differs from the others in its effect on plant growth. We then apply Tukey's HSD as our post hoc test and obtain the following results:

This indicates that fertilizer A differs significantly from fertilizer B (p = 0.01), and fertilizer B differs significantly from fertilizer C (p = 0.03). However, there is no significant difference between fertilizer A and C (p = 0.85).

Limitations and Considerations

• Assumptions: Many post hoc tests rely on assumptions about the data, such as normality and homogeneity of variance. Violations of these assumptions can affect the reliability of the results. Consider using robust methods if assumptions are violated.
• Multiple Testing: Even with post hoc tests, the more comparisons you make, the higher the chance of finding a statistically significant difference by chance alone. Always consider the context of your findings and the practical significance of the results.
• Effect Size: While statistical significance is important, it's crucial to consider the effect size. A statistically significant difference might be small and not practically meaningful.

Frequently Asked Questions (FAQ)

Q: What is the difference between a t-test and a post hoc test?

A: A t-test compares the means of two groups. Post hoc tests are used after an ANOVA (or similar test) to compare the means of three or more groups while controlling for the inflated Type I error rate that arises from multiple comparisons.

Q: Can I use post hoc tests if my ANOVA is not significant?

A: No. Post hoc tests are only meaningful if the ANOVA (or similar omnibus test) initially reveals a statistically significant difference between the groups.

Q: Which post hoc test should I use?

A: The choice depends on your data characteristics (sample size, homogeneity of variance) and research question. Consider the assumptions of each test and choose one that is appropriate for your data.

Q: What does "familywise error rate" mean?

A: The familywise error rate is the probability of making at least one Type I error (false positive) when conducting multiple statistical tests. Post hoc tests help control this rate.

Q: How do I interpret the p-values from post hoc tests?

A: A p-value less than your chosen significance level (e.g., 0.05) indicates a statistically significant difference between the groups being compared.

Conclusion: A Powerful Tool for Precise Analysis

Post hoc tests are invaluable tools in statistical analysis. They allow researchers to move beyond the general finding of an ANOVA and pinpoint the specific differences among multiple groups. Choosing the appropriate test and interpreting the results correctly is crucial for drawing meaningful conclusions from your data. By understanding the various types of post hoc tests and their underlying principles, you can enhance the rigor and precision of your research findings. Remember to always consider the assumptions, limitations, and the practical implications of your results alongside statistical significance. This comprehensive understanding of post hoc testing will empower you to conduct more robust and insightful statistical analyses.