The Difference Between Alpha and P-Values

The Difference Between Alpha and P-Values In conducting a test of significance or hypothesis test, there are two numbers that are easy to get confused. These numbers are easily confused because they are both numbers between zero and one, and are both probabilities. One number is called the p-value of the test statistic. The other number of interest is the level of significance or alpha. We will examine these two probabilities and determine the difference between them. Alpha Values The number alpha is the threshold value that we measure p-values against. It tells us how extreme observed results must be in order to reject the null hypothesis of a significance test. The value of alpha is associated with the confidence level of our test. The following lists some levels of confidence with their related values of alpha: For results with a 90 percent level of confidence, the value of alpha is 1 - 0.90 0.10.For results with a 95 percent level of confidence, the value of alpha is 1 - 0.95 0.05.For results with a 99 percent level of confidence, the value of alpha is 1 - 0.99 0.01.And in general, for results with a C percent level of confidence, the value of alpha is 1 - C/100. Although in theory and practice many numbers can be used for alpha, the most commonly used is 0.05. The reason for this is both because consensus shows that this level is appropriate in many cases, and historically, it has been accepted as the standard.  However, there are many situations when a smaller value of alpha should be used. There is not a single value of alpha that always determines statistical significance. The alpha value gives us the probability of a type I error. Type I errors occur when we reject a null hypothesis that is actually true. Thus, in the long run, for a test with a level of significance of 0.05 1/20, a true null hypothesis will be rejected one out of every 20 times. P-Values The other number that is part of a test of significance is a p-value. A p-value is also a probability, but it comes from a different source than alpha. Every test statistic has a corresponding probability or p-value. This value is the probability that the observed statistic occurred by chance alone, assuming that the null hypothesis is true. Since there are a number of different test statistics, there are a number of different ways to find a ​p-value. For some cases, we need to know the probability distribution  of the population.​​ The p-value of the test statistic is a way of saying how extreme that statistic is for our sample data. The smaller the p-value, the more unlikely the observed sample. Difference Between P-Value and Alpha To determine if an observed outcome is statistically significant, we compare the values of alpha and the p-value. There are two possibilities that emerge: The p-value is less than or equal to alpha. In this case, we reject the null hypothesis. When this happens, we say that the result is statistically significant. In other words, we are reasonably sure that there is something besides chance alone that gave us an observed sample.The p-value is greater than alpha. In this case, we fail to reject the null hypothesis. When this happens, we say that the result is not statistically significant. In other words, we are reasonably sure that our observed data can be explained by chance alone. The implication of the above is that the smaller the value of alpha is, the more difficult it is to claim that a result is statistically significant. On the other hand, the larger the value of alpha is the easier is it to claim that a result is statistically significant. Coupled with this, however, is the higher probability that what we observed can be attributed to chance.