8.4 Level of Significance and the p-Value – Simple Stats Tools (2024)

The conceptlevel of significance is used to adjudicate whether the probability (of our results if the null hypothesis is true) is too high to dismiss the null hypothesis or low enough to allow us to reject the null hypothesis. In other words, the level of significance is what we use to proclaim results as statistically significant (when we reject the null hypothesis) or not statistically significant (when we fail to reject the null hypothesis).

Think about it this way: recall that with confidence intervals we had selected 95% certainty and 99% certainty as meaningful levels of confidence. What is left is 5% and 1% “uncertainty”, as it were, which we agree to tolerate. These 5% or 1% are distributed equally between the two tails of the normal distribution (2.5% on each side or 0.5% on each side, respectively). They also correspond to z=1.96 and z=2.58. Following the logic of Example 8.2 (A) from the preivious section, in order to reject a null hypothesis, we want the probability to be lower that these 5% or 1% (so that we can “feel confident enough”).

And this is exactly it: When we put it that way, saying that we want the probability (of the null hypothesis being true) — called a p-value — to be less than 5%, we have essentially set the level of significance at 0.05. If we want the probability to be less than 1%, we have set the level of significance at 0.01. We can go even further: we might want to be extra cautious and to wanta “confidence” of 99.99%, so that we want the probability to be less than 0.01% — then we have set the level of significance at 0.001.

These three numbers — 0.05, 0.01, and 0.001 — are the most commonly used levels of significance. The level of significance is denoted by the small-case Greek letter a, i.e.,α, thus we usually choose one of the following:

You can think of the significance level as the acceptable probability of being wrong — and what is acceptable is left to the discretion of the researcher, subject to the purposes of the particular study.

Following the logic presented in Example 8.2(A) then, if the probability of the result under the null hypothesis — the p-value — is smaller than a pre-selected significance levelα, the null hypothesis is rejected and the result is considered statistically significant[1]. This is denoted in one of the following ways:

p ≤ 0.05

p ≤ 0.01

p ≤ 0.001[2]

To summarize, when a hypothesis is tested, we end up with an associated p-value (again, the probability of the observed sample statistics if the null hypothesis is true). We compare the p-value to the pre-selected significance levelα: if p≤ α, the results are statistically significant and therefore generalizable to the population.

So far so good? Good. However, unfortunately this isn’t all (sorry!). What I have presented above is the most conventional treatment of how to use and interpretp-values. It is attractively straightforward — but it’s also arbitrary, and its true interpretation is subject of an ongoing debate. As an introduction to the topic, I will leave it at that but you should be aware that there’s more to the p-value, and that its usage has been (rightfully) questioned and/or challenged in recent years.[3].

Going back to our example from the preivious section, let’s see how the p-values can change due to particular features of the study, like the sample size. Example 8.2(B) illustrates.

Again, Example 8.2 is a heuristic device, used only to explain the logic of hypotheses testing. Of course, normally we wouldn’t have information about population parameters and we will be using sample statistics (i.e., we would use not only the sample mean but also the sample standard deviation s, to calculate the estimated sampling distribution ). (Not to mention that we would have two different standard deviations, one for the trained group and one for the not-trained group of employees.) As you learned in the previous chapter, this moves us from using the z-distribution to the t-distribution with given degrees of freedom. Recall that with a sample size of about 100 — i.e., with df=100 — the two distributions converge.

Here then is a quick-and-dirty method you can use as a preliminary indication of whether something will be statistically significant. Since z=1.96 corresponds to 5% probability (2.5% in each tail), and z=2.58 corresponds to 1% probability (0.5% in each tail), even without knowing the exact p-value associated with a given z-value, you can guess that getting a z<1.96 will be non-significant while a z>1.96 will be significant atα=0.05; similarly, getting a z>2.58 will be statistically significant atα=0.01[4]. As samples used in sociological research are commonly of N>100, the same insight applies to the corresponding t-values with df≥100. Understand, however, that this is not an official way to test hypotheses or report findings: to do that, you always need to report the exact p-value associated with a z-value or a t-value with given df[5].

One-tailed tests. Finally, a note on one-tailed tests. While at the beginner researcher level, I advise you against using them yourself, it is not a bad idea to know they exist and what they are. Briefly, the idea is that if we have a good reason to suspect not only a difference/effect but a difference/effect with a specific direction (i.e., positive or negative), we can specify the hypotheses accordingly. To use Example 8.2(A) again, say, we think there is no possibility that the training course decreased productivity scores. Then we can state the hypotheses as:

• H₀: The training course either did not affect productivity or decreased it; ≤.
• H_a: The training course increased productivity; >.

This is a stronger claim (that’s why it needs to be well-justified) — we test not a difference (that can be either positive or negative) but an increase. Thus, we move the significance level to only one of the tails, as it were, the positive (right) tail, so instead of 2.5% being there, 5% are.

This change in probability essentially “moves” the z-value corresponding to significance closer to the mean; now a smaller z-value will have the p-value necessary to achieve statistical significance. To be precise, 5% (2.5% in each tail) corresponded to z=1.96; all 5% in the right tail corresponds to z=1.65[6]. This obviously “lowers the bar” of achieving statistical significance without changing the level of significanceαitself, and makes rejecting the null hypothesis easier, hence my description of the two-tailed test as more conservative (and my insistence on using it instead of a one-tailed test).

Before we move on to the last section of this theoretical chapter, the promised warning about the meanings of the term significance.

When testing hypotheses, I defined the significance level as sort of probability of being wrong we are willing to tolerate. This implies that a likelihood of making anerroneousdecision about the null hypothesis (to reject it or not) exists. The next and final section deals with just that.