" Hey John, did you like the model?" asked Fred. John - " Why do you think it is a good model?"; "Because it has p-value at 0.03 !!" replied Fred.
"Can you explain what that means?" asked John who heads sales for a mycompany.com where Fred works in the ‘Ilovedata’ department as a Data Scientist. Fred replied to John that he will get back to him. Have you ever been in a similar situation when someone asks about p-value as their future decision is based on its interpretation? If so, then this article is for you.
Many times, I have seen data science professionals who know very well what is p-values but they struggle to explain it. I have also seen many professionals who don’t want to get into technical details but want to know more and understand why it is so important.
With these thoughts in mind, I tried my favorite Wikipedia and the initial definition is greatly written but needs little decoding for others.
Let me say here that I run a supermarket named "mystuff" and we do online home deliveries with a promise of 1 hour deliveries. I am confident that we never miss our scheduled delivery slots and often deliver much faster. How can we prove this, we need data?
We started to collect data on delivery times from a random selection of deliveries in order to conduct a test. This is a test that has just one variable, do we deliver in 1 hour or not, as less than an hour is good while greater is not good. So we will use a one-tailed test.
What we have done is create a theory that "mystuff" makes its deliveries within in an hour. This is called the Null hypothesis and what if we are unable to do this? What if we deliver beyond 1 hour? If this is the case, then this is called an alternate hypothesis.
So now the idea is to check if my claim is right or wrong. And just how right or wrong I am.
After collecting some sample data and performing a one tail test we produce an example Z score which helps us to understand where the overall data lies as compared to the average in the sample. Our test scores provides us with a p-value of 0.04. We will cover a detailed article on the Z score in a future post. Now that we have a test score how do we measure it for significance.
Depending upon the domain, I can define the significance level, which is also called an alpha level. For example, consider it as minimum pass marks required to accept the claim. FOR example, I can say if I get the test result with a 90% confidence then I will consider it successful and can accept the claim. This is an important factor that could lead to our decision in hypothesis testing.
Here I will be more surprised to see the test result much below 90% and will not be much surprised by 90 and above so that the key. P-value answers this by creating a probability and we can convert our percentage to a scale of 1 where 90% can be converted as 0.10. Anything below 0.10 or lower will be adding weight to my claim or reinforcing it.
In our case a score of 0.04 does not means there is a probability of 4% that result of the test is due to a chance or random. Ideally, a p-value doesn't give any proof to a claim if provides an indication of how likely something is to occur e.g. its probability of occurring.
1. It is significant if it is below the significance level (alpha value). In our case, we defined 0.10 while generally, it is 0.05 again it is dependent upon the domain and problem area.
2. P-value gives an indication of probability
3. It is used to challenge our initial claim when the result is statistically significant.
4. Always defining a null and alternative hypothesis and considering an alpha value to accept or reject the claim or belief will prove more beneficial for the experiment or test.
5. While a low p-value favors rejecting the null hypothesis, it does not address the likelihood of rejecting it
If you found this Article interesting, why not review the other Articles in our archive.