Statistical Significance in Decision-making

Business Analytics
Knowledge Center


Aniket Deolikar
Consultant, India

Statistical Significance in Decision-making

🔥NEW Companies are increasingly using business analytics and are increasingly relying on data to make operational, tactical and even strategic business decisions. Obviously, when making decisions, managers should not rely on calculations and models they don't understand. That's why understanding statistical significance is becoming more important for managers.
First of all, you should realize that in statistics, "significant" does NOT mean something is big or important! Rather it means that the result we see in some sample also exists in the (total) population.

What is statistical significance?
Statistical significance is the likelihood that a relationship between two or more variables we find in our results is NOT caused by random chance. If you draw conclusions out of a lot of data, you have to use a sample. A sample is a set of individuals / objects collected or selected from a bigger statistical population by some defined procedure. But in any experiment or observation that involves drawing a sample from a larger total population, there is always the possibility that an observed effect occurred due to sampling error (coincidence). Statistical significance is a way of showing to what extent that the answers which were obtained from the data are not totally due to luck. It is the confidence level you may have in a mathematical way that some result or conclusion is reliable. The more critical your decision based on the data is going to be for your firm, the higher you will want the statistical significance to be.

Statistical hypothesis testing
Statistical hypothesis testing is the method by which an analyst determines that the results in the data are not explainable by chance alone. This test provides a p-value, which is the probability of observing results as extreme as those in the data, assuming the results are truly due to chance alone. A p-value of 5% or lower is normally considered to be statistically significant.
But when the p-value is large, then the results in the data are explainable by chance alone. So when the p-value is higher than 5% (0,05) we consider the results statistically NOT significant as they could easily be explained by chance alone.

Mistakes and errors made while working with statistical significance
  • SAMPLING ERRORS (too small or non-representative). Statistical significance tells us that if something works for a smaller sample, then with how much confidence can we say that it will also work the same for a large population. But many times people make mistake of taking a sample which does not represent the total population. For example the objects in the sample might be biased towards some specific color, say red, while the total population might be a variety of colors. The statistical significance becomes low when such errors are made. Choosing a good diversified sample which represents the total population accurately is very important for getting a high statistical significance.
  • NON-SAMPLING ERRORS. There are other errors that may occur that are as important as the sampling errors. Non-sampling errors include the ones where the measurement and experiment protocols were broken. Sometimes the data may get lost. Sometimes the people in the sample may have lied in the survey. Etcetera. These factors often are difficult to be controlled, but a careful analysis will help to reduce these non-sampling errors.
Applications of statistical significance
Statistical significance basically tells us whether we should trust the results of some experiment or test like for example an A/B test. Before we send a special offer to 10 million customers, we cold first send two variants to 50 thousand or 500 thousand of them to see what percentage subscribes to it. Or we might test landing page conversions, website calls to action, customer reactions to product launches, etc.
If we need to be sure about a small difference between these two variants, we would need the large sample to be sure what we find is statistically significant. And if we take only a small example, but the effect we find is very big, than these results are still statistically relevant.
Amy Gallo, "A Refresher on Statistical Significance", Harvard Business Review, 2016, February 16.
"Statistical Significance: What is it, how to calculate it, and when to use it", Mixpanel


Bob R
Manager, United States

Common Mistake in not Using Statistical Significance: Type 1 Errors

Good article and much needed in business decision making. The most common mistake I see is people as... Sign up


What is a Type 1 Error?

@Bob R: Agreed. A Type 1 error (or type I error) is a statistics term used to refer to a type of err... Sign up

Gandhi Heryanto
Management Consultant, Indonesia

Practical versus Statistical Significance in Decision Making

In decision making we must also look at the practical significance, which refers to the empirical im... Sign up


More on Business Analytics:
Discussion Topics
Big Data Analytics: a Management Perspective
Software for Mass Data Analysis?
👀Statistical Significance in Decision-making
Data Visualization: When Does it Work and When it Doesn't?
How to Use Machine Learning in Business? 3 Steps
Special Interest Group

Do you know a lot about Business Analytics? Become our SIG Leader

Business Analytics
Knowledge Center

About 12manage | Advertising | Link to us / Cite us | Privacy | Suggestions | Terms of Service
2023 12manage - The Executive Fast Track. V16.1 - Last updated: 28-3-2023. All names of their owners.