Statistical Significance in Decision-making

Statistical Significance in Decision-making

🔥NEW Companies are increasingly using business analytics and are increasingly relying on data to make operational, tactical and even strategic business decisions. Obviously, when making decisions, managers should not rely on calculations and models they don't understand. That's why understanding statistical significance is becoming more important for managers.
First of all, you should realize that in statistics, "significant" does NOT mean something is big or important! Rather it means that the result we see in some sample also exists in the (total) population.

What is statistical significance?
Statistical significance is the likelihood that a relationship between two or more variables we find in our results is NOT caused by random chance. If you draw conclusions out of a lot of data, you have to use a sample. A sample is a set of individuals / objects collected or selected from a bigger statistical population by some defined procedure. But in any experiment or observation that involves drawing a sample from a larger total population, there is always the possibility that an observed effect occurred due to sampling error (coincidence). Statistical significance is a way of showing to what extent that the answers which were obtained from the data are not totally due to luck. It is the confidence level you may have in a mathematical way that some result or conclusion is reliable. The more critical your decision based on the data is going to be for your firm, the higher you will want the statistical significance to be.

Statistical hypothesis testing
Statistical hypothesis testing is the method by which an analyst determines that the results in the data are not explainable by chance alone. This test provides a p-value, which is the probability of observing results as extreme as those in the data, assuming the results are truly due to chance alone. A p-value of 5% or lower is normally considered to be statistically significant.
But when the p-value is large, then the results in the data are explainable by chance alone. So when the p-value is higher than 5% (0,05) we consider the results statistically NOT significant as they could easily be explained by chance alone.

Mistakes and errors made while working with statistical significance

SAMPLING ERRORS (too small or non-representative). Statistical significance tells us that if something works for a smaller sample, then with how much confidence can we say that it will also work the same for a large population. But many times people make mistake of taking a sample which does not represent the total population. For example the objects in the sample might be biased towards some specific color, say red, while the total population might be a variety of colors. The statistical significance becomes low when such errors are made. Choosing a good diversified sample which represents the total population accurately is very important for getting a high statistical significance.
NON-SAMPLING ERRORS. There are other errors that may occur that are as important as the sampling errors. Non-sampling errors include the ones where the measurement and experiment protocols were broken. Sometimes the data may get lost. Sometimes the people in the sample may have lied in the survey. Etcetera. These factors often are difficult to be controlled, but a careful analysis will help to reduce these non-sampling errors.

Applications of statistical significance
Statistical significance basically tells us whether we should trust the results of some experiment or test like for example an A/B test. Before we send a special offer to 10 million customers, we cold first send two variants to 50 thousand or 500 thousand of them to see what percentage subscribes to it. Or we might test landing page conversions, website calls to action, customer reactions to product launches, etc.
If we need to be sure about a small difference between these two variants, we would need the large sample to be sure what we find is statistically significant. And if we take only a small example, but the effect we find is very big, than these results are still statistically relevant.
Sources:
Amy Gallo, "A Refresher on Statistical Significance", Harvard Business Review, 2016, February 16.
"Statistical Significance: What is it, how to calculate it, and when to use it", Mixpanel