Information scientists are within the enterprise of decision-making. Our work is concentrated on find out how to make knowledgeable selections beneath uncertainty.
And but, in relation to quantifying that uncertainty, we regularly lean on the thought of “statistical significance” — a device that, at greatest, offers a shallow understanding.
On this article, we’ll discover why “statistical significance” is flawed: arbitrary thresholds, a false sense of certainty, and a failure to handle real-world trade-offs.
Most necessary, we’ll discover ways to transfer past the binary mindset of great vs. non-significant, and undertake a decision-making framework grounded in financial influence and threat administration.
Think about we simply ran an A/B check to guage a brand new function designed to spice up the time customers spend on our web site — and, consequently, their spending.
The management group consisted of 5,000 customers, and the remedy group included one other 5,000 customers. This provides us two arrays, named remedy
and management
, every of them containing 5,000 values representing the spending of particular person customers of their respective teams.