How dependable are possibilities predicted by a machine studying mannequin? What does a predicted likelihood of 80% imply? Is it much like 80% likelihood of an occasion occurring? On this newbie pleasant put up, you’ll be taught the fundamentals of prediction possibilities, calibration, and the way to interpret these numbers in a sensible context. I’ll present with a demo how one can consider and enhance these possibilities for higher decision-making.
As an alternative of calling mannequin.predict(information), which provides you a 0 or 1 prediction for a binary classification drawback, you may need used mannequin.predict_proba(information). This provides you with possibilities as a substitute of zeroes and ones. In lots of information science circumstances that is helpful, as a result of it provides you extra insights. However what do these possibilities really imply?
A predicted likelihood of 0.8 implies that the mannequin is 80% assured that an occasion belongs to the optimistic class. Let’s repeat that: the mannequin is 80% assured that an occasion belongs to the optimistic class. So it doesn’t imply: there may be an 80% real-world probability of the occasion occurring…