Maximizing Revenue with Data: The Value of Upgrading Your Prediction Model

Introduction

Should you invest in a better prediction model? How would you decide if it's worth the effort?

Prediction models can be very valuable to a business. But how do you know how much you should invest in improving your model's performance? The analysis comes down to two crucial concepts: the confusion matrix and expected value.

A Case Study: Subscription Churn

Consider a scenario where a company experiences 5,000 subscription renewals monthly. Each renewal is valued at $1,000 annually, but 10% result in churn, leading to a significant revenue loss. The company contemplates offering a $100 discount to high-risk customers to mitigate churn. However, it's essential to avoid unnecessary discounts to those likely to renew without the incentive. Suppose offering the discount has an 80% success rate in converting potential churn into renewals¹.

Understanding the Confusion Matrix

A confusion matrix is a tool to evaluate the performance of a classification model by comparing its predictions against actual outcomes. It's essential not only to consider the model's accuracy but also to understand the financial implications of each prediction outcome. In its simplest form, this is a 2x2 grid that compares model predictions to an evaluation data set that the prediction model has not seen.

In terms of accuracy, we could think about calculating the number of True Positives and True Negatives that the prediction model got right and dividing by the total number of cases predicted. However, doing so misses the business context so we need to know the value of each of the boxes in the 2x2 grid.

Cost-Benefit Matrix

Let's think about what happens in each case. In the True Positive case, the model correctly predicts that a customer will churn. In that case, we offer the incentive and 80% of these churn cases are "saved" and will renew. So $1,000 x 80% - $100 incentive equals $700 in expected revenue per case. In the False Positive case, we offer the incentive to customers that were going to renew anyway, so the value is -$100. In the False Negative case, we failed to predict that these customers would churn (and did not offer the incentive), so lost out on the opportunity in the True Positive case and instead receives $0 in revenue (we might be tempted to put a negative in this box but that would double-count the value of a True Positive case). For the True Negative case, we correctly predict that the customer will not churn and we do not offer the incentive so the incremental revenue is $0.

Establishing a Baseline Model

When building a predictive model (even for the first time), it's useful to create a baseline prediction to compare our new predictive model against. This could be something simple like always predicting the most frequent outcome (i.e., not churning) or a prediction based on some heuristic rule-of-thumb. In this case, let's suppose we predict that a customer will churn if they have submitted a customer support ticket in the past quarter that contains a complaint about our product. This is a crude measure of course because many customers who are unhappy or plan to churn don't submit tickets and many tickets that are resolved positively actually increase the customer's likelihood of renewal.

The following is the confusion matrix performance of our baseline model.

Calculating Expected Value

If we multiply our baseline model's confusion matrix times the cost-benefit matix, we will get the expected value of the baseline model predictions. The calculations are as follows: (5% x $700) + (5% x -$100) + (5% x $0) + (85% x $0) = $30. Even though the crude model misses half of the "actual" churn (p) and further has as many false positives as true positives, the expected value of this model is still positive due to the asymmetry of the cost-benefit matrix.

Developing a More Advanced Model

Suppose we want to develop a new, more complex model that takes into account a customer's tenure, industry, number of users, product tiers, and other relevant factors that we believe could have an effect on predicting customer churn. As an assessment tool, let's examine how much better a new model could be across the confusion matrix so that we can estimate the value of a new model. Suppose that the new model was able to more accurately predict True Positives as specified in the following confusion matrix:

Even though the False Negative/False Positive and True Negative rates are not all that different, the fact that this model is able to accurately predict 60% more True Positives means that the expected value of this model is much higher than our baseline model. How much more expected value? (8% x $700) + (6% x -$100) + (3% x $0) + (83% x $0) = $50 per case. So for each predicted customer event, our new model is 67% more valuable!

Calculating the Total Value

Considering the expected value per case is important, but assessing the aggregate impact over all cases provides a clearer picture of the model's overall financial impact. If we are predicting churn events for 5,000 customers a month, then over the course of a year we are making 60,000 predictions. The baseline model would be worth an estimated 60,000 x $30, or $1.8MM in incremental revenue. The new model would be worth 60,000 x $50, or $3.0MM in incremental revenue. The difference between the two models is $1.2MM in revenue per year. The analysis suggests that there is considerably potential value in building a more complex prediction model to address customer churn.

Conclusion

Investing in a more sophisticated prediction model has the potential for significant financial returns. However, it's crucial to carefully analyze the expected improvement in accuracy and the corresponding financial implications.

Next Steps

Do you have a prediction model situation you would like to analyze? Get in touch with me and we can discuss.