WPS Analytics WoE Transform Editor:
a universal modelling gem
By: Natasha Mashanovich, Lead Data Scientist at World Programming, UK
There are two common beliefs about predictive modelling we frequently come across:
Machine Learning models are superior to logistic regression.
What if we can make logistic regression with magical variable transformations as good as it gets when using Machine Learning (ML) models? Would you still choose an ML model and take the risk of having a hassle with endless model training time, numerous parameter optimisations and almost impossible model explanation?
Scorecard models can only be used in credit risk.
The traditional credit scorecard risk model is a special form of logistic regression. Its rank ordering is identical to the rank ordering of the logistic regression model. However, its scoring formula is simpler and more intuitive than the probability formula of its model equivalent. What if you were tasked to present your model to a non-technical audience? Would you rather talk about estimates, probabilities and log-odds or walk them through the scoring system and total sum of points?
For us, these are only modelling myths. Let’s prove it!
Here is a ‘simple’ problem to solve. For a given decision space shown in Figure 1 build a model to predict if a coordinate (x, y) lies within the yellow circle. This problem can be easily solved using the Pythagorean Theorem. However, the objective of this exercise is to test the predictive power of different modelling techniques based on the complex decision problem with the quadratic relationship between decision variable (that is, in/out of the circle) and (x, y) coordinates.
The decision space is simulated by creating 3 different datasets (train, valid and test) using uniform distribution (Table 1). The simulated data has 3 variables: 2 input variables representing x and y coordinates and a binary variable inCircle indicating if a (x, y) coordinate lies within the circle (value 1) or outside the circle (value 0). The simulated datasets are balanced with roughly 50/50 proportion of coordinates in and out of the circle.
Three modelling techniques are applied, and their predictive power compared (Table 2). All three techniques are binary classifiers with inCircle being the dependent variable.
The first modelling technique is logistic regression with x and y inputs. The second approach is also logistic regression with the same setting as the previous one; however input variables are WoE-transformed x and y values (more about this in the next section). The third technique is based on neural networks, specifically Multi-Layered Perceptron (MLP) with the structure shown in Table 3.
The model performance is assessed using two metrics: Area Under Curve (AUC) and misclassification (error) rate with 0.5 probability threshold for in/out class prediction. As expected, the MLP model has the highest AUC value and lowest error rate, however, what comes as a nice surprise is the superior performance of the logistic regression model when combined with WoE-transformed independent variables. So what is WoE transformation?
Worse than random prediction
Almost ideal prediction
Instead of Wikipedia
Weight of Evidence (WoE) transformation is a special form of binning, where each bin is given a numeric value calculated using the formula below.
Any binning option can be applied (manual, equal width, equal height, winsorized or optimal), however it is optimal binning which typically leads to better model performance. In theory, WoE values can be calculated for any number of bins. In practice, to prevent model overfitting, choose no more than 10 bins and avoid bins with a small number of observations.
WoE transformation requires two variables: a binary dependent variable (DV), which is generally the focus of a predictive model, and an independent variable (IV). The IV is binned and binning is guided by the DV categories, one of which must be selected as the target category, as per Equation 1.
In Figure 3, the independent variable interest_rate is binned in relation to the dependent variable default and its target category ‘Y’, using optimal binning. The WoE values assigned to the bins become risk carriers indicating the level of risk associated with each bin; the lower the WoE value the riskier the bin category. In this example, the interest rates greater than 23% are associated with the highest risk and given a negative WoE value calculated as follows:
Although the binning process itself transforms any variable to binned categories, the numerical values (that is, WoE) assigned to each bin actually transforms the variable to a numeric one – hence the WoE transformation transforms all variables, including categorical variables, to numeric, the desired feature for any modelling algorithm. Smashing!
Figure 4 is an example of WoE transformation of a categorical variable occupation. There are several points to emphasise here.
Firstly, since the target category is ‘Y’ – individuals with higher income – the negative WoE values are associated with higher income. In this case, occupations ‘Exec-managerial’ and ‘Prof-speciality’ have the highest concentration of individuals with higher income.
Secondly, missing values for occupation are associated with positive WoE value of 1.01. This is the second-highest WoE value, meaning that in general individuals who did not state their occupation typically have lower income. This is a valuable finding, where missing values could be a useful piece of information. In situations like this one, WoE transformation is invaluable as it can not only effectively deal with imputation of missing values but assign the right value so the WoE value for the missing occupation would be an indication of lower income.
Thirdly, the WoE transformation has eliminated the need for any dummy coding, with additional 14 indicator variables, and effectively performed a 1-to-1 mapping between categories and WoE values. Any categorical variable, regardless of the number of categories, is transformed to a single numeric variable.
Best friends forever
WoE transformations and logistic regression are compatible. The combination of formulas included in both techniques makes the absolute value of the parameter estimate for logistic regression with a single WoE-transformed variable equal to 1 (Table 4).
Further calculations based on WoE, logistic regression and scaling parameters convert a logistic regression model to an additive model with points assigned to all bins, where the probability estimates given by the logistic regression are transformed to scores, calculated as a total sum of points across different bins (Figure 5). It is the WoE transformation formula that provides the perfect match to the logistic regression model so the model estimates can be translated to the scoring points using adequate scaling.
The miracles of WoE transformation
Transforms non-linear into linear relationships. With adequate binning, WoE can transform non-linear relationships into linear or at least monotonic relationships with log-odds. This is one of the main assumptions when training a logistic regression model; often challenging to satisfy with other transformations or more transformations being required to prevent violation of the modelling assumption. Figure 6 (the second graph) shows the result of transforming variable age to be a monotonic function of the WoE value.
Transforms logistic regression model to the additive model with scoring points. This is the ideal concept for a non-technical audience to understand, provide feedback and state the requirements that can be easily merged into the model, if required. (Figure 5)
Applicable to any data type – numeric or text. (Table 6)
Applicable to any data measurement scale – interval, ordinal or nominal. (Table 6)
One-to-one transformation regardless of the number of bins (Table 6). Alternatively, one can use indicator variables (that is, dummy variables). Many predictive models perform internal transformation of categorical variables into dummy variables. This could be a problem with a large number of categories. Specific models such as logistic regression also require specification of a reference category.
Logistic regression with WoE-transformed variables is a fully transparent model. This is a superior advantage when compared with ML models. (Figure 5 and Figure 8)
Model performances comparable to ML models. See the Prologue section or read the rest of the blog!
Provides implicit imputation of missing values. Missing values can be binned separately, and a WoE value calculated (Figure 4). Alternatively, they can be merged with other bins with a similar WoE.
Provides implicit treatment of outliers. Like missing values, outliers are the beast in many modelling techniques. They need to be treated to prevent them from spoiling a model. The binning process involved in WoE transformation implicitly solves this problem without any further interventions. For example, any extreme salaries would be encapsulated within a bin with the highest salary range and have the same WoE value across that bin.
The fellowship of the scorecard model
Despite the growing popularity of machine learning models for superior performances there is a significant drawback. These are ‘black-box’ models and the lack of ability to explain what constitutes a machine learning model is a key obstacle to acceptance by business users. Hence, World Programming data scientists have an imperative quest – to come up with a simple, justifiable solution that is, intuitive and easy to interpret whilst still being as predictive as a good black-box model. The modelling fellowship of WOE transformation, logistic regression and model scaling, often referred as the scorecard model, has become our preferred modelling methodology, not only for credit risk models but for many other business models developed for our clients. Under champion-challenger scrutiny our scorecard models consistently outperform other candidate models on performance criteria.
In 2019, we published a blog about our modelling solution for predicting the 2019 Rugby World Cup winner. Several data scientists from our solutions team were tasked to create a predictive model on the emerging topic of the Rugby World Cup, which coincided with the WPS annual event organised at London for our existing clients and prospects. This was a perfect opportunity to present the latest additions to the WPS analytical platform and solutions through the prism of the World Cup prediction and engage our audience with the hot topic that everyone was talking about.
We opted for a scorecard model – a quick and simple model that is easy to present to our audience but still powerful. Using the WoE transformation with optimal binning and monotonic behaviour for selected input variables we translated a logistic regression model to the additive one based on the scoring system presented in Figure 8.
Without compromising the model performance, we were able to accurately reflect historical behaviour with data patterns extracted during the model training and portray the exact logic for prediction of future events. This provided an ideal setting for discussion with our attendees. Any ML model might have given us the same or possibly better performance but would have created uncertainty and reservation within the audience. Moreover, our modelling approach enabled better synergy with the expert’s opinion by modifying bin definitions and weights – something we could have hardly controlled with ML models.
Another serious drawback of ML models is hyper-parameter optimisation. It is almost impossible to produce a good model without optimisation. The problem becomes even more significant for neural networks with many hyper parameters and ultimately a large search space. Extra care is required when selecting parameters for optimisation and their values. As the total number of possible combinations is determined by an exponential function, if we are not careful, model training could be very long. On the other hand, WoE transformations and logistic regressions run very quickly and there are no hyper parameters to optimise, which is an extra bonus.
As for model deployment, scorecard models are probably the easiest to deploy. Many commercial and free analytical tools have automatic deployment into different programming languages. But even without this capability or limited coverage of deployment languages, scorecard models can be effectively implemented in virtually ANY programming language either manually or programmatically with a simple script. All we need is a series of IF-THEN-ELSE statements in a target programming language (Table 7).
Last but not least, ML models have a steep learning curve and only well experienced data scientists are able to use them. On the other hand, scorecards are easy to build and with the right modelling suite and visual programming interface, the entire spectrum of data modellers, including citizen data scientists, would be able to use them.
Upgrading the fellowship
Use WoE transformations to increase the performance of ML models.
One of our ‘Rugby World Cup’ model candidates correctly predicted the top three teams, although sadly not in the right order. Nevertheless, the point to bring up here is to test if a neural network model (given the same model inputs) would outperform the existing scorecard model. We created a couple of model challengers using MLP and tested all three models on the 2019 Rugby Cup results (Table 8). Both MLP models, with and without WoE transformations, had the same network architecture: 10 inputs and 2 hidden layers with 25 neurons each. All three models had a similar AUC on the validation sample. Interestingly, they all performed even better on the actual 2019 World Cup results. Overall, MLP with WoE transformation had the best performance, followed by the logistic regression with the same WoE transformations and MLP without WoE being the worst among them.
WPS Analytics WoE Transform Editor: a precious commodity
Applying WoE transformation is a tough mission without a proper analytical tool. The first and obvious step is to search for a WoE package in one of the open source languages, such as Python or R. Even though such packages do exist, the problem occurs on application. For example optimal binning on categorical variables; or monotonic WoE transformation on numeric variables; or to control bin size; or to assign a WoE value to missing values; or to force optimal binning, and monotonicity, and bin size, and missing values at the same time. Additionally, we may want to choose between different binning measures such as information value, entropy variance, Gini or chi-square. In an ideal world, we would like to visualise binning and transformations and for a premium feature we would really want to use all of that interactively!
To the best of our knowledge there is no such commodity available as an open source tool. Several commercial products offer WoE editors with more-or-less similar functionalities. Our WPS Analytics WoE Transform Editor is one such precious commodity, offering the following functionality:
- Setting the dependent variable, target category and frequency variable
- Selecting variables to transform
- Assigning variable treatment: interval, nominal, ordinal
- Force monotonicity option
- Binning types: optimal, equal-width, equal-height, winsorized and manual
- Optimal binning measures: information value, Gini variance, entropy variance and chi-square
- Ability to optimise all selected variables with a single click
- Ability to fine-tune variable one-by-one
- Settings for fine and coarse classing
- Include or exclude missing values
- Bin size settings for both count and percentage
- Information value of transformed variables
- Visual graphs for WoE values, bin sizes and characteristic analysis
- WoE table with relevant column values
- SQL and SAS language deployment code
- Fully interactive editor
Figure 10. WPS Analytics WoE Transform Editor
With this little gem your modelling becomes a real pleasure. 😊