TheMReport

MReport October 2018

TheMReport — News and strategies for the evolving mortgage marketplace.

Issue link: http://digital.themreport.com/i/1035854

Contents of this Issue

Navigation

Page 39 of 67

38 | TH E M R EP O RT FEATURE In markets where credit risk- scoring models are regulated and scrutinized, there is a strong re- quirement for the models and the credit decisions derived from them to be explainable. The impact each variable has on the credit score must be traceable (transparent), clearly explained and palatable (understandable and acceptable) to lenders, regulators, and consumers. These requirements are guar - anteed to be met by the current FICO ® Score model construct. In contrast, ML-only models can be difficult to explain, requiring considerable simulation effort to even approximate how the models compute their scores. Even if ex - planations can be computed, they may not be palatable in all cases. Research in Action T o assess whether the latest ML technologies yield im- provements over FICO ® Scores calculated via time-tested score- card models, our researchers explored the different models' tradeoffs between: • Performance: The efficacy in iden- tifying individuals of acceptable credit risk. • Palatability: The acceptable ex- plainability of the score, in order to pinpoint the impact of specific risk factors on a credit score. Our team performed an A/B test of analytic models—a FICO ® Score model and two ML-only models—against the same data set, a nationally representative sample of millions of credit files. The test revealed that ML-only models offer only very small predictive improve- ments. We also tested the ML-only model and FICO's Scorecard ap- proach on unscorable populations to determine whether the ML-only model may be able to squeeze additional predictive information out of the sparse credit data that is available on these files. Using a sample of millions of unscorable records, FICO compared the predictive power of credit bureau-based "research scores" built via scorecard tech - nology to those solely based on ML techniques. Compared to the scorecard model, the ML-only model did initially produce a small improvement on the "derog- atory info in credit file" segment. However, the scorecard-based model was engineered for palat- ability, while the ML model was unconstrained, and when palat- ability constraints were removed, the scorecard model yielded nearly identical levels of predictive performance. The upshot of these results? All models struggle similarly when there is only sparse credit bureau information available. Any predictive lift attributable to the ML-only approach appears to be derived from sacrificing palatabil - ity for a modest improvement in model performance. Palatability is a key element of credit scoring. Paying down credit card debt improves credit scores, an axiom in the world of credit risk management. Consumers must be able to understand not only the factors that go into a score but also what actions they can take to improve their credit. An unconstrained ML-only model we tested would result in 9.2 percent of consumer records receiving a lower score after debt was paid off (all other factors held equal). This effect is deeply unpalatable; con - sumers and lenders in high-stakes, high-stress credit situations, such as applying for a mortgage, would be confounded by such a deviation from their mutual expectations. Separately and together, these test results provide evidence that ML-only credit risk models are unsuitable as the primary determinant of creditworthiness in mortgage and other types of lending. The ability to impose pal - atability constraints within FICO ® Scores provided by scorecard technology ensures that counter- intuitive results are minimized, greatly improving the experience for consumers and lenders. Anatomy of a Machine-Learning Model Figure 1: The gradient boosting approach uses training data to generate thousands of trees, which are combined to produce a predictive credit risk score. Paying down credit card debt improves credit scores, an axiom in the world of credit risk management. Consumers must be able to understand not only the factors that go into a score, but also what actions they can take to improve their credit.

Articles in this issue

Archives of this issue

view archives of TheMReport - MReport October 2018