Validation Report Writing | High-Level Framework for the Identification of Findings

A model validator independently evaluates the underlying model and identifies model risk issues. A model validator raises findings corresponding to the model risk issues. This post outlines a high-level framework for the identification of model risk issues.

For providing practical insights into the high-level framework, take an example of an EAD (Exposure at Default) model meant for the CCAR (Comprehensive Capital Analysis and Review) purposes. In other words, the objective of the illustrative EAD model is to project the EAD under baseline and severely adverse scenarios for the next nine quarters from a given jump-off date. Further, assume that one of the model segments having 5% of the portfolio size has the only macroeconomic variable in the model as statistically insignificant with a p-value of 75%.

Why a model risk issue is an issue in the first place?

On this point, a model validator must think through holistically. For the illustrative EAD model, using a model with the only macroeconomic variable being statistically insignificant indicates limited/negligible sensitivity of the model to the macroeconomic environment. This lack of macroeconomic sensitivity could, in turn, result in flat EAD projections under a severely adverse scenario.

What is the category of the model risk issue?

Classifying a model risk issue into suitable categories (e.g., model design, model performance, modeling data, etc.) is testimony to a structured approach. Further, a bank’s model risk management policy might require such classification. For the illustrative EAD model, the identified model risk issue is design-related. Specifically, it is a model specification-related issue typically classified as model design.

Model users are facing what risks because of the identified model risk issue?

This part is critical and helps in determining the severity of the finding. For the illustrative EAD model, using a model that does not show adequate sensitivity to the macroeconomic conditions undermines the purpose of the CCAR exercise. Further, using presumably flat projections from such a model, especially under a severely adverse scenario, could result in underestimation of the EAD, ceteris paribus. To that extent, there are risks of non-compliance with the spirit of the regulation, model not in line with the desired use, and underestimation.

What business risk corresponds to the identified model risk issue?

An identified model risk issue might theoretically be a big issue. However, assessing the expected impact of that model risk issue is extremely important. The anticipated impact of the model risk issue determines the severity of the corresponding finding. For the illustrative EAD model, only 5% of the portfolio is affected by the identified model risk issue impact. This 5% impact could be considered immaterial.

Are there any mitigants available for the identified model risk issue?

This question also helps in determining the severity of a finding. For the illustrative EAD model, overlays applied to the final EAD projections to make them somewhat sensitive to the macroeconomic environment could serve as a suitable mitigant.

Comments

Post a Comment

Popular posts from this blog

Interview Tips – Multiple Linear Regression

In my experience, multiple linear regression is a commonly used algorithm in the credit risk modelling space, e.g., in the PPNR models (I will not talk about lack of stationarity issues with time series data used in the PPNR models 😊 ). In this post, I have shared a high-level overview of the practical aspects of the multiple linear regression typically assessed during quants’ interviews. I expect folks to consider these high-level inputs and build/enhance their understanding of the multiple linear regression model. Understand the mathematics behind the multiple linear regression It is imperative to have a holistic understanding of the mathematics behind multiple linear regression. I belong to a school of thought that one must be thorough with the underlying mathematics behind any algorithm. That understanding helps to comprehend the underlying limitations of the algorithms. Folks typically fail to answer fundamental questions, e.g., the possibility of a negative adjusted R-squar...
Read more

Validating the modeling data

Image
Introduction Organizations use various models for varied purposes, e.g., credit risk models, pricing models, operational risk models, stress testing models, fraud analytics models, marketing mix models, etc. In principle, every model follows a life cycle involving stages such as development, validation, implementation, monitoring, etc. As per various regulatory guidelines (e.g., SR11-7  [1] , TRIM  [2] , etc.) and based on internal guidelines on model risk management, model validation entails assessment of various components, e.g., modeling data, conceptual soundness, model performance, etc. This article provides practitioner views on validating the modeling data component during a model validation exercise. In principle, remarks made on the modeling data in this article apply to the production data as well. Practitioner views on validating the modeling data Figure 1 depicts components of modeling data validation. Figure 1: Components of Modeling Data Validation 1.  ...
Read more

Tips for Technical Document Writing

In this post, I will not write about technical concepts on model development or model validation per se. I would write about  elementary aspects  that most folks overlook while writing technical reports, e.g., model validation reports. I have seen even seasoned professionals ignoring these aspects! If you are writing a validation report, try to be in the shoes of someone who relies on the validation report to understand the model and corresponding model risk issues. One might have spent weeks understanding the model and model risk issues. As such, while writing, you are more likely to  presume  that a reader will know it all. Don't err on that side.  Agree on a writing style , e.g., using passive voice, present perfect tense, etc., and use it consistently throughout the report. Agree on terminologies and use them consistently throughout the report. For example, do you want to write model validation, model validati...
Read more