Validating the modeling data

Introduction

Organizations use various models for varied purposes, e.g., credit risk models, pricing models, operational risk models, stress testing models, fraud analytics models, marketing mix models, etc. In principle, every model follows a life cycle involving stages such as development, validation, implementation, monitoring, etc. As per various regulatory guidelines (e.g., SR11-7 [1], TRIM [2], etc.) and based on internal guidelines on model risk management, model validation entails assessment of various components, e.g., modeling data, conceptual soundness, model performance, etc. This article provides practitioner views on validating the modeling data component during a model validation exercise. In principle, remarks made on the modeling data in this article apply to the production data as well.

Practitioner views on validating the modeling data

Figure 1 depicts components of modeling data validation.

No alt text provided for this image

Figure 1: Components of Modeling Data Validation

1.     Data sources and controls

The model validator must assess the usage of a well-governed data source(s) for the modeling data. A well-governed data source, in this context, means a substantially error-free data source. Naturally, as a model validator, one must have evidence (e.g., user acceptance testing, separate review/validation report on the source system, etc.) demonstrating proper governance of the data source(s).

The model validator must understand and assess the data aggregation process in case data is sourced from multiple data sources. Every data source has its limitations. An understanding of those data limitations entails a better assessment of model risk. Practically speaking, this is very relevant in the case of IRB (Internal Rating Based) models, wherein regulation [3] specifically requires the addition of margin of conservatism (MOC) to account-for data deficiencies.

Data sourced via an automatic process leads to lower operational risk than that sourced via a manual process, ceteris paribus. Moreover, additional data controls shall be in place for manually extracted data, which the model validator must assess.

In practice, simple models are developed based on the feeder models. For example, Expected Credit Loss (ECL) is mathematically the product of Probability of Default (PD), Loss Given Default (LGD), and Exposure at Default (EAD). In such cases, the model validator must assess if uncertainty in the feeder models (e.g., because of known model design limitations, performance issues, etc.) leads to additional uncertainty in the main model. Typically, most validation teams end up raising findings in such instances.

2.     Data quality

ECB guide to internal models (refer to link) outlines various data quality dimensions, as mentioned below:

1.      Completeness (values are present in any attributes that require them);

2.      Accuracy (data are substantively error-free);

3.      Consistency (a given set of data can be matched across the institution’s different data sources);

4.      Timeliness (data values are up-to-date);

5.      Uniqueness (aggregate data are free from any duplication arising from filters or other transformations of source data);

6.      Validity (data are founded on an adequate and rigorous classification system that ensures their acceptance);

7.      Availability/Accessibility (data are made available to the relevant stakeholders);

8.      Traceability (the history, processing, and location of the data under consideration can be easily traced).

The model validator must thoroughly assess data quality against each of these dimensions. Typically, model developers provide descriptive statistics (e.g., count, missing values, mean, median, standard deviation, values at different percentiles, etc.). Practically speaking, data quality assessment is much beyond assessing descriptive statistics. For example, although descriptive statistics potentially helps in assessing completeness and accuracy dimensions, however, it does not help in the assessment of other dimensions such as Consistency, Traceability, etc.

3.     Data representativeness

In various practical scenarios, it becomes necessary to use external data and/or data from other internal products/portfolios. For example, consider a case where model developers intend to create a PD model for a recently originated portfolio, resulting in limited historical data. In such cases, model developers typically leverage external data and/or data from other similar products/portfolios. Another example could be a case where an organization has acquired a portfolio of another organization. In addition, there are cases where underlying policies/procedures of an organization have changed in the recent past resulting in subjecting old historical data to potential lack of data representativeness issues.

As a model validator, one must understand the rationale behind external data usage and/or other internal data. Further, only checking statistical similarities (e.g., whether the mean/standard deviation of the internal data is statistically similar to that of external data?) does not suffice. Similarities should also be assessed from qualitative aspects, e.g., in the case of an acquired portfolio, whether underlying credit policies are similar across two organizations. Reviewing a policy change log often informs the model validator on policy changes and timings thereof, which, in conjunction with the product/portfolio knowledge, in turn, helps in assessing the lack of data representativeness. Practically speaking, internal audit teams and regulators critically look at the lack of data representativeness.

4.     Data imputation

It is practical to impute missing data during the model development phase. As a model validator, one must assess the missing value imputation technique and its impact on the model outcome. Consider a case where model developers use a mean value imputation. In such a case, the model validator could assess the change in the model specification by using a different mean value imputation.

5.     Data exclusions

It is practical to exclude data during a model development process, e.g., because of data limitations. As a model validator, one must critically assess if all the data exclusions are intuitive. Further, it should be assessed if the data exclusions are before the missing value imputation or vice versa.

6.     Data period and frequency

As a model validator, one must assess that the start period of the modeling data is commensurate with the model objective. Take an example of a PD model developed for stress testing purposes. In such case, model development data shall include data from the global financial crisis (GFC) period (2007-2009), ceteris paribus. Take another example of a PD model used for IRB purposes. In such a case, as required by the regulation (refer to footnote 3), the modeling data must have a “likely range of variability.” Naturally, intricacies are involved, e.g., if the underlying portfolio originated post-crisis period and hence does not have any data of the GFC period, etc.

As a model validator, one must assess that the end period of the modeling data is commensurate with the model’s objective and in line with the availability of the latest data. Given that most organizations have a rigorous model governance process, it takes at least six months for all the internal approvals and the model implementation. The pertinent question is that model developers might have used the recent data during the development phase, but is it the latest available data when model validation commenced.

As a model validator, one should assess whether the frequency of the modeling data creates additional limitations to the model. For example, consider a PPNR (Pre-Provision Net Revenue) model whose objective is to predict the quarterly balance for the next nine quarters. Assume that monthly historical data is available with the model developers. The pertinent question is whether it is prudent to use quarter-end data when the monthly data is available.

7.     Data sampling

In various models, especially for marketing mix models, a development sample is created using various sampling techniques. Mostly, simple random sampling and stratified sampling are used in practice. As a model validator, one must assess whether sampling has resulted in any bias. For example, if in the original development data, there was a 5% hit rate, and after sampling, it has changed to 6%, is it acceptable? In certain cases, e.g., for a low default portfolio, it is atypical to keep all the defaults even after sampling. In such cases, a pertinent aspect to assess is its downstream impact on the model estimation, as inherently sampling is biased.

In case development data is divided into different samples, e.g., development sample, validation sample, out of time sample, etc., the model validator should assess whether all such samples are similar.

References

[1] SR11-7 – refer to link

[2] TRIM – Targeted Review of Internal Models (refer to link)

[3] EBA guidelines on PD estimation, LGD estimation, and the treatment of defaulted exposures – refer to link

Disclaimer

The ideas, views, and opinions expressed in this article represent my views and not those of any of my current or previous employers.

Comments

Post a Comment

Popular posts from this blog

Interview Tips – Multiple Linear Regression

In my experience, multiple linear regression is a commonly used algorithm in the credit risk modelling space, e.g., in the PPNR models (I will not talk about lack of stationarity issues with time series data used in the PPNR models 😊 ). In this post, I have shared a high-level overview of the practical aspects of the multiple linear regression typically assessed during quants’ interviews. I expect folks to consider these high-level inputs and build/enhance their understanding of the multiple linear regression model. Understand the mathematics behind the multiple linear regression It is imperative to have a holistic understanding of the mathematics behind multiple linear regression. I belong to a school of thought that one must be thorough with the underlying mathematics behind any algorithm. That understanding helps to comprehend the underlying limitations of the algorithms. Folks typically fail to answer fundamental questions, e.g., the possibility of a negative adjusted R-squar...
Read more

Tips for Technical Document Writing

In this post, I will not write about technical concepts on model development or model validation per se. I would write about  elementary aspects  that most folks overlook while writing technical reports, e.g., model validation reports. I have seen even seasoned professionals ignoring these aspects! If you are writing a validation report, try to be in the shoes of someone who relies on the validation report to understand the model and corresponding model risk issues. One might have spent weeks understanding the model and model risk issues. As such, while writing, you are more likely to  presume  that a reader will know it all. Don't err on that side.  Agree on a writing style , e.g., using passive voice, present perfect tense, etc., and use it consistently throughout the report. Agree on terminologies and use them consistently throughout the report. For example, do you want to write model validation, model validati...
Read more