Issues and Challenges of Model Risk

Quantification
In April of 2011, regulatory changes stemming from the 2008 financial crisis spurred the Federal
Reserve/OCC to issue the Supervisory Guidance on Model Risk Management. The failure of
models to adequately describe and convey the risks they were designed to represent was one of
the major causes of the crisis. New regulations, such as the Dodd-Frank Act stress test and the
Comprehensive Capital Analysis and Review (CCAR) have introduced literally hundreds of new
models to the risk management profession. The proliferation of and greater focus on models has
made it more important than ever for risk management professionals to understand how to
adequately identify, measure, manage and monitor the specific risks inherent in their models and
management understand the impact models have on their business. Recognizing that models
need to be independently reviewed and validated, banks need to establish a model risk
management framework to govern the entire lifecycle. This includes inventory of risk, tools to
manage risk, risk tracking and measurement, risk mitigation or transference, continuous
monitoring, testing of control processes, validation reporting and boardlevel communication of
risk. The Rise of Model Risk Awareness Interest in model risk first peaked in the late nineties
following the collapse of Long-Term Capital Management, according to Ravi Chari, manager of
Americas Risk Practice at SAS Institute. At the time, model risk management was primarily
concerned with quantitative models for trading losses and market risk, since both can directly
impact profitability. LTCM had a lot of complex models that worked very well for a long time,
says Chari. Then you had the Russian crisis [in 2008] and after that all these complex models
suddenly did not seem to work. Recent regulations such as the Dodd-Frank Act and CCAR
introduced requirements for quantitative modeling not only of market risk, but also credit risk,
operational risk and even areas such as growth rates, employment expenses and pre-payments.
The sudden creation of so many new models, each with their own set of assumptions and
operating parameters, introduced an entirely new set of model risks, which must now also be
managed. Since the creation of many of the new models have stemmed from regulatory
requirements, it has become necessary not only to get them right, but also be able to prove that to
regulators. Organizations now use models to analyze everything from pricing, planning,
marketing, and even operational issues. Given that models are used in multiple places, the
question isn't whether or not there are inadequacies in the models, but how do you manage those
inadequacies? says Suresh Gopalkrishnan, Principal, Business Information Management at
Capgemini Financial Services. Gopalkrishnan describes the model lifecycle as a five-stage
process, beginning with the decision by the intended model user to write a requirement for a new
model. The model is then developed, validated, put into production and, finally, replaced and
retired. The circumstances may change sufficiently so that the model no longer works well,
Gopalkrishnan says. When the input data changes, the model may need recalibration.

Under this framework, there are three lines of defense: the individual writing the original model
requirements, the model validation process and the auditing process. Each of these is responsible
for identifying and analyzing potential risk sources, starting with data selection. Sources of Risk
During the model development pro-cess, the designer must make decisions about which data the
model will be built upon. There are some exclusion choices that you make, Gopalkrishnan
says. These need to be looked at to make sure theyre appropriate for the model purposes. As
the use of models has expanded beyond its traditional role in analyzing pricing and market risk,
organizations have had to experiment with new development methodologies. Since standard
methodologies have not yet been established for many of the new risk areas, special attention
must be paid to the assumptions made as part of the model specification process to ensure the
organization is developing the best model possible. A second potential source of risk arises in the
implementation phase, as the model is put into production. A model developed in SaaS may be
implemented in Java, creating a potential source for errors. Once the model has been successfully
put into production, risk managers must ensure that the data being analyzed by the model fit
within its design parameters. Despite the focus usually given to the development and validation
processes, organizations often fail to create sufficient controls on the data that feeds into their
models. A model is built for a purpose, Gopalkrishnan says. But it might get used for other
purposes. That's a potential source of risk. Other risks, such as regulatory acts like the Equal
Credit Opportunity Act or the Fair Housing Act, which prohibit the use of discriminatory
variables, may also be present. Inventorying and Measuring Risk Once the potential sources of
risk are known, organizations can begin to quantify risk levels within their models. That process
begins with creating an inventory of every model being deployed.
You need to document each and every model, the purpose of the model, whether its a model for
loan losses or whether its a model for pre-provision net revenue, says Chari. For something like
the CCAR process, that can be an enormous challenge. There are hundreds of models in this
process, Chari says. And if that is not enough, banks are expected to have two models for each
forecast that is material. Under CCAR, these are known as the Champion and Challenger
models. While the Champion is used for the stress-test analysis, the Challenger acts as a
safeguard against potential inadequacies in the Champion model. In order for the ChampionChallenger system to work however, the two models must proceed from fundamentally distinct
sets of philosophies and assumptions. Documenting how material each model is to the
organization is another critical component of the model inventory stage. This analysis may be
required not only by regulators, but by senior management, particularly when a specific model
may signifi- cantly impact capital calculations. In the CCAR process, the Fed lists 26
macroeconomic variables to be included in the analysis. Beyond that list, banks may need to
incorporate additional variables that are material to their particular situation, such as state GDP
or additional interest rate forecasts. All of these need to be incorporated into forecasts extending
nine months out under three different scenarios: baseline, adverse, and severely adverse. These
forecasts, and the models that produce them, must then be validated through the traditional
validation process described by the Enterprise Risk Management framework. It tells you

whether the numbers you're using from these models, the outputs, are meaningful or not, Chari
says.
Finally, attention must be paid to implementing the appropriate governance policies. Factors such
as the design quality of the model development process, potential conflicts of interest, or the use
of any non-validated models, must be documented and analyzed. Ensuring the Best Models
Although model risk receives a significant amount of blame for the financial crisis, regulators are
pushing for more quantitative modeling than ever before. While this trend is sure to increase the
overall amount of model risk in the financial system, the most important issue is the extent to
which that risk is recognized and managed. As long as management and the folks developing
the models are aware of all the assumptions and limitations of the model, using quantitative
models works quite well, Chari says. Conclusion Banks are expected to ensure that their model
risk management policies, procedures, and practices are consistent with the Supervisory
Guidance on Model Risk Management. Best practices mandate the development of a model risk
governance charter and systematic support for the model validation and governance processes,
ratified at the executive level for continued success. By introducing a robust model risk
management framework banks can take a significant step to reducing model risk and in doing so
improve decision making, financial performance and meet the demands of the regulators.