Application Provision
1
Introduction
1.1
This Prudential Regulation Authority (PRA) supervisory statement (SS) sets out the PRA’s expectation as to the model risk management practices firms should adopt when using stress test models. It supports firms’ development and implementation of policies and procedures to identify, manage and control the risks inherent in the use of stress test models.
- 01/06/2018
1.2
This SS is relevant to PRA authorised banks, building societies and PRA-designated investment firms only. Credit unions are not in scope and there is currently no proposal to extend the principles to insurance and reinsurance firms1.
Footnotes
- 1. Insurance and reinsurance firms are currently not in scope. Requirements on model management for firms subject to Solvency II are set out in Commission Delegated Regulation (EU) 2015/35 Title I, Chapter VI, and PRA expectations are set out in SS17/16: ‘Solvency II: internal models – assessment, model change and the role of non-executive directors’: www.bankofengland.co.uk/prudential-regulation/publication/2016/solvency2-internal-models-assessment-model-change-and-the-role-of-non-executive-directors-ss.
- 01/06/2018
1.3
Adopting a proportionate approach, the PRA expects the larger firms that participate in the Bank of England’s (the Bank) annual concurrent stress testing to apply the principles contained in this SS in full, while firms not participating in the Bank’s annual concurrent stress testing should apply the principles on a proportionate basis, taking into account their size, complexity, risk profile and the relevance to them of stress test models.
- 01/06/2018
1.4
The expectations in this SS will take effect from Friday 1 June 2018.
- 01/06/2018
2
Model risk management principles for stress testing
2.1
The PRA’s model risk management principles for stress testing are centred on four key principles:
- Principle 1 – Banks have an established definition of a model and maintain a model inventory.
- Principle 2 – Banks have implemented an effective governance framework, policies, procedures and controls to manage their model risk.
- Principle 3 – Banks have implemented a robust model development and implementation process and ensure appropriate use of models.
- Principle 4 – Banks undertake appropriate model validation and independent review activities to ensure sound model performance and greater understanding of model uncertainties.
- 01/06/2018
Scope of application, proportionality and model materiality
2.2
The PRA expects firms to adopt a risk-based approach to determine the materiality of models focusing on two factors: coverage (eg size of a portfolio) and impact (eg financial, capital or risk), with due consideration given to the model risks associated with the models, ie criteria such as complexity, purpose or strategic importance.
- 01/06/2018
2.3
The PRA expects firms participating in the Bank’s annual concurrent stress testing to apply the principles for all stress test models. All other firms should take into account their size, nature, scale, complexity of business activities and use of stress test models when seeking to apply the principles. For these firms the PRA expects at a minimum:
- implementation of Principles 1 and 2 (ie establish a model definition, maintain a model inventory and implement an effective governance framework, policies and procedures), and
- application of Principles 3 and 4 (ie implement a robust model development process and undertake validation and independent review) to those models they consider material and pose the most significant risks to their business.
- 01/06/2018
2.4
While the concepts of materiality are articulated for certain principles, all firms should focus their validation and independent review activities commensurate with the overall use, complexity and materiality of models across the model life cycle to ensure those models that pose most significant risks – financial, capital or other – are adequately managed.
- 01/06/2018
Assessment
2.5
The assessment of the model risk management practices of firms participating in the Bank’s annual concurrent stress testing will form part of the Bank’s qualitative review of the annual concurrent stress tests.
- 01/06/2018
2.6
In line with previous concurrent stress test qualitative review practices,2 feedback on the status of firms’ practices will be provided confidentially to each firm. In addition, the PRA will retain the flexibility to publish a high level (anonymised) overview of findings together with the stress test results publication.
Footnotes
- 2. ‘The Bank of England’s approach to stress testing the UK banking system’, October 2015: https://www.bankofengland.co.uk/news/2015/october/boe-publishes-approach-to-stress-testing-the-uk-banking-system.
- 01/06/2018
2.7
For firms not participating in the Bank’s annual concurrent stress testing the PRA will review their stress test model risk management practices as part of the supervisory review and evaluation process (SREP). The PRA’s assessment will form part of the supervisory assessment of risk management and controls and management governance and culture as set out in ‘The PRA’s approach to banking supervision’, March 2016.3
- 01/06/2018
2.8
All firms applying the principles are expected to do a self-assessment of their stress test model risk management practices against the principles as part of their internal capital adequacy assessment process (ICAAP) and report the findings in the ICAAP documents from Tuesday 1 January 2019 onwards.
- 01/06/2018
Detailed principles
2.9
The four principles on model risk management for stress testing are detailed in the boxes below.
Principle 1 – Banks have an established definition of a model and maintain a model inventory
P1.1 Definition of a model: Banks should establish their own definition of a model. When identifying models banks are expected to take into consideration:
- (a) Calculation methods or systems that are based on statistical, financial or economic assumptions (eg impairment models, income models).
- (b) Calculation mechanisms used to transform a set of parameters or values into a quantitative measure (eg scenario expansion models, probability of default models).
- (c) Frameworks or systems where qualitative judgement is applied to generate quantitative results (eg where adjustments are made to address known model limitations).
- (d) Calculation mechanisms where outputs of other models are used to calculate financial/risk measures (eg expected loss which uses the output of probability of default, loss given default and exposure at default models).
In cases where calculation mechanisms are not classified as models, banks should ensure the risks associated with the implementation and use of such calculations are adequately understood, controlled, and documented as part of an established management control process.
P1.2 Model inventory: Banks should maintain a comprehensive set of information on models ‘implemented for use’, ‘under development’, or ‘recently retired’. The information should clearly identify model owners and users, and should also include all model uses and direct or material dependencies, ie models that depend or use the output of other models. A designated internal party4 should be responsible for maintaining the bank-wide inventory of all models. Any variation of a model which requires separate validation and approval should be classified as a separate model.
Principle 2 – Banks have implemented an effective governance framework, policies, procedures and controls to manage their model risk
P2.1 Board of directors and senior management responsibility:5 The board of directors should establish a framework for the management of model risk and this should be adequately documented. Senior management is responsible for the execution and maintenance of the framework and should designate the roles and responsibilities for the framework to model owners, model users, and control and compliance functions.6 The board of directors and senior management are expected to provide challenge to model outputs and understand model capabilities, the model limitations, and the potential impact of model uncertainty for the most material models and the aggregate outputs.
P2.2 Model risk management policies: These should cover all aspects of model risk management, including model definitions; model development standards; model change; implementation; use; validation; review; and management sign-off. The policies should set out appropriate governance and challenge frameworks, and the roles and responsibilities of model owners, model users, and control and compliance functions. The prioritisation, scope and frequency of validation, review, and monitoring activities should also be set out in the policies.
P2.3 Model developers, owners, users and control functions: Model developers should be responsible for the development, evaluation and documentation of models and may be involved in model monitoring and a reassessment of already implemented models. Model owners should have accountability for model use and performance. Model owners should be responsible for ensuring that models are appropriately developed, conceptually sound, implemented, used as intended, have undergone appropriate validation and approval, and are recorded and maintained in the model inventory. Model users should ensure that models are used consistently with the model’s intended purpose, and that any model limitations are understood and taken into consideration when using the output of the model. Control functions should have the authority to restrict the use of models and monitor any limits on model use.
P2.4 Role of Internal Audit (IA): IA should assess the overall effectiveness of the model risk management framework. IA should evaluate and independently verify whether model risk management practices are comprehensive, rigorous, and effective.
P2.5 Use of external resources: If external resources are used for any model development, validation, or review activities, banks should be able to verify that these are conducted in accordance with their model risk management standards. Designated internal staff should be responsible for the work delivered by the external party, and should be able to address any issues identified either with model development or as a result of model validation.
Principle 3 – Banks have implemented a robust model development and implementation process and ensure appropriate use of models
P3.1 Model purpose and design: The purpose, design, choice of parameters, mathematical theory, and underlying assumptions of a model should be appropriately documented and conceptually sound (appropriate for the intended purpose), and supported by published research, where available, and generally accepted industry practice where appropriate. Particular emphasis should be placed on model limitations and, where appropriate, model results should be supported by a comparison with alternative theories/approaches, or by assessing the sensitivities of changes in model inputs.
P3.2 Use of data: The data used to develop a model should be assessed for quality and relevance. Where adjustments are made, proxies are used, or where the data are not representative of the bank’s portfolio or asset mix, the impact should be justified and documented so that users are aware of the potential model limitations.
P3.3 Testing: Appropriate testing of models should be conducted to take into account potential limitations, assess their robustness and stability over time, and across a variety of economic and market conditions, in particular those relating to periods of stress. Testing activities should be appropriately documented.
P3.4 Documentation: Banks should have sufficiently detailed model documentation so that an independent third party with relevant expertise is able to understand how the model operates, identify its key assumptions and limitations, and replicate any parameter estimation and model results. Where a bank uses vendor models, it should have appropriate documentation on the approach to be able to validate the model.
P3.5 Use of judgement: Any judgements or model overlays that are used to modify the parameters, inputs and/or outputs of a model due to known model limitations should form a part of the development process, should be appropriately understood and documented, and should be subject to review and challenge by independent parties.
P3.6 Supporting systems: Model calculations should be implemented in information systems or environments which have been thoroughly tested for this purpose. The findings of any system and/or implementation tests should be documented.
P3.7 Business involvement: Business experts should play an integral part in the design and testing of models and should challenge the methods, the underlying assumptions, and the output of the models – both at inception and on an ongoing basis.
P3.8 Model uncertainty:7 Banks should implement a model risk appetite framework and demonstrate that model uncertainties are adequately understood, managed, monitored and reported, both on an individual level as well as in aggregate. Model uncertainty should be accounted for in the results and where conservatism is used to mitigate model uncertainty, banks should justify and document any such adjustments and demonstrate that the adjustments are intuitive from a business and economic perspective.
P3.9 Monitoring: Banks should perform periodic monitoring of model performance to ensure parameter estimates and model constructs remain fit for purpose and use when sufficient new observations are available and to ensure model assumptions remain valid. The frequency of model monitoring should be commensurate with the nature and materiality of the models and risks, with due consideration given to model complexity.
Principle 4 – Banks undertake appropriate model validation and independent review activities to ensure sound model performance and greater understanding of model uncertainties
P4.1 Scope of validation and review: All model components (inputs, calculations and reporting outputs) should be subject to independent validation for both in-house developed models and vendor models. Any validation work undertaken by model developers and users as well as any material changes to already validated models or overlays should be subject to review by an independent party. The nature and extent of validation and independent review should be appropriate with the overall use, complexity, and materiality of the models, model components, adjustments to model results or changes to a model.
P4.2 Independence: The people performing model reviews should be independent of the model development process to be able to provide a robust and objective view, evidenced by the quality of issues identified and the actions taken to address them.
P4.3 Competence and influence: Banks should consider whether people performing validation have: the necessary knowledge, skills, and expertise to perform model validations; an adequate degree of familiarity with the business, product, risk, and intended use of the model; and sufficient influence and stature within the bank to ensure that issues and deficiencies are escalated and addressed in a timely manner.
P4.4 Treatment of model issues and/or deficiencies: When significant model deficiencies and/or errors are identified during the validation process, banks should consider whether the use of models should either be prohibited or only be permitted under strict controls and mitigants. The process of managing identified model issues should include the tracking of the outstanding issues and should be adequately documented.
P4.5 Frequency of model validation: Banks should undertake regular revalidation of models to track known limitations and to identify potential new issues. Periodic reviews should be carried out with a frequency and level of rigour commensurate with the overall use, complexity, and materiality of the models.
Footnotes
- 4. ‘The Bank of England’s approach to stress testing the UK banking system’, October 2015: https://www.bankofengland.co.uk/news/2015/october/boe-publishes-approach-to-stress-testing-the-uk-banking-system.
- 5. When assigning the responsibilities of the management of model risk to senior management functions, firms should consider the relevant prescribed responsibilities in ‘Allocation of Responsibilities’ 4.1 or 5.2 in the PRA Rulebook.
- 6. In the context of model risk management frameworks control and compliance functions refer to the independent model review function and model risk governance, or any other related governance function involved in the model risk management framework.
- 7. Model uncertainty should be understood as the inherent uncertainty in the parameter estimates and results of statistical models, including the uncertainty in the results due to model choices or model misuse.
- 01/06/2018