A Beginner’s Guide to Model Validation

Models inform business decisions by using a variety of techniques to explain relationships among variables, entities, or events. In addition to actuaries, other stakeholders in the healthcare industry such as hospitals, providers, and insurance companies rely on models to manage the volatility of pricing and demand, which characterizes healthcare. While models are an effective tool to quantify and manage risk, it is critical to also recognize that a model itself constitutes a risk. In fact, a defective model can become one of the costliest sources of risk without proper scrutiny.

Model risk can manifest in several forms but generally pertains to misrepresenting the intended relationship, a flawed implementation, or misuse. Common causes include over- or under-specifying the model, insufficient data, inappropriate model choice, and more. As technical and healthcare risk experts, Axene Health Partners offers a model validation process to mitigate model risk. This can be generalized and applied to models built on a variety of software for actuarial or non-actuarial purposes.

What is model validation?

According to the American Academy of Actuaries, model validation is “the practice of performing an independent challenge and thorough assessment of the reasonableness and adequacy of a model based on peer review and testing across multiple dimensions.”¹ It is best practice to perform the validation after the model is developed and prior to implementation, with repeated and proportional validations following any subsequent updates to the model.

AHP performs formal model validations consistent with the eight core validation principles defined by the North American CRO Council²:

1. Model design and build is consistent with its intended purpose
2. Model validation is an independent process
3. Designate an owner of the validation process
4. Ensure appropriateness of established model governance
5. Make validation efforts proportional to areas of materiality and complexity
6. Validate the model components (input, calculation, and output)
7. Address limitations of model validation
8. Document the model validation

A thorough validation based on these principles will assure model users and management that the model achieves its intended purpose, performs appropriately across a variety of scenarios, and meets relevant standards.

Model Design and Build is Consistent with the Intended Purpose

In order to develop an effective model and perform a validation, there must be a clear definition of the business problem being solved. Without this clarity, the model cannot inform business decisions reliably. The validation examines the model construction to determine if there is a more effective alternative. Reasonable questions might include the following: Is the model stochastic or deterministic? Is tail risk or average risk the primary concern? What risk measures are used? Is this a projection model? What software is used to develop the model? Validation inspects the model logic and data flow to ensure the relevant business problem is addressed most effectively. For example, aggregate risk models are intended to demonstrate the potential losses over a wide range of scenarios and need not provide granularity into individual member or loss data.

Model Validation is an Independent Process

A self-defeating approach would be to mix responsibilities and require the model developer(s) also perform the validation. The importance of peer review has been well documented and is especially important for quantitative projects. The model validation team should possess relevant expertise and can be employees of the same company, but preferably from a different department or area of the company. A particular benefit of hiring an independent firm such as Axene Health Partners to perform the validation is the breadth of perspective and experience available. In-house employees likely do not have the industry knowledge that comes from working with a wide array of healthcare stakeholders and projects. AHP is not merely a team of actuaries; we are healthcare experts that also possess a quantitative and technical expertise. On the other hand, licensed models such as AHP’s IBNR reserving model can be validated by the internal team that plans to use it.

A particular benefit of hiring an independent firm such as Axene Health Partners to perform the validation is the breadth of perspective and experience available.

Designate an Owner of the Validation Process

It is best practice in any project to designate a single individual to be held accountable for the results and be the point of contact. While a qualified team will most likely be required to sufficiently validate a complex model, the validation owner is responsible for making important decisions and ensuring the process is following core principles. Any invalid results produced by the validation or other issues that inevitably arise are escalated to the owner for final authority and communication with the model development team.

Ensure Appropriateness of Established Model Governance

Every model should be subject to some degree of a governance framework. For less complex or critical models, the governance might be unwritten and informally managed by general “rules of thumb.” However, for other important models a governance framework is critical to ensure the ongoing reliability of results. While many papers have been written on the aspects of model governance, it typically specifies the segregation of duties and defines the role of all model users, maintainers, IT, and other functions. The validation team should review the governance framework taking into consideration the complexity and importance of the model.

Make Validation Efforts Proportional to Areas of Materiality and Complexity

Not all validation efforts are equal. A complex model that drives critical business decisions requires more assurance of its results than a lesser one. Furthermore, not all components of a particular model are equally important for providing accurate output. Under time and resource constraints, it is up to the validation owner to decide where to focus the process and where it is acceptable to provide a more cursory validation.

Validate the Model Components (Input, Calculation, and Output)

Validating the input, calculation, and output components of a model is the central purpose of the validation process and likely the area of most concern to the model developers and users.

• Input
  The input component consists of assumptions and data that drive the model calculations. Depending on the source of data, it should be reconciled to its source and compared to industry benchmarks and internal experience. If the data comes from another model, that model’s output component should be validated as well. Expert judgement should be used to validate any assumptions and back-testing can also be employed.
• Calculation
  The validation team should carefully examine the model logic to determine the reasonableness of calculations and ensure that input is correctly incorporated. Two key methods for testing the calculation component include sensitivity testing and dynamic validation. These tests will provide assurance of the stability and reasonableness of the calculations.
• Output
  In addition to the quantitative results of the calculations, the output component includes its format and presentation. The validation ensures that the presentation of results is clear and does not mislead the user. If a similar model is available, the results should be compared for consistency. Historical back-testing and version control are other methods for validating the output component of a model.

Address Limitations of Model Validation

Validation comes with time and resource costs. While outsourcing the validation can save on time costs, it is still likely there will be limitations to the process. This can potentially be addressed by performing frequent high-level reviews to supplement periodic thorough validations. Furthermore, the validation might primarily be concerned with one component of a model and less focus is given to the others. Ultimately, it is impossible to remove model risk and the validation will attempt to mitigate those that are most critical. The validation owner should clearly disclose the extent to which the validation was performed in each aspect of the model.

Document the Model Validation

Whether validation is performed internally or by an external consultant, the processes, key findings, and limitations of the validation process should be documented. Although it is impossible to identify all scenarios, the documentation should also address when additional validation might be needed in the future. Lastly, the documentation should identify the validation owner and justification for any recommendations to improve the model.

Conclusion

The potential impact of model risk should not be understated. One of the most infamous examples involves JP Morgan losing $6.2 billion in 2012, which the U.S. Senate concluded was the result of a systemic failure in risk limits. These risk limits were breached hundreds of times and eventually loosened based on a spreadsheet error that underestimated the risk by half.³ Although it might seem superfluous in the near term to invest in a validation process, a simple and independent validation process can avoid a future tail risk to which senior management is blind. Contact Axene Health Partners for an independent model validation by technical and healthcare risk professionals.

Endnotes

1. https://www.actuary.org/sites/default/files/files/publications/ModelGovernancePracticeNote_FinalDraft_10.30.2016.pdf

2. http://www.crocouncil.org/images/CRO_Council_-_Model_Validation_Principles.pdf

3. https://www.bloomberg.com/quicktake/the-london-whale