Backtesting: Effective Ongoing Due Diligence through Ongoing Backtesting

When an organization sets up the Perpetual KYC framework this is an achievement that commands respect and admiration, both internally as externally. A lot of work, in various departments, has been done to make it a reality. This can be an exhausting effort. There is however no time to relax, because after the framework and model(s) have been put into production the organization needs to verify whether or not the framework is performing as expected.

The are various elements of the Perpetual KYC framework that need backtesting;

• The performance of Transaction Monitoring rules, e.g. are customers circumventing known thresholds and patterns
• The performance of Transaction Filtering rules, e.g. combining data-elements to hibernate hits which you can ‘whitelist’
• The performance of Client Filtering rules, e.g. better matching of customers to detected profiles
• The performance of Client Monitoring rules, e.g. lineage of customer behavior versus point in time
• The performance of the Model and (Automated) Risk Classification as a whole, e.g. impact on up-and-down grading of risk to operations inflow
• Risk Appetite, e.g. change in societal norms can influence your risk appetite such as sustainability.

The first question that needs to be answered is whether or not a risk appetite per control has been developed by the organization. When does the model or do the rules perform within risk appetite and when do they not. It is necessary to establish a measurable boundary that can be tested. If there is no risk appetite statement this either means that the risk appetite is 0%, so everything needs to be correct; or that testing cannot be performed and a risk appetite needs to be developed before testing can begin.

Methodoloy of testing
The second question that needs to be answered is what the methodology of testing is. How does the organization prove that the rules work based on a ‘sample’. How large is the sample size and what is the margin of error and the confidence level? This should be based on the answer of question 1, the risk appetite regarding the control. If these inputs are used for establishing the sample the organization uses, it is statiscally supported and not a normative guess. If the the sample size is large enough to be able to derive correct conclusions; you can start looking if the rule or model is effective. If a different (smaller) sample size is used this either influences the risk appetite or the accurateness of the conclusion. There are many variables that can make up a testing strategy next to sampling, such as periodicity and events within the bank. If you want to know more, continue reading.

Types of backtesting
The third question that needs to be answered is which types of backtesting are suitable for the testing of a specific control. Is it below the line testing, above the line testing, testing of false positives of peergroups, testing of the risk classification or are other types of backtesting necessary. It is important to chose the right backtesting types for a specfic rule or model.

Management information dashboard
The fourth question is if there is a proper management information dashboard that enables management and employees to target specific backtesting and focus on low or exceptionally high performing rules. Insight in the performance of rules is essential to establish effective backtesting. Therefore the outcome of rules (performed by analysts in the operation) need to be made available via a management information dashboard. These dashboard can support performing certain backtesting activities such as below the line testing and above the line testing. If there is no accurate management information about the performance of rules, this needs to be arranged as soon as possible.

Procedures and moment of backtesting activities
The last but not the least questions are when and how do you perform backtesting? Is there are discription of the procedures of backtesting activities available for management and personal, including the necessary approval procedures and audit trails. It is also important to design the right place and moment of performing backtesting. Are you able to perform backtesting as part of the production process or do you need to design a process in which you perform backtesting on samples separate from production. Are you able to automate certain parts of backtesting and how do you make sure there is no bias in the backtesting process (f.e. because employees know it are false positives that need a re-assessment).
Backtesting of the model and its rules is just as essential as the building of the model and the rules itself. Make sure you incorporate that in your design and product or there will be unpleasant surprises afterwards.

Interested in more information about backtesting or how to implement backtesting in your organization? Contact us through the options on this website or directlty through silven@coduce.nl

Want to know more about our services, feel free to browse Services – Coduce or follow us for updates on Linkedin