Financial innovations often involve complex quantitative models that can be used to perform difficult computations, like valuing complex derivatives, and to enable great new products and capabilities, like passive index fund investing, hedging interest rate or credit risk, or developing a sustainable financial plan.  But you have to be careful.  Sometimes the complexity inherent in a model can backfire and become the source of major financial losses or failure.

Best practice in model risk management is, in addition to utilizing competent and diligent people to create the models, to submit all models to an independent process of testing and assessment.  This process is called “model validation” or more simply model vetting.  However, sometimes even this process does not prevent debacles.

I recently read¹ about an example of model error that struck home because I knew one of the principals involved.  Barr Rosenberg is one of the top experts in quantitative finance on the planet.  He took a PhD in Economics from Harvard and for many years was a professor of finance at the University of California at Berkeley (where I once took a class from him, back in the early 1970s).  In the mid-1970s, Professor Rosenberg (Barr) started a consulting firm called BARRA that supplied sophisticated risk management products to the institutional investment management industry.   These products included detailed assessments of portfolio risk.  Security market risk can be decomposed into two pieces – systematic risk that is associated with one or more risk factors that are common to many securities and idiosyncratic risk that is unique to a particular security.  The state of the art in academic finance is to explain systematic risk with one or just a few risk factors.  The BARRA risk models included literally dozens of risk factors.

After running BARRA for about ten years, Barr became convinced that his risk models could be used to identify under and over-valued securities, and thus could form the basis for an actively managed equity portfolio business.    So in 1985 Barr sold BARRA to investors and started a new company, Rosenberg Institutional Equity Management (RIEM) to manage equity portfolios.  Given his stellar reputation among institutional investors, his new company quickly acquired several large pension fund clients.  Barr claimed that his models would generate portfolio returns 3-5% greater than market indexes.  The models turned out to work just as well as expected and RIEM produced market beating returns for several years.

Ultimately, a large money manager based in Europe, AXA Investment Management, purchased a 50% interest in RIEM and Barr stayed on as Chairman of the new entity AXA Rosenberg.  This firm was also highly successful and eventually assets under management grew to exceed $100 billion.  But then a bug crept into the model.  Specifically, a programmer installing a new version of the code introduced an error by, according to news reports, treating a percent as a decimal.  That is, suppose the volatility of a certain risk measure is 30%.  If you represent this in decimal form as .3, while the computer expects the input in percentage form, then your volatility measure will be off by a factor of 100 (and the variance, the square of the volatility, will be off by a factor of 10,000).

A lengthy period of underperformance caused the technical staff to search carefully for the dead rat buried in the code, and eventually they found and corrected the error.  But in the meantime portfolio returns lagged behind benchmarks, clients became uneasy and assets under management plunged dramatically.  The consequences of model error in this case, while material to the principals involved, did not include large effects on the overall economy or financial system.  Still, the lesson is that it can happen to the best in the business.  Again, you have to be careful.

¹Larker and Tayan, “Risk Management Breakdown at AXA Rosenberg,” Stanford Closer Look Series, 2013.