Backtesting a decision model: method, data and pitfalls

What is backtesting in the context of a decision model?

Backtesting is a term used in modelling to refer to testing a predictive model on historical data (en.wikipedia.org, Backtesting). For a decision model, the historical data describes cases whose outcome is known, such as projects that succeeded and projects that failed. The model scores each case as it stood at a past date, and the analyst compares the score with what happened next.

The method differs from testing on live decisions, where the outcome takes months or years to appear. A backtest compresses that wait into an afternoon, which is its strength. The same compression carries limits, since the past case cannot show how the decision would have changed events, and the sections below explain how to keep the test honest.

How do you design the sample for backtesting a decision model?

Build a balanced sample. Include failures and successes, from several sectors and several years, with at least 10 cases when possible. A sample of only collapses can show that the model raises alarms, and it cannot show how often those alarms are false. Write down the selection rule before choosing, for example every listed company in a sector with a given date of event.

Freeze one analysis date per case, and keep a separate group for the final test. A common split uses part of the cases to design the method and the rest to test it. Do not look at the test group until the design is closed. A model that performs on cases it has never seen carries far more weight than one that performs on the cases it was built on.

What data does backtesting a decision model need?

Backtesting needs detailed historical data, dated at the moment it became available. For each input, record the source and the publication date. Annual reports, regulatory filings, press articles and public registries serve well, since their dates can be checked. The back-test pages of EQOS Asia use data that was public at the time, such as filings and central bank reviews (EQOS Asia 2008 case study, as of October 2026).

Point-in-time data matters because many sources are revised. A figure restated two years later reflects knowledge that the analyst did not have on the analysis date. Store each input as it appeared then, and not as it reads today. If only the revised version exists, mark the input and test the model with and without it.

Which metrics score the backtest of a decision model?

Two metrics carry most of the verdict. The hit rate is the share of failures that the model flagged before the outcome. The false alarm rate is the share of successes that the model also flagged. Take 6 failures and 4 successes in an illustrative sample. If the model flags 5 of the 6 failures and 1 of the 4 successes, the hit rate is about 83% and the false alarm rate is 25%.

A third metric, lead time, measures how long before the outcome the alert came. A warning one week ahead helps little, and a warning 12 months ahead gives room to act. Report all three together with the number of cases, because a hit rate from 3 cases weighs less than one from 30. The figures here are illustrations, not EQOS Asia results.

Why does look-ahead bias threaten a backtest of a decision model?

Look-ahead bias occurs when the test uses information that was not available on the analysis date. It makes a model look sharper than it was, because the inputs already contain hints about the outcome. Examples include revised financial statements, later ratings, and labels written after the event. The page on look-ahead bias in back-tests covers the checks in detail.

The practical defence is a dated source for every input and a second reviewer who checks the dates before any score is computed. Many back-tests fail this check quietly. A strict team keeps a log with the date of each document and rejects any input whose date falls after the analysis date, whatever its value to the argument.

How does overfitting show up when you backtest a decision model?

Overfitting is the production of an analysis that corresponds too closely or exactly to a particular set of data (en.wikipedia.org, Overfitting). In a backtest, it appears when weights or thresholds are adjusted until the past cases fit. Backtesting itself carries this risk, since it is often possible to find a strategy that would have worked well in the past (en.wikipedia.org, Backtesting).

The warning sign is a gap between the cases used to design the method and the cases held back. Strong results on the first group and weaker results on the second suggest that the model memorised the answers. The page on overfitting in decision models explains how to spot it and how to keep the equation fixed before the test.

How do the EQOS Asia back-tests apply this method?

EQOS Asia publishes 3 retrospectives. The Theranos page uses public data from 2013 and reports a φ(S) of 0.085, the WeWork page uses public data from 2017 and reports 0.189, and the 2008 page uses public data from 2006 and 2007 and reports 0.234. The 2008 page states that the scores come from the Codex φ v3.1, the same equations used for clients today, and that all computations are reproducible (EQOS Asia back-test pages, as of October 2026).

The Codex behind them is described on the method page. Apply the checklist of this page to them: sample balance, dated inputs, fixed equation and held-back cases. Three failures show direction, and a wider set with successes would strengthen the case. To discuss how a retrospective would apply to your own decision, a 30-minute pre-diagnostic by video, scheduled in Singapore time (SGT), is available at no commitment (EQOS Asia contact page, as of October 2026).

Frequently asked questions

About the author

Stéphane Jambu

President, EQOS Forecast Asia

Stéphane Jambu leads EQOS Forecast Asia and works as a consultant in e-commerce, search visibility, and AI. LinkedIn profile