Overfitting in decision models: how to spot it

What is overfitting in a decision model?

Overfitting is the production of an analysis that corresponds too closely or exactly to a particular set of data (en.wikipedia.org, Overfitting). The model begins to memorise the cases it has seen instead of learning what generalises. An overfitted model has more parameters than the data can justify, and it performs well on the past and poorly on new cases.

In a decision model, the parameters are the weights of the dimensions, the thresholds of the zones and any rule for combining ratings. Each setting can be tuned. Tuning them until every past case is ranked correctly feels like progress, and it is a risk, because the settings then describe the sample and not the world.

How does overfitting happen in a decision model?

It happens when the freedom of the model is large compared with the number of cases. Take a model with 9 dimensions: 8 free weights plus 1 threshold make 9 settings. Test it on 6 past cases and the settings can usually be arranged to rank all 6 correctly. The fit is guaranteed by the arithmetic of the setup, so it proves nothing about the equation.

Backtesting makes the temptation stronger. It is often possible to find a strategy that would have worked well in the past (en.wikipedia.org, Backtesting). The analyst runs a test, sees a miss, adjusts a weight, and runs the test again. After 20 rounds, the model has absorbed the answers. The remedy is to decide the settings first and to keep cases aside.

Which warning signs reveal overfitting in a decision model?

Four signs appear again and again. The results on the cases used for design are much stronger than the results on held-back cases. The weights change a lot when one case is removed. The weights look odd, with one dimension at a very high weight and no business reason. And the method contains many special rules that each fix one past case.

The first sign is the clearest. A common illustration in the general literature shows validation error rising while training error keeps falling (en.wikipedia.org, Overfitting). In a decision model, the equivalent is a hit rate that stays at 6 of 6 on design cases and drops to 2 of 4 on held-back cases. The table shows this illustrative gap.

How do you test for overfitting with held-back cases?

Split the cases before any tuning. Keep one group for design and one for the final test, and lock the test group. A time split works well: design on earlier cases and test on later ones. Fix the equation, the weights and the thresholds, write them down with a date, then score the test group once and record the result.

When cases are few, rotate. Leave one case out, design on the rest, score the case that was left out, and repeat for every case. This leave-one-out approach is a form of cross-validation, a technique that the general literature names (en.wikipedia.org, Overfitting). The average result across the rotations estimates how the model will behave on cases it has not seen.

How many cases does a decision model need before its settings deserve trust?

The more settings a model has, the more cases it needs. A working rule for this page is to aim for at least 10 cases per free setting that was tuned. A model with 9 tuned settings then asks for about 90 cases, and a model with no tuned settings needs far fewer, since there is nothing to fit. The rule is a convention for planning, and not a law.

Fewer settings help more than more cases. Fix the weights from reasons stated in advance, and tune nothing on the test group. A model whose weights come from first principles can be checked on a handful of cases, while a model whose weights come from the sample needs a large held-back set. The general literature also advises verification on bigger or completely new datasets (en.wikipedia.org, Overfitting).

How does a fixed, deterministic equation help against overfitting?

The EQOS Asia Codex Universalis is described as a computation core with no AI, no learning and no hidden probabilities, so no training on past cases takes place (EQOS Asia method page, as of October 2026). The 2008 page states that its scores use the Codex φ v3.1, the same equations used for clients today (EQOS Asia 2008 case study, as of October 2026).

A fixed equation removes tuning on the test cases, which is the main route to overfitting. The original choice of weights still deserves a question, and the weighting coefficients are proprietary, available under NDA on request. A reader can ask how the weights were set, when, and on which cases. The answer shows whether the retrospectives were a test or a fit.

Which questions expose overfitting in a vendor decision model?

Ask five questions. How many cases were used to design the model, and how many were held back? Were the weights and thresholds fixed before the final test? How did the result change when one case was removed? Are failures and successes both in the test group? Can a third party recompute the scores from the same equation?

A 30-minute pre-diagnostic by video, scheduled in Singapore time (SGT), applies the EQOS Asia equation to a real case, and you can bring these five questions to the call at no commitment (EQOS Asia contact page, as of October 2026). The pricing page lists the tiers for a fuller study, from a Reading at US$358 (EQOS Asia pricing page, as of October 2026).

Frequently asked questions

About the author

Stéphane Jambu

President, EQOS Forecast Asia

Stéphane Jambu leads EQOS Forecast Asia and works as a consultant in e-commerce, search visibility, and AI. LinkedIn profile