“Why Is My Model 100% Accurate?” — Your Decision Tree Just Confessed

Read time: 2.5 minutes

When 100% Is a Red Flag, not a Win!

1. If you have trained your model to have perfect accuracy, then it's probably over-fitted.

• The model has memorized all the noise in the training data and everything that was specific to that dataset.

• The model was memorizing the training data rather than generalizing to unseen data.

In other words, it's not intelligent; it's just recalling every entry from the training data.

2. Decision trees are particularly vulnerable to memorization.

• The trees will continue splitting until they give "perfect" splits for every sample in the data.

• Each leaf node is a lookup for a sample of data.

To change this, we need to go from depth to being controlled by complexity.

3. You used the wrong metric to optimize your model.

• Training accuracy is very impressive on paper,

• But the only thing that counts is how well your model performs in the real world.

To change this, we need to go from using training metrics to validation metrics and testing performance.

4. Complexity without constraint causes a model to break.

• Too many splits (leaves).

• No pruning takes place.

• No regularization takes place.

In other words, more information does not always improve a model's predictive power.

5. The objective is not perfection but generalizability.

• It is perfectly acceptable to have slightly lower training accuracy when improving a model to better perform on unseen data.

• The great achievement is to beat a model prediction on unseen data.

💡Key Takeaway: 

A perfect fit for your training data does not necessarily translate into the same level of performance when applied to the real-world data your model will encounter.