“Let’s Use a Neural Network.” — The Data Science Mistake Everyone Makes with 800 Rows

Read time: 2.5 minutes

5 Brutal Truths About Small Datasets in Data Science

1️⃣ Bigger Datasets = More Complexity
Large datasets provide neural networks with many observations.
When working with small datasets, use logistic regression, Decision trees, and gradient boosting.

2️⃣ Overfitting is Extremely Likely to Happen
The model can memorize patterns rather than learn them.
When working with a small dataset, use cross-validation and regularization to catch instabilities early on.

3️⃣ The Importance of Interpretability is Magnified with Small Data
When dealing with a small dataset, understanding the interrelationships will be vitally important.
Choose and develop models with visible Feature Importances and Coefficients.

4️⃣ Data Quality is More Important than the Complexity of the Model
Cleaning features usually improve the algorithm's performance more than changing the algorithm.
Prioritize Feature Engineering over multiple sources of Data and use Domain Knowledge as a Primary Source of Data.

5️⃣ Simple Models Win More Often Than People Think
Simple solutions tend to outperform fancier solutions when the amount of data is small.
Start simple, then add complexity if the amount of data warrants it.

💡Key Takeaway: 

The best Data scientists do not ask: "Which model has the most capabilities?" Instead, they inquire which model better interprets the data we now possess.