5 Brutal Truths Principal Data Scientists Learn When January Forces Re-Explanation

Read time: 2.5 minutes

5 Ruthless Realities Chief Data Scientists Must Face as January Brings Re-clarification

1. Your model’s much-vaunted approval was contingent upon its being a part of an organizational structure.

The previous approval of a model ended when your company changed its organizational structure.
Fix: If there is no current individual who has been assigned to make decisions regarding a model, regardless of how the model performs, it cannot be used.

2. Your assumptions existed in people, not systems.

The model’s actual functioning in practice is different from what was assumed by the developers of the model.
Fix: If there is no documented evidence of any prior confirmation of an Assumptions Contract, the model cannot go into a production environment.

3. Just because a model produces accurate results doesn’t mean a decision can be made.

AUC doesn’t answer the only problem that executives have: Do I decide to take action?
Fix: Each model must present one tradeoff that can be explained in simple English.

4. If the executive can’t understand it quickly enough, they won’t build a relationship with it.

The model's trust only lasts 10 minutes after a reorganization, after which it must be rebuilt.
Fix: If the model can’t pass the 10-minute reorganization test, do not implement it until you can demonstrate that there is an actual business case for using the model.

5. If the model can’t function without you, it’s not working.

Operations processes and procedures that can be implemented don’t require you to be present for the implementation.
Fix: A model that must have a person in the room is no longer available because it is being rebuilt.

💡Key Takeaway: 

If a model will require someone to touch it to exist past January, then what you have built is a dependency, not leverage. So when your models become disconnected from the context, those dependencies will become visible.