5 Brutal Truths About How Data Science Is Forcing Operating Model Change (And Making AI Actually Work)

Read time: 2.5 minutes

5 Operating Model Truths that Enable Successful AI Implementation

1. Without a Decision Contract, a Model Becomes a Toy

A model with no identified boundaries for who can use it will likely be treated as optional. Thus, if there are no established boundaries, models are not used in production.
How to Fix: Identify decision boundaries and escalate procedures prior to training. No decision boundaries = no production usage.

2. The Lack of Ownership for Features Causes Unseen Risks

No one owns the input or output of a model. As such, there is no one responsible when it is no longer valid. This is not an issue related to machine learning but a failure of the governance process.
How to Fix: Centralize features in a feature store with clear documentation of ownership. Block models from executing against features or targets that lack defined owners.

3. Honesty in Offline Metrics is Required

AUC is not equivalent to value. AUC is basically used to compare how models rank in a controlled environment.
How to Fix: Link offline metrics to online metrics (e.g., lift, cost saved, cost of error). Document the FALSE POSITIVE / FALSE NEGATIVE trade-off for each model.

4. Optional Models Will Be Ignored at Scale

If a model's output is available to the organization, it will be considered dead. If it is part of the way that work is done, it will be considered alive.
How to Fix: Integrate the outputs of machine learning models into the execution plan. Monitor usage, override rates and outcome difference.

5. Changes to Organizational Structures Will Accelerate the Decay of Models

Models are changing without being detected. A reorganization will quickly destroy the decision-making process of the models. Both will act to destroy the value of a model.
How to Fix: Create a process for creating drift detection, retraining triggers and kill criteria for models. Reaffirm all models after an organizational change, including strategy changes.

💡Key Takeaway: 

Your responsibilities as a staff member are not solely to create better models, but to create enduring systems that can persist through organizational changes.