Machine learning model management

Models are the central output of data science, and they have tremendous power to transform companies, industries, and society. Amazon and Netflix — two of the most successful businesses of this decade — are just two examples of companies using models to create new products and drive ongoing operational improvement. Despite the advantages of being model-driven, many companies are stuck trying to get there.

Companies are treating models like software when they are, in fact, very different — we call this the Model Myth. Even though models look like software and involve data, models have different input materials, different development processes, and different behavior. To overcome the Model Myth, companies need to develop a new organizational capability called Model Management. Previously, model management referred to monitoring production models, but we believe it should encompass a much broader capability. Just as companies have built capabilities in sales, marketing, people management, finance, and so on, they need an equivalent capability in data science. Model Management is a new category of technologies and processes that work together to enable companies to reliably and securely develop, validate, deliver, and monitor models at high velocity to create a competitive advantage.

Organizations that successfully build a Model Management capability will reap exponential rewards as more models drive better customer experiences and better margins. As models build on each other, more models leads to more data and capacity for organizations to invest in new and better models. Those organizations will also better navigate common pitfalls that stymie the impact of models such as ethics, bias, and compliance risk. Ultimately, the haves and the have-nots of this next era of computing will be determined by the quality of an organization’s Model Management.

Machine learning models — what are they and why do they matter?

In less than a decade, data science has risen from a niche function to a board-level focus. Why is that? What is it that data scientists do that is so valuable? Beyond all the hype, all the buzzwords, and all the talk about AI and machine learning — at the heart of data science, the source of its power, is the model. Models are what data scientists make — they are where data scientists create their value. A model is a special type of algorithm. In software, an algorithm is a hard-coded set of instructions to calculate a deterministic answer.

But in data science, models are algorithms whose instructions are induced from a set of data and are then used to make predictions, recommendations, or prescribe an action based on a probabilistic assessment.

Many people have described data as the new oil. If that’s the case, then models are the engines. Models make things happen. They initiate action. They can predict things before they happen more accurately than humans, such as catastrophic events or who is at risk of imminent death in a hospital. Models can build on each other. One model’s output acts as the input to another, more complex model and then creates a living, connected, trainable army of decision-makers. And for better or worse, models can do so autonomously, with a level of speed and sophistication that humans can’t hope to match. Models started in finance and certain areas of risk management, but they are now proliferating to almost every industry. The forces of digital transformation are capturing more data about how businesses operate and are thereby creating more opportunities for data scientists to create models to improve how things are done.

Why models are important

Whether it’s called data science, machine learning, or AI, the atomic unit of this era is the model. Models drive new breakthroughs and operational improvements. The privileged few businesses that run on models are already ahead and accelerating. Everyone else faces an existential risk.

According to a recent McKinsey study, organizations that leveraged models extensively showed a 7.5% profit margin advantage over their peers, whereas those that did not use models had a 2.5% profit margin deficit compared to their peers.

How models create value

Models dramatically lower the cost of prediction, similar to how semiconductors dramatically lowered the cost of arithmetic. They also improve the quality of prediction. This change makes models the new currency of competitive advantage, strategy, and growth. But how exactly do models translate into business success? There are two fundamental mechanisms by which models drive business value.

1. Models are the foundation for breakthrough products, killer features, or even entirely new revenue streams.
2. Models allow companies to create operational efficiencies that compound through constant incremental improvement.

Models are a matter of existential risk

Companies like Allstate, Johnson and Johnson, Lockheed Martin, and other early movers among model-driven businesses create flywheel effects. They are able to build a few models, gather additional data, improve those models, spread learnings from one area to another, increase the probability of breakthroughs, and drive ever-more efficiency. While flywheel effects started in digital-first businesses with rapid model feedback loops (e.g., online retail, digital advertising, and finance), the underlying forces of digital transformation give nearly every business the opportunity to be model-driven.

Similarly, it means no business is immune to the threat from a model-driven competitor. The McKinsey study previously mentioned showed model-driven leaders are adding as much as 20% of their earnings before interest and taxes through the use of AI and ML. This advantage compound to push winners further ahead while laggards fall further, and eventually, impossibly far behind.

Some businesses may comfort themselves that they have strong competitive advantages in the form of their infrastructure, people, or data. However, the competitive advantages many organizations have previously relied upon are already eroding:

• Proprietary algorithms and infrastructure are increasingly threatened by open and cheap competitive offerings due to the rise of open source and cloud computing.
• Unicorn hires are increasingly transitive, with the median tenure of a data scientist less than 2 years.
• Proprietary data is increasingly accessible through sharing arrangements, aggregators, or regulated by rules like GDPR.

However, what remains a competitive differentiator is an organization’s ability to develop, validate, deliver, and monitor models at scale.

The model myth holds us back

As the previous analysis shows, there is tremendous power when a company is model-driven. Unfortunately, only a quarter of organizations are able to pull it off. According to a 2022 survey of senior data and technology executives by New Vantage Partners, only 26% of respondents have AI in widespread production. Why is that? If being model-driven is so important, why aren’t companies making more progress?

Many companies seem stuck in their journey to become more model-driven. Said differently, it’s not that companies aren’t trying to become model-driven; it’s that their approach (for some reason) isn’t working. Here are just a few of the things we have heard in our interactions with people at many companies:

• Data scientists face day-to-day issues with accessing tooling. One organization described how data scientists covertly brought their personal laptops to work because it took months to get new Python packages installed
• Data science managers struggle with tracking the institutional knowledge generated during model development. One data science manager mentioned, “It’s so bad that everyday is like my first day.”
• Decision-makers who use data science-generated insights seldom understand the assumptions and background of those insights. This lack of understanding leads to decision-makers ignoring or misinterpreting the insights.

Models are different

But what’s the deeper issue here? Given the importance of what’s at stake and how much companies are investing in these capabilities, they’re certainly not trying to get it wrong. Our view is that data science leaders and other executives at these companies are treating models like things they’ve seen before — software, data, or business intelligence — rather than recognizing that models are fundamentally different.

We call this mistake the Model Myth — the misconception that because models involve code and data, you can treat them the same way you treat software or data. Models are different and failing to treat them as such results in companies getting stuck on their path to be more model-driven.

The solution — model management

Businesses have developed organizational capabilities for all essential activities: sales, marketing, people management, software engineering, among others. Organizational capabilities allow a business to perform a key function reliably at scale and move that function beyond a dependence on individual heroics or silver bullet tools. Model Management is the name of that organizational capability when applied to data science.

What is machine learning model management?

Model management is a new capability comprised of technologies and processes that work together to enable companies to reliably and securely develop, validate, deliver, and monitor models that create a competitive advantage. We believe it is the solution to the Model Myth. It is the way to unlock the transformative potential of models.

In the past, “model management” has been used as a technical term of art for tracking and monitoring models running in production. We mean it as something much broader. In this section, we offer a framework for this new organizational capability and a vision for how it can change the way we do data science.

The pillars of model management for ML

There are five pillars to model management. The diagram below shows the blueprint of model management.

Most organizations today have invested in building out a capability in two or maybe three of these pillars. Typically, they have focused on model development (the ability to build models), model technology (the ability to equip data scientists with technology), and, hopefully, model production (a way to productionize those models). However, the leaders in this space recognize those alone are incomplete and unreliable as a team’s size and expectations scale. We believe that the full potential of models can only be unlocked with a complete solution that unifies all of these capabilities — a solution that embraces the unique traits of models, including the ability to govern the entire system and manage the knowledge generated throughout.

Model technology

Model technology encompasses the compute infrastructure and software tooling that gives data scientists the agility they need to develop and deploy innovative models. The input materials for models are different from software or BI. Models are constantly evolving, with unprecedented innovation in both the open-source and commercial ecosystems. Models also use more computationally-intensive algorithms, so they benefit from scalable compute and specialized hardware like GPUs. Model technology provides the backbone of the complete data science system, allowing organizations to use the appropriate tools at each step, from early data exploration to ongoing monitoring of mission-critical model products.

Model development

Model development allows data scientists to rapidly develop models, experiment, collaborate, and drive breakthrough research. The process to build a model is much more experimental than software. Data scientists explore data interactively and programmatically test many different permutations of features and algorithm types. Also, data scientists share insights with colleagues and stakeholders as well as store insights for later use. Often the insights gained in model development are just as important to the success of an organization as the model itself and should be categorized and retained

Model production

Model production is how a data science team’s work is operationalized. It is how it goes from an innovative project to a live model product integrated into business processes, affecting decisions and driving value. Since a model behaves differently than software, model production enables organizations to consume models in ways that allow humans to easily apply necessary judgment or provide feedback. Model production also tracks how a model performs (model monitoring and model risk management), how and where it is used, and ensures a closed-loop to drive iterations and improvements.

Model governance

Model Governance is how a company can keep a finger on the pulse of the activity, cost, and impact of data science work across its organization as well as understand what’s going on with projects, production models, and the underlying infrastructure supporting those. Governance of the whole model system is far more complex than other systems because of the confluence of previously described unique traits of models: rapidly evolving toolkits, research-based development, and probabilistic purpose

Model context

Model Context is all the knowledge, insights, and artifacts that are generated while building or using models. It represents the complete provenance of a model which can be modularized (features, datasets, environments, code, subject matter experts, validation checks, monitoring plans) and reassembled into new models. This is often a company’s most valuable intellectual property, and the ability to find, reuse, and build upon it is critical to driving rapid innovation. Like in science, bottoms-up cross-pollination of knowledge is the most effective and sustainable to scale the impact of models. Model Context is also essential to proactively addresses concerns around compliance and auditability of models.

The future of model management

Model Management is the next great paradigm shift for businesses. The organizations that become model-driven will continue to distance themselves from laggards. Much of the frustration and disappointment to date stems from shoehorning models into existing organizational paradigms like software development or data management. By recognizing what is unique about models and building an organizational capability that takes the learnings of other capabilities and embraces these differences, businesses can generate more breakthroughs and consistently improve their core operations. This is the key to remaining competitive in this next era.