Model velocity: the key to accelerating your model-driven business

Most data scientists will tell you that success lies in developing models iteratively, but iterative development alone is not sufficient. No amount of iterative development will drive sustained business outcomes if your model is deployed once and never touched again (or, as so frequently happens, not deployed at all). Not only will your model degrade the moment it goes into production, but more often than not, you only discover the true business requirements for your model once people use the production version, i.e., after your model is deployed. And when, sooner or later, you face systemic shifts in your data – e.g., due to a pandemic – you will need to throw out your model no matter how many iterations went into its development.

Model velocity is a must for model value and volume

The key to driving transformative impact lies instead in your ability to iterate across the entire lifecycle of your models. How quickly and effectively you can do this is a measure of your Model Velocity, and how it changes over time helps you track your progress toward becoming a model-driven business. Model Velocity captures your ability to traverse the end-to-end model development and deployment process rapidly, repeatedly, and consistently – from data engineering, feature engineering, algorithm selection, hyperparameter tuning, validation, deployment, to monitoring. It gauges your ability to implement new models in the same way that Feature Velocity helps developer teams gauge their ability to deliver new features.

You need to maintain and accelerate your Model Velocity to tackle every opportunity to drive business value using new models, but just as importantly, keep your models up-to-date so they reflect current market conditions, customer preferences, and more. You need to continuously adapt your models, improving them with additional training data and adjusting them for drift. Your Model Velocity tracks how quickly you can iterate continuously through the data science lifecycle. Combined, your Model Velocity determines how quickly you can tackle new model-driven projects, how many you can sustain simultaneously, and how effectively you can maintain successful outcomes with them.

Measure your models and your model velocity

Every data science team struggles with measuring their effectiveness, tracking progress, and benchmarking themselves. Measuring model performance – e.g. precision, recall, or F1 score – or the resulting business outcomes, while extremely important, provides limited insight into your capabilities. A drop in model performance could easily mask strong capabilities because the drop would have been larger if the team hadn’t deployed an updated model.

Measuring the number of projects completed or models deployed is just as problematic. It is commonly assumed that more is better, but if you aren’t maintaining these models then more models just means more risk, and it can just as well signal that your teams are stretched too thinly and can’t support the current portfolio, let alone undertake new initiatives.

Measuring Model Velocity gives you a more complete view of your data science capabilities, maturity, and performance because it measures the success of the activities that went into those models. It is thus less affected by external factors (such as changes in business conditions or IT systems) and gives you a better view, not just of the health of your deployed projects, but a view forward of the capacity, capability, timing, and frictions that you need to plan for the future.

Accelerate your model-driven business

What company today doesn’t want to be more model-driven? The challenge lies in scaling beyond ad-hoc projects and siloed successes to reliable, repeatable capabilities to grow, maintain, and manage model-driven projects across the organization. The road forward is littered with speed bumps and potholes that limit your Model Velocity and can stall your model-driven ambitions altogether. Too frequently, data scientists end up with inadequate access to data, outdated tools, insufficient computational horsepower, and slow, manual processes for getting models into production and maintaining them over time. Accelerating your model velocity requires enterprise-grade machinery to drive collaboration and automation with the most up-to-date tools and infrastructure that support the end-to-end lifecycle.

Model Velocity gives us a new way to measure the model lifecycle holistically – capturing the people (roles and skills), processes (collaboration), and technology (integrated platforms) factors that determine success. Model Velocity is a new concept – you will need to adapt the exact formula for your needs, and it is not a replacement for most of the metrics we use today, but with it we now have an accelerometer for our model-driven journeys.