Wednesday in the Woods 002: Goals and Metrics in Machine Learning Maturity Models

You wood have to admit that there were some production issues last week with the first episode, so I decided to aim for a higher level of production. Unfortunately, as happens so often when you branch out into something new, I forgot to leaf something out of the shot. I promise I'll tree-t you to the perfect fall production next time.

But back to the content - because how else are you going to fit a micronap into your day. To be fair, as Jillian Bommarito pointed out, 13 ½ minutes is more of a power nap (you can delete Calm and Headspace from your phone now). Anyway, last week, we discussed goals and metrics and I left off by promising a real example of goals and metrics across the maturity scale. Today, we discuss the machine learning maturity journey and what it might look like for an online dress shop.

Maturity is obviously a journey, but how can you figure out where you are on the path? When we work with organizations, we tend to focus our discovery on evidence of certain activities or concepts. These hallmarks of maturity exist in more than one dimension, but when you put all the clues together, the picture helps identify where an organization is today, how and why it got there, and where and how it might improve.

Our maturity assessments generally cover a range of dimensions, some of which depend on the context or industry, but almost every client we work with gets to talking about goals and metrics. These two dimensions of an analysis are, in my opinion, the most important, which is why we cover them in detail to start our series.

Goals begin with, well, nothing; in the beginning, these ideas are unknown to an organization. You could think of this as maturity level zero, as there isn’t really any activity or idea to assess.

At the first stage of maturity, companies become aware of a concept like data science or machine learning and decide that they want to associate with that label. There’s nothing actually going on yet - no capabilities, no dedicated resources, no purposefully-designed systems - but there is a spark of intention directed towards a goal. Sometimes this spark comes from the top, lit by an airport magazine or HBR article, and sometimes this spark comes from the bottom, with a curious and engaged employee. Either way, it needs fuel to grow.

At the second stage of maturity, the flame begins to burn and goals begin to form around capabilities. Goals sound like specific techniques, resource roles, or vendor types. For example, management might seek to hire a data scientist or machine learning engineer, or a self-driven employee might want to build a clustering model. The capabilities are possibly connected to discrete tasks in their day-to-day business life, but are generally fragmented. Data collection is ad-hoc, pipelines are not replicable, and systems for research and productionization are in the Pre-Cambrian stage - but there is life in the world.

In the third stage of maturity, organizations begin to align activities with operational goals. Goals start to mirror overall processes or financial metrics, especially at the department or business unit level. We see multiple types of data science roles, including an understanding of the difference between data collection and annotation, data governance and management, research and development, and productionization, though some of these functions are typically still externalized to outside vendors. During this stage, organizations typically experience the most immediate and demonstrable financial impact from their activities. Sometimes, these activities even take on *too* much life of their own, as the capability to use machine learning has outstripped the organization’s governance or compliance of that capability.

Finally, in the fourth stage, organizations can focus on strategic goals, including areas like customer experience and brand, business sustainability, or governance. Organizations in this stage have typically established dedicated, internal roles and teams, including management, for those from stage three. More importantly, these organizations dedicate resources towards compliance and ethics related to their activities, such as incorporating analysis of these activities into risk and ESG programs or monitoring models for training data bias. Capabilities around regulatory research, lobbying, and policy communication feature at the top end of the scale.

Psst - wake up. Micronap time is over. You earned it!

For Tomorrow and Friday, we’ll be posting two more example illustrations - one for an online event management and ticketing company and another for a manufacturing company. And as always, we hope you enjoyed a moment of serenity if you made it through (or 12:38, for those of you who are just here for the forest soundtrack)!
Be the first to comment