Building a Capability Roadmap: The Maturity Stages of Data & AI

Enterprises spend an average of $15M annually on data & AI initiatives. Yet, last year, 90% of AI investments by enterprises saw zero return, according to VentureBeat. This means a lot of money and effort is going into advancing data & AI capabilities, but companies are still struggling to see the business value. CDOs and CDAOs must ruthlessly prioritize their focus to demonstrate value so they can justify additional investment. They need to keep the lights on (address urgent operational issues) and make the foundational improvements to continue expanding their abilities. To manage this, most leaders have created a plan of attack in the form of a capability roadmap that addresses those challenges in order of priority and dependency.

Organizations want to mature quickly and carefully. It is tempting to start with technology improvements and lean on existing frameworks, like Google’s MLOps framework, to show progress and demonstrate value. These improvements are important but without alignment to the business and everyone productively contributing to the ecosystem, this initiative may not be successful. More often, data & AI leaders face people challenges; talent acquisition and retention, culture change, and buy-in from business stakeholders to run experiments or adopt new solutions. These less technical hurdles tend to have fewer industry-recognized playbooks readily available, but they are being talked about. Accenture recently published a maturity framework that highlights the critical areas that must be present to move up the maturity curve, listing capabilities that range from tooling availability to upskilling and education. What if we could actually address cultural and change management challenges throughout the process of improving technical capabilities?

Developing a set of standards and policies on each tactical capability is critical to ensure consistency across teams and adherence, measurement of adoption, and measurement of business value. Let’s start with an example. A high amount of effort is spent organizing data and creating reliable metrics the business can use to make better decisions. This creates a daunting backlog of data quality improvements and, sometimes, a graveyard of unused dashboards that have not been updated in years. This is usually due to a bottom-up approach of only building everything that is feasible to build today, regardless of need. The worst metric to track productivity is the “number of dashboards deployed.” This incentivizes teams to build with volume in mind instead of value. Technical challenges usually bubble up during this process which starts the conversations on making platform improvements that are not guaranteed to unlock ROI for the company.

A top-down approach looks at the business use cases in priority of potential value. Assessing each use case for feasibility, solution design, and necessary technologies to solve those business problems. This ties tactical capabilities (like data stewardship) to real business value. Capability improvements should be prioritized this way. Without a steady intake of business problems to solve, technical teams end up building a platform or solution for the sake of building it. Eventually, the CDO is tasked with justifying additional investment into the data or AI organization without demonstrating a tangible return.

One of the other challenges of leveraging data and AI is the level of dependencies needed to build and maintain valuable and relevant solutions. For example, you may start with wanting to solve the customer churn problem but end up uncovering a nasty data quality issue or lack of tools to build the most effective solution. This discovery may distract you with an initiative to overhaul the entire data capture system and data ingestion pipelines. Or make long-term improvements to managing data pipelines and storing data in the warehouse. Explaining this to your business teams without getting into the technical nuance is complex. Nobody wants to tell the business, “It is going to take another 6 months to answer that question.” However, this one use case may not justify that level of investment.

Intentionally planning these capabilities allows you to proactively address the needs of the business and technical teams. It is important to avoid the desire to do a “big bang” release. Focus on making incremental improvements to the entire ecosystem while focusing on the improvements that unlock the most valuable use cases. This often requires a few key elements:

1. A well-curated repository of use cases that can be helped through analytics or AI and their associated potential business impact.
2. The impact that each fundamental capability has on each use case.
3. The technical or business process challenges for each capability that contribute to the quality of solutions.
4. The effort to change or update each capability and the layers of dependencies.
5. The level of effort required to change behaviors and underlying workflows and sustain those changes

Once it is determined which capabilities will yield the highest impact and business value, it is time to determine how to make those changes effectively and efficiently. Each incremental improvement typically undergoes four key phases in order to ensure proper change management: Design, Pilot, Rollout, and Sustain. Below are typical high-level activities to consider in each phase.

Design

• Determine the root cause of the challenges prohibiting productivity (might be continuous data quality issues experienced in production dashboards)
• Understand the impact this issue has on the business (how many instances occur and what is the user doing with these quality issues?)
• Design a few approaches that could solve this issue (create low, medium, and high-effort options)
• Weigh the impact with the level of effort to solve it (example: set up new monitoring on all data pipelines that proactively exposes quality issues before they hit the dashboard)

Pilot

• Highlight the best practices or framework changes that need to be implemented (which dimensions of quality should we monitor and at what point?)
• Select a high-impact use case to run a pilot with
• If the new approach requires new tooling, create a list of potential tools to solve the problem
• Run a proof of concept for those tools and determine the appropriate tool
• Finalize the results of the pilot that highlights the change in approach and the business value on the use case

Rollout

• Determine all individuals that will be impacted by the change (technical teams, end users, business users)
• Build a robust communication and educational plan that includes expected process changes and knowledge checks to ensure the change will be sustainable
• Continuously follow up with everyone impacted and compare the actual outcomes with expected outcomes

Sustain

• Monitor value metrics, business impact, and adoption over time
• Onboard additional use cases, make tweaks to the approach, and support the technical ecosystem
• Ensure resources and referenceable materials are also up to date and being utilized appropriately
• Revise and adjust the capability roadmap based on newly prioritized business needs

If your organization has not structured improvements in this way, there are simple steps to get started. Track down your version of a capability map and understand the prioritization logic, look for the list of prioritized business use cases along with the potential impact, and look for internal existing standards or policies. Finally, it is critical to outline a mechanism to manage those iterative changes and sustain those changes over time.

Originally posted on OpenDataScience.com

Read more data science articles on OpenDataScience.com, including tutorials and guides from beginner to advanced levels! Subscribe to our weekly newsletter here and receive the latest news every Thursday. You can also get data science training on-demand wherever you are with our Ai+ Training platform. Subscribe to our fast-growing Medium Publication too, the ODSC Journal, and inquire about becoming a writer.