the health strategy . institute
research and strategy institute
for in-person health strategy
and digital health strategy
Joaquim Cardoso MSc
Chief Researcher and Strategy Officer (CRSO)
for “the health strategy” — research unit
Chief Editor for“the health strategy” — knowledge portal
Senior Advisor — for “the health strategy” — advisory consulting
July 11, 2023
ONE PAGE SUMMARY:
Organizations that use AI to improve or create new KPIs see more business benefits than those that adjust their KPIs without AI.
- According to a survey, 7 out of 10 respondents agree that enhancing KPIs is critical to business success.
Companies are using AI to evolve their KPIs and move away from legacy metrics.
- Examples include Lyft, which transformed its revenue KPI by optimizing conversion rates, and Tokopedia, which used AI to create a new KPI around merchant quality.
Companies relying solely on human judgment to adjust KPIs are less likely to see improvements.
- AI-informed KPIs offer benefits such as increased efficiency, better alignment, and greater financial benefit.
We reseachers have identified three ways to enhance strategic metrics with AI:
- Improve existing KPIs.
- Create new KPIs.
- Establish new relationships among KPIs.
AI can be used to improve existing KPIs by reexamining and redefining parameters, such as Wayfair’s analysis of lost-sales KPIs.
- AI can also be used to create new KPIs, such as predicting sudden cardiac death or identifying indicators of future performance.
- Establishing relationships among KPIs is important for better managing businesses, as seen in DBS Bank’s integration of KPIs using AI.
Leadership should invest in AI to improve KPIs, create new ones, and establish interrelationships among KPIs.
- KPIs should be transparent, explainable, and trusted to avoid organizational misalignment.
- Data quality is crucial for AI-driven KPIs, and organizations should invest in cultivating data as an asset.
Monitoring and oversight of AI-influenced KPIs will be necessary, including evaluating their performance and alignment with desired outcomes.
Infographic
DEEP DIVE
Improve Key Performance Indicators With AI
Organizations that use AI to improve existing KPIs or create new ones realize more business benefits than organizations that adjust their KPIs without AI.
MIT Sloan Management Review
Michael Schrage, David Kiron, François Candelon, Shervin Khodabandeh, and Michael Chu
July 11, 2023
About the Research
This article series presents findings from the seventh annual global research study on artificial intelligence and business strategy by MIT Sloan Management Review and Boston Consulting Group.
In the spring of 2023, we fielded a global survey and subsequently analyzed records from 3,043 respondents representing more than 25 industries and 100 countries. We also interviewed 17 executives leading AI initiatives in a broad range of companies and industries, including financial services, media and entertainment, retail, travel and transportation, and life sciences.
Our research examines how managers and leaders use AI to enhance strategic measurement to advance strategic outcomes. It explores how organizations have adapted — and even generated new — KPIs with AI to define and deliver measurably better performance.
Improving key performance indicators is a clear mandate for most organizations.
According to our seventh annual global executive AI survey, 7 out of 10 respondents agree that enhancing KPIs — not just improving performance — is critical to their business success.
As one executive notes, “We need to evolve our KPIs all the time so we don’t run our business on legacy metrics.”
A growing number of companies now use AI — in a variety of ways — to accelerate that evolution. “I’m very excited about what machine learning can do in terms of having our senior leaders move away from metrics that look backward to metrics that can look forward,” says Avinash Kaushik, chief strategy officer at digital marketing agency Croud and a former senior director of global strategic analytics at Google.
Early on at Lyft, engineers designed an algorithm to maximize revenue by matching driver supply and customer demand. “It looked at all the possible combinations of riders and drivers and picked the combination that — based on the ride being requested, where the driver was located, all of the system dynamics — would maximize revenue,” says Elizabeth Stone, former vice president of science at Lyft. Then, as data scientists began testing other objectives, something interesting emerged. One AI solution discovered that optimizing conversion rates — how often a user ordered a ride after opening the app — would, in turn, deliver more ride requests in the future. More ride requests ultimately mean more revenue. As a result of using AI, Lyft transformed its revenue KPI from one focused on ride and driver combinations to one that also focuses on optimizing conversion rates. 1
At Tokopedia (part of GoTo Group), one of Indonesia’s largest marketplaces, AI sifts through petabytes of data to detect signals that are correlated with credibility and reliability. These are key considerations, given that 86.5% of its 14 million merchants — selling 1.8 billion products — are new entrepreneurs. Having more-credible merchants makes the marketplace more appealing, effective, and efficient. “They might have good products to sell, but they don’t know how to manage their stock,” says Herman Widjaja, Tokopedia’s CTO. “With AI, we connect our customers to the right product that is served by the right merchants that they want.” The company synthesized millions of possible signals into a scoring system that represents a new KPI around merchant quality.
While most respondents understand the need for enhanced KPIs, a clear majority currently rely on inadequate tools and technologies to manage their metrics.
Even as machine learning algorithms and generative AI transform enterprise capabilities, human judgment remains the overwhelmingly dominant approach to KPI enhancement. Two-thirds of survey respondents affirm that managers make judgment calls when adjusting their organization’s KPIs. While common, this approach often fails to yield the desired results: Barely a third of survey respondents relying only on human judgment see their KPIs improve.
Even as machine learning algorithms and generative AI transform enterprise capabilities, human judgment remains the overwhelmingly dominant approach to KPI enhancement.
In contrast, companies that use AI to inform their KPIs are far more likely to see improved metrics.
Ninety percent of respondents who use AI to create new KPIs say they see their KPIs improve.
These AI-informed KPIs offer business benefits and demonstrate new capabilities: They often lead to more efficiency and greater financial benefit and are more detailed, time sensitive, and aligned with organizational objectives. (See “Creating New KPIs With AI.”)
We observe a growing awareness among executives that KPIs need to become smarter and more predictive.
Creating New KPIs With AI
All of the leaders we interviewed for our research voiced similar rationales for enhancing KPIs. As legacy metrics become static and outdated, their value as tools for defining and achieving organizational goals diminish. In fact, they become measurably less useful. Both individually and collectively, KPIs need to be updated and enhanced to ensure that they advance desired organizational outcomes. Improving performance without enhancing KPIs creates competitive risk. Companies that focus on — or align around — suboptimal measures are at a competitive disadvantage. Focusing on performance without a commensurate focus on its measure creates an inherent imbalance that can sabotage a company’s efforts to compete effectively.
Based on our research — which includes results from a global survey of more than 3,000 managers and qualitative analysis of more than a dozen executive interviews — we have identified three ways to enhance strategic metrics with AI:
- Improve existing KPIs.
- Create new KPIs.
- Establish new relationships among KPIs.
This article explains how this Improve-Create-Establish, or ICE, framework can help leaders and managers repurpose KPIs to make their people, processes, and technologies more effective.
Our research shows that AI-enhanced KPIs are associated with strategically valuable business benefits, including increased efficiency, better alignment, and greater financial benefit.
Integrating AI into the strategic measurement process has enormous implications for the future of capital allocation, customer engagement, employee experience, EBITDA, and every other executive metric. (See “Benefits From AI-Adjusted KPIs.”)
Benefits From AI-Adjusted KPIs
We offer several practical steps on how to use AI to enhance their KPIs.
Improve Existing KPIs With AI
Companies that initially use AI to boost performance numbers tend to find that the technology creates opportunities to revisit and review key performance parameters.
The Substitution Effect at Wayfair
Wayfair’s AI approach makes that case. While losing a sale may appear to be a straightforward measure, the online furniture retailer used AI to reexamine the fundamentals behind its lost-sales KPI. As CTO Fiona Tan recalls, “We used to think that if you lost the sale on a particular product, like a sofa, it was a loss to the company. But we started looking at the data and realized that 50% to 60% of the time, when we lose a sale it is because the customer bought something else in the same product category.”
This AI-enabled analysis allowed Wayfair to experiment with substitute offers based on customer concerns such as price points, shipment times, and other factors. Demand for these substitute products provided a new lens on what had been previously measured and interpreted as lost sales. Recognizing this “substitution effect” led Wayfair to adjust pricing across its entire sofa domain, revamp product configurations in its fulfillment centers, and reprioritize legacy KPIs.
Companies that initially use AI to boost performance numbers tend to find that the technology creates opportunities to revisit and review key performance parameters.
At one level, Wayfair used AI to transform its legacy lost-sales KPI into a more valuable metric that differentiated truly lost sales from actual sales. But at another level, the AI behind this upgraded KPI also enabled more sweeping changes. Tan’s team used the substitution effect to develop a “profit-awareness framework,” which changed the way the company interacted with customers. Wayfair’s furniture recommendations began to factor in delivery times and shipment costs, as well as product incidents and product profitability, when making next-best offers. Explicitly measuring and incorporating the substitution effect reframed lost sales as sales opportunities and led to pricing shifts. This reframing consequently aligned product placement decisions with the needs and capacities of distribution centers and warehouses, which improved both employee and customer experiences.
Companies across the industry landscape are finding ways to use AI to improve existing KPIs. Some use AI to deepen their understanding of factors that drive KPI outcomes, whereas others use AI to identify and prioritize which KPIs deliver the most value to the organization. While our survey results suggest that most companies will satisfice around improving KPIs (that is, rely on human judgment), we also see companies recognizing that AI makes possible altogether new KPIs that can achieve, and drive, next-level performance.
Create New KPIs
In addition to improving existing performance metrics, AI offers the potential for managers to algorithmically discover and generate entirely new KPIs. While our survey shows that only 16% of respondents’ organizations use AI to generate new KPIs, 90% agree that their use of AI has measurably improved their KPIs.
For example, hunting for indicators that can help physicians preempt sudden cardiac death, which afflicts 300,000 people in the U.S. every year, is an ongoing effort for physicians and researchers. These types of deaths come out of nowhere; doctors can’t predict them, since patients don’t present as high risk. It’s an especially frustrating condition because there is a known effective treatment — installing a cardiac defibrillator — if doctors can identify at-risk patients in time. One of the researchers in this hunt, Ziad Obermeyer, a physician and professor at the University of California, Berkeley, is working with Region Halland Health System in Sweden whose electronic health record data can be linked to a variety of government data. He and his collaborators trained an algorithm to predict sudden cardiac death in the year after an electrocardiogram (ECG) is performed, using death certificates, and a variety of other data points taken from government records and electronic health records.
“The algorithm became quite good at predicting who’s going to succumb to sudden cardiac death in the year after an ECG is taken. Every time someone gets an ECG, it generates a risk score that measures the probability the individual will die from sudden cardiac arrest,” says Obermeyer. That ECG score can also function just like a key performance indicator: With the score in hand, a doctor might observe how it changes if the patient is prescribed a medication such as a beta blocker or an ACE inhibitor. While the research is at a very early stage, it might one day empower doctors and patients to work together to reduce the risk of sudden cardiac death — and identify new interventions that might decrease the risk. “Having the ability to turn these very complicated biological signals into indicators is very powerful,” Obermeyer says, “and I think that’s going to be something that we’ll see a lot more of, not just for sudden cardiac death but for diabetic complications and all sorts of other preventable, high-stakes conditions.”
Obermeyer’s use of AI to create a new KPI for sudden cardiac death promises to improve patient outcomes, reduce costs, and enhance physicians’ sense of their own efficacy. Developing and discerning new key performance indicators with AI is an emergent phenomenon across the business landscape and among large companies we interviewed, such as DBS Bank, General Electric, General Motors, Sanofi, Schneider Electric, and Walmart. These companies are using AI to identify indicators of future performance at earlier and earlier stages of corporate activity. This capability facilitates better situational awareness and a more effective response to changing market conditions, among other benefits.
Establish New Relationships Among KPIs
No KPI is an island. Many executives we spoke with emphasized that better managing their business requires that they bring local KPIs together into a more integrated set of metrics. Executives across industries explicitly remarked on the inherent organizational and computational tensions between maximizing local KPIs and optimizing higher-level, more macro KPIs. As KPI features and parameters evolve, the importance of anticipating, modeling, and coordinating multiple KPI interactions with AI becomes more critical.
DBS Bank Integrates Its KPIs With AI
Singapore-based DBS Bank once relied on independent KPIs for each function touching different points along a customer journey. That is, for a given product, marketing would, for instance, have its own customer engagement metrics, product would have its own attrition metrics, finance would have its own revenue metrics, etc. Over the past three years, however, the multinational banking and financial services company has replaced that vertical model with a horizontal one. DBS created a value map that ties the use cases together into a single customer journey with outcomes in four categories: customer experience, employee experience, profitability, and risk. “We call it ‘managing through journeys,’” says Sameer Gupta, DBS’s chief analytics officer. “Each customer journey would have multiple drivers, and each driver would in turn have multiple metrics to measure. You can imagine how quickly these scale, and it would be humanly impossible to optimize such a large number of drivers and metrics. AI is a force multiplier that gives us the ability to analyze large numbers of drivers and metrics and also identify those that need to be acted on now.”
In the new model, the outcome data is visible to cross-functional squads, whose members all have a stake in optimizing results in all four categories. “Everyone is looking at the same data, and everyone is accountable for the same outcome,” Gupta notes. The squads use experimentation, data analytics, customer immersion, and AI to continually review the factors driving different outcomes. In addition, they retain a sharp focus on choosing the correct metrics.
Establishing these interrelationships among their KPIs has been, in Gupta’s words, “a fundamental shift.” He anticipates that AI will have an increasingly prominent role as the organization continues to coordinate its metrics in a drive toward continuous improvement. “We are starting to unpack to ask, ‘Do we currently make that decision through data and AI enablement? Is there an opportunity to make it through data and AI enablement? If there is, how can we do it?’” Gupta says. Discerning new relationships among KPIs with AI and other analytical techniques can yield new business opportunities, new insights, and a valuable set of data around which to manage organizational behaviors.
Leadership Takeaways
The future of strategic measurement belongs to smarter KPIs. Our research suggests, however, that few legacy organizations are strategically using AI to improve their KPIs. This makes both AI and KPIs undervalued assets. Leadership needs to invest accordingly. Using AI to improve KPIs, create new KPIs, and establish new relationships among KPIs creates an opportunity to capture measurably greater value. The ICE framework invites executive and boardroom discussions that should lead to management investment in improving strategic measurement.
Our research also emphasizes the need for leaders to align not only on which KPIs to pursue but, even more fundamentally, on the purpose of those KPIs. Will KPIs be primarily retrospective, or will they predictively lean into the future?
If the latter, organizations need road maps for improving existing KPIs; a commitment to exploration and experimentation around creating new KPIs; and a recognition that relationships among KPIs may matter as much as, if not more than, the individual KPIs themselves. Intelligent KPIs should not be primarily managed as independent performance silos; rather, their interdependencies and interrelationships must be understood and addressed. As our examples have shown, these shifts represent operational, organizational, and cultural challenges to leadership.
As has been the case with almost every successful AI initiative, KPIs, like the algorithms that comprise them, must be transparent and explainable. KPIs — organizationally, culturally, and operationally — cannot be seen as black boxes. They should be designed and deployed to be trusted. This cultural component should not be minimized or taken for granted. Otherwise, people at all levels of the organization will seek to game the metrics that guide them, ensuring organizational misalignment.
Underlying this new emphasis on AI and KPIs, and on strategic measurement more broadly, is an acknowledgment that quality KPIs depend on quality data. KPIs trained or supervised on biased, incomplete, and flawed data sets are likely to be biased, incomplete, and flawed. Data quality matters more wherever KPIs matter more. We see organizations committed to AI-driven KPIs becoming even more committed to cultivating data as an asset. They see investments in data reflected in KPI improvements.
Finally, as AI-influenced KPIs become more influential in the enterprise, their own performance will require new levels of monitoring and oversight. Leadership will need to be able to evaluate how well their KPIs are doing, individually and collectively. Are they the right KPIs for sustainable success? Are they appropriately anticipating the future? Are they making the right decisions — and helping humans make the right decisions — for achieving desired and desirable outcomes? Simply put, KPIs will require their own KPIs to help answer those important questions.
About the Authors
Michael Schrage is a research fellow with the MIT Sloan School of Management’s Initiative on the Digital Economy. His research, writing, and advisory work focuses on the behavioral economics of digital media, models, and metrics as strategic resources for managing innovation opportunity and risk.
David Kiron is the editorial director, research, of MIT Sloan Management Review and program lead for its Big Ideas research initiatives.
François Candelon is a senior partner and managing director at Boston Consulting Group and the global director of the BCG Henderson Institute, where his research focuses on the impact of technologies on business and society. He can be contacted at candelon.francois@bcg.com.
Shervin Khodabandeh is a senior partner and managing director at BCG and the coleader of BCG’s AI business in North America. He is a leader in BCG X and has over 20 years of experience driving business impact from AI and digital. He can be contacted at shervin@bcg.com.
Michael Chu is a partner and associate director at BCG who focuses on applying AI and machine learning to business problems in commercial functions, including optimizing pricing, promotions, sales, and marketing. He can be reached at chu.michael@bcg.com.
References
1. S. Ransbotham, S. Khodabandeh, D. Kiron, et al., “ Expanding AI’s Impact With Organizational Learning,” MIT Sloan Management Review and Boston Consulting Group, Oct. 20, 2020, https://sloanreview.mit.edu.
Originally published at https://sloanreview.mit.edu on July 11, 2023.
