Gartner
Top Trends in Data and Analytics, 2022
By Rita Sallam, Ted Friedman, and 37 more
11 March 2022
Top trends in data and analytics technology and practices can help anticipate change and transform uncertainty into opportunity.
Data and analytics leaders must factor these trends into key investments that drive new growth, efficiency, resilience and innovation.
OVERVIEW
Opportunities
- Connections between diverse and distributed data and people create truly impactful insight and innovation.
These connections are critical to assisting humans and machines in making quicker, more accurate, trustworthy and contextualized decisions while taking an increasing number of factors, stakeholders and data sources into account. - CEOs’ highest priority is to return to and accelerate growth, but they must do so in an extremely uncertain and volatile environment.
Capabilities that enable navigating and responding to accelerated disruption across all aspects of the geopolitical environment, business, government and society are foundations of success. - Prioritizing trust and security in these unprecedented times of global chaos is fundamental to the strategic role of data and analytics to realize new sources of value.
Recommendations
Data and analytics leaders looking for new opportunities for their D&A programs and practices should:
- Improve situational awareness to rapidly adjust to disruption and uncertainty by prioritizing investment in data and analytics diversity and dynamism, including
– adaptive AI systems,
– expanded data sharing and
– data fabrics. - Drive new sources of innovation and value for stakeholders by implementing context-driven and domain-relevant analytics to be composed from modular capabilities by the business.
Addressing the scarcity of skills and hireable D&A talent is a top existential priority. - Institutionalize trust to achieve pervasive adoption and value at scale by managing AI risk and security,
and enacting connected governance across distributed systems, edge environments and emerging ecosystems.
What You Need to Know
The past year has seen an accelerated pace of disruption across all vectors of business, government and society with the potential for even more profound shockwaves to come as a result of the Russian invasion of Ukraine.
The global health crisis has been displaced for the moment by a geopolitical one, and the combination continues to shift people’s priorities, their values and their roles as family members, customers, employees and citizens.
These conditions are driving extreme uncertainty, but also opportunity.
This will only accelerate.
CEOs’ highest priority is to return to and accelerate growth while faced with uncertain and highly fluid global political, economic and health realities and their impacts. 1
For organizations to thrive in this environment and realize value at scale, they need to optimize for a new value equation.
One which enables them to respond more quickly than their competitors to shifts in customer and employee values and accelerates new product, channel and business model innovations, particularly in response to macroeconomic and political disruptions.
At the same time, organizations must factor in new stakeholder demands for a response to geopolitical, sustainability and social justice issues.
These create new market dynamics, heated competition for talent, acute supply chain challenges, and renewed macroeconomic unknowns, such as inflation, new regulation and political shifts.
Increasingly frequent weather disruptions introduce yet another set of challenges and new risks.
Combined, these issues have made managing radical uncertainty the purview of the D&A leader, and a critical core competency for success.
It is vital for data and analytics leaders responsible for strategy and innovation to take action along three tracks set out in our analysis of the Top Trends in Data and Analytics for 2022 (see Figure 1):
- Activate diversity and dynamism by leveraging the rise of adaptive AI systems to drive growth and innovation while coping with fluctuations in global markets.
Innovations in data management for AI, automated, active metadata-driven approaches and data-sharing competencies, all founded on data fabrics, unleash the full value of data and analytics. - Augment people and decisions to deliver enriched, context-driven analytics created from modular components by the business.
To make insights relevant to decision makers, organizations must also prioritize data literacy and put in place strategies to address the scarcity of hireable data and analytics talent. - Institutionalize trust to achieve value from D&A at scale by managing AI risk and enacting connected governance across distributed systems, edge environments and emerging ecosystems.
Figure 1. Gartner’s Top Data and Analytic Trends for 2022
The top D&A trends represent business, market and technology dynamics that you cannot afford to ignore (see Table 1).
They have the potential to transform your enterprise, and will accelerate in their adoption over the next three years.
You should decide whether to proactively monitor, experiment with or aggressively invest in key trends based on their urgency and alignment to strategic priorities.
The top D&A trends and technologies do not exist in isolation; they build on and reinforce one another.
We have selected our trends for 2022 in part based on their combined effects. Taken together, our top data and analytics technology trends for 2022 will help you meet your organization’s top strategic priorities to anticipate, adapt and scale value.
This, in turn, will enable you to encourage innovation and put in place success metrics and incentives that emphasize learning and reward innovation.
Activate Dynamism and Diversity
Adaptive AI Systems
Analysis by: Erick Brethenoux, Soyeb Barot, Ted Friedman
SPA: By 2026, enterprises that have adopted AI engineering practices to build and manage adaptive AI systems will outperform their peers in the number of operationalized AI models by at least 25%.
Description: Adaptive AI systems aim to continuously retrain models and learn within runtime and development environments based on new data, in order to adapt more quickly to changes in real-world circumstances that were not foreseen or available during initial development. AI engineering orchestrates and optimizes applications to adapt to, resist or absorb disruptions, facilitating the management of adaptive systems. AI engineering provides the foundational components of implementation, operationalization and change management at the process level in order to enable adaptive AI systems. Adaptive AI systems support a decision-making framework centered around making faster decisions while remaining flexible to adjust as issues arise.
Why Trending:
- Flexibility and adaptability are now a fundamental business requirement — many organizations unfortunately learned this the hard way during the global COVID-19 pandemic. Reengineering systems has significant impacts on employees, businesses and technology partners. For many enterprises, these changes demand resilience by design and adaptability by definition.
- The value of fully industrialized AI lies in the ability to rapidly develop, deploy, adapt and maintain AI across different environments in the enterprise. Given the engineering complexity and the demand for shorter time to market, it is critical to develop less rigid AI engineering pipelines.
- Automated, yet resilient, adaptable systems will require composable D&A architectures with application development to assemble intelligent decision-making solutions.
- Decision making is a core capability, and it is becoming more complex. Decisions are becoming more connected, more contextual and more continuous. Hence, D&A leaders need to reengineer decision making enabled by adaptive systems in the future.
Implications:
- Adaptive AI systems will require processes to be reengineered for automated decision-making. Increased automation will, in turn, require business stakeholders to ensure the ethical use of AI for compliance and regulations.
- Adaptive AI systems, while enabling new ways of doing business, and by leveraging generative AI capabilities, will result in the creation of new business models, products, services and channels.
- While adapting to context, real-time changes fostered by adaptive capabilities will promote new collaboration between organizations; this is how adaptive AI systems will enable cross-organizational change. It is possible to only change a few internal functions using adaptive AI systems, but that would amount to local optimization, which defeats the purpose.
- The bringing together of the D&A, AI and software engineering practices will be critical in building adaptive systems. AI engineering is going to play a critical role: building and operationalizing composable architectures.
Recommendations:
Data and analytics leaders should:
- Make it easier for business users to adopt AI and contribute to managing adaptive AI systems by incorporating explicit and measurable business indicators through operationalized systems and institutionalizing trust within the decisioning framework.
- Maximize business value from ongoing AI initiatives by establishing AI engineering practices that streamline the data, model and implementation pipelines to standardize AI delivery processes.
- Place limits on the amount of time a system takes to make a decision in response to disruptions, while focusing on the critical components of the decision-making framework. The new decision-making process should encourage the use of flexible initial decisions that can be amended as more information is gathered about the environment.
Changes Since Last Year
As part of the Top Data and Analytics Trends, 2021, we introduced a trend called XOps (see Top Trends in Data and Analytics for 2021: XOps). We also expanded on this trend in Gartner’s Top Strategic Trends research by evolving XOps into AI engineering (see Top Strategic Technology Trends for 2022: AI Engineering). AI engineering is a discipline that streamlines the AI development and operationalization life cycle by leveraging DataOps, ModelOps and DevOps, paving the way to build automated adaptive AI systems. Over the past 10 years, AI-based systems have been built for efficiency and autonomy, but operationalization has remained brittle, even if AI engineering practices are on the rise.
Gartner Recommended Reading Adaptive AI Systems Executive Brief
Data-Centric AI
Analysis by: Svetlana Sicular, Ted Friedman, Mike Fang, Erick Brethenoux
SPA: By 2024, organizations that lack a sustainable data and analytics operationalization framework will have their initiatives set back by up to two years.
Description: Data-centric AI disrupts traditional data management and prevalent model-centric data science by addressing AI-specific data considerations, such as data bias, labeling and drift, in order to improve the quality of models on an ongoing basis.
Why Trending:
- The AI community is facing a bifurcation of “model-centric” and “data-centric” AI, because data quality and consistency improve AI accuracy more efficiently than tweaking models.
- Most commonly delivered AI solutions depend on data availability, quality and understanding, not just AI model building. However, many enterprises attempt to tackle AI without considering AI-specific data management issues. The importance of data management in AI is often underestimated.
- Traditional data management is ripe for disruption, to support AI efforts. Without the right data, building AI is risky and possibly dangerous. In most organizations, AI-specific considerations, such as data bias, diversity and labeling, are addressed haphazardly.
- The AI community remains largely oblivious to data management capabilities, practices and tools that can greatly benefit AI development and deployment.
- AI reflects and amplifies bias originating in the choice of algorithms, data interpretation and labeling, and human bias recorded in the data. Bias mitigation is an acute, AI-specific problem. People interpret data, curate it and put algorithms and data together. For this reason, AI-centric risk management, including data governance, is also a related trend.
Implications:
- Data-centric AI is evolving, and should include relevant data management disciplines, techniques and skills, such as data quality, data integration and data governance, which are foundational capabilities for scaling AI.
- Data management activities don’t end once the model has been developed. Deployment considerations and ongoing monitoring of the relevance of the model require dedicated data management activities and practices.
- Organizations that invest in AI at scale will shake up their data management practices and capabilities to preserve the evergreen classical ideas and extend them to AI by making them AI-centric in two ways:
- Add capabilities necessary for convenient AI development by an AI-focused audience that is not familiar with data management.
- Use AI to improve and augment evergreen classics of data governance, persistence, integration and data quality. For example, by making augmented classical profiling, cleansing, visualization and entity resolution available to AI teams.
- Leading enterprises are building out data fabric and active metadata, while the tooling for data management is about to be reinvented. They must decide what to build themselves now, and what to wait for. For example, enterprises are aggressively implementing graphs and extending their governance to AI.
Recommendations:
Data and analytics leaders should:
- Formalize data-centric AI and AI-centric data as part of your data management strategy. Implement active metadata and data fabric as a key foundational component of this strategy.
- Leverage an ecosystem of AI-centric data management capabilities that combine traditional and new capabilities to prepare the enterprise for the era of decision intelligence.
- Promulgate policies about data fitness for AI. Define and measure minimum standards, such as formats, tools, metrics, etc., for AI-centric data early on. This will prevent the need to reconcile multiple data approaches when you take AI to scale.
- Seek diversity of data, algorithms and people to ensure AI value and ethics.
- Establish roles and responsibilities to manage data in support of AI. Leverage AI engineering and data management expertise and approaches to support ongoing deployment and production uses of AI. Include data management requirements when deploying models.
Metadata-Driven Data Fabric
Analysis by: Robert Thanaraj, Melody Chien, Ehtisham Zaidi, Mark Beyer, Mayank Talwar
SPA: By 2025, active metadata-assisted automated functions in the data fabric will reduce human effort by a third, while improving data utilization fourfold.
Description: Metadata is “data in context” — the “what,” “when,” “where,” “who” and “how” aspects of data. The data fabric listens, learns and acts on the metadata. It applies continuous analytics over existing, discoverable and inferenced metadata assets. By assembling and enriching the semantics of the underlying data, the data fabric generates alerts and recommendations that can be actioned by people and systems. It improves trust in, and use of, data in your organization as a result.
Why Trending:
- Practitioners are now able to experiment with the data fabric design:
- Advancements in technology, such as graph analytics, graph-enabled machine learning, automated data content analysis and profiling, have increased the level of automation that can be introduced to data management overall.
- Cloud capacity has enabled the expansion of data assets in terms of volume and variety, while at the same time offering significantly more complex resource allocation and utilization models in an on-demand, elastic environment.
- Metadata-driven data fabric has significant business value potential to:
- Reduce by 70% various data management tasks, including design, deployment and operations. The city of Turku in Finland found its innovation held back by gaps in its data. By integrating fragmented data assets, it was able to reuse data, reduce time to market by two-thirds and create a monetizable data fabric.
- Accelerate adoption through timely and trusted recommendations, enabling business experts to consume data with confidence. It also enables less-skilled citizen developers to become more versatile in the integration and modeling process.
- Optimize costs, because data fabric designs are built on the foundations of balancing collect-and-connect data management strategies. It does not require you to rip out and replace existing systems. Data stores and applications participate by providing metadata to the data fabric. Then, by analyzing the metadata across all participating systems, the data fabric provides insights on effective data design, delivery and utilization, thereby reducing costs.
Implications:
- Metadata analysis can expose hidden insights into business demand, metadata sharing can speed up integration and decision making, and metadata can reinvent governance and reduce risk. The existing data management systems, analytical platforms and systems of record are mere participating systems in the data fabric design — they feed metadata to the data fabric.
- By assembling and enriching the semantics of the underlying data, and by applying continuous analytics over metadata, the data fabric generates alerts and recommendations that can be actioned by both humans and systems. Such a high degree of automation drives effective data design, delivery and use, reduces human efforts and yields a high ROI.
- Semantic modeling skills will have a profound impact on several roles in an enterprise:
- Application developers building customer-facing applications increasingly use graph databases as the storage and execution back-end.
- Data architects designing knowledge-graph-based solutions for content management, personalization and semantic data interoperability.
- Data scientists performing higher-order exploration into connections and relationships between data points for better insights through graph visualizations, queries and algorithms.
- Database designers seeking alternative solutions to growing volumes of semistructured data.
Recommendations:
Data and analytics leaders should:
- Start monitoring how data is used, and leverage discovery tools to look for new and unexpected uses of data. This might imply new opportunity, or an emerging risk that warrants some attention.
- Target known opportunities and pain points by investing in experimentation and innovation with metadata. Assess ways to capture system logs, user logs, transaction logs and current data locations from your existing systems. There will be initial pushbacks from your application owners — create a shared benefit statement.
- Initiate a pilot effort to build a “limited” data fabric by identifying an intersection of data used, use cases, users and systems performing the data management, and the affected business units.
Changes Since Last Year
The data fabric trend progresses toward augmented data management principles by generating recommendations and alerts to its participating systems and individuals. We have published a few case studies on initiatives that have already begun to show real benefits, such as that of Montefiore (see Gartner Recommended Reading).
Always Share Data
Analysis by: Lydia Clougherty Jones, Eric Hunter, Malcolm Hawker, Jason Medd
SPA: By 2026, applying automated trust metrics across internal and external data ecosystems will replace most outside intermediaries, reducing data sharing risk by 50%.
Description:
Data sharing includes sharing data internally (between or among departments or across commonly owned and controlled parties such as subsidiaries or sister companies) and externally (between or among parties outside the ownership and control of your organization). The longstanding calculus that data sharing is not worth the risk of data misuse is obsolete. We observe that a risk of failure now inures to those organizations that do not share data automatically, or “always share data,” unless there is a vetted reason not to. Data sharing is a business-facing key performance indicator that an organization is achieving effective stakeholder engagement and providing enterprise value.2 D&A leaders at high-performing organizations promote data sharing or increase access to the right data aligned to the business case.3
Why Trending:
- Organizations need more and more singular types and complex combinations of data to feed their increasingly voracious use of data and analytics in order to drive business value. This can often be found in silos within the organization and increasingly, external to it.
- D&A leaders responsible for building a data-driven enterprise also require expansive data sharing, yet they struggle to identify and then locate the right data for their business case. They also face internal data hoarding, external data hijacking and privacy shaming.
- The global COVID-19 pandemic created urgency to share data in order to accelerate independent and interrelated public and commercial digital business value, as well as improvements to surrounding agility and resilience.
- Global data strategies highlight data sharing as a key priority for increasing government efficiency and generating public value. They also encourage industry data sharing with the purpose of producing market growth.
- External data has an increased level of relevance for D&A leaders in support of predictive models, as models trained exclusively with internal or first-party data have seen model drift due to phase shifts in customer behaviors.
- There is a lack of relevant available data for AI training, as well as sustainability and cost pressures for processing large amounts of AI training data.
- We observe increased demand for more robust predictive analytics generated from more diverse data sources to drive relevant, unique or otherwise unknowable insights and data-driven innovation.
Implications:
- D&A leaders know that data sharing is a key digital transformation capability, but they lack the “know how” to share data at scale and with trust.
- &A leaders will change their investment priorities to automate the identification and acquisition of the most relevant data, whether from big, small, personal, synthetic or yet to be created data, to match their business case.
- Remedying interoperability challenges and adopting a data fabric design will become a priority, contributing to environmental sustainability through data centralization, reuse and resharing, while meeting or exceeding stakeholder value and business outcomes, including composable business objectives.
- Automation and open-data programs ease the investment burden; machine-readable metadata provides automatic discovery of datasets and services, and open standards for metadata lower the barriers for their discoverability, reuse and resharing.
Recommendations:
Data and analytics leaders should:
- Collaborate across business and industry lines, promoting data sharing to create individual and aggregate stakeholder value. This will accelerate buy-in for increased budget authority and investment in data sharing.
- Establish trust in internal and external data, metadata, data sources, data sharing technologies, and downstream reuse and resharing ecosystems through automated mechanisms and metrics from active metadata insights, augmented data catalogs and automated data quality metrics
- Overcome data sharing hurdles by focusing on, and then quantifying, the risk of not sharing data, including business failure.
- Consider adopting data fabric design to enable a single architecture for data sharing across heterogeneous internal and external data sources.
Changes Since Last Year
As part of the Top Data and Analytics Trends, 2021, we introduced a trend called “D&A as a Core Business Function,” which noted that smarter data sharing increasingly plays a key business role. Smarter data sharing has evolved from “don’t share data unless” to “share data always,” unless there is a well vetted reason not to.
Augment People and Decisions
5.Context-Enriched Analysis
Analysis by: Afraz Jaffri, David Pidsley, Sumit Pal
SPA: By 2025, context-driven analytics and AI models will replace 60% of existing models built on traditional data.
Description: Contextual data originates in multiple sources, including image, text, audio, log, sensor and associated metadata. It is needed to build a richer knowledge-based-model of business entities and relationships. Holding this information in a graph structure enables deeper analysis utilizing the relationships between data points as much as the data points themselves. Graph analytics exploits this structure to identify and create further context based on similarities, constraints, paths and communities. This forms the basis for the utilization of active metadata that drive data fabrics, personalization of automated insights and data stories, powerful feature sets for machine learning models and a wider field of vision for improved decision making.
Why Trending:
- In an ever-changing world full of uncertainty, context is a necessity, not just “nice to have.” Utilizing contextual data enables building a richer knowledge-based model of business entities and relationships to meet the challenge of accelerated internal and external business dynamics.
- Analytics and data science techniques are applied to data where the properties of each data item are taken in isolation. Adding contextual information gathered from the relationships between entities uncovers new patterns of behavior that reveal facts and better insights and generate more predictive power.
- Advances in techniques for audio, image and video recognition, natural language processing (NLP) and speech-to-text are increasing the accessibility of a wider set of data for use in advanced analytics and decision intelligence.
- Augmented analytics tools capture contextual information about the user, environment, datasets and analytics outputs to generate automated insights and data stories, as well as personalized news feeds in social-media-style timelines.
- There is an overload of analytics content and insights. Preserving context enables targeted insights to be provided on demand, enabling consumers to concentrate on core insights that require exploration and action.
- Graph data models naturally accommodate the context of data through relationships and graph structural properties. Data is represented in a form that is easily understood by business users and domain experts.
- Increasing the usage of data fabrics within data management exposes metadata that captures the context and usage of data assets.
Implications:
- Capturing, storing and utilizing contextual data demands capabilities and skills in building data pipelines, X analytics techniques and AI cloud services that can process different data types and provisioning use cases that include this data for end-user applications.
- Automated insight generation using greater contextual insights enables analysts who produce BI reports and dashboards to focus on the more complex needs of business users. It also enables them to answer critical business questions and improve insights that are used in decision making. This will shift the traditional skills required for analysts from technology-centric to business-focused, and give rise to the importance of business translators who can identify decision-making points and priorities.
- Identifying and accessing contextual data that lies outside an organization requires the enablement of data sharing services and exchanges and the creation of policies that promote and regulate internal and external data sharing.
Recommendations:
Data and analytics leaders should:
- Identify multiple data sources and formats that can be used to augment data that is already used for analytics, data science and AI, and enable the extraction of key entities and relationships.
- Utilize the power and flexibility of graph data models to connect data that exists across the organization into graph structures that can be fed into downstream data management, analytics and AI processes.
- Augment existing machine learning models with features extracted from graph representation using X analytics, graph analytics and data science techniques.
- Identify key personas that can benefit from shorter time to insight and assess analytics and BI platforms that deliver contextual insights.
Changes Since Last Year
The “Context-Enriched Analytics” trend builds on previous top D&A trends on graph technologies as a key enabler for data and analytics innovation. This year, the focus is on bridging the gap between structured, unstructured and metadata with graphs and the utilization of graph analytics for an increasing number of business use cases, both stand-alone and within packaged applications.
Business-Composed D&A
Analysis by: Julian Sun, Yefim Natis, Shaurya Rana
SPA: By 2025, 50% of embedded analytics content will be developed by business users leveraging a low-code/no-code modular assembly experience.
Description: Business-composed D&A refers to the active building of D&A capabilities by business technologists using low-code composition technology along with collaboration tools to assemble the modular and rich capabilities of organization-level value streams. “Business-composed D&A” shifts application development power to the business users or business technologists, enabling them to craft business-driven D&A capabilities collaboratively.
Why Trending:
- Historically, building custom analytics applications or embedding data and analytics in a workflow or business process required traditional IT-led embedded D&A capabilities, such as APIs and developer software development kits (SDKs). This is slow and expensive, and requires a high level of technical skills.
- Per a recent Gartner Building Digital Platforms survey, D&A capabilities are the most common and fastest growing capabilities to be integrated with digital business platforms within an enterprise, enabling agile decision-making.4
- IT-led embedded D&A is time consuming to deploy and lacks business impact. Business users expect to close the loop of D&A and build the last mile of it by turning insights into business actions.
- Fully decentralized D&A causes siloed insights that achieve local goals at the expense of achieving organizationwide goals.
Implications:
- D&A leaders will increasingly foster and build collaborative D&A experiences and processes with cross-functional teams.
- The adoption of business-composed D&A, facilitated by fusion teams, will bridge the insights with actions and decrease business outcome challenges, which wasn’t possible with fragmented embedded D&A deployed by IT.
- Organizations will develop a business-driven data and analytics architecture by composing packaged business capabilities (PBCs) aligned with value streams.
- The D&A catalog will evolve as business technologists create PBCs as reusable D&A assets for others to easily compose and recompose, beyond traditional datasets, reports or dashboards that need slow and expensive integration.
- The expanded capabilities would bring more complex governance concerns for both D&A and application development, requiring more Ops practices to manage production. A more comprehensive Citizen-X program might emerge within the community to share skills, best practices and governance rules and processes
Recommendations:
Data and analytics leaders should:
- Prepare for growth in the number of citizen developers and business technologists using low-code/no-code technology to broaden their use of self-service analytics by providing training in basic software development practices.
- Evaluate your existing analytics and data science tools and innovative startups offering low-code enabled composition experience to build rich analytics-infused applications within the business process.
- Foster collaboration and build more PBCs by creating cross-functional fusion teams of both centralized and decentralized D&A users that are aligned with the corporate-level value streams.
Changes Since Last Year
As part of the Top Data and Analytics Trends, 2021, we introduced a trend called “Composable Data and Analytics.” Business-composed D&A builds on previous trends in low-code-enabled composition from modular ABI and DSML capabilities as a key enabler for D&A applications. This year, the focus is on the people side, which is shifting from IT to business, composing D&A capabilities that are aligned with value stream management and powered by automation workflow.
Gartner Recommended ReadingBusiness-Composed D&A Executive Brief
Decision-Centric D&A
Analysis by: Gareth Herschel, W. Roy Schulte
SPA: By 2023, more than 33% of large organizations will have analysts practicing decision intelligence, including decision modeling.
Description: Careful consideration of how decisions should be made (the discipline of decision intelligence — see Note 1) is causing organizations to rethink their investments in D&A capabilities. The application of D&A to organizational decisions requires contextual understanding that this application can take several forms, such as decision support, augmentation or automation. The initial focus of decision-centric D&A is on the insight that decision makers need to govern D&A investments, rather than on the available data or the analysis that can be carried out. This is a sign of increasing maturity of D&A strategies and organizations’ ability to reengineer decisions to drive business outcomes.
Why Trending:
- Access to a growing variety of data, combined with a continually increasing variety of analytical techniques, means organizations have an overwhelming permutation of decisions to make about the analysis they could carry out. Relevance to organizational decisions is an easily identifiable filter for these options.
- Forty-seven percent of organizations believe that the decisions they face will be more complex, increasing demand for connected, continuous and contextual D&A and explainable decision processes.5 Decisions are increasingly interconnected and based on continuously changing data about the business context. Without D&A, the growing complexity of decisions is impossible for organizations to manage.
- Only 58% of organizations report that they consistently and formally define decision ownership. Decisions are made in several different ways. Personal decision making based on experience, training or intuition is the most common, but also the most difficult to combine with D&A.
- Improving the quality of decisions is not a new goal, but organizations are not good at building the systems and processes needed for it to be possible. This is the intent behind growing interest in the discipline of decision intelligence.
Implications:
- Decision-centricity requires a deep understanding of human psychology and behavior. Embedding D&A into collaborative or individual decisions is most likely to be successful when supported by NLP or data visualization capabilities.
- Decision-centric D&A needs to be aligned with enterprise architecture and technical development teams. D&A can be embedded into decisions in several different ways. For example, embedding analytic decision models (e.g., machine learning) into business processes requires technical integration with business applications, as described in the Business-Composed D&A trend.
- Time and complexity are the main criteria for determining which decisions are most suited to automation with D&A. Simple problems that can be confronted quickly are the most appropriate candidates for automation, while highly complex problems that need time to solve require decision support and augmentation, rather than automation.
Recommendations:
Data and analytics leaders should:
- Approach D&A projects by considering which decisions you are seeking to influence, rather than what data you possess or how to analyze it.
- Create new decision-making habits by training decision makers to apply best practices, such as critical thinking, trade-off analysis, recognizing bias and listening to opposing views.
- Consider creating a role for decision engineers by hiring or upskilling experts who can work with decision makers to identify opportunities that would benefit from the rigor of decision intelligence practices.
Changes Since Last Year
The application of D&A to decisions has often been an add-on; we have the data and analysis, then we think about how to accelerate its deployment to real time, or contextualize it for a collaborative decision. We are seeing a switch to a decision-first approach: using decision intelligence disciplines to design the best decision, and then delivering the required inputs (possibly including D&A). This flip from D&A-driven decisions to decision-driven D&A is subtle, but fundamental.
Gartner Recommended ReadingDecision-Centric D&A Executive Brief
Skills and Literacy Shortfall
Analysis by: Alan D. Duncan, Eric Hunter, Jorgen Heizenberg, Peter Krensky, Joe Maguire, Sally Parker
SPA: Through 2025, the majority of CDOs will have failed to foster the necessary data literacy within the workforce to achieve their stated strategic data-driven business goals.
Description: The 2021 Gartner CDO Survey shows that organizations that deal with the human elements of D&A are more successful than organizations that only consider technology. With a human focus, the mission of D&A is to foster broader data literacy and digital learning, rather than simply delivering core platforms, datasets and tools. Organizations (and society) will need to learn how to learn, and they will need a lot of help to do so.
Why Trending:
- Virtual workplaces and the heightened competition for D&A talent have exposed weaknesses in organizations lacking content-neutral and enterprise-specific data literacy strategies
- Increasing talent acquisition efforts require D&A leaders to take an increasingly agile approach to data literacy and upskilling investments. New hires are increasingly dependent on this investment to accelerate their contribution of critical delivery and support roles across the organization.
- Cultural aversion to change is a prevailing and recurring roadblock to the success of D&A programs. Expecting data literacy in the workforce as a default is a false presumption.
- Momentum is being driven by vendors addressing customer enablement, advocacy and adoption as part of their go-to-market (e.g., Tableau, Qlik, Alteryx, Collibra). More and more providers are offering learning solutions for D&A. These solutions include not just technical learning, but also the “soft” skills for curiosity, critical thinking and communication (e.g., Pluralsight, Skillsoft, The Center for Applied Data Science, Udacity).
Implications:
- CDOs must distinguish transferrable/domain-neutral skills from domain-specific knowledge and experience of the organization’s own processes. Those who are carrying out succession planning are not referring to it as such.
- Data literacy and technical upskilling strategies will increasingly embrace the concept of “just-in-time” channels and options for the provision of key skills and concepts as talent becomes increasingly fluid (new hires, contingent workers, etc.).
- D&A leaders will reduce their selectivity criteria for new hires as competition for talent intensifies, but will compensate for this through diverse upskilling strategies to complement new hires’ potential.
- The cost of investing in data literacy and employee upskilling will lead employers to insert “claw-back” or “payback” clauses into contracts with new hires in order to recover costs in the event that an employee departs the organization.
- Traditional succession planning discussions will incorporate data literacy performance.
- Platform and technology selections will increasingly prioritize capabilities for prebuilt content and customer literacy training and assessment as a part of key workflows and platform usage
Recommendations:
Data and analytics leaders should:
- Take the lead in creating a narrative that sets a strong vision for the desired end state and business outcomes, particularly with respect to innovation opportunities and use cases that have not been identified by others.
- Work with line-of-business leaders to trace measurable business outcomes back to supporting analytics output and underlying data.
- Monitor the effects of improved data literacy among the workforce by using data literacy assessments and measuring associated improvements to data-driven business outcomes.
- Distinguish between competition for people who already have D&A skills and educating/training those who currently do not. Develop a multispeed approach: different approaches for the most enthusiastic, the slower on the uptake and the outright resisters.
- Collaborate with HR and business leaders to run data literacy pilot projects in business areas where there is a high likelihood of achieving measurable business outcomes. Use quick wins to build momentum and incentivize staff to use data in their interactions. Using workers’ stated pain points as stimuli can get them to identify the changes necessary to address those pain points.
- Go beyond vendor product training to focus on people’s roles. Use a mix of training delivery methods by considering the times, locations, roles and skills differences to improve overall learning effectiveness and experiences for new analytics capabilities.
- Formulate a succession plan for key D&A roles both inside and outside IT. Budget for 25% turnover. Incentivize teaching colleagues what you know, rather than engaging in self-interest-based protectionism.
- Coordinate data literacy initiatives with overall data governance programs.
Changes Since Last Year
The limited pool of D&A talent has become acute, with competition for skilled staff escalating amid the “great resignation.” There is an increasing recognition that this cannot be solved by reorganization, but by development of the workforce.
Institutionalizing Trust
Connected Governance
Analysis by: Andrew White, Saul Judah, Ted Friedman, Guido De Simoni
SPA: By 2026, 20% of high-performing organizations will use connected governance to scale and execute on their digital ambitions.
Description: Connected governance is a framework for establishing a virtual D&A governance layer across organizations, business functions and geographies, to achieve cross-enterprise governance outcomes. Connected governance provides a means to connect disparate governance efforts, including D&A governance, across different organizations, both physical and virtual, as well as geographies. This approach is similar to virtualized governance efforts, but with at least one difference: virtualized governance organizations are new and discrete teams with an independent objective. Connected governance does not exactly create a new team, but helps connect those that are already established. It also provides a means to align and link efforts so that complex outcomes shared by the organizations and geographies can be achieved.
Why Trending:
- As the anticipated transformation across most industries materializes, organizations that are unable to address cross-organizational, cross-geographical challenges in a flexible and agile way will be left vulnerable to competition or fail to meet political pressures in the public sector.
- The accelerating pace and complexity of digitalization is putting pressure on senior leaders across multiple business functions to respond to business and mission demands. Similarly, governments are consolidating organizations that are often driven by a need to provide more effective and comprehensive services or regulatory regimes.
- Organizations need effective governance at all levels that not only addresses their existing operational challenges, but is also flexible, scalable and highly responsive to changing market dynamics and strategic organizational challenges.
Implications:
- New demands for cross-enterprise responses and initiatives will require accountability from multiple executive leaders. This will require knowledge of local governance accountability and how decision rights are implemented.
- Connected governance will be a mechanism to bring together and coordinate diverse business areas to avoid time-consuming, watered down or even conflicted decisions and mediocre action and performance.
Recommendations:
Data and analytics leaders should:
- Identify the business scenarios and outcomes that are most difficult to govern in the ecosystem because of, for example, geographic and organizational diversity, complexity and autonomy.
- Build trust in data-driven decision making by setting up connected governance of all data assets in the enterprise.
- Consider how connected governance can help to address these challenges more effectively.
AI Risk Management
Analysis by: Avivah Litan, Bart Willemsen, Svetlana Sicular, Sumit Agarwal, Farhan Choudhary
SPA: By 2026 organizations that develop trustworthy purpose-driven AI will see over 75% of AI innovations succeed, compared to 40% among those that don’t.
Description: The speed of AI innovation is increasing pressure to keep pace while keeping businesses operating, tempting organizations to cut corners on AI trust, risk and security management (TRiSM). This will lead to potentially harmful outcomes. Organizations must spend time and resources now on supporting AI TRiSM. Those that do will see improved AI outcomes in terms of adoption, achieved business goals and both internal and external user acceptance.
Why Trending:
- Almost half of AI models developed by experienced organizations do not make it into production, and users cite security and privacy as a primary reason for this negative outcome (see Survey Analysis: Moving AI Projects From Prototype to Production).
- AI is becoming more pervasive, yet most organizations cannot interpret or explain what their models are doing. They struggle to ensure trust and transparency in their models.
- Most organizations developing AI:
- Are not precise on what they want to achieve when developing models; and often expand the scope when they see what is possible, making operationalization almost impossible.
- Have no processes, tools or measurements to govern and manage model trust, risk and security.
- Tend to gather AI training data without deliberate data selection goals. Data is often biased and inadequate for training models.
- Are driven mainly by regulatory compliance when it comes to model governance. However, compliance does not necessarily lead to trustworthy models.
- The increasing dependence on, and scale of, AI escalates the impact of misperforming AI models with severely negative consequences.
- AI regulations are proliferating across the globe, mandating certain auditable practices that ensure trust, transparency and consumer protection.
- Organizations are not prepared to manage the risks of fast-moving AI innovation and are inclined to cut corners around model governance.
- Organizations that properly and continuously govern their models have much improved AI model outcomes.
Implications:
- Increased focus on AI TRiSM will lead to:
- Controlled and stable implementation and operationalization of AI models.
- Far fewer AI failures, including incomplete AI projects, and a reduction in unintended or negative outcomes.
- Positive societal impact if AI is trustworthy and has a legitimate purpose, leading to reduced discrimination, more fairness and protection of human autonomy and privacy.
- In contrast, AI models that are not trustworthy or transparent and do not have a legitimate purpose can eventually lead to severely negative consequences.
Recommendations:
Data and analytics leaders should:
- Engage all internal and external stakeholders from the start to foster transparency and trust.
- Define in full the primary purpose of AI, and use AI engineering and governance to develop metrics to continuously assess the intended impacts of the models.
- Account for the parameters of the entire ecosystem in which change is to be made, in addition to focusing on the primary purpose of the AI model itself.
- Make sure the right amount of the right data is available to train the model to achieve balance, improve accuracy and mitigate bias.
- Consider tools such as synthetic data to generate more useful training data when trusted sources do not suffice.
- Reach beyond compliance by involving the ethics board or other interested parties in creating purposeful unbiased models.
Changes Since Last Year
The rising interest in privacy-enhancing computation (PEC) techniques (see Top Strategic Technology Trends for 2022: Privacy-Enhancing Computation for an introduction) is a promising support factor for this trend. Particularly, the expedient adoption of generative AI-based synthetic data (see Hype Cycle for Privacy, 2021) will aid in providing non-identifiable, privacy risk-free data to train AI models on.
Vendor and Region Ecosystems
Analysis by: Rita Sallam, Julian Sun, Adam Ronthal, Carlie Idoine, Sumit Agarwal
SPA: By 2024, 50% of new system deployments in the cloud will be based on a cohesive cloud data ecosystem, rather than on manually integrated point solutions.
Description: D&A ecosystems consisting of increasingly comprehensive D&A capabilities are growing in availability and capabilities. Both CSPs and ISVs are offering less costly, less complex architecture requiring little or no integration of products and cloud services. At the same time, many global organizations are assessing the implications of building parallel regional D&A ecosystems to comply with local regulation. Data sovereignty, financial governance and orchestration across components are key ecosystem considerations.
Why Trending:
- CSPs increasingly view compute- and data-intensive D&A workloads as an attractive vehicle for overall cloud services revenue growth.
- Most CSPs and ISVs are using varying degrees of workflow integration with the promise of composability, AI-driven automation and capabilities convergence across D&A categories. The aim is to deliver lower cost, shorter time to deployment and agility as a key competitive strategy to respond to buyer needs.
- Cloud data ecosystems redefine the best-of-breed and best-fit engineering versus suite debate. By resolving disparate and increasingly siloed D&A, they can alleviate some of the challenges of multicloud, intercloud and hybrid deployments. Unlike with the on-premises stacks of the past, with cloud ecosystems, the vendor lock-in challenges of using a single vendor may be outweighed by the potential benefits.
- Regional ecosystem pull is accelerated by regional data security laws, which intensify data gravity and require better regional composability between vendors. This pushes global organizations to consider migrating and duplicating some or all parts of their D&A stack within specific regions.
Implications:
- Over the past several years, many once-separate D&A software markets have been converging, driven by increased adoption in the cloud and buyer demands for faster deployments and lower cost. Greater workflow integration between previously separate products has enabled this trend.
- Although there is a need for third-party tools to fill in elements of the D&A ecosystem, this should diminish over time via innovation in the D&A ecosystem and through acquisitions and cross-category organic development.
- Unlike stack-centric market swings in the past, innovation in the market will thrive, but not just to the benefit of small, innovative vendors. CSPs might not be the main drivers of innovation; because of cloud agility and composability, they are fast followers and acquirers of innovation introduced into the market by smaller, nimble vendors.
- Most large organizations will, by design or by default, have to manage a multicloud and multivendor strategy. However, because it requires additional skills, time and cost to build and manage applications built from multiple vendor product components, multicloud and intercloud tooling will evolve to make it far easier to deal with.
- Regions not able to create or sustain their own D&A ecosystems will have no choice but to leverage capabilities created in other regions (hyperscalers) and resort to legislation and regulation to maintain some level of control and sovereignty.
Recommendations:
Data and analytics leaders should:
- Begin by evaluating the extensibility and broader ecosystem offerings for vendor solutions already in use and consider aligning with them.
- Lower costs and improve organizational resilience by evaluating D&A vendors on the availability, strength and — importantly — the integration of their broader ecosystem of D&A capabilities.
- Evaluate not only extensibility, cost, agility and speed, but also coherence and consistent ease of use and workflow across the CSP-based D&A ecosystem.
- Reevaluate policies favoring a best-of-breed or best-fit strategy for end-to-end D&A capabilities in the new cloud world by weighing the benefits of a single vendor ecosystem in terms of cost, agility and speed.
- Consider regional composability while assessing the global deployment of D&A solutions by understanding their technology partnerships and integration with regional D&A ecosystems.
Originally published at https://www.gartner.com
About the authors
By Rita Sallam, Ted Friedman, Erick Brethenoux, Donald Feinberg, Soyeb Barot, Lydia Clougherty Jones, Malcolm Hawker, Eric Hunter, Jason Medd, Robert Thanaraj, Melody Chien, Mark Beyer, Ehtisham Zaidi, Mayank Talwar, Pieter den Hamer, W. Roy Schulte, Paul DeBeasi, David Pidsley, Sumit Pal, Joe Maguire, Yefim Natis, Shaurya Rana, Guido De Simoni, Afraz Jaffri, Alan D. Duncan, Julian Sun, Gareth Herschel, Avivah Litan, Bart Willemsen, Svetlana Sicular, Farhan Choudhary, Sumit Agarwal, Mike Fang, Adam Ronthal, Andrew White, Carlie Idoine, Jorgen Heizenberg, Peter Krensky, Sally Parker
