Enterprise intelligence platforms: accelerating the path from data to decision
December 4 2020
by Matt Aslett
At this time a year ago, we described the emergence of a new product category in the analytics space that combines functionality from what have previously been considered three distinct product categories: data integration, data storage and processing, and analytics. While it is still early stages for what we termed enterprise intelligence platforms, ongoing vendor activity and a variety of converging market trends point to growing interest in the concept.
The 451 Take
We believe that enterprises need to invest in data and analytic products, services and functionality that support business agility and accelerate the path from data to decision to remain competitive amid fast-changing socioeconomic conditions and increased data-driven decision-making. While data ingestion and integration, data storage and processing, and data visualization and analysis remain the three fundamental steps on the path from data to decision, there are potential opportunities to deliver enhanced efficiency and reduce data friction. Modern data projects require approaches that are not burdened by traditional assumptions about these being served by stand-alone products and services.
Acceleration of business insight relies on a combination of capabilities that deliver more agile approaches to data ingestion and integration: avoiding the limitations of predefined schema and data models; enabling multiple analytic projects to be performed against data (both structured and semi-structured or unstructured) in a single environment; incorporating the results of queries performed on data in other locations; and accelerating multiple BI and analytic tools (including advanced analytics and machine learning functionality) by multiple users for multiple purposes.
Data and analytics have always had a critical role to play in business decision-making but the importance of data was highlighted in 2020 as COVID-19 provided senior executives and decision-makers with a vivid reminder of the importance of agile, data-driven decision-making in enabling enterprises to respond to fast-changing market conditions and competitive threats. The pandemic, and the way in which enterprises have responded to it, appears to be exacerbating the divide between the 'haves' and 'have-nots' when it comes to data. While some enterprises are retrenching, others have seized on the outbreak as an opportunity to accelerate transformational change.
Technology priorities driven by the C-suite amid coronavirus include improving efficiency and cost-cutting, understanding evolving business processes, and accelerating digital transformation. All of these rely on data, along with agility and pragmatism, to make rapid decisions as socioeconomic conditions evolve. In particular, the new pace of business demands a faster path from data to decision to ensure that data and analytic plans are aligned with – and central to – transformation initiatives. This path from data to decision involves multiple steps, with the most significant being data ingestion and integration, data storage and processing, and data visualization and analysis.
The Path from Data to Decision
Enter the enterprise intelligence platform
To propel true transformational change, business teams need to reduce the time needed to investigate, analyze and take action on business data. Doing so involves rethinking traditional assumptions about data management and analytic roles and technologies.
A year ago, we detailed how a new category of products was emerging in the analytics space that combined data integration, data storage and processing, and analytics functionality into a single enterprise intelligence platform (EIP), noting that almost three-quarters of respondents to 451 Research's Data & Analytics, Data Management & Analytics 2019 survey agreed that their organization would be interested in adopting an EIP.
Our ongoing research demonstrates that there are multiple trends spurring this level of interest. The first of these is the desire to enhance data-driven decision making, and invest in new products to accelerate that. Respondents to our Data & Analytics, Data Management & Analytics 2020 survey told us that the two most popular steps taken to improve data culture are investment in new data management products and services (44% of respondents), and investment in new analytic products and services (40%).
Despite enormous innovation in the data and analytics sector in recent years, for many organizations there remains a gap between what is theoretically possible with the latest data and analytics technology and a practical, meaningful impact on business decision-making. One of the reasons for this is that in many companies, data and analytic projects remain built around data pipelines and analytic processes that assume the use of stand-alone data integration, processing and analytic products – especially ETL data integration and centralized data warehousing – as steps on the path from data to decision.
The need to integrate with existing data architecture was cited as a barrier organizations face in attempting to be more data-driven by 34% of respondents to our Data & Analytics, Data Management & Analytics 2020 survey, making it the second-most-significant barrier, behind only limited budget.
While data integration, data processing and analytics remain the core steps on the path from data to decision, enterprises are increasingly seeking a more cohesive approach that enables agile self-service and advanced approaches to analyze large data volumes, as well as collaboration across departments and functions to facilitate shared discovery and momentum.
Converging trends drive convergence
In addition to the overall shift toward data-driven decision-making, there are a variety of technological trends that are driving demand for a more direct path from data to decision. While each of these is individually important, they are also themselves converging.
Load data first, ask questions later. The prevailing approach for decades has been to transform data to match the analytical requirements of the business by using a dedicated staging zone and stand-alone ETL tools to apply predefined schema and data models prior to loading the transformed data into the target data warehouse. In recent years, best practices have swung in the other direction toward the benefits of extracting and loading data into the target, automatically creating data models based on the retained source schema, and then using the processing power of the data storage and processing platform to apply the required schema at query time. The primary benefit of this approach is agility – the avoidance of upfront development and testing, and data model design can potentially lower the time taken to create an environment ready for analysis, while it also results in an environment that is flexible to changing data sources and queries.
All the data, all the time. More than one-quarter (27%) of respondents to our Data & Analytics, Data Management & Analytics 2020 survey cited the number of data silo departments as one of the biggest analytic challenges faced by their organization, while the same proportion noted that they had invested in programs to reduce data silos and data duplication to enhance data culture. The data lake minimizes data migration and movement complexity by enabling multiple analytic projects to be performed against data from multiple sources in a single environment, particularly in enabling analytic queries to be applied to both structured and semi-structured or unstructured data. Ensuring that business value is generated from data lake projects can be easier said than done, however. One approach to alleviate barriers to value is to automate the identification and tagging of data as it is ingested into the data lake. While the data may retain its original schema, identifying and mapping relationships between data based on the source metadata can accelerate the analysis of that data.
Take the query to the data. While the data lake architecture is better suited to evolving data and use cases, advances in data virtualization and federated query functionality have also allowed enterprises to embrace the advantages of data being generated and stored in multiple locations, rather than forcing it into a centralized data warehouse or data lake. This is especially important as a greater volume of enterprises data is being generated in the cloud and stored deploying inexpensive cloud storage services. Many enterprises are increasingly taking the query to the data: adopting distributed query engines and an abstracted data architecture, as well as data virtualization or query federation capabilities to analyze data in, and across, multiple locations without having to first move and transform it. In addition to enabling enterprises to process data persisted natively in cloud storage, enterprise integration platforms should enable the incorporation of the results of queries performed on data in other locations, if required.
Reduce data friction. Allowing access to data in different locations for different purposes and use cases can reduce the need for data duplication and related data quality issues. This also means that people in different roles (e.g., data analytics, data science, data governance) have access to the same data. This reduces the potential for disagreements about data, but also lowers the potential for friction between data consumers (such as data analysts, developers and senior decision-makers) and data operators (e.g., data management and IT professionals). Nearly one-third (29%) of respondents to our Data & Analytics, Data Management & Analytics 2020 survey reported lack of collaboration between departments as one of the biggest analytic challenges faced by their organization, while almost one in five (18%) indicated that they had fostered collaboration between data owners, providers, operators and consumers to improve data culture. EIPs should enable multiple users to have access to – and make use of – the same data for multiple purposes.
Different tools for different roles. To support the use of the same data for multiple purposes, EIPs should facilitate self-service access to data. This means enabling business analysts, data analysts and data scientists alike to access and prepare data (where appropriate) without having to wait for new data marts or reporting environments to be created for them, while also recognizing and allowing for the fact that different users will be taking advantage of different BI and analytic tools for different purposes. Senior decision-makers may be accessing recommendations and alerts provided by automated decision intelligence, while business and data analysts may be employing more advanced self-service analytic and visualization tools, and the highest-skilled analysts and data scientists will likely be taking advantage of data science tools and functionality. While we increasingly see analytics and visualization functionality being bundled with data ingestion and data-processing functionality in EIPs, these environments should also provide integration with analytic tools from multiple providers, serving multiple roles, and accelerate analytics insight regardless of the tools and interfaces deployed by different groups of users.
Embedding advanced analytics. With enterprises looking to be more analytics-driven, and with a rise in the adoption of machine learning to help spur business decisions, many organizations have implemented separate systems for data science activities. While this functionality is primarily used by data scientists to experiment and develop new analytic applications and business processes, it is also increasingly being employed by business and data analysts via analytic tools and services that mask the complexity of more advanced analytic approaches. As such, we note many organizations warming to the idea of embedding machine learning (ML) capabilities within their core data platforms as a complement to – or perhaps instead of – using stand-alone AI/ML-based systems or cloud services. Enterprise intelligence platforms need to support this innovation by providing support for advanced analytics functionality without the need for users to spin up separate environments for data science and advanced analytics.