Overwhelming amounts of data are used by businesses to optimise operations and transactions on a daily basis. Often, large volumes of unstructured data needs to be extracted from a number of disparate systems before it is of any value to a business. This process of gathering, structuring and integrating data is commonly referred to as Extract, Transform and Load (ETL); however, for massive volumes of data from many different sources, a classic batch ETL process is not sufficient. That’s where SAP Data Intelligence comes in.

SAP Data Intelligence equips you with data integration, data innovation, and data compliance.

WHAT IS SAP DATA INTELLIGENCE?

SAP Data Intelligence is a comprehensive data management solution that connects, discovers, enriches, and orchestrates disjointed data assets into actionable business insights at enterprise scale.

It enables the creation of data warehouses from heterogeneous enterprise data, simplifies the management of IoT data streams, and facilitates scalable machine learning. SAP Data Intelligence allows you to leverage your business applications to become an intelligent enterprise and provides a holistic, unified way to manage, integrate, and process all of your enterprise data.

With SAP Data Intelligence you can:

• Discover and connect to any data, anywhere, anytime from a single enterprise data fabric;
• Transform and augment data across complex data types and curate a robust, searchable data catalog;
• Implement intelligent data processes by orchestrating complex data flows enriched with scalable, repeatable, production-grade machine learning pipelines;
• Integrate any data, from any source;
• Reuse any engine and orchestrate any SAP or third-party data processing engine;
• Democratise intelligence by making machine learning easier to implement, scale, operationalise, and govern;
• Ensure data quality by discovering, preparing, and governing data assets in the same tool;
• Deploy on any mix of hyperscalers, whether hybrid or on premise.

WHO BENEFITS FROM SAP DATA INTELLIGENCE?

• Data stewards can use an organisation’s data governance processes to ensure fitness of data elements – both for content and metadata.
• Chief data officers can better ensure enterprise-wide governance and use of information as an asset through data processing, analysis, data mining, information trading, and other means.
• Data warehouse administrators can support the development and maintenance of data warehousing and data mart systems through the entire data development lifecycle, including data profiling, design and development, testing, and support.
• Business analysts can help guide businesses in improving processes, products, services, and software through data analysis, helping bridge the gap between IT and the business to improve efficiency. They can also use data analytics to assess processes, determine requirements, and deliver data-driven recommendations and reports to executives and stakeholders.
• CIOs can help their organisations manage requirements they may not understand as well as deploy and scale data science with confidence and at a lower cost.

IS SAP DATA INTELLIGENCE ANOTHER ETL TOOL?

SAP Data Intelligence goes beyond classic batch ETL or real-time streaming. It scales these functions and leverages these technologies across the enterprise, which cannot be accomplished with traditional ETL engines. It also focuses on the integration of new technologies (for example, Docker and Kubernetes) operating in distributed landscapes. The main paradigm is to bring the logic to where the data resides and leverage the computing power. ETL cannot handle the volume of data and the constantly increasing number of different systems, sources, and consumption points in these complex data landscapes. It will continue to serve its current function while SAP Data Intelligence allows scalability across these modern landscapes.

By supporting specific AI-related expert work and at the same time linking all of it together in an integrated manner, Data Intelligence allows organisations to move from a fractured approach – often dominated by data sprawl and disparate tools – to a continuously unified operation – one where machine learning models can be continuously repurposed and reused, accelerating efficiency like never before.

BEFORE SAP DATA INTELLIGENCE

The architecture of traditional enterprise resource planning (ERP) platforms is designed for the management of structured data. Structured data is data that is already modelled and stored in a pre-defined format, for example, in rows and columns in a database. Unstructured data is a combination of different data types in their native format, for example, audio/video files or document collections like emails and invoices. Structured data is commonly stored in data warehouses and unstructured data is stored in data lakes.

Unstructured data haphazardly scattered across a business IT landscape can be difficult to find, never mind analyse. Traditional data management software is simply unable to efficiently replicate so much data from so many sources. What businesses need is intelligent technologies like automation and machine learning to be able to more efficiently integrate data for easy access, analysis and insight.

However, the question remains: how do we achieve this?

THE FUTURE WITH SAP DATA INTELLIGENCE

SAP Data Intelligence enables a business’s IT department to manage unstructured data through metadata governance and data quality processing, along with discovery, lineage, preparation, transformation, integration and finally orchestration.

DATA ORCHESTRATION = DATA INTEGRATION + DATA PROCESSING (ML) + AUTOMATION

When you bring comprehensive access to all data together (Data Integration), with the application of data processing engines like ML (Machine Learning), and connect the results to business processes and workflows (Automation), you get true data orchestration.

SAP Data Intelligence doesn’t stop there though. It is able to leverage metadata assets to make sorting and searching for data that much easier. Metadata (data about data) is the summary and description of your data that is used to classify, organize, label and understand data – the key to extracting maximum value out of your data.

With SAP Data Intelligence, metadata assets from related source systems can be made available in the Metadata Explorer catalog for labeling, searching, and data lineage functionalities.

Finally, through machine learning, SAP Data Intelligence is able to:

• Automatically classify extracted metadata assets based on text analysis, image processing, or natural language processing.
• Automate the mapping of predefined entities (like sales orders or customer industries) as well as establish and maintain the relationships of extracted entities originating from various connected systems.

Looking ahead, SAP developers are working on “self-learning metadata governance” which will include a combination of metadata management and machine learning enriched with business semantics, which are extracted from SAP business systems and applications. It represents the metadata management of the future and will enhance the productivity of involved subject matter experts.