Webinar : Contextualized Data : September 14, 2022 09:00AM Pacific Time
Register
A Unified Analytics Framework is an architecture designed to centralize and standardize the process of transforming raw data into actionable information distributed across an organization.
An information model defines the key data an organization focuses on and establishes governance for how that data is created, managed, and used. It includes the rules for transforming data into useful information as well as the rules for distributing it. Utilizing such a model provides many benefits, including:
The Information Model in the UAF is designed to be uncoupled from any specific data source or application, allowing for flexibility and scalability. This ensures that data from multiple sources can be seamlessly integrated, transformed, and governed without dependencies, providing a unified and consistent view across the organization.
By abstracting data from various sources, the UAF creates a cohesive and unified view. This abstraction layer simplifies data management by concealing the complexities of underlying systems, enabling users to interact with data through a standardized interface, and ensuring consistency and reliability across the entire organization.
The UAF offers a structured framework for data management giving organizations the freedom to follow existing standards like ISA 95 or build custom solutions that work best for their unique requirements. This balance between structure and flexibility enables efficient data operations and supports continuous improvement and innovation.
Leveraging predefined templates, the UAF streamlines data management and governance. This templatized approach allows for quick deployment of consistent data models and standardized rules and transformations. It also enables the extension of previously completed work, creating base templates that can be further divided into various template subsets. This reduces the need for custom coding, accelerates implementation, ensures scalability, and makes it easier to maintain and extend the data infrastructure.
In manufacturing, context often extends beyond immediate process parameters and machine statuses. The UAF enriches data with comprehensive contextual layers, including temporal data (timestamps and shifts), spatial information (locations and relationships), event logs (alarms and key events), and product details (batch and specifications). This broader scope ensures that data is not only accurate but also actionable, leading to deeper insights and more informed decision-making across the organization.
The Execution Engines in the UAF are specialized components designed to centralize data processing, provide real-time notifications, and automate data flows within manufacturing environments. These engines ensure that data is handled efficiently and accurately, enabling seamless integration and distribution of information across the organization.
The ability to leverage years of data stored in time series historians as well as transactional records in SQL databases is a key consideration when selecting a data engine. Deploying an information model with backfilling capabilities means you can immediately validate your expressions and ensure that your calculations are providing the results you expected. This becomes crucial as you train models and look towards advanced analytics.
Manufacturing data often changes, arrives late, and needs to be versioned. If engineers and the enterprise can't trust that the data is accurate or up to date, they won't use it. For that reason, the Data Engine must be capable of automatically handling late or modified data values, version changes, and dependencies. It ensures data integrity by backfilling and rerunning calculations, creating a reliable and consistent data foundation crucial for confident decision-making.
Storing only new insights or results is paramount to a successful UAF strategy. Data that has already been validated and is stored in other databases should be left there. Rather, only the new insights, that is events, KPIs, and the contextualized meta data around them should be stored. Raw data should only be called from the original sources as the engines require it for processing.
The Messaging Engine should be used to integrate data into the organization's existing notification and communication tools, such as email, SMS, Microsoft Teams, and Slack. By seamlessly delivering real-time notifications and updates through these familiar platforms, it ensures that stakeholders receive critical information in a timely and efficient manner, enhancing communication and responsiveness across the organization.
This engine is used to automate, on schedule or on trigger, the streaming of data to various databases, data lakes, and BI tools, either on a triggered or scheduled basis. By matching the schema of target systems, it facilitates seamless data integration, ensuring that all enterprise systems are synchronized with the latest information.
Unifies access to both raw and transformed data as well as all definitions held within the Information Model via a single, queryable access point. Ideally this component supports multiple API technologies, such as Graph and REST, and may integrate SQL support to ensure flexibility and compatibility with various applications and systems.
Information Gateway offers a single point of entry to all connected data sources, simplifying the way users interact with the Information Model. This centralized access streamlines data retrieval and integration processes, making it easier to work with complex data sets.
The Information Gateway includes a registry for data silos, allowing users to query any underlying databases attached to the Information Model via the UAF without needing to understand the structure or nature of those databases. This simplifies access to disparate data sources and enhances data integration capabilities.
Helps users of all abilities to create data-driven applications, perform analyses, and generate insights without needing deep technical expertise or knowledge of multiple systems and databases. By lowering barriers to data access, it fosters innovation and agility, enabling more team members to contribute.
The UAF must account for a variety of historian vendors and technologies to ensure cross-platform compatibility and free the enterprise from historian vendor lock-in. This includes supporting:
Unifying data from these diverse systems necessitates the ability to connect to and pull data from each of them. This integration must encompass both established and legacy providers, as well as modern solutions, to offer comprehensive and flexible data management and operational capabilities. By doing so, organizations can achieve seamless data integration and efficient operations across their entire manufacturing environment.
In many manufacturing environments, crucial real-time data is not currently being archived, which limits the ability to analyze and transform this information. To address this gap, the UAF should include the capability to collect and store real-time data, playing a role similar to a traditional data historian but specifically for data that is not already being stored. When connecting to a Unified Namespace (UNS) as a real-time source, it is essential to understand what data is already being historized elsewhere. By identifying and capturing only the data that is not currently stored, the UAF ensures that all relevant information is available for comprehensive analysis and transformation, thereby enhancing decision-making and operational efficiency. This approach prevents redundancy, optimizes storage, and ensures a complete and accurate data set for actionable insights.
Common technologies and protocols that should be supported include OPC servers, MQTT brokers (vanilla and Sparkplug), web APIs, Kafka streams, and other real time data sources.
Transaction-based data is crucial in providing the necessary context to slice and interpret time series data effectively. This type of data helps in understanding the events and transactions that occur within the manufacturing process, offering a detailed view of operational activities and their impact. Incorporating transaction-based data into the UAF allows for a more comprehensive analysis, enabling better decision-making and insights.To support this integration, the UAF must be capable of connecting to and pulling data from various SQL databases. This includes widely-used technologies such as:
Entering data, categorizing events, and capturing comments and context is vital in ensuring that all relevant information is included in the UAF, especially when certain data points are not automatically collected by sensors or systems. This type of data often includes critical insights from human observations, quality checks, or maintenance activities that provide additional context and depth to the automated data streams.
To effectively integrate manual data, the UAF should support various methods for capturing and categorizing this information. This can be achieved through web forms, mobile applications, and by importing CSV files.
The value of data already transformed and heavily encoded with operational process knowledge and context cannot be overstated. For enterprises, such data is a gold mine, offering rich insights and actionable intelligence. The UAF ensures that this data is not only accessible but also integrated seamlessly into various systems, saving data teams significant time and effort. This feature is especially crucial for data teams overseeing multiple sites, as it enables them to make informed decisions quickly and efficiently, leveraging a unified, context-rich dataset. By structuring the data to match the existing schemas of these systems, the UAF facilitates smooth integration and usability.
Standard connectivity methods will ensure that existing enterprise architectures and data strategies are met. This means ensuring solutions like AWS, Azure, Snowflake, Databricks, Google BigQuery, and Oracle can be fed data streams from the UAF.
Supports integration with standard BI tools such as PowerBI and Tableau to facilitate data visualization and business analytics.
Connects to advanced analytics platforms and machine learning and AI tools through standard methods, creating a plug-and-play environment with structured, contextualized, and even preprocessed data in wide table formats with normalized timestamps, greatly increasing the time to value for these projects.
Publishes processed and contextualized data back to the Unified Namespace (UNS) to maintain a continuous and updated data flow, ensuring the integrity and accuracy of the entire data ecosystem.
The UAF builds on the foundation provided by the UNS by governing the transformation and additional contextualization of the data collected.
The Unified Analytics Framework (UAF) was conceived by industry leaders Graeme Welton, Leonard Smit, Walker Reynolds, Allen Ray, and Jeff Knepper, with contributions from manufacturing user groups like IntegrateLive! and the Industry 4.0 community. They recognized a common issue: fragmented data systems and inefficiencies due to disconnected data silos, particularly the spread of data transformation processes across the application layer, making them impossible to govern. Determined to centralize integration and transformation work, they set out to create a framework that would unify data sources while prioritizing OT's requirements.
Through collaborative efforts, discussions, and workshops, the concept of the UAF took shape. The aim was to develop a framework that could integrate, contextualize, and govern data, providing a single, auditable source of truth. The UAF was designed to make OT's job easier and ensure reliable data governance through its ability to log changes, version calculations, and maintain a full dependency map.
The leadership at CESMII, including John Dyck, Jonathan Wise, and John Louka, played a key role in highlighting the problems facing manufacturers and the need for centralized governance, helping to shape the understanding and necessity of the UAF. Today, the UAF continues to evolve with ongoing contributions from the manufacturing community, delivering improved decision-making and operational efficiency across the industry.
At Flow Software, we exist to give teams decision-making superpowers.
We’re a driven, curious, and dedicated global team, committed to turning raw data into actionable insights since 2011.
Guided by our core values, we empower businesses to elevate their operations, embrace innovation, and achieve lasting success. Remote by design and lean by choice, we deliver solutions that simplify complexity and inspire smarter decisions.
When information flows, insights grow.