Blueprint: The Layers of IIoT Analytics Architectures
The ITU defines the “Tactile Internet” as a network that delivers high availability, reliability, and security at very low latency. Modern-day network engineers associate these characteristics with 5G communications. But operational technologists have been deploying Tactile Internet networks for years using various flavors of Industrial Ethernet.
Unlike in the past, networked industrial devices must now interact directly with enterprise applications. Organizations looking for greater insight into factory floor operations are even deploying some of these applications directly onto industrial devices themselves.
Whether this software consists of analytics engines or full-blown AI algorithms, it can quickly compromise the integrity of real-time equipment if not designed properly.
Today’s industrial engineers need a blueprint for integrating IT and OT technologies into a single piece of machinery. These systems must integrate the power of IT networking and processing, while remaining deterministic, safe, and reliable.
Fortunately, Proof of Concept (PoC) system designs that provide an outline for these system architectures are now becoming public. And they’re based on the principles of workload consolidation and Ethernet Time-Sensitive Networking (Ethernet TSN).
An Upgrade for Industrial Ethernet
Industrial engineers have traditionally created deterministic versions of IP networks using protocols such as EtherCAT, PROFINET, and POWERLINK. These Ethernet-based technologies carry IP packets to devices like programmable logic controllers (PLCs) at the network control layer. Beyond that, traditional fieldbus communications protocols are required.
Now, TSN extends Ethernet packets to subsystems like I/O modules, sensors, and actuators with sub-millisecond latency (Figure 1).
Ethernet TSN and Workload Consolidation
Ethernet TSN is based on the IEEE 1588 Precision Time Protocol (PTP) standard, which timestamps packets based on the device clocks of network endpoints. One centralized master source is used as a reference to keep the network synchronized.
PTP networks are capable of achieving sub-millisecond latencies. But just as important, PTP packets can be encapsulated in IP. Therefore, operational devices can essentially become a part of enterprise networks. For more about TSN, read Time-Sensitive Networking: Hype vs. Reality.
Meanwhile, industrial devices are being asked to incorporate enterprise functionality like real-time edge analytics.
Known as workload consolidation, industrial edge systems that have traditionally performed fixed functions are now being designed around multicore processors. Virtualization technology allows individual cores to execute processes like data analytics while other cores can be reserved for data acquisition or control tasks.
With more powerful, flexible systems at their disposal, organizations can reduce costs, minimize points of failure, and even use enterprise software upgrades to extend deployment lifecycles. (Read Embedded Virtualization Powers Mixed-Criticality IoT Systems to learn more).
The convergence of workload consolidation and TSN can unlock the true potential of multilayered, real-time data analytics architectures. With deterministic, low-latency enterprise communications and multipurpose compute platforms at each layer of the network, factory operators can deploy AI to oversee various domains and subdomains (Figure 2).
Decisions are made by AI and analytics engines at each tier based on the response time required and the compute resources available. But the end-to-end information and decision-making hierarchy is connected across a single, unified, time-sensitive communications protocol. Such architectures promote technology reuse, interoperable networking equipment and services, and common data exchange formats.
Bringing Tiered Industrial Data Analytics Architectures to Life
A microcosm of this architecture can be found in a real-time industrial control server PoC from congatec and Real-Time Systems (Figure 3). The PoC is built around:
- The 35W conga-TS370 Type 6 COM Express module based on a six-core Intel® Xeon® processor
- An Intel® Arria® 10-based PCIe FPGA card
- A Basler machine vision camera
- The RTS Hypervisor
To demonstrate workload consolidation, the RTS Hypervisor partitions Xeon processor cores into multiple virtual machines (VMs).
The first VM occupies three cores and runs a Linux OS. The OS supports a 300 fps Basler industrial camera and Arria 10 FPGA card. AI algorithms developed using the Intel® OpenVINO™ toolkit are executed on the FPGA card for on-device object detection and recognition.
Two additional VMs host real-time Linux OSs. These control servo motors keep an inverted pendulum balanced upright. And the final VM hosts a third OS type that handles enterprise security and communications back to the cloud.
All of these applications run concurrently on the same hardware platform. But with the hypervisor, non-real-time VMs can be rebooted without interrupting the real-time partitions. This means that software updates can occur while control tasks continue to execute. And security vulnerabilities can be contained to enterprise VMs and not impact embedded systems as well.
A live demonstration of the system can be seen in (Video 1).
Video 1. Proof of Concept demonstration from embedded world 2019
Another key feature of the conga-TS370 is that it includes an Intel® i219-LM GbE LAN Controller. This discrete networking chip natively supports IEEE 1588 PTP.
In platforms like the congatec/Real-Time Systems PoC, the i219 determines the source and destination of PTP packets and synchronizes them using integrated timers. The controller works in conjunction with an OS to schedule packets during transmission windows that are dedicated to TSN traffic.
Release 4.20 of the Linux kernel added support for the Time-Aware Priority Scheduler (TAPRIO) and Earliest Transmit Time First (ETF). These modules allow developers to configure the system so that Ethernet TSN packets receive priority regardless of their position in the transmission queue.
The RTS Hypervisor also offers a privileged mode that allows guest OSs to access system networking resources in just a few microseconds. TSN packets from guest VMs therefore receive the same low-latency priority scheduling.
IIoT Analytics on the Shoulders of Standards
One characteristic that stands out about the PoC is that all of the core functionality is based on open source technology and industry standards. Linux and OpenVINO are open-source software technologies. IEEE 1588 PTP and the Ethernet TSN umbrella of standards are industry-driven efforts that will keep costs down, interoperability high, and ensure longevity.
COM Express is one of the most popular embedded hardware specifications available that supports server modules, gateway controller, and sensor processing use cases. It provides a stable, affordable, long-lifecycle deployment platform for new tiered industrial data analytics architectures.
What will you do with increased intelligence?