The Metaverse and eXtended Reality with Distributed IoT

Theo Kanter
November 16, 2021

 

The center of gravity of our private and work lives has shifted to the Internet, facilitated by technological advances such as the Internet of Things. This Internet now allows us, ideas, and things to interact and collaborate in new ways and on a large scale. This shift has also accelerated the search for untethered ways for people and things to interact in real and virtual places. However, at least one lesson learned from using the Internet in our daily lives is that – while it is very good at enabling people to interact with other people – there is much room for improvement in the technology to allow people to interact with things to improve their lives.

Smart homes give us the ability to control our environment while increasing energy efficiency quickly. Connected vehicles in Smart Transport can reduce the overall energy footprint combined with increased safety. The increasing demand and costs for providing Elderly Care are global challenges that can be met through Ambient Assisted Living solutions involving remote presence, improving emotional and social well-being. Advances in interaction in Extended Reality (i.e., modalities of untethered presence in augmented and virtual reality) are perceived to be vitally important to advancing Social Networks, Performing Arts, Media, and Gaming Industries. Facebook is at the forefront of attempting to solve the technology gap to create a Metaverse (an extended-reality space in which users can interact) [1].

Challenges

There are several challenges involved in creating metaverses that are efficient with a good user experience. First, security must be addressed, as highlighted in the July and September 2021 IEEE IoT Newsletters. Smart solutions mentioned above aggregate data at an accelerating rate due to increased speeds in wireless networks. The smartness in solutions presumes algorithms to operate on a vast body of accumulated data in what is referred to as the Digital Universe. Although cloud computing has proven to be very effective in offering scalable computing power, this cannot keep up with the demand for timely responses to where data originates and is consumed, such as smartphones.

A simple example is location-sharing, where the response times are slow and unpredictable — far from real-time. Actors such a Google, Netflix, and others have responded to this challenge by taking advances of so-called Edge Computing, which allows computing decisions to be handled by servers in access networks. The disadvantage is that these servers become the next bottleneck. Edge servers are commonly tailored to optimize a specific task (e.g., caching streams) or handle data requests between many devices. As Cisco and others [2] proposed, Fog computing is focused on providing multiple paths of connectivity between (wireless) devices, similar to capabilities proposed for 5G.

Problem Statement

However, neither Edge computing nor Fog Computing are a sufficient answer to the challenge of enabling people and things to interact in real-time seamlessly. Their focus is to off-load the cloud concerning data connection to end devices. In contrast, the solution to enabling people and things to seamlessly interact in real time in Extended Reality or the Metaverse requires them to communicate and collaborate directly with each other, leveraging the computing power amply available on devices and sensors.

The ability to interconnect the behavior of people and things is vitally important to rethink and advance the above scenarios involving interaction in Extended Reality. This newsletter focuses on the four key aspects required to successfully address the challenges above (Distributed IoT, Reachability, Delegation, Self-organization) implemented in a distributed IoT platform – 7th Sense™. This newsletter then summarizes the merits of the distributed IoT platform as demonstrated in EU projects and, most recently, the Digital Futures program in Sweden.

Figure 1: The 7thSense Architecture.

Figure 1: The 7thSense Architecture.

Distributed IoT: a key target of the 7thSense distributed IoT platform is to enable people and things to interact seamlessly and in real-time. People and things (electronic devices) are represented as objects handled by a software agent running in a device collocated with people or hosted in the things we use (a node). Agents interconnect automatically to the nearest node, where the first node acts as a bootstrap with a well-known name. Interaction between objects requires a separation between different layers. The network layer provides connectivity and an IP network identity. Connectivity may employ technologies to establish connectivity between devices (Ethernet, 5G, OpenFog, etc.). The sensor and actuator layer provides abstractions of their interfaces. As a result, the 7thSense distributed IoT platform offers a simple but powerful programming model for context-aware applications, enhanced through add-in capabilities. Rather than residing in a cloud IoT platform, the 7thSense distributed IoT platform allows context-aware services applications (with multiple nodes) to operate independently of cloud IoT services in-line with requirements for autonomous operations.

Reachability: Similarly, it is paramount that objects reach each other and exchange information to operate independently of cloud IoT services. A Lookup service finds the location of the objects across the peer network of nodes by name only. Objects use a dissemination service to exchange context information with subscribers in an information-centric network of nodes to make local decisions based on global context information. Information centricity means that the reachability of nodes and access to global context information is achieved without knowing the path.

Delegation: it is mutually important for the Distributed IoT and a cloud-based IoT platform to have an interworking function. One of the nodes can be hosted in a device collocated with, e.g., an Edge server and act as a gateway between them. This way, the cloud-based IoT platform can express and delegate complex operations to sets of sensors based on criteria rather than identities. In return, the central platform can either receive the combined results (similar to map-reduce [3]) or enable the parameterized local execution of scripted operations. Among the further advantages are avoiding unnecessary communication and data transfer with the cloud IoT platform. Still, the cloud IoT platform can tap into the computing power of an increasing and vast number of end devices in return. Therefore, delegating software updates or finding the occurrence of a particular event matching criteria is greatly simplified.

Self-organization: common-sense tells us that objects in a distributed IoT should not require updates from any other object. That would waste computing and communication resources. Instead, objects in applications are likely to subscribe to context information from objects matching specific characteristics. Similarly, searches for objects include specific requests. To this end, nodes include added clustering algorithms to rank objects and their context information based on similarities. As a result, nodes can self-organize and form publish-subscribe relations based on specified characteristics.

Figure 2: Delegation and Self-organization with 7thSense agents.

Figure 2: Delegation and Self-organization with 7thSense agents.

Results

The 7thSense Distributed IoT Platform has been used and evaluated in several field trials. The Smart-City testbed in Stockholm demonstrated delegation for different service providers varying from Smart-Grid to smart home services via a single access infrastructure; thus, avoiding duplication of access infrastructure. The EU MOBIS-project demonstrated Intelligent Traffic Control and Smart Home Solution with self-organization between subscribers and consumers of context information contributing to reducing the environmental footprint. The EU AAL SALIG project has shown distributed awareness in Ambient Assisted Living scenarios increasing safety and social interaction between elderly and next of kin.

Recently, the PROMPTER project of the Digital Futures program [4] showed how 7thSense distributed IoT enabled people, and things can meet in a multiplayer cultural event in which players could be present either in augmented reality, virtual reality or be present (and placed in VR) as spectators as shown in Figure 3 and Figure 4.

Figure 3: Prompter video for the opening of Digital Futures showing the synchronized views of participants (https://youtu.be/8mkUjm0veTg).

Figure 3: Prompter video for the opening of Digital Futures showing the synchronized views of participants (https://youtu.be/8mkUjm0veTg).

Figure 4: Prompter Prototype demonstrates the alignment in time and place of experiences of real and virtual objects.

Figure 4: Prompter Prototype demonstrates the alignment in time and place of experiences of real and virtual objects.

The objective was to demonstrate the alignment in time and place of experiences of real and virtual objects in augmented and virtual reality. 7thSense enables end-devices to self-interconnect via different networks, thus achieving reachability between objects for sharing context data. Since experience in real-time is paramount, the choice of wireless network access was limited in this implementation to 802.11 and LTE/5G, but the principle of heterogeneity and handling via 7thSense applies. 7thSense also enables introducing new objects in the scene and visualization, in this case, participants in an online audience (here via the server). A key component for the latter is the interworking capability of 7thSense between centralized management of (IoT) assets and delegating this to 7thSense agents that have access to local interfaces. But the reverse-path also applies.

Conclusion

Advances in interaction in Extended Reality (i.e., modalities of untethered presence in augmented and virtual reality) are vitally important to developing our societies. This may be expressed by advancing phenomena such as Social Networks, Performing Arts, Media, and Gaming Industries and, Distance Healthcare, to name a few. This is evidenced, e.g., by Google, Facebook, and Microsoft initiatives related to the metaverse. The technologies needed to make the metaverse a reality, equal to Extended Reality, are discussed in this newsletter, which brings us in a feedback loop with virtual reality objects through interaction devices and sensors. Extended Reality requires a distributed IoT platform with characteristics and key features of 7thSense. This mandates further research and innovation to integrate distributed IoT platforms in the development process and solutions of various usage scenarios.

References

  1. Heath A., Facebook is planning to rebrand the company with a new name, The Verge, Oct 19, 2021: https://bit.ly/3BKsHqA
  2. "IEEE Standard for Adoption of OpenFog Reference Architecture for Fog Computing," in IEEE Std 1934-2018, vol., no., pp.1-176, 2 Aug. 2018.
  3. Borthakur D., The Hadoop Distributed File System: Architecture and Design. The Apache Software Foundation; 2007.
  4. Digital Futures - Vision and Strategy: https://bit.ly/3bN9OJ4

 

 

Theo G KanterTheo G. Kanter earned a technical doctorate in Computer Communications from the Royal Institute of Technology (KTH) in Stockholm, Sweden, in 2001. He is a Professor of Computer Science in Sensor-based Services at the Department of Computer and System Sciences (DSV) within Stockholm University. His research focuses on concepts, architectures, and technologies concerning mobile context-aware services and applications. In parallel, he drives innovation related to "7thSenseTM".