The irresistible marriage of AI and XR

AI connects real-world data with virtual applications

According to the Technology Hype Cycle by analyst Gartner, AI itself now has entered the Trough of Disillusionment, the moment when “interest wanes as experiments and implementations fail to deliver”.

Andrew Anagnost, CEO at design software company Autodesk, acknowledges that the hurdles AI is currently facing are the hurdles that every technology encounters at the beginning of commercialisation. 

“When new technologies rise up to change the way we work and live, they inevitably come with a lot of hype” he says, with the “period of hype before utility leaves a lot of us with unmet promises” referring to the current state of the AI market landscape.

“[But] that hype always seems to exceed the usefulness of the technology until – and it happens quite suddenly – it doesn’t, and the utility becomes obvious,” Anagnost says, adding that this maintains that now the time has come that the technology is moving “beyond experimenting”.

Yet here are a number of companies keen to emphasise the benefits that AI can provide to XR environments. Jussi Mäkinen, CMO and founder at Distance Technologies; and Lotta Syrmä, head of marketing at Younite AI, point to the ability of AI to preprocess and transform data for the use in XR environments, creating responsive, interactive and real-time interpretations of physical dynamics and phenomena.

Mäkinen believes that AI, and especially GenAI, has tremendous potential to be an essential part of virtual reality. He highlights its use for mission-critical and real-time applications where embedding and visualisation of crucial data and information is of importance. Distance Technologies, for instance, is working on leveraging AI for glasses-free XR systems for the automotive, aerospace and defence sectors. He asserts that the combination of computer vision, mixed reality and AI has the power to reshape awareness and understanding of the physical surroundings in real time.

Syrmä, meanwhile, states: “Simply put, AI seamlessly connects humans with data.” Younite AI is using AI to create digital twins with photorealistic properties. She notes that users of XR-enabled applications can “interact naturally with data using speech, gaze, hearing and touch – enabled by AI”. While XR technologies serve as interaction tools, AI will link humans to data in real time.

AI empowers digital twins

However, experts at management consultancy McKinsey believe that many organisations are already implementing GenAI and digital twins separately.

The authors of the report Digital twins and generative AI: A powerful pairing say: “GenAI can structure inputs and synthesise outputs of digital twins, and digital twins can provide a robust test-and-learn environment for GenAI. By combining these technologies, organisations could produce synergies that reduce costs, accelerate deployment and provide substantially more value than either could deliver on its own.”

The authors see digital twins as being able to recreate scenarios of real-world situations and provide an early warning system that can anticipate future events. Digital twins also serve as digital laboratories in which designs and options can be tested, efficiency increased, and schedules and sequences optimised in a risk-free environment.

GenAI can create content in form of audio and text, images and videos, and entire simulations. Both enabling technologies are currently changing business approaches and operational processes, with the report authors stating: “Digital twins and GenAI can be used in combination to unlock insights and elevate their respective value.”

According to McKinsey, developers of digital twins can leverage AI for a number of key purposes:

• Creating a universal model: The creation of highly specialised twins for well-defined applications can take a long time and substantial investment, but LLMs can create “a foundational, universal model”.
• Collecting, transferring and augmenting data: Since digital twins require massive amounts of data to leverage their full potential, embedding data – often in real time – LLMs can “significantly compress data while retaining essential information”.
• Serving as an interface: LLMs can interpret vast amounts of data to synthesise insights. This capability can function as an interface to generate various scenarios for simulation efforts.
• Using digital twins to validate GenAI applications: With their representation of real-world data and virtualised dynamics, digital twins can offer a secure and safe environment to train AI, fine-tune results, validate AI’s capabilities and increase AI’s capabilities.

The experts conclude: “The symbiotic relationship between digital twins and GenAI increases their combined scalability, accessibility and affordability.”

Commercial companies already make use of AI’s capabilities to create virtual representations. Autodesk chief technology officer Raji Arasu suggests that because of Autodesk’s “industry expertise and a 10-year head start on AI research, the company can develop foundation models that deliver CAD geometry with high accuracy and precision”. She thereby outlines the importance AI will have in future XR applications and – given Autodesk’s product portfolio – its relevance for digital twins of the future.

AI will require thoughtful consideration

One aspect that deserves attention, particularly in safety and security sensitive applications, is the level of trust users can currently put into AI-generated information. The issue of hallucinations, incorrect or misleading results, has seen substantial media coverage.

But even perfectly working AI models can be problematic. Wrong assumptions regarding the described phenomena will lead to technically correct outcomes that don’t reflect real-world dynamics.

Similarly, the omission of crucial parameters or introduction of parameters that side-track the algorithm are issues too. The problem “garbage in, garbage out” is a real concern, particularly when modelling highly complicated – even complex – operations or environments.

Security issues exist too. Algorithms can function as an entry point for nefarious purposes. Safety considerations are an issue in applications that connect to real-world operations. Privacy concerns are of concern as AI will collect, process and respond to personal information, behaviour and even physiological input as advanced interface solutions become more common. Ethical aspects will be an issue, and bias and discrimination of algorithms need addressing.

Finally, widespread use of AI and GenAI will not only make development of virtual worlds more efficient and increase productivity, but also drastically change the required talent pool and employment opportunities in industries. The need for content creation will be replaced by the skill of prompting algorithms and curating AI output, for instance.

AI, XR make for an uneasy marriage

Advantages of combining AI and XR are manifold, but for most upsides there can also be powerful downsides. Artificial intelligence will find use to detect, prevent and mitigate threats and attacks in virtual environments. Unfortunately, criminals will also leverage the technology to create and execute sophisticated attacks. An AI arms race is likely – not only in XR-related applications, but they are enticing targets.

GenAI similarly has the potential for good and bad. AI can create adaptable, interactive, and personalised environments and objects to create truly immersive and beneficial landscapes and sources of entertainment and information. But the technology will also create increasingly realistic deepfakes and could result in entirely spoofed environments.

As noted in Computer Weekly in 2022, in spoofing attacks, perpetrators trick victims by pretending to be somebody else or by falsifying data. In the metaverse, entire environments could become what might be called ‘reality spoofs’ – representations of legitimate companies, shops or marketplaces to extract money or gain access to valuable data and information.

AI will also find use to analyse user behaviour and interactions in extended realities to predict preferences and create customised experiences. AI can provide the data users expect in a form users prefer. AI will customise environments and populate them with personalised objects and avatars. AI will adjust haptic sensations, sound and visual impressions to individuals’ desires and requirements. Such adjustments not only create immersive landscapes but also inclusive environments.

But according to the FCC’s Wheeler: “The online challenges with which we wrestle today, such as privacy, competition and misinformation, will be supercharged by the intrusive, immersive, individually identifiable and manipulative nature of the metaverse. On top of this, the metaverse expands the problems inherent in unsupervised online communities such as harassment, bias, manipulation and threats to personal safety and the safety of children.”

Wheeler sees immediate needs for putting a regulatory framework in place in his commentary AI makes rules for the metaverse even more important, adding: “Failure to develop meaningful policies now – on an international basis – will mean we did not learn the lesson of multiple decades of internet exploitation by a handful of companies, and how the companies write their own rules at the expense of the public interest.” 

Martin Schwirn is the author of Small data, big disruptions: How to spot signals of change and manage uncertainty (ISBN 9781632651921). He is also senior adviser for strategic foresight at Business Finland, helping startups and incumbents to find their position in tomorrow’s marketplace.