Digital Assistants in the Metaverse Might Be Better Conversationalists

Our very own Jarvis

Key Takeaways

  • Meta is betting big on AI to deliver the Metaverse.
  • Project CAIRaoke is a framework that’ll help developers build better context-aware digital assistants.
  • Meta envisions Project CAIRaoke-powered assistants built into VR headsets and AR glasses.
Close up of a family using a smart speaker while having breakfast in the morning

Marko Geber / Getty Images

If Meta has its way, our struggles with digital assistants could be a thing of the past.

In a virtual event in February, Meta showcased a new neural model for digital assistants called Project CAIRaoke, which it claims will be capable of having much better contextual conversations.

"The major issue with digital assistants is [they're] supposed to adapt to user behavior and environment, but they work the other way round," Vivek Khurana, Head of Engineering, Knot Offices, told Lifewire over email. "Project CAIRoke appears to be the right step in the direction of an assistant adapting to user behavior and conditions."

Real Assistants

In a technical post, Meta's AI tech lead, Alborz Geramifard, argued that the current generation of digital assistants, both text- or voice-based, leave much to be desired since they lack contextual awareness.

Despite their complex AI-powered brains, they can't comprehend simple requests that would make sense to a 10-year old. For instance, a request to "silence all notifications except if it's a call from my mom" would stump any current digital assistant. 

To overcome this hurdle, Geramifard said it's building Project CAIRaoke as a combination of the four Artificial Intelligence (AI) speech models used in assistants today. He wrote that, unlike most assistants that are programmed to respond to certain words and phrases, Project CAIRaoke is designed to understand context better and have the ability to recognize different phrases used to say the same thing. This approach enables it to have a more natural and flowing conversation. 

Digesting the technical information in Geramifard's post, Khurana said that through Project CAIRoke, developers would be able to build assistants that can easily have a dialog with the user since they can take decisions by looking at a wide range of information, and not just the model they've been trained with. 

He illustrated the addition of context to conversations through an example of a digital assistant taking food orders, which will be able to suggest newly introduced menu items based on the user's preferences or past orders. "This opens a whole new bunch of options for the developer, to build self-service assistants for customer service," opined Khurana.

First-Person Perspective

Meta has been betting big on AI to help deliver its vision of the Metaverse as the evolution of the internet that is a lot more interactive and immersive, and Project CAIRoke is a major component of that experience.

In a video presentation, Meta CEO Mark Zuckerberg said its new and vastly improved digital assistant framework combines the approach behind its open source chatbot called BlenderBot, with the latest in conversation AI to deliver better dialogue capabilities.

Updated in July 2021, BlenderBot 2.0 is unique for its ability to make note of pertinent information in a conversation and store it in long-term memory, which it then relies upon to have meaningful interactions in the future. Furthermore, the "knowledge" is stored separately for each person BlenderBot interacts with to deliver customized experiences.

Young man preparing food in pan on stove at kitchen island. Woman is looking at boyfriend cooking meal

Morsa Images / Getty Images

According to Zuckerberg, Project CAIRaoke extends the BlenderBot 2.0 technology to support task-oriented conversations.

The system is currently being tested with Meta's Portal family of video-calling devices, which have an AI-powered smart camera that pans and zooms to follow users as they move freely. Portal also functions as a digital assistant, currently relying on Amazon's Alexa to interface with smart home accessories and for other tasks.

There's no timeline for when Project CAIRaoke will be available on the Portal devices, but the company stated it hopes to eventually roll it out to virtual reality (VR) headsets and Augmented Reality (AR) glasses as well.

"The major issue with digital assistants is [they’re] supposed to adapt to user behavior and environment... "

Geramifard envisions the new era of digital assistants will be able to redefine interactions between people and devices. For instance, you could ask a Project CAIRaoke-powered assistant built into AR glasses for shirt suggestions that go with a particular pair of pants, based on your favorite color, and even tweak its suggestions based on your current tastes. 

"On devices like VR headsets and AR glasses, we expect this type of communication to eventually be the ubiquitous, seamless method for navigation and interaction, much as how touch screens replaced keypads on smartphones," Geramifard forecasted.

Was this page helpful?