Languages in the metaverse: Why AI is critical for communication in the brave new world

By the end of 2021, the word “metaverse” had generated a lot of hype. The topic was covered by almost every existing media outlet. But what is a metaverse and what is its purpose? To put it simply, it is cyberspace, a virtual world that should serve human connection. 

In a metaverse, a person is represented by an avatar, just like in a video game. People are supposed to enter a metaverse with the help of virtual reality headsets and be able to communicate with other people no matter where they physically are. It is a technophile’s nirvana that enables any person to travel through various 3D surroundings without leaving their couch. Something that sci-fi writers have been dreaming of for a long time.  

Bill Gates predicts that in two to three years, all virtual meetings will move from 2D to 3D worlds in metaverses. Microsoft, Meta, Amazon, Tiktok, Apple and other tech giants are planning to invest money and efforts in creating their own metaverses. Fashion and beauty brands seem to be interested in the opportunity too. JPMorgan estimates the metaverse market at $1 trillion in yearly revenues.

Probably one of the closest experiences to what a metaverse should be is Minecraft. Since the pandemic started, this video game has been used as a social gathering space multiple times. For instance, a South Korean University recently held its entrance ceremony within the virtual space of the game. In spring 2020, there was a music festival in Minecraft with the pop star Charli XCX as one of its headliners. In spring 2021, Brown University also tried its hand at organizing music events in Minecraft.  

As you can see, the metaverse is meant to be a social experience. The awesomeness of metaverse is that you can communicate, work, play or make art with people who are not in the same physical space as you are. You don’t have to be in the same room, city or even country. It makes the concept of metaverse sound like something that can unite the whole world by overcoming the restrictions of geographical and political borders. But it also feels totally utopian because we don’t have the superpower of universal understanding even in real life. But what if it’s indeed possible to achieve it in cyberspace? 

Though metaverse is a virtual space, we are still going to approach language and communication there according to the same principles as in the physical world. Speaking languages, showing emotions, listening, being polite, respectful and empathetic. In general, communications inside and outside digital reality will be similar in many ways. It’s a good thing, because we don’t have to learn from scratch, but the difficulties are the same too.

If we are speaking about a global cyberspace that is supposed to unite humanity, we have to solve the problem with the language barrier. Meta, Mark Zuckerberg’s company that owns Facebook and Instagram, is planning to develop its own metaverse. Currently, Meta gathers on its social media platforms users from different countries and they are mainly staying within their language group without reaching out to the ones who speak a different language. As of 2019, there were 41 languages officially supported by Facebook. It’s not much, because there are hundreds of languages and dialects, but it’s still a lot.

This doesn’t sound unifying, because, obviously, users are able to communicate and build connections only within a limited group of people in these circumstances. It narrows their views and possibilities. At the same time, it would be a tragedy to lose the diversity of languages by choosing only one of them as a universal language. That’s why metaverse needs AI translation. 

AI translation in the metaverse

In the last decade, machine learning and artificial intelligence have been improving by leaps and bounds. In the past few years, machine translation has progressed significantly, as well as voice recognition and other language technology. Now, social networks take advantage of these breakthroughs through the instant translation of scriptory texts. You might have noticed this in Facebook posts and comments, or in video subtitles like the ones that are automatically generated by YouTube and Tiktok. But in the metaverse, we need to go further with AI and language. There, people will use their voices to speak, like in the physical world. The language barrier problem should be solved as seamlessly as possible to make conversations effortless and inclusive.

It looks like Mark Zuckerberg understands the importance, too. In February 2022 Meta CEO announced that the company was working on building an AI-powered universal speech translator for languages in its metaverse that would work for everyone in the digital world. It will use AI to provide instant translation of widely spoken languages. “The ability to communicate with anyone in any language — that’s a superpower people have dreamed of forever and AI is going to deliver that in our lifetimes,” Zuckerberg said.

Meta’s universal speech translator is said to be able to translate speech directly to speech without transcribing it first. It will make the translation process unnoticeable, which will help conversations to feel natural. If the project succeeds, people won’t even remember they are talking to someone with no understanding of their language. It will be like a native-to-native conversation. This level of translation can’t be achieved seamlessly in the physical world.   

AI translation also helps save language diversity, as it motivates people to keep preserving their native languages because it no longer will be a barrier for international communications. In the future, there will be no need to learn a foreign language to be a part of the international community. 

It takes a lot of work before we get there, of course. Meta’s universal speech translator has to learn hundreds of languages and achieve the ability to translate from one to another in real-time. The engineers standing behind the project need to find training data, which is not easy to get for many languages that are not popular on the internet. It’s a complex and highly ambitious project, but it has enormous potential to bring a lot of good to the world.   

Petr Malyukov is a serial IT entrepreneur and CEO/cofounder of YOUS.