Metaverse Lessons From Hollywood – The New York Times

Decades ago, Steve Perlman’s technology helped kick-start an era in which figments of Hollywood imagination feel increasingly authentic.

Now, Mark Zuckerberg and other technologists are dreaming of a future in which we spend more of our time in immersive digital worlds, called the “metaverse.” And Perlman has advice for metaverse creators: Focus on the face.

What Perlman means is that in movies and sophisticated video games, ingenious technology and creative professionals have combined to create increasingly lifelike virtual creatures. They do that by focusing on detailed facial expressions and body movements.

Think about Gollum from the “Lord of the Rings” movies, many Marvel movie superheroes or Brad Pitt as Benjamin Button, a man who ages in reverse. They aren’t real, but we can connect to them and their emotions.

Perlman, a longtime inventor who also helped to develop digital video technology at Apple, said that what he has seen so far from the metaverse feels flat compared with fully fleshed out Hollywood and video game characters. He says that technologists should be laser-focused on recreating the human face and emotional expression. It’s extremely difficult technology, but one that Perlman says is necessary to tackle to give virtual interactions a sense of intimacy.

“The face is the most subtly expressive part of the body,” Perlman told me. “And the face is a part of the body that none of the VR/XR headsets accurately capture,” he said, referring to virtual reality goggles and technologies like Snap’s Spectacles that mix digital images with the real world — often known by the umbrella term “extended reality” or XR.

That’s just one person’s opinion, and Perlman isn’t a disinterested observer. He is working on a wireless technology that he believes is a necessary first step for the metaverse or other immersive virtual interactions.

I wanted to talk to Perlman to mine lessons from the long marriage of technology and imagination in Hollywood and beyond. Even if the metaverse never pans out, we’re spending more time interacting through screens, and we’re hunting for more authentic ways to connect. It’s both a technological challenge and a human one.

Perlman has a history of helping to bring virtual characters to life. In the mid-2000s, his start-up made a breakthrough that helped us feel connected to two-dimensional characters. The Mova Contour system used phosphorescent makeup on actors’ faces and custom-made camera gear and software to turn those performances into computer data.

This and similar approaches, now typically called motion capture, have become fairly common in movies and sophisticated video games to create fantasies that feel realistic. Those of us who have laughed at and cried along with characters who aren’t human owe a debt to Perlman and other technologists. (Perlman’s firm is involved in litigation that claims that other companies improperly profited from its intellectual property.)

If you’ve tried virtual worlds like Fortnite, Second Life or Meta’s Horizon Worlds, you know that the digital beings there don’t yet look or feel human. They’re cartoonish, and sometimes people in the metaverse don’t have legs. For now, that’s fine. But Perlman says that for immersive digital worlds to catch on, metaverse people will need to feel like people.

“In my view, the metaverse becomes interesting when it’s at least as good as advanced video games today, and likely even better,” he said. “It won’t have a wide appeal until it is a universe of humans.”

That’s where Perlman imagines playing a role. His company has worked for years on a wireless technology that promises to handle a cascade of digital information — including from virtual reality — and to translate within fractions of a second a head movement of someone wearing V.R. goggles into that same action onscreen.

We’ll see if Perlman’s latest invention takes off, but there is a long history of creative ideas being enabled by bedrock and often dull technology inventions.

Without Amazon’s cloud computing service and others like it, there would have been no iPhone age. Without advances to shrink down computer chip components or put GPS in every smartphone, most of the technologies that we use daily wouldn’t be possible.

Perlman looks back at his work with Hollywood characters and hopes that a similar breakthrough is possible within this decade for virtual humans that we can feel in our bones.

Before we go …

• President Biden said that Russia was considering cyberattacks on U.S. companies in retaliation for sanctions imposed on the country because of its invasion of Ukraine, my colleague Zolan Kanno-Youngs wrote.

• Are you covered in that Uber ride? Protocol examines some of the country-by-country differences in the accident insurance for people who take a trip in an Uber. For one man, a crash left him unable to return to his job, and he blamed Uber for not taking more responsibility for passengers’ safety and drivers’ errors.

• “You can have a real conversation without being distracted.” Hilary Reid found people who are embracing the landline telephone as an antidote to spending so much time multitasking and in front of screens.

  Related from The Atlantic in 2015: “The telephone used to be truly great, partly because of the situation of its use, and partly because of the nature of the apparatus we used to refer to as the ‘telephone’ — especially the handset,” Ian Bogost wrote. (A subscription may be required.)

Hugs to this

Bask in the ingenuity of these college cheerleaders from Indiana University, and the fans who love them.

We want to hear from you. Tell us what you think of this newsletter and what else you’d like us to explore. You can reach us at ontech@nytimes.com.

If you don’t already get this newsletter in your inbox, please sign up here. You can also read past On Tech columns.