Friends App Goes Cutting Edge with AR People Occlusion

The one where the Friends anniversary app uses bleeding edge mobile technology

Lance Ulanoff sitting in Central Perk

 Lifewire / Lance Ulanoff

A product of its time, Friends, the classic sitcom that’s now celebrating its 25th anniversary, is virtually tech-free. I can’t remember a single computer (maybe one at Chandler’s work desk—what did he do anyway?), let alone a cellphone.

Cellphones existed in 1994, but they were still relatively bulky, expensive, and did little more than let you call your friends (not those Friends) and family. Imagine how different Season One, Episode 7, “The One with the Blackout,” would’ve been if Chandler had access to a cellphone when he was trapped in the ATM with model Jill Goodacre. He could've called Joey. Instead, the writers resorted to letting us hear Chandler’s inner thoughts.

Friends also left the air before the advent of the iPhone and apps and remains a 1990s-to-early aughts snapshot of our society, cultural norms, and tech advancements. The show is remarkably popular with Millennials and Gen Z, many of whom were children or toddlers when Friends left the air and not even born when the pilot first aired on NBC. They also experience Friends in a way not possible in the late 20th century: by bingeing on the Netflix streaming services (often with closed captioning on).

The ability to consume mass Friends quantities over and over (and over) again has both built and dismantled a massive fan base. Some see Friends as a beloved cultural icon, while others now worry its jokes and cultural insensitivities have aged badly.

For that latter group, what’s coming next—the merging of a beloved cultural icon with some of the most cutting-edge mobile technology on the planet—is not the balm they were hoping for.

The One Where Friends Pushes the Bounds of AR

In June during Apple’s Worldwide Developer Conference, Apple introduced ARKit 3, an iOS 13 developer update to the augmented reality toolkit developers use for integrating augmented reality into their apps. It’s what gives Snapchat filters their face-forming reality, how you can use your iPhone to virtually place realistic looking furniture in your home before you buy it, and how, through your iPad, you can create a virtual LEGO apartment building on your real desktop.

ARKit 3 adds two important updates to the developers' augmented reality bag of tricks. One is motion capture, which lets app developers track human body movement in real time, and the other is people occlusion, which can put real people behind—fully or partially—virtual objects.

The Friends 25 App
The Friends App is full of friends references that fans will love, including stickers, Marcel, and this Ichiban 3D filter lipstick.

I recall seeing some pretty awesome demos of these technologies at WWDC but to be honest had mostly forgotten about them. At least until I got my hands on an update to the Friends 25 Years app.

Developed for Warner Brothers, which owns the Friends brand, by Double A Labs, the Friends app is an entertaining collection of show-related activities. There are virtual stickers featuring characters, objects, and show quotes that you can place on real photos, a couple of 3D filters, like purple lipstick face and a shoulder-sitting Marcel the Monkey, recipes (Joey’s Tots), and trivia. Starting later this month (and coinciding with the launch of new iPhones and impending release of iOS 13), however, Double A Labs adds new game that leverages people occlusion technology found only in iOS 13 and supported by Apple’s A12 Bionic CPU (introduced for the iPhone XS series).

Called “Point-N-Pivot” (Friends fans will get the reference), the game is essentially a digital version of Twister, but with a real twist of its own.

In the game, you hold up your iPhone and, in each show scene-themed round (“Smelly Cat”), you’ll see a trio or more of stickers floating in space. With another player standing in front of you, you look through the phone screen to direct them where to place their arms, hands, legs, feet, head and any other body part they can easily control to touch some floating stickers while avoiding others.

People Occlusion
This is, essentially, how people occlusion works in in ARKit 3.  Apple

What’s exciting about the game and its technology is that these objects are not just floating in a flat plane. Thanks to people occlusion and ARKit3’s ability to see the depth of a space (it uses what’s called a “Depth Buffer”), the game places the stickers almost anywhere in 3D space.

One of the game’s developers told me the game is only on iOS because Android doesn’t currently support AR people occlusion.

Reaching For It In Space

While the game tries to ensure that one person can effectively touch all of the stickers at once within the 60 second time limit, one Double A Labs developer admitted that the poses can get downright awkward. The game will even include a health and fitness warning, basically telling you not to try anything you can’t physically handle.

The first time I played an early build running on an iPhone XS with the iOS 13 beta, I didn’t have anyone to put in front of the iPhone camera and instead, as directed by the developer, I used the fingers of my hand to touch the right stickers (which appeared tiny and distant) while avoiding the ones that didn’t fit the theme.

I noticed immediately that it wasn’t enough just to place my fingers in the right place. I had to move them forward and back in space to reach each sticker. The final app will not support playing the game with just your fingers.

I also got access to a debug screen that showed me exactly how the system interpreted my fingers in 3D space.

Friends App
In the beta, I could play by myself. That won't be the case in the final version.

Later at the downtown Manhattan Friends 25th Anniversary Pop-up space, I played again with Double A Labs CEO Amber Allen.

After we found a clear space adjacent to the Central Perk set recreation, Allen gamely got in front of the camera and bent her arms akimbo as she tried to reach, with my direction, the right stickers. Then we switched places and I followed her direction, lifting my right arm into a Saturday Night Fever disco pose, while stretching out my left arm and then bending my head so far forward that I thought I might wrench my back.

Gameplay is automatically recorded so players can then come back behind your iPhone screen and watch them making fools of themselves.

Twisted Indeed

Playing a game in which the person with the phone has all the control and has to perfectly describe how the other players must pose themselves to win is relatively new in the mobile phone gaming space. It’s basically the opposite of Ellen’s Headspace app, where you hold the phone up to your head with a subject that everyone can see except you, and you have to guess the answer based on clues they shout at you.

Friends 25 Point-N-Pivot App
How the Friends 25 Point-N-Pivot App will look when it's released.  Warner Bros.

As Double A Labs was building the game, they had to ask themselves, Allen told me, “How do we get people to be comfortable with stuff? 'Oh, it’s digital twister.'" An early beta even had “Twist” in the name, but I’m assuming Hasbro, which owns Twister, would’ve had something to say about that.

Allen told me they did consider building (and may do so later) on ARKit 3’s body-tracking capabilities to create a Monica Dance contests, where players would have to match Monica’s dance moves in the episode called “The One with the Routine” to win.

We may no longer all agree that Friends is one of the best sitcoms in American television history, but it is, I think, permanently woven into the cultural fabric of our society. So, it makes sense that it’s just as connected to our ever-changing and often surprising digital experience as we are.

Point-N-Pivot is scheduled to launch on September 19.