A day after leaving the “Fast and Furious” franchise, Justin Lin got a curious phone call. Lin, who had directed four of the street racing-themed action movies, was asked by Google to collaborate on a very different kind of film project: one that would be available only on mobile phones, and force him to give up a huge amount of creative control. “I couldn’t have asked for a better challenge or medium,” Lin recalled during an interview this week.
Fast forward two years, and the result of Lin’s cooperation with Google is being released at Google’s I/O developer conference in San Francisco this week. “Help,” as Lin’s five-minute short film is called, tells the story of an alien attack in Los Angeles, complete with explosions, a subway car being torn apart and a dramatic showdown in the dry concrete bed of the Los Angeles River — all of which is being shown with 360-degree spherical video that can be freely explored by the viewer.
“Help” is being distributed via a mobile app that allows the viewer to explore every angle of the film by simply moving the device around freely, as if it was equipped with a camera pointed at an invisible stage. Shift the screen to the left, and the viewpoint shifts to the left as well, where the alien monster is ready to smash up a subway car. Pan to the right, and you see the scared faces of train passengers scrambling for safety. Move it up, and you can catch a glimpse of helicopters flying over the Los Angeles River. It’s immensely engaging, and feels unlike any action movie you’ve ever seen.
No one took advantage of a phone’s true potential
“Help” is just the latest incarnation of a new genre of spherical videos that Google calls Spotlight Stories. Before “Help,” Google partnered with animation legend Glen Keane and other well-known Pixar talent for three animated shorts, all playing out on a 360-degree stage. “Your phone is the window into that world,” explained Regina Dugan, whose Advanced Technologies and Projects (ATAP) group at Google has been developing these Spotlight Stories.
ATAP is one of Google’s experimental research and development units, tasked with advancing the state of mobile technology. ATAP’s best-known project involves building a modular smart phone, while another is all about advanced computer vision for mobile devices. So why, of all things, did Dugan decide to make movies with filmmakers like Justin Lin? Asked this question, she said that it was all about leveraging the true potential of mobile devices.
“We reached an inflection point,” Dugan said during a recent interview at Google’s ATAP headquarters in Mountain View. By 2014 cell phones had become just as powerful as game consoles, said Dugan, but few if any apps took advantage of all that computing power. Her team began to experiment with combining the graphics rendering capabilities of modern cell phones with its built-in sensors, and quickly honed in on the potential to reinvent storytelling — in part because it seemed like a worthwhile challenge. “The level of technical execution is crazy hard,” Dugan said.
Everything was jerry-rigged
That’s doubly true for shooting live action, as Lin soon learned first-hand. The ATAP team, Lin and his production company Bullit had to reinvent all of their tools for Help, starting with the camera. Previous experiments with spherical video had often been based on multiple GoPro cameras, arranged to shoot 360 degrees and then stitch the result together in post-production. Lin wasn’t satisfied the results of that approach, so his team began to experiment with higher-end cameras — a process that alone took four months. “We had to jerry-rig everything ourselves,” said Lin.
In the end, they had a custom rig of four Red cameras, each shooting in 6K, capturing gigabytes and gigabytes of video. “The amount of data was huge,” explained Bullit CEO Todd Makurath, adding that it required a 48 terabyte RAID setup on set. All that data was then stitched together in real time to give Lin and his crew a live view of everything being captured simultaneously.
Pointing cameras in all directions also means that everything is always in focus, and there is no place to hide anything, requiring the crew to get creative with lighting and post processing. But not all challenges could be solved with better cameras or more advanced production tools. A lot also had to do with the way a story is told for an audience that has the freedom to look in any direction it wants. “You do have the risk that someone is not going to go where you want them to go,” Makurath said. That’s why “Help” uses a lot of subtle cues to get a viewer’s attention, including binaural sound, light and effects as well as the story itself.
Even the timing is tricky for a project like this one: Move too fast, and a viewer may miss a key moment because he was looking in the wrong direction. Wait too long, and the film would become boring. “It’s real life, and real life is not waiting for you,” explained Karen Dufilho-Rosen, another Pixar alum who has joined Google for Spotlight Stories.
The production was “slightly more cumbersome” than a traditional shoot, Makurath admitted with some understatement. “The technical challenges were immense,” said Lin more frankly, adding: “We were building the road as the car was moving.”
Spotlight Stories: A bit like VR, without the reality
The idea of spherical videos that viewers can explore to their own liking, scenes that you can look around in, has become en vogue with the resurgence of virtual reality, or short VR. Facebook’s $2 billion purchase of Oculus has put a huge spotlight on immersive and virtual storytelling. Samsung has started to invest some money of its own in narrative VR content for its Gear VR headset, and Google has an iron in the fire as well with its Cardboard project, which like Gear VR combines a 3-D viewer or headset with a cell phone display.
However, everyone involved in Spotlight Stories was quick to point out that there are notable differences between stories like “Help” and VR. For one, Spotlight Stories don’t require you to use a headset. Facebook’s Oculus Rift headset won’t even be available to consumers until 2016. Spotlight Stories, on the other hand, can be accessed with most current smart phones, reaching a potential audience of hundreds of millions of viewers.
But relying on the phone as the viewer isn’t just about power in numbers. Spotlight Stories uses the mobile phone as a window: Viewers use it to peek into a different world. In VR, they expect to be part of that world, which comes with its own challenges for storytelling. For example, VR users often complain about dizziness when thrown into scenes with too much motion. “On the creative side, VR is a lot more constrained,” El Guerrab told me, quipping: “This is easier.” “It’s different,” added Dufilho-Rosen diplomatically.
Now on Android, iOS and YouTube
Spotlight Stories had a bit of a coming-out party at Google’s developer conference in San Francisco this week. Not only did the company release Lin’s film, it also for the first time made iOS and Android apps widely available, after previously distributing Spotlight Stories only on Motorola’s Moto X handset. Spotlight Stories are also being made available through YouTube’s mobile apps, reaching an even wider audience, and Google has started to make the Spotlight Stories SDK available to select partner studios, including Aardman Animation, Evil Eye Studios and Nexus Productions, with Dugan explaining that the goal is eventually to give creatives complete access to tools for spherical video productions.
All of that is happening as Google is still keeping mum on its long-term vision for the project — a question the company may have to answer very soon. Google’s ATAP unit has been modeled after the Defense Advanced Research Projects Agency (DARPA), a government agency that is using cutting-edge science and research to keep the country safe.
Regina Dugan served as DARPA’s director before joining Google to head ATAP, and one of the rules she brought with her from the defense agency is that projects have a two-year lifespan. That two-year mark is just about up for Spotlight Stories, which means that Google has to decide if it wants to graduate the project and make it part of one of its product groups with a goal to generate revenue, spin it out or give up on it altogether. Dugan declined to elaborate on those options, but said that she sees a lot of potential for Spotlight Stories.
Dugan has found at least one believer in Lin. The famed director told me that making Help was “kind of addictive,” and he’s already dreaming to one day shoot an entire feature film as a Spotlight Story. In fact, he explained that he’d love to make another film with Google as soon as he’s done with “Star Trek 3.” Said Lin: “I’m ready to jump back in.”