At the Royal Shakespeare Company in Stratford upon Avon, Sarah Ellis has the difficult job of figuring out where theater of the 1500s fits into the 21st century. As Director of Digital Development, a title which might seem out of place in an industry ruled by live, human performances, Ellis represents a recent seachange on the hallowed grounds of the RSC, to incorporate new virtual and augmented technologies with the work of a centuries-old playwright.
This spring, the RSC released its latest project made in partnership with creative technology company Magic Leap – a re-enactment of Shakespeare’s Seven Ages of Man from “As You Like It,” a soliloquy whose famous line “All the world’s a stage” encapsulates the company’s intentions for the piece. The RSC refers to this as “Tabletop Theatre,” pint-sized productions which can be staged on any surface.
“We wanted to start with a scene from Shakespeare and use it as a statement of intent for a wider ambition in this space,” Ellis tells “Variety.” “It’s a starting point for us to explore how we make theatre with the new technologies found in gaming and other industries, such as motion capture, photogrammetry, and volumetric capture.”
“Seven Ages of Man” begins after pulling on the Magic Leap One augmented reality headset, then unfolds on a small pedestal-size stage in front of you. The three-minute monologue is performed by Shakespearean actor Robert Gilbert, miniaturized for the size of the tiny set but still pacing the scene with the intensity of a live performer. Gilbert’s performance was initially recorded using volumetric capture, a process that takes video and creates a digital, three-dimensional space from it, making it possible for a live staged performance to be translated into augmented reality and played anywhere. Behind the actor grows a large tree, which ages as the performance pushes on, turning from a spring-time green to autumnal red. Leaves start to fall off its branches, leaving the tree barren as the monologue comes to an end.
“Theatre and technology have always had a relationship,” says Ellis. “And so this technology adds to our theatre-making tool kit and extends the possibilities for performance through the tools we have today. The experience of telling a story has always been through the best technology there is, which is the live actor. What’s important to us is how these new technologies add to that performance, take the storytelling further, give new options for directors and designers, allow us to shape, refine and redefine the experiences for our audiences is key.”
“What we’ve found with mixed reality is that it offers a shared experience whilst having a digital world presented to you through the technology. This allows us to think about the real world and the digital world simultaneously and bring those realities together. We’ve found audiences focus on the performance but also look at each other through the device and enjoy that collective and connected experience.”
The RSC first began exploring this space in collaboration with Intel for the 2017 production of “The Tempest,” which featured a full sensor-laden motion-capture performance for the play’s spirit-servant Ariel. Two years later, mixed reality entertainment is becoming more prevalent in the mainstream as the technology becomes more widely available.
“Over the past decades, we’ve seen our storytelling industries work in silos,” says Ellis of the often-dueling creative industries of cinema, television, theater, music and the Internet. “Audiences over time have been crossing these silos and defining their own parameters for content with their own storytelling channels such as YouTube, social media, online platforms such as Netflix and iPlayer for Broadcast, and now “Fortnite” has entered the entertainment space with their latest Marshmallo performance. These are the spaces audiences congregate and it’s how we position theatre in those spaces.”
Senior Director of Developer & Creator Relations at Magic Leap Greg Rinaldi imagines a future where all entertainment is consumed through mixed reality. No screens, no controllers, no mediation beyond a headset.
“I’m a huge sci-fi nerd,” he says. “I think that’s why I joined the company.”
Prior to joining Magic Leap, Rinaldi worked in the video games industry with Electronic Arts creating massively multiplayer games, enormous online worlds for thousands of players to congregate. But after 15 years in a landscape mediated by controllers and screens, he now views these as impediments to virtual entertainment.
“Most of us experience the world with our faces in our phones. I don’t want to be doing this for the next 20 years. That doesn’t have to be the way I interact with the digital world. Our hope is that as the technology evolves, as it gets smaller and more powerful, that it will be natural to just wear a pair of glasses or interact with some new visual medium that gets created.”
“What we’re trying to work out here is how we make the interaction more natural where there isn’t a steep learning curve.”
Mixed reality, it should be said, isn’t exactly like its virtual counterpart. For one, there are technical differences. Unlike VR which uses digital stereo images to create its virtual worlds, the Magic Leap One harnesses “digital light fields” – a technique that provides the wearer with depth cues similar to those generated by objects in the real world. But perhaps the biggest difference is philosophical. VR envelopes you in another world; XR injects virtuality into what’s already really there.
Since its founding in 2010, Magic Leap has led a somewhat mysterious existence. The technology company is the brainchild of Rony Abovitz, a bioengineer involved in the creation of Mako Surgical Corp.’s surgery-assisting robot arms, and boasts staff members like Science-fiction author Neal Stephenson who maintains the curious job title of “Chief Futurist”. The technology company has also found success among the investor class, going on to rake in an estimated $1.4 billion in investments from the likes of Google and J.P. Morgan.
The design for Magic Leap One has evolved wildly throughout that period. Early on, during a time in the development when the team was referring to the device as The Beast, the Magic Leap One was the size of an industrial refrigerator and required users to stick their heads inside of it as if having their eyes checked by a monolithic optometrist. But the fundamental technology has always remained the same, says Rinaldi.
“I think VR is a fantastic technology frankly. It’s just fundamentally different. It’s about escapism,” says Greg Rinaldi, head of Magic Leap. “You can create really rich worlds in VR.”
“In virtual reality, you can create the space. You have complete control as the creator to make this other world. But with mixed reality, you have to respect the space that you’re in and you have to really understand how the digital content is interacting with the space, and then how the viewers are interacting with that space.”
“I think the reason we’re so excited about mixed reality is because you don’t have to remove people from the world,” he continues. “But I wouldn’t consider them competing technologies, they’re just different immersive mediums. It’s like film and radio almost.”
“What our technology allows you to do is we can map your space with sensors and cameras. We can create a three dimensional model of the room in real time. So the character will understand when they’re moving through the space: there is a table here, I can’t walk through the table, I have to stay on this side of the table, there’s a chair here, I can sit in the chair.”
“Storytelling over the next 10 to 20 years will be defined by the events of the day, what the generation before us has to say, and Shakespeare plays will continue to be entwined in society’s discourse of the day,” says Ellis.
“Over the next ten years, we as industries, sectors, genres will respond to audiences and their new meeting spaces. And we will see those stages converge, and the future of entertainment will be less siloed or defined by a single platform or experience. Content will become distributed, circulated and curated around the spectacle of an event. Theatre will hold this space as strongly as it always had, and it will be how theatre holds the multiple platforms and stages that are opening up to it. What will be important is to remember that with this opening up comes how we curate and architect those stories not forgetting the rituals that have been so intrinsic to how we experience a story for centuries.”
In January, the RSC announced the Audiences of the Future program, which explores the future of live performance with Marshmallow Laser Feast, Magic Leap, Intel, Epic Games, and others. The RSC launched its next exploration into theater and spatial computer in March, with the addition of six fellowships with the University of Portsmouth, the University of Goldsmiths, and i2Media Research.
“I think the possibilities for the Magic Leap device coming into how we make theatre are many, and we are currently exploring those avenues together,” Ellis says. “Exploring this collaboratively gives us an opportunity to imagine what a future production or performance could be whilst thinking about the many ways audiences now connect with content and storytelling. How theatre can exist in those spaces and also how can theatre drive some of the innovation in these areas.”
“As theatre-makers, we’ve been working in virtual reality for centuries so you could argue that the technology has caught up.”