The Blood and Bone Behind Magic Leap and Its Mixed Reality Goggles

Rony Abovitz was inspired by a little bit The Matrix and a little bit Texas Chainsaw Massacre

The Blood and Bone Behind Magic

To understand the mixed reality goggles that Florida tech company Magic Leap has created and the potential impact it might have on society, it helps to know the journey the company’s founder took from building robotic surgical assistants to rethinking the way humanity perceives reality. Along the way, there was blood and bone, billion dollar backers, rocket scientists, film-makers and comic book creators, and an expensive hunt for the impossible which was one pixel shy of failure.

“First and foremost it was just really cool. I don’t have a better explanation than that. The things that we read about in books, the things that we see in film or television, I wondered, ‘How could that become real?’”

Rony Abovitz seems at home in his office, a sizable cube made up of three glass walls and one constructed of an array of oddly-shaped shelving seemingly custom designed to hold its collection of models, toys, books and other pop culture jetsam. There are two others in the room with us, but after I ask Abovitz about his inspiration, he drops into a monolog that lasts, with few interruptions, about an hour. In it, he tells me a bit of his childhood, his college days, the growth and sale of his first company and finally the technology that empowers everything they do at Magic Leap.

What started Abovitz down his unique path was a love of the creator – of the science fiction, authors, and painters who would later have a tremendous impact on his life. Those creations, he says, put an idea in your head, those ideas become dreams and those dreams inspire a strong desire to figure out how to turn them into reality. “At least, for me,” he says. “I think I grew up watching a lot of cartoons, a lot of science fiction, also reading a lot of great novels from Elie Wiesel and all kinds of people.”

Soaking in the science fiction and fantasy of his youth began to normalize certain tropes of his childhood, he says. Like how there is always a Jiminy Cricket in a Disney movie, or how the droids in Star Wars are a part of that world.

“I think a lot of people who want to do those things, they write books; they do art, they write music,” he says. “That’s where you can do everything you want, you can imagine it and it can happen over there. But I was like, ‘What if that stuff could happen in the world?’”

Lazy loaded image

That early love of science fiction and of things that may one day possibly be is what likely drove Abovitz to study biomedical engineering in college. After working on his Masters, a project that involved the creation of shape-memory aortic implants, Abovitz’ interested shifted to orthopedic surgery and neurosurgery.

“I was imagining this very high-tech thing going on, where there are people in space suits and like infrared stuff. You just wave something over and you automatically have this miniaturized healing process with tiny little droids,” he says. “But it was the opposite of that; it was like Texas Chainsaw Massacre. I literally had blood and bone sprayed all over me. I was in a hip surgery and it was just horrifying.

“I was like, ‘My god this is still in like the early 1900s, with saws and stuff spraying around and Home Depot-like carpentry and eyeballing things, this is the state of the art.’”

His first thought after the surgery was: “This needs robots.”

So he and a team of like-minded friends and colleagues set about building an operating system for the operating room and a robotic arm to help improve surgeries. That first company later became Mako Surgical Corp. While the introduction of robotic arms into orthopedic surgery was a breakthrough, it brought along with it a number of other key inventions and gave Abovitz his first taste of being bringing science fiction into the real world.

The robot arm was licensed from a friend at MIT who had created it to throw baseballs and was looking to design it to clean bathrooms. Mako took that robot and re-engineered it with an incredibly high-speed haptic system.

“Haptics is basically volumetric touch, where you can feel stuff that isn’t there,” he says. “People thought that was the much harder problem than VR.”

The solution allowed surgeons to feel the texture of objects, like the smoothness of glass or the grit of a bone, in a completely real way. The company saw so much success that they were eventually purchased by another company for $1.65 billion.

Lazy loaded image

“If Mako didn’t go public and we didn’t get purchased there probably wouldn’t be a Magic Leap because I needed to be the first investor,” he says.

His experience also helped inspire some of the fundamental ways that Magic Leap would go on to rethink the problems facing mixed reality and his push to bring science fiction into the real world. He remembers sitting in on a procedure and watching people put technology into the brain to help a patient with Parkinson’s.

“This is happening a little before the movie The Matrix or some of it during the Matrix,” he says. “That looked like it is 100 years away, from what I was seeing. We are at the very crude beginning of dealing with how to understand the brain.

“I had that in the back of my mind, something like in The Matrix we can actually visually integrate anything you want to in the brain, that was sort of the end game. Computers and displays are abstractions, but we have the best built-in display.”

The way The Matrix used this advanced integration with the brain and its neurons seems both scary and very far away to Abovitz, but he thought at the time that there must be another way forward that isn’t as primitive as the current thinking in virtual reality.

That way forward, it turned out, was the combination of a spatial-sensing computer with a lightfield-generating pair of goggles. Abovitz calls the system the Magic Leap One.