×

Apple Patents Gesture, Expression Tracking for Mixed Reality Headset

Apple’s upcoming mixed reality headset could include a number of sensors to track the eyes, gestures and even facial expressions of its users. The company applied for a patent to track these kinds of inputs, and combine them with information gathered from outward-facing sensors for mixed reality experiences.

The patent application in question, simply titled “Display System Having Sensors,” was first filed in March of this year, and published last week. It describes in detail plans to use a range of sensors to gather data from the wearer of a mixed reality headset.

Such sensors would make it possible for Apple to more realistically reproduce a user’s facial expression in mixed reality. Apple has already developed facial tracking software for Animoji, the company’s animated AR emoji. Animoji make use of an iPhone’s selfie camera to track facial expressions, and then translate those movements to animation.

The challenge with that approach is that you can’t simply film a user’s face if they’re wearing a headset. That’s why Apple is looking to combine data from separate sensors, including some used for eyebrow and jaw tracking, as well as eye tracking cameras. The latter could also be used for biometric authentication, as the patent application notes. The company also may use cameras for gesture tracking, according to the patent application.

Popular on Variety

Apple has been working on its own headset for a couple of years now. Cnet reported last year that this headset would combine augmented and virtual reality, meaning that it would overlay virtual objects and worlds over a view of the real world.

The patent application filed last week further outlines how the company is likely going to do that: by capturing the real world with outward-facing cameras, and then displaying these images on a display — an approach that’s very different from the way Microsoft’s and Magic Leap’s augmented reality headsets work. From the patent application:

“In some embodiments, the world sensors may include one or more “video see through” cameras (e.g., RGB (visible light) video cameras) that capture high-quality video of the user’s environment that may be used to provide the user with a virtual view of their real environment.”

Apple has yet to publicly comment on its headset plans, including when it intends to ship any such product. In March, Apple analyst Ming-Chi Kuo estimated that the company may start to produce its headset by Q4 of this year, and then publicly introduce it in 2020.

More Digital

  • Vudu

    NBCUniversal in Talks to Buy Walmart's Vudu

    Comcast’s NBCUniversal is looking to add some Vudu into its streaming-video mix. The media conglomerate is in talks to buy Vudu, the Walmart-owned entertainment rental, download and free-streaming service, sources confirm to Variety. It’s unclear what the terms of the pact would be or the timing. News of NBCU’s interest in Vudu was first reported [...]

  • Tubi

    Fox in Talks to Acquire Free-Streaming Service Tubi for Over $500 Million (Report)

    Fox Corp. is in discussions about acquiring Tubi, the ad-supported free streaming service, in a deal worth more than $500 million, according to a Wall Street Journal report citing anonymous sources. With Tubi, Rupert Murdoch’s TV broadcasting and cable company would be adding a dedicated streaming component — offering over 20,000 older TV shows and [...]

  • Zombies 2 Disney Channel

    How Disney Channel's 'Zombies 2' Production Team Made Monsters Kid-Friendly

    Traditionally spooky creatures like zombies and werewolves get the Disney treatment in “Zombies 2,” the follow-up to the 2018 hit TV-movie musical of the same name. While the undead have now assimilated into the community of Seabrook, they’re confronted by a new set of outsiders: werewolves. Milo Manheim and Meg Donnelly return as Zed and [...]

  • Baby Yoda - The Child Animatronic

    Hasbro's Adorable Baby Yoda Animatronic Toy Is Already Sold Out on Disney's Online Store

    The Force remains strong for toys based on Baby Yoda, the breakout star of Disney Plus original series “The Mandalorian.” Less than a day after becoming available for pre-order, Hasbro’s new $59.99 Baby Yoda animatronic toy is no longer available on Disney’s official online store: As of Friday morning, Shop Disney listed it as “sold [...]

  • BTS Leads Spotify’s New Music Friday

    BTS Leads Spotify’s New Music Friday Playlist Rebrand

    Spotify has launched a global rebrand of its popular New Music Friday playlist, which this week features BTS, The Weeknd, Noah Cyrus, Trippie Redd, Kenny Chesney, Rei Ami and others. The revamped playlist, which has 43 versions worldwide, has more than 3.5 million followers in the US and eight million globally. New elements include a [...]

More From Our Brands

Access exclusive content