×
You will be redirected back to your article in seconds

Apple Patents Gesture, Expression Tracking for Mixed Reality Headset

Apple’s upcoming mixed reality headset could include a number of sensors to track the eyes, gestures and even facial expressions of its users. The company applied for a patent to track these kinds of inputs, and combine them with information gathered from outward-facing sensors for mixed reality experiences.

The patent application in question, simply titled “Display System Having Sensors,” was first filed in March of this year, and published last week. It describes in detail plans to use a range of sensors to gather data from the wearer of a mixed reality headset.

Such sensors would make it possible for Apple to more realistically reproduce a user’s facial expression in mixed reality. Apple has already developed facial tracking software for Animoji, the company’s animated AR emoji. Animoji make use of an iPhone’s selfie camera to track facial expressions, and then translate those movements to animation.

The challenge with that approach is that you can’t simply film a user’s face if they’re wearing a headset. That’s why Apple is looking to combine data from separate sensors, including some used for eyebrow and jaw tracking, as well as eye tracking cameras. The latter could also be used for biometric authentication, as the patent application notes. The company also may use cameras for gesture tracking, according to the patent application.

Popular on Variety

Apple has been working on its own headset for a couple of years now. Cnet reported last year that this headset would combine augmented and virtual reality, meaning that it would overlay virtual objects and worlds over a view of the real world.

The patent application filed last week further outlines how the company is likely going to do that: by capturing the real world with outward-facing cameras, and then displaying these images on a display — an approach that’s very different from the way Microsoft’s and Magic Leap’s augmented reality headsets work. From the patent application:

“In some embodiments, the world sensors may include one or more “video see through” cameras (e.g., RGB (visible light) video cameras) that capture high-quality video of the user’s environment that may be used to provide the user with a virtual view of their real environment.”

Apple has yet to publicly comment on its headset plans, including when it intends to ship any such product. In March, Apple analyst Ming-Chi Kuo estimated that the company may start to produce its headset by Q4 of this year, and then publicly introduce it in 2020.

More Digital

  • Sunil Doshi

    ATF: Sunil Doshi Launches Content Development Platform All Things Small (EXCLUSIVE)

    Sunil Doshi, whose company Alliance Media & Entertainment produced 2017 Tokyo film festival title “Bioscopewala” and distributed Jafar Panahi’s 2018 film “3 Faces” in India, has launched a new content development platform at the ongoing Asian TV Forum & Market in Singapore. The Mumbai-based All Things Small (ATS) is founded by Doshi, journalists Sidin Vadukut, [...]

  • YouTube-Rewind-2019

    YouTube Rewind 2019 Scraps Old Format After Last Year's Torrent of Dislikes

    YouTube has taken a brand-new approach with this year’s YouTube Rewind mashup video — dropping its tradition of highly produced original videos segments. Instead, the 2019 version is a montage of what the platform says were the biggest moments and milestones from around the world on YouTube. The move came after the 2018 YouTube Rewind [...]

  • PewDiePie

    PewDiePie Is the Most-Watched YouTube Creator of 2019 With 4 Billion Views

    PewDiePie’s Bro Army remains stronger than ever: The top YouTuber is the most-viewed creator of 2019 on the platform year-to-date, with more than 4 billion views this year, according to YouTube. The comedian, game vlogger and meme aficionado, whose real name is Felix Kjellberg, is also the most-subscribed individual YouTube creator, currently with 102 million [...]

  • Variety Summit Examines Streaming Options and

    Variety Summit Examines Streaming Options and Innovation

    Analytics, monetization and the expanding landscape of entertainment will be the focus of Variety’s Innovate Summit: Discovery & Data Science in Media on Dec. 5 in West Hollywood. The event is presented by PwC. The summit will feature 32 speakers, each considered industry leaders in the emerging world of streaming. Each panel will speak to [...]

  • Talas World Xandra

    Amazon Alexa Gets First Interactive Audio Original, Courtesy of Xandra (EXCLUSIVE)

    First there was “Bandersnatch,” now there is “Tala’s World.” Interactive media is making the jump to smart speakers, starting with a new serialized young adult adventure series produced by audio startup Xandra, and licensed exclusively by Amazon. “Tala’s World” consists of seven episodes, which invite listeners to come along on an audio adventure that involves [...]

  • Billie Eilish Dazzles Crowd With Intimate

    Billie Eilish Dazzles Crowd With Intimate Acoustic Set at Apple Headquarters

    “This is really insane. Nothing like this has ever been done on this stage.” Billie Eilish was addressing the packed crowd midway through her set at the Steve Jobs Theater at Apple Park in California’s Silicon Valley, marking the first time an artist has given a public concert at the iconic home to Apple’s keynote [...]

  • James-Charles-YouTube

    YouTube Reveals 2019 Top-Trending Videos, Most-Viewed Music Videos

    YouTube is out with its year-in-review lists for 2019, documenting the top videos, trends, creators, music, games and more. Leading the list of U.S. top-trending videos was beauty influencer James Charles’ “No More Lies” (pictured above), his video response to accusations of betrayal and misbehavior by vlogger Tati Westbrook. After the controversy blew up in [...]

More From Our Brands

Access exclusive content