×
You will be redirected back to your article in seconds

Apple Patents Gesture, Expression Tracking for Mixed Reality Headset

Apple’s upcoming mixed reality headset could include a number of sensors to track the eyes, gestures and even facial expressions of its users. The company applied for a patent to track these kinds of inputs, and combine them with information gathered from outward-facing sensors for mixed reality experiences.

The patent application in question, simply titled “Display System Having Sensors,” was first filed in March of this year, and published last week. It describes in detail plans to use a range of sensors to gather data from the wearer of a mixed reality headset.

Such sensors would make it possible for Apple to more realistically reproduce a user’s facial expression in mixed reality. Apple has already developed facial tracking software for Animoji, the company’s animated AR emoji. Animoji make use of an iPhone’s selfie camera to track facial expressions, and then translate those movements to animation.

The challenge with that approach is that you can’t simply film a user’s face if they’re wearing a headset. That’s why Apple is looking to combine data from separate sensors, including some used for eyebrow and jaw tracking, as well as eye tracking cameras. The latter could also be used for biometric authentication, as the patent application notes. The company also may use cameras for gesture tracking, according to the patent application.

Apple has been working on its own headset for a couple of years now. Cnet reported last year that this headset would combine augmented and virtual reality, meaning that it would overlay virtual objects and worlds over a view of the real world.

The patent application filed last week further outlines how the company is likely going to do that: by capturing the real world with outward-facing cameras, and then displaying these images on a display — an approach that’s very different from the way Microsoft’s and Magic Leap’s augmented reality headsets work. From the patent application:

“In some embodiments, the world sensors may include one or more “video see through” cameras (e.g., RGB (visible light) video cameras) that capture high-quality video of the user’s environment that may be used to provide the user with a virtual view of their real environment.”

Apple has yet to publicly comment on its headset plans, including when it intends to ship any such product. In March, Apple analyst Ming-Chi Kuo estimated that the company may start to produce its headset by Q4 of this year, and then publicly introduce it in 2020.

Popular on Variety

More Digital

  • Illustration of the video streaming company

    Netflix Q3 Preview: The Calm Before the Streaming-Wars Storm?

    Wall Street is expecting Netflix to hit or slightly beat subscriber and financial targets when it reports third quarter 2019 results on Wednesday after market close. But then what? Debate among analysts continues to rage about what will happen to Netflix’s market-leading position once it’s facing big new rivals in Disney Plus and Apple TV [...]

  • countdown

    Apple Takes Down Viral 'Countdown' Horror App, STX Brings It to Android Instead

    An app that can predict the date and time of your death may have been too much for Apple to stomach: The iPhone maker removed a viral app inspired by the upcoming horror movie “Countdown” a few days ago, putting an end to its meteoric rise. Before its removal, “Countdown” had surpassed TikTok, YouTube and [...]

  • daydream view

    Google Ships Pixel 4 Without Daydream VR Support, Stops Selling Daydream Viewer

    Google is effectively phasing out its Daydream virtual reality (VR) platform: The company’s latest flagship Pixel 4 flagship phone, which Google unveiled at a press event in New York Tuesday, won’t support Google’s Daydream mobile VR platform anymore, a spokesperson confirmed to Variety. The company will also stop selling the Daydream viewer, but continue to [...]

  • Google Wifi

    The New Google Wifi Ships in Three Colors Because Good-Looking Routers Work Better

    When Google revamped its Wifi router product, it extended the devices reach and doubled their speed, while also adding a full-blown smart speaker with Google Assistant. But the company paid just as much attention to Google Wifi’s design, down to the decision to ship the product in three colors. The surprising reason for this: Google [...]

  • Nest Mini

    Google Introduces New $49 Nest Mini Speaker With On-Board Machine Learning, Stereo Pairing

    Google introduced a revamped version of its entry-level smart speaker at a press event in New York Tuesday: The new $49 Nest Mini speaker effectively replaces its Home Mini predecessor with bigger sound, a built-in machine learning chip for faster responses, ultrasound for proximity detection and the ability to pair 2 speakers for inexpensive stereo [...]

  • Google Stadia Game Streaming Service Launching

    Google’s Stadia Game Streaming Service Will Launch Nov. 19

    Google hardware chief Rick Osterloh gave us an update on the company’s game streaming service Stadia during the company’s fall hardware event in New York Tuesday: Stadia will be available to the public on Nov. 19. Stadia promises to stream games directly from the cloud, with no need to buy a full-blown game console. Consumers [...]

More From Our Brands

Access exclusive content