×

Apple Patents Gesture, Expression Tracking for Mixed Reality Headset

Apple’s upcoming mixed reality headset could include a number of sensors to track the eyes, gestures and even facial expressions of its users. The company applied for a patent to track these kinds of inputs, and combine them with information gathered from outward-facing sensors for mixed reality experiences.

The patent application in question, simply titled “Display System Having Sensors,” was first filed in March of this year, and published last week. It describes in detail plans to use a range of sensors to gather data from the wearer of a mixed reality headset.

Such sensors would make it possible for Apple to more realistically reproduce a user’s facial expression in mixed reality. Apple has already developed facial tracking software for Animoji, the company’s animated AR emoji. Animoji make use of an iPhone’s selfie camera to track facial expressions, and then translate those movements to animation.

The challenge with that approach is that you can’t simply film a user’s face if they’re wearing a headset. That’s why Apple is looking to combine data from separate sensors, including some used for eyebrow and jaw tracking, as well as eye tracking cameras. The latter could also be used for biometric authentication, as the patent application notes. The company also may use cameras for gesture tracking, according to the patent application.

Apple has been working on its own headset for a couple of years now. Cnet reported last year that this headset would combine augmented and virtual reality, meaning that it would overlay virtual objects and worlds over a view of the real world.

The patent application filed last week further outlines how the company is likely going to do that: by capturing the real world with outward-facing cameras, and then displaying these images on a display — an approach that’s very different from the way Microsoft’s and Magic Leap’s augmented reality headsets work. From the patent application:

“In some embodiments, the world sensors may include one or more “video see through” cameras (e.g., RGB (visible light) video cameras) that capture high-quality video of the user’s environment that may be used to provide the user with a virtual view of their real environment.”

Apple has yet to publicly comment on its headset plans, including when it intends to ship any such product. In March, Apple analyst Ming-Chi Kuo estimated that the company may start to produce its headset by Q4 of this year, and then publicly introduce it in 2020.

Popular on Variety

More Digital

  • Moviepass

    MoviePass Confirms Security Issue With Customer Records

    MoviePass, the struggling movie ticket subscription service, has confirmed a security issue may have exposed customers’ records. In a statement, MoviePass said Wednesday that the security lapse was recently discovered and its system was immediately secured. Reports of the data breach first surfaced Tuesday through the Tech Crunch site, which alleged that tens of thousands [...]

  • Tycho weather app

    This Website Creates Spotify & Apple Music Playlists Based on Your Local Weather

    Electronica musician Tycho launched a clever promotional website for his new album “Weather” this week: Visitors of Tycho’s website can use a web app to generate a playlist based on their local weather. Playlists can be saved to both Spotify and Apple Music, and consist of 25 songs, both from Tycho’s catalog as well as [...]

  • AT&T TV

    AT&T TV: An Unskinny Streaming Bundle That Looks a Lot Like Traditional Pay TV

    AT&T this week launched AT&T TV, a new subscription streaming television service that uses an Android-based internet set-top, in 10 markets. But the way it’s priced and packaged looks very similar to cable and satellite TV services — in other words, AT&T TV isn’t targeted at the cord-cutter crowd. It’s basically designed as a way [...]

  • Sofia Wylie arrives at Variety's Power

    Disney Channel Enlists Duplass Brothers for YouTube Hip-Hop Dance Series Starring Sofia Wylie

    Disney Channel’s first original scripted series made exclusively for release on YouTube is “Shook,” a hip-hop dance short-form series starring Sofia Wylie, produced by Mark Duplass and Jay Duplass. “Shook” will debut Saturday, Sept. 28, on Disney Channel’s YouTube channel. The scripted single-camera show centers on 15-year-old Mia (Wylie), who yearns to dance professionally but [...]

  • YouTube logo

    YouTube Is Getting Rid of Messaging Feature

    YouTube is killing off a feature that allowed users to share videos and other messages with each other. YouTube Messages, which was first introduced in January of 2017, will be turned off by September 18, the company announced this week. “We’re constantly reevaluating our priorities and have decided to discontinue YouTube’s native direct messaging feature [...]

  • Yara Shahidi

    Audible Teams Up With Big Names as Audio-Only Storytelling Expands

    Hobnobbing with the likes of Reese Witherspoon, Kate McKinnon and Yara Shahidi isn’t typically the gloss applied to the cerebral audiobook set, but Audible is putting the humble audio story through its own movie-makeover montage, inking high-profile development deals with entertainment names like “The Walking Dead” comics creator Skybound Entertainment, Witherspoon’s Hello Sunshine and Lorne [...]

  • Geneva Wasserman

    Condé Nast Taps Film Veteran Geneva Wasserman as SVP of Motion Pictures

    Condé Nast Entertainment hired Geneva Wasserman as senior vice president of motion pictures, overseeing development of the media company’s slate of feature film properties. Wasserman, a nearly 20-year veteran of the entertainment industry, most recently served as co-founder and executive producer of production firm Project Z Entertainment. She takes over the role at CNE after [...]

More From Our Brands

Access exclusive content