You will be redirected back to your article in seconds

Inventing the Future Through Serious Games

Adam Berenzweig considers the gesture recognition technology of “Minority Report” an ergonomic disaster. The constant whipping around of arms. Fingers that strum and tap at the air as if you’re conducting an orchestra. For all the cool gadgets that Hollywood science fiction can produce, the reality is they’re usually just impractical and uncomfortable.

“People have known this for a long time,” says Berenzweig. “The story is, when they were filming ‘Minority Report,’ Tom Cruise actually had to stop filming that scene a bunch of times because his arms were getting tired holding his hands up for so long.”

As director of research and development at CTRL-labs, a New York-based start-up that aims to create a world where computers are natural extensions of thought and movement, Berenzweig is working to build the actual controllers of the future. The team’s prototype technology, described during its announcement last year as a way of converting biological neurons into computational neurons, is called the CTRL-kit. It’s a wearable wrist-worn arm band fitted with a series of electrodes.

With this device, CTRL-labs is harnessing human intention. It’s an ambitious goal but with the CTRL-kit, Berenzweig can already win a game of “Asteroids” without ever touching a controller.

If he wants to steer the retro game’s tiny 2D ship and bypass a dozen or so galactic rocks, he can do so with the speed and effectiveness of an arcade joystick while sat seemingly motionless in front of the laptop on which it has been refitted. When he fires the ship’s cylinders or shields himself from the blast of asteroids, he is motionless albeit for a very slight twitch of his hand which lays otherwise flat on a table, fingers stretching outward. He can play it with his arms folded or in his pockets. You’d be forgiven for thinking of it as a Jedi parlor trick.

“It’s super fun,” Berenzweig says of the “Asteroids” experiment. “We found it was definitely possible to [play the game] without moving your hands. And in fact, it was in some ways better, more comfortable. And you could just do these tiny little wiggles that are imperceptible who’s standing right next to you, but for you, it still feels very clearly like what you’re doing: I’m going to press my hand this way and my ship is going to go to the right.”

The CTRL-kit uses EMG or Electromyography, a method of recording the electrical activity produced by skeletal muscles. It’s an old technique. First developed in the 19th century when early neuroscientists working to understand how the human neurosystem, the EMG device reads bursts of electrical activity like zeroes and ones as they emerge from the wearer’s motor cortex, down their spinal cord, through their muscle fibers, and to their hands and fingers.

“Imagine your arm as a pipe of information from your brain to the world,” Berenzweig explains. “Your motor nervous system is the way that any animal affects the world. Control, in the context of what we’re building, is the process of transforming intent in the mind into action in the world. You have some desire, you want something to happen, you want to change something. And the only way you have to effect some change in the world is moving some muscles.”

It’s a technique you might recognize as being similar to the EEG or Electroencephalogram, which also reads the electrical output of neurons. Experiments with EEG record this activity using an electrode-covered cap worn on your head to take readings from the scalp. But this can be picked up elsewhere on the body, says Berenzweig, and this way offers an even louder and clearer signal.

“When [the signal] enters the muscle, it then is amplified by the muscle because there are several if not dozens or hundreds of muscle fibers attached to that one nerve and they all act in concert. It’s a bigger event. We’re looking at events that are in the millivolt range as opposed to a single neuron in the brain which would be like microvolts. Not to mention the fact that there’s just fewer of them to worry about in the arm than the billions of neurons in the brain that are all on top of each other that you’re trying to pick apart through the skull.”

“So what we’re doing on one level is tapping that pipe of information and shortcutting it so that we can get all of that information about what your brain is trying to do directly and then plug that directly into the input of any kind of technology.”

The result is a way of gaming without the need for a bulky controller or external cameras or even physical movement at all. It’s a strange idea, but it offers a glimpse of how our relationship with technology is likely to transform in the coming years.

This future is being designed not in Hollywood but in labs on the edges of academia. Serious Games – a field that studies the potential for games beyond play – have been slowly, quietly building to a crest over the past decade. In 1999, the U.S. Army founded the USV Institute for Creative Technologies Mixed Reality Lab with a $45 million contract. Last year’s Global Game-Based Learning Market report forecast the serious games industry will reach over $17 billion by 2023. This fall the University of Santa Cruz is introducing its first Master’s Degree program in Serious Games.

“The lions share of development funding has gone towards gaming for entertainment,” says Todd Richmond, Director at the Tech & Narrative Lab at Pardee RAND and member of IEEE. “[But] there has been a parallel effort on using games for training and education – so-called ‘serious games.’”

At the Mixed Reality Lab, where Richmond is also a director, his team began exploring user intent in a project known as ESP or Early Synthetic Prototyping. In ESP, Richmond and his team studied members of the military as they performed exercises in a virtual world, experiments that put players into a multiplayer match to search for specific target objects. The ultimate aim, for Richmond, is to accurately reflect humanity in technology – what he calls being an analog soul in a digital world.

“Current digital games typically log events during the game,” says Richmond. “The player clicked on this, they moved here, they swung their sword there. Game companies, who are in the business of making money by keeping people playing, evolved the game engines and applications to collect data that would help them understand what players were doing, and how they might keep them playing and develop new capabilities that would further expand the player engagement.”

“Entertainment games have one main goal: keep players playing. Thus a game designer has to create an application that engages and entertains,” he continues. “Commercial developers want to know what a player does so that they can design better and more engaging experiences. For serious game development, knowing what is insufficient.”

“For the Army, training is a significant investment in time and effort and is ongoing throughout a Soldier’s career. When a person is performing tasks, there typically is a ‘right’ and a ‘wrong’ way to perform said task. Current simulations and training are designed to understand what a person does, and sometimes how they do it, but not why. As a trainer, if I don’t understand why someone made a mistake (or got something right), remediation becomes formulaic and might not be effective – you can be treating a symptom instead of an underlying cause.”

Richmond’s ESP program uses an eye tracker to track where on the screen the user is looking at any given time. A webcam is trained on their face as they’re playing, which is also recorded and analyzed using machine learning and facial recognition. In some experiments, players wore an EEG electrode-covered cap, which would track when the user had seen and recognized the object they were looking for. The match is recorded and then analyzed using software that breaks down every action performed. Players appear on the screen with lines sprouting out from them, representing their next move, and the next one, and the next.

In return, the team was able to gather surprisingly specific information, ranging from “smile level” and “speech rate” to numerical values that are used to estimate the player’s emotion as angry, happy, or surprised, positive or negative.

The result is a virtual world of cause-and-effect. When a vehicle collides into a wall and rolls into the path of bullets, the team rewinds to unknot the events that occurred in the seconds and minutes beforehand which may have influenced events. If the players went left, should they have gone right instead? Did they double back early on when searching for their target object? Were they looking the wrong way or talking? Were they potentially angry or surprised? Were they smiling?

“Tacit knowledge often is what drives our behaviors. What we’re trying to glean is some of what is going on unconsciously, likely through biometrics, and in the future, through EEG and other brain sensing. If someone turned left instead of right, why did they make that choice?”

“We know the data isn’t particularly accurate at this point. But we also know that these types of computer vision systems and algorithms, particularly when combined with Machine Learning techniques, will become very accurate and reproducible. So part of our work was to think about how one would engage that information once it does become better.”

Over the next decade, the intersection between technology and human intention will continue to be blurred. The implications for this are broad. Perhaps more than just enabling a world without buttons, the technology of the future will be an extension of us, bypassing our slow and error-prone muscles and opening up the doors for pure expression. In other words, says Berenzweig, to control technology with the adeptness of a musician playing an instrument.

“Really everything we do, all the technology we use, should be as expressive and fluid and natural as playing an instrument. Musical instruments are in some ways the most evolved controllers that humans have built as far as expressivity capturing the nuance of what humans can do with the output of their brain.”

“If we do what’s right, what we’re building could be an amazing musical instrument.”

More Gaming

  • Raising Kratos Documents the Making of

    'Raising Kratos' Documents the Making of 'God of War'

    An upcoming documentary, “Raising Kratos” looks at the five year long effort that was the making of “God of War,” PlayStation revealed via an announce trailer on its YouTube channel on Saturday. The documentary reveal comes just after the one year anniversary of “God of War”. “Facing an unknown future, Santa Monica Studio took a [...]

  • “Mortal Kombat 11” game review

    Video Game Review: "Mortal Kombat 11"

    “Mortal Kombat 11” is like a slasher film, a bloody mess that evokes emotion in the same ways that the horror genre does—specifically, the more campy horror flicks. Developed by NetherRealm Studios, “Mortal Kombat 11” doesn’t try to scare players with jump scares or psychological horror, but is instead blatant in its unrestrained gore. NetherRealm [...]

  • Arc System Works and WayForward Revive

    Arc System Works and WayForward Revive Classic 'River City' IP

    Notable fighting game developer Arc System Works is teaming up with “Shantae and the Pirate’s Curse” developer WayForward to create a brand-new game based on the classic “River City” intellectual property, they announced on Friday. The new project is called “River City Girls.” While the two studios won’t release full details about the project until [...]

  • Panthers Schedule Promo References a Slew

    Panthers Schedule Promo References a Slew of Video Games

    The Carolina Panthers’ released its 2019 schedule promotion video on Wednesday, which makes callbacks to classic games, like “Pitfall” and “Oregon Trail” among other games, shared via YouTube. The Carolina Panthers are a professional football team in the National Football League (NFL). The team is based in Charlotte, North Carolina. The promo video shows various [...]

  • Gearbox CEO Defends Epic Games Store

    Gearbox CEO Defends Epic Games Store and 'Borderlands 3' Exclusivity

    Earlier this month, developer Gearbox Software revealed the PC version of its anticipated co-op shooter “Borderlands 3” is coming exclusively to the Epic Games Store. This created some blowback from fans who prefer Valve’s digital storefront Steam. Gearbox founder and CEO Randy Pitchford defending the decision in a lengthy Twitter thread on Saturday (helpfully recreated [...]

  • Inside the Music of Netflix's 'Ingress:

    Inside the Music of Netflix's 'Ingress: The Animation' (EXCLUSIVE)

    “Ingress: The Animation” hit last fall in Japan, but the global release doesn’t hit Netflix until April 30, fortunately, it’s bringing a little something new to the show based on Niantic’s other popular video game. The global version of the show will feature a new musical score created by Jacob Yoffee (“Free Meek,” “Andi Mack”). [...]

  • Vic Mignogna

    Accused of Sexual Harassment, Vic Mignogna Sues Funimation

    Vic Mignogna, known for voicing Broly in the “Dragon Ball” series of games, films, and tv shows, filed a lawsuit claiming defamation and other charges against Funimation and other voice actors on Thursday, according to public court documents. Fellow Funimation voice actors Monica Rial and Jamie Marchi, along with Rial’s fiance Ronald Toye, are also [...]

More From Our Brands

Access exclusive content