Netflix is obsessed with testing. The video service did more than 150 so-called A/B tests last year, showing different user interfaces and experiences to a small subset of its subscriber base, and then measuring their response. But Netflix also realized that everyday testing can significantly increase video viewing as well.
Case in point: Those images you are seeing on Netflix’s website and within its apps for movies and TV shows? They’re fine-tuned with testing to make you watch more.
Netflix designers and engineers realized a few years ago that Hollywood’s traditional box art wasn’t really the best way to advertise the videos in its catalog. “We have 30 seconds, 60 seconds or 90 seconds to capture your interest,” said Netflix chief product officer Neil Hunt in an interview with Variety at CES in Las Vegas this week. If consumers don’t find something to watch during those first 90 seconds, they may just close the app altogether and decide to do something else.
That’s why the company introduced a new user interface in 2013 that replaced DVD box cover art with wider, more expressive imagery taken from both studio assets as well as still frames from movies and TV shows. These new images are being used as tiles to list titles, as well as big background images to tease their content — a change that already led to a lot more interaction with the app.
But more recently, Netflix went even further. The company now prepares a half dozen images for each and every video, and tests these images themselves with a subset of its user base. The winners are then used as imagery for all of Netflix’s members. It’s a simple trick, but one with a massive impact: Video viewing increases between 20 and 30 percent for titles with images that have been chosen through these kinds of tests, Hunt said.
However, tests don’t always have that big of an impact. One other area that Netflix’s engineers have been spending a lot of time on is personalization. As part of these efforts, the company has also been exploring ways to respond to timing and context. The assumption: You just don’t watch the same things while eating breakfast as during a Saturday movie night.
To test this assumption, Netflix sliced up the day of some of its subscribers into half-hour chunks, analyzing their viewing behavior during each of those small time slots and trying to serve them relevant recommendations based on that behavior. This was done on a per-household basis to make sure that someone working a night shift gets a different experience than someone going to bed at 10 p.m. every night.
But in the end, all of that didn’t matter: Timing recommendations this way just didn’t move the needle. Not enough anyway to warrant a close to 50 time increase in computation to generate recommendations that change every half hour, explained Hunt. He attributed this to the fact that we’ve all been trained by Netflix to watch a certain way, no matter what time it is. “So much of Netflix is binging on a series,” Hunt said, adding: “We don’t need magic for that.”