Inside Netflix’s Plan to Boost Streaming Quality and Unclog the Internet (Exclusive)

Netflix
Twin Design / Shutterstock

The company is on a quest to re-encode its entire catalog

Over the past few months, Netflix has dared some of its employees in its Los Gatos offices with a special kind of challenge: Two TVs mounted side-by-side were playing the same TV show episode. One was coming straight from Netflix’s existing service, the other was based on a new bandwidth-saving technology that the company has been working on for four years. Anyone capable of pointing out the difference could win a bottle of champagne. But in the end, even eagle-eyed employees had to give up, and the prize went unclaimed.

Encouraged by these results, as well as months of additional testing, Netflix now has begun to embark on one of the biggest changes to its streaming technology since it launched its online video service in 2007. If all goes according to plan, the switch could help consumers get better-looking streams while also saving up to 20 percent of data — which is significant in North America, where Netflix usage single-handedly accounts for more than a third of all data consumed during peak times, and an even bigger deal in all those countries with relatively slow internet speeds that the company is looking to enter in 2016.

‘My Little Pony’ Is Not the Same as ‘The Avengers

Netflix has been working on this new technology since 2011, when members of its video algorithms team realized that they had gotten it all wrong. Like practically everyone else in the online video world, Netflix had been preparing its video files for streaming based on the bandwidth available to consumers. Some Netflix subscribers were accessing the service with slow DSL connections, others had faster cable connections, and a lucky few were already online with super-fast fiber speeds.

Based on these use cases, Netflix’s video algorithms team had developed a number of quality levels, or recipes, as they’re called in the world of video encoding. Each video file on Netflix’s servers was being prepared with these same recipes to make multiple versions necessary to serve users at different speeds. At the lowest end was a file encoded with a bitrate of 235 kbps, which would work even on very slow connections, but also only deliver a resolution of 320 by 240 pixels. Somewhere in the middle was a 1750 kbps file for a resolution of 1280 by 720, and the best quality was a 5800 kbps version for a great-looking 1080p experience.

Netflix’s service has been dynamically delivering these versions based on a consumer’s bandwidth needs, which is why the quality of a stream occasionally shifts in the middle of a binge-watching session. But across its entire catalog of movies and TV shows, the company has been using the same rules — which didn’t really make sense. “You shouldn’t allocate the same amount of bits for ‘My Little Pony’ as for ‘The Avengers,’” explained Netflix video algorithms manager Anne Aaron.

To a computer in charge of encoding, ‘My Little Pony’ just isn’t that complex.
Courtesy of Hasbro / Netflix

That’s because a show like “My Little Pony” is not at all like the ‘Avengers’ — especially to a computer in charge of preparing media files for streaming. Animated shows like “My Little Pony” can be reproduced with relatively little data. An animated sky, for example, tends to be filled with the same shade of blue, and an animated pony that just stands around talking isn’t very complex to a computer either. The “Avengers” on the other hand is full of fast-paced action, which plays out in front of cityscapes and other environments with lots of visual details. And, of course, explosions, rubble, smoke, and lots of it. Or as video engineers like to call it: noise. “Noise is hard to encode,” said Aaron.

It’s a much more complex movie, at least from a purely visual perspective. This means that Netflix’s encoding servers have to work a lot harder to get it down to the same file sizes as they do with “My Little Pony,” leading to more compression and possibly more visible artifacts at low bitrates. That’s why in 2011, Netflix’s video algorithm engineers realized that they shouldn’t apply the same encoding rules to these two very different titles. “A one-size-fits-all model doesn’t give you the most optimal quality,” said Aaron.

Instead, they decided that each title should get its own set of rules. This allows the company to stream visually simple videos like “My Little Pony” in a 1080p resolution with a bitrate of just 1.5 Mbps. In other words: Even someone with a very slow broadband or mobile internet connection can watch the animated show in full HD quality under the new approach. Previously, the same consumer would have just been able to watch the show with a resolution of 720*480, and still used more data.

But Netflix’s new per title approach doesn’t just improve things for animated TV shows. Fans of other fare are also set to benefit, and see significant bandwidth savings when streaming the highest quality 1080p video. Netflix’s video algorithms team set up a test for Variety in its offices in Los Gatos last week, streaming two episodes of “Orange is the New Black” in 1080p on two TVs mounted side-by-side, similar to the tests the company has been doing with its employees. The images on both TVs looked virtually identical — but one streamed with 5800 kbps, using Netflix’s legacy encoding scheme, whereas the other displayed the show with 4640 kbps. The difference? 20 percent in bandwidth savings.

Each episode of a TV show can have different settings

Over the past couple of years, Netflix’s video algorithm engineers have worked on perfecting this more flexible approach towards video encoding. Initially, they assumed that they would just have to categorize all of their titles to develop new encoding rules for animated shows, action movies, slow indie dramas and so forth. But they quickly realized that no two indie movies are alike, and that there can even be considerable visual differences within a season of the same TV show. “Each episode could be very different,” said Aaron. The result is a true title-by-title approach, where every single movie and TV show episode gets its own encoding settings.

SEE MORE: Why Netflix Is Pushing ‘Breaking Bad,’ ‘House of Cards’ in 4K Ultra HD

In recent months, Netflix has been silently testing the new encoding scheme by sending out streams with new and improved bitrates and resolutions to randomly selected customers. The company did keep a close eye on completion rates and streaming duration, but it also wanted to make sure that all devices out in the field were able to deal with the new bitrates.

Separately, Netflix has been testing the new and improved videos in person both with its own employees as well as with a number of customers. And earlier this month, Netflix quietly added a first batch of videos encoded with the new per-title approach to its catalog. Beginning with some of the most popular videos, the company aims to have a thousand re-encoded titles in its catalog by the holidays, and complete the entire process by the end of Q1 2016.

Movies are re-encoded while you’re asleep

This whole endeavor couldn’t have been possible without some considerable technical advances. First of all, Netflix needed a good way to analyze the visual quality of a video file when encoded with different settings. The company doesn’t really talk about the size of its catalog, but it’s fair to assume that the entirety of its catalog, including all TV show episodes, movies and trailers across all of its markets around the world is in the six digits, and new titles are constantly added.

It’s impossible for Netflix’s employees to visually inspect every single one of these videos in all available resolutions and bitrates. That’s why Netflix partnered with researchers at the University of Southern California, the University of Nantes and the UT Austin to develop technology to automate this process.

But knowing which quality would be best for a video and actually redoing Netflix’s entire catalog are still two very different things. The company has been using two tricks to make this approach even more efficient and prepare itself for the massive re-encoding project: It cuts every title into numerous slices, making it possible for multiple servers to crunch away on it at the same time, significantly speeding up encoding.

Netflix has long used cloud computing instead of its own data centers to run its service, and the encoding is also done on Amazon’s web services. However, the sheer amount of computing power necessary for a redo of its entire catalog is still significant, even in a cloud world where capacity can be added on demand. That’s why Netflix is shifting around its encoding jobs on the server instances it rents from Amazon to make use of any idle time. Netflix’s Amazon servers may help you to binge on “Jessica Jones” in the evening, but at night, when everyone is asleep, those very same servers are busy re-encoding the company’s catalog.

New markets require better-looking videos at lower speeds

Netflix’s new per-title approach is good news for its subscribers, who now get to watch many of the service’s videos with a better quality while saving bandwidth at the same time. But it could be an even bigger deal for phone and cable companies and the internet at large.

Netflix is now responsible for 37 percent of all internet traffic going into people’s homes in North America during those peak times when everyone is in front of their TVs, according to recent data from Sandvine. Reducing that amount of data by up to 20 percent could help to alleviate internet bottlenecks. “We want to be good stewards of the internet,” said Aaron.

Netflix’s impact on the internet has been a bit of a touchy subject in the past. Cable companies like Comcast and Time Warner Cable wanted Netflix to reimburse them for sending so much traffic to their customers. Netflix argued that this would amount to double billing because those customers are already paying for their service. The internet companies disagreed, and a public stand-off temporarily led to Netflix speeds going down for many customers. In the end, Netflix gave in and agreed to pay up. But as the company is expanding around the world, similar issues are likely to pop up again. Anything that helps Netflix to slow down the growth of its traffic is likely going to make things a lot easier for the company.

SEE MORE: Netflix Wants the World: Can It Really Expand to 200 Countries in 2 Years?

What’s more, many of Netflix’s future markets have much slower internet speeds. Netflix has said that it wants to be in all countries around the world by the end of 2016. Expanding to India, Africa and the Middle East in many cases means taking a mobile-first approach, or dealing with much slower wired internet speeds. Both will benefit significantly from the new encoding scheme, which will allow Netflix to deliver better-looking videos at slower speeds.

Once Netflix’s video algorithms team is done with re-encoding the existing catalog, it already has a number of new challenges ahead. There is 4K, and there are a number of other ways to improve image quality, including higher frame rates and HDR.

And then there is another crazy idea that could require the company to re-encode the entire catalog all over again: After finding the best setting for each single video, Aaron’s team is now thinking about even encoding each scene of a movie or TV show with different settings to account for higher information density during fight scenes and lower demands during slow moments of introspection.

For Netflix employees, this could mean more chances to finally win that unclaimed champagne bottle.

Filed Under:

Want to read more articles like this one? SUBSCRIBE TO VARIETY TODAY.
Post A Comment 38

Leave a Reply

38 Comments

Comments are moderated. They may be edited for clarity and reprinting in whole or in part in Variety publications.

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

  1. ed llorca says:

    have 30mbs service and Netflix quality is still substandard. Blocking occurs on every show.

  2. Nana18kim says:

    netflix has been blocked in my country

  3. The sooner the better, even with a premium internet connection I still have problems.

  4. cjplay93 says:

    The main pushes here use more than simple VBR adjustment. Having actually spoken to some of these people in the past, I know it’s not just a statmuxed encoding scheme, and the Netflix blog post shows it. Unlike Dish/DirecTV/TWC/Comcast/Charter/etc, you cannot know what content is going to be pushed to what pipe in an SVOD/Buffet video environment. What Netflix is doing is going beyond simple VBR to actual resolution reduction with some special sauce thrown in. Did you know that Bullwinkle and most of the Hanna Barbera’s from the pre-90s can be encoded at 12fps with minimal work and motion quality loss? I’ll bet Pokemon and other anime have similar framerate reduction possibilities. I’m glad Netflix is doing something, but saying here that they’re doing what’s been done before is incorrect.

  5. pre user says:

    IOW’ what they have developed is a [variable bit rate ] encoder something like you would find in an .AAC audio encode instead of constant bit rate mp3 but presumably with proportionally less severe across the board compression from what they have now to the new product . I don’t think it’s a new concept just a new proprietary variable bit rate encoder for to Netflix. IOW a new Netflix VBR video encoder .

    There are various FFmpeg encoders that support variable bit rate / constant quality encoding. This gives you more efficient encodes within a limited bandwidth .

    That stuff can get real tricky though hopefully the encoders will make the right decisions and not depreciate the video quality on the variable action and complex video content.

    • kaizeng045 says:

      I agree.
      But once you have a reliable general quality measurement metric, it won’t be just hope for the best. It’s an optimization problem. Maybe with a little tricky about what’s the mimimum quality level to choose, which high depend on what level of your service you want to provide to your customers.

  6. stylebitesbernd says:

    welcome to an approach Flowplayer has implemented two years ago.

    • kaizeng045 says:

      What quality measure that Flowplayer used?
      It’s easier to implement at the player side, but needs no-reference quality model and put more burden at the client side.

  7. americanussr says:

    I just turn it on, and it works. Who would have thought.

  8. kaizeng045 says:

    For those people who disappointed and criticized here, it’s always easier said than done. Dealing with such huge amount of videos with multiple formats and levels everyday, it’s an extremely hard job to determine what bitrate to use for each title.
    The so called “constant quality” mode was claimed to generate constant quality bitstream, but what quality level to pick across titles? For example, in x264, CRF 24 would still leads to drastically different perceptual quality when applied to “My Little Pony” and “Avengers”. You can always categorize the content into different types and apply same settings within each category, but you can imagine the limitation.
    If Netflix can adjust bitrate level for each individual title and meanwhile achieve the same quality level for all of their videos, it IS a big deal! The core part is actually the video quality assessment technology.

    • Jon Greenhalgh says:

      Assesment can be done with a simple diff. Netflix’s encoding and QA should have had this lined up years ago. These problems were just ignored. This is NOT a big deal AT ALL particularly when you look at the stock price and the massive resources available to the company. Much smaller companies have done this for many years. Many folks uploading to YouTube do this when optimizing video. A company called Loudeye (now defunct) was doing his back in 1996. VBR, some batch scripts and some procedural and low wage QA work is all that is required. Nothing here is new or impressive. Netflix has a responsibly to be on top of tech and they clearly weren’t even in the right decade when they built their encoding farm.

      • kaizeng045 says:

        A simple diff, like PSNR, MSE, or SSE, would ruin your video quality assessment system. We need a metric that is consistent with human eyes and universally applied to all kinds of content. That’s the reason Structural SIMilarity got such famous in the recent decade. This blog may help you understand: http://ssimwave.com/2015/11/29/why-ssim-got-popular/
        Because compression is all about Rate-Distortion(Quality) optimization, a more effective quality mode will have huge impact to the final bitstream and always desired, just like you always want have a correct map in your navigation system.
        Hope now you can imagine the science behind this Netflix story.

  9. Pit says:

    This has been extremely obvious to anyone even a little bit interested in video material for the last 1000 years now.
    Every encoder has a “constant quality” mode (a little more complex but basically that)

  10. mArS says:

    A better and with >>20% gain is the other direction Netflix will be experimenting with: P2P streaming (WebTorrent)

  11. Jon Greenhalgh says:

    Nothing new or newsworthy here – this has all been done for decades. It used to be you had to have a big show deal to get a mention in Variety. Now you just need to be doing something ‘techy sounding’ that the average joe doesn’t understand or care about.

  12. David Schembri says:

    I love Netflix however, the best solution for reduce data consumption is to provide the option to download videos. Amazon now had this feature and this feature alone has reduced my Netflix access significantly…at least 50%. Get with it Netflix!!!!

  13. My wife is a stay at home mom and watches tons of netflix with the kids all day. I’m constantly going over my cap. This will be a godsend for us.

  14. Kudos to Netflix for trying to solve a very hard problem.

  15. Jeffrey Parker says:

    Umm I have been working with video codecs for quite a while and though I know it does not adjust everything, but this customizing the compression for each video is kind of the idea of multipass encoding and quantizers. Using these techniques you can get a highly customized bitrate and file size.

    I guess the biggest thing was the other codec settings which are mostly about quality per bitrate, and/or quality for encoding time. I never thought it was that big of a deal to determine the best settings for these things as most of it should be based on how much changes from frame to frame, scanning the frames in a movie and getting the average and peak amount of change would give you a good starting point for deciding on what settings to use. Then there are some considerations for color saturation and scene changes, but most of that I thought would be fairly easily calculated as well.

    • Joel Emmett says:

      Do you think they’re just reducing the number of keyframes on more-static videos? Reducing color count?

      Twenty percent doesn’t seem like much of a savings here. It’s been a while since I was regarded as an encoding expert, but my memory is that many updates to a more recent codec could result in savings close to that.

      It’s good they’re working on this, especially when Netflix is consuming most of the Internet’s bandwidth on any given day.

  16. You mean x264’s CRF based encoding vs 2-pass bitrate encoding???

  17. keith says:

    These techniques are nothing new, except possibly the automated process of reviewing visual quality. Over 10 years ago I used off-the-shelf software to compress a DVD video on my pc, and it used these sorts of methods to reduce the file size, particularly the adaptive bitrates so simpler scenes (i.e. a cartoon) automatically used less data.

    The article should really go into WHY Netflix is just now starting to use these techniques. I’m sure there’s a good reason.

    • Grover says:

      Well up until relatively recently, variable bitrate encoding was a terrible idea for streaming video. If you have a 3Mbit connection and your 2Mbit stream jumps to 4Mbit for 20 minutes because you just got to the sweet car chase, you’re kind of screwed. Most people don’t really notice if the quality drops, but EVERYONE notices if the stream starts buffering. Keeping your stream at a nice, consistent bitrate gave the most people the best experience.

      But these days,the base cable-internet package is 25Mbit, so it’s less of an issue.

      • Kris Rehberg says:

        @Grover Not really. Since these are all recorded, you set the maximum bit rate to the target profile. Therefore, when the sweet car chase happens, all that occurs is more data is pumped down up to the bandwidth specified by the profile.

        It really isn’t rocket science. And, since all of Netflix content is pre-recorded, there is really no reason they shouldn’t already be doing this. It’s baffling that they don’t do this since day one.

        What *is* newsworthy is that Netflix, a video content delivery company, has been so blithely ignorant of old and well-established transcoding concepts already in use in the industry.

  18. Matt Smith says:

    Variable encoding sounds like it’ll help, but how about an option for pre-downloading a higher-resolution version than could be streamed live on a lower-bandwidth connection? Without pausing for buffering every few minutes.

    • nan says:

      Smarter video codecs could buffer upcoming change while you’re watching a quieter scene – that is, use the available (constant) bandwidth with a little intelligence. Netflix is generally watched end-to-end, unlike Youtube where people skip a lot.

      I get from this article that Netflix’s encoding until now has been very dumb indeed, and it’s now being incrementally improved. Still constant bitrate, and a lot of energy spent precoding whether or not media is ever watched.

  19. Kris Rehberg says:

    I always wondered if Netflix was still sending cartoons at the same bitrate as live action. DirecTV and DiSH Network have been using this technology since 1994, over twenty years ago. I’m very disappointed to hear that Netflix wasn’t aware of this technology until just now. I am also very surprised and disappointed that Netflix had not been doing this since day one of their streaming service.

  20. Mark Fowler says:

    I’m unclear about the “Netflix is shifting around its encoding jobs on the server instances it rents from Amazon to make use of any idle time” statement. AWS servers are rented by the hour…how are these servers “idle” while “we sleep” – wouldn’t Netflix just stop renting them?

    • bensawyer says:

      Certain versions of their offerings are more on-demand as you think. Others are not and are billed differently such that the instances can be available but not active.

  21. Ecarson says:

    “Noise is hard to encode”. Noted that compression artifacts seem to be most evident in two different ways. 1. A flickering campfire onscreen rapidly pixelates. 2. A flashlight shining through a smoke filled room, or fog. Distinct bands of shading form that indicate the compression just batched subtle variations into ” numerical buckets”. Campfire flame patterns cannot be predicted (random), and continuous analog variation (lights through fog) are two problems that challenge video data compression algorithms.

  22. Martin Cohen says:

    I would like NF to offer the option of poorer images with lower bandwidth, so when Comcast wants to cap, I can still watch NF.

  23. Justin says:

    My problem with Netflix is the crappy *selection* of instant-streaming content, not the picture quality. And that blasphemous scene in “House of Cards” where Spacey spit on the statue of Jesus.

    • “And that blasphemous scene in “House of Cards” where Spacey spit on the statue of Jesus.”

      Nowhere near as heinous as God sending down two magical space bears to maul 42 kids over making fun of an old man’s baldness.

  24. frank deniro says:

    How about a promise not to passive aggressively punish subscribers who pay for both streaming and (in the mail) DVDs?
    At present, if you stream even one movie in any given week they’ll mess w/ your DVDs, insuring that you never-ever get first (or even ninth) choice from any of the movies on your queue.

    • nerdrage says:

      Uh, you may just be a little paranoid there. I have streaming and DVDs, watch streaming almost every night, never noticed anything amiss about my DVDs other than very high profile titles might have a wait, but that’s been true since before Netflix launched streaming. It’s no worse now than it was then.

More Digital News from Variety

Loading