Category Archives: Netflix

Netflix to retire on-prem datacentres by summer’s end

Netflix is making big changes to how it architects its service

Netflix is making big changes to how it architects its service

Netflix said it plans to move its last remaining on-prem systems to the cloud in a move aimed at streamlining its datacentre strategy.

According to a recent report in The Wall Street Journal’s CIO Journal, while its entire customer-facing business runs on AWS, Netflix said the company is planning to completely retire its own datacentres later this summer.

While most of its internal applications also run in the public cloud the company still uses its own infrastructure to store backups of its video collection, and for persistent failover.

It is clear Netflix has until very recently continued to invest in that infrastructure. Earlier this year the video streaming giant swapped 16 existing storage systems for three XIV systems to reduce datacentre floor space used by about 80 per cent and boost its database transactions-per-minute.

It was also testing IBM’s recently announced Spectrum Storage software, which is designed to optimise storage and ease management within hybrid cloud environments.

Moving all of its systems and applications to the cloud will complement a massive architectural overhaul announced earlier this year.

The company said rising demand for its service, which is mostly deployed on AWS infrastructure from multiple locations (initially just in the US) has prompted an effort to simplify its architecture so that it can scale more rapidly and reduce outages.

“Over the past 7 years, Netflix streaming has expanded from thousands of members watching occasionally to millions of members watching over two billion hours every month.  Each time a member starts to watch a movie or TV episode, a “view” is created in our data systems and a collection of events describing that view is gathered.  Given that viewing is what members spend most of their time doing on Netflix, having a robust and scalable architecture to manage and process this data is critical to the success of our business,” the company said at the time.

Will Chicago’s “cloud tax” affect enterprise cloud services elsewhere?

Chicago is taxing some cloud services, an increasing trend in the US in recent years

Chicago is taxing some cloud services, an increasing trend in the US in recent years

Financially troubled and looking to raise funds to plug a swelling hole in the city’s budgets, Chicago recently extended its existing tax laws to levy a 9 per cent surcharge on cloud-based entertainment streaming services like Netflix and Spotify as well as certain software services hosted on cloud platforms in the city. But will the tax laws in Chicago – and elsewhere – soon be stretched to include other cloud services?

The tax law, an extension of existing laws, came from two separate rulings from the City’s Department of Finance. One covers “electronically delivered amusements”, which relates to music, TV and video streaming services like Netflix and Spotify, and another covering “nonpossessory computer leases,” which effectively includes rented storage and compute resources.

The law covering “electronically delivered amusements” doesn’t require those services to be hosted locally (only consumed locally), but the law relating to “nonpossessory computer leases” does, which means local cloud providers are due to collect 9 per cent on their transactions (the exception being when streaming data is in question / interaction with the “rented equipment” is minimal).

The reasoning for the legal reform is simple enough. Cloud is becoming the dominant means by which software and media are being delivered and consumed, and as a result web-based vendors are dominating brick-and-mortar outfits, with the city feeling the pressure from a loss of related sales and property tax revenue. Naturally, the city is looking to compensate that loss with more cash.

Some have suggested this sets a worrying precedent for the way cloud services could be taxed in the US going forward, but some legal experts believe it is not yet clear how the ruling will apply to a wide range of different kinds of cloud services in practice.

“It is likely that we will see more State and local government adopting a tax for certain services to compensate from the loss of revenue from other services that are not generating as much revenue as they did in the past,” Francoise Gilbert, managing director of the IT Law Group told BCN.

A number of US States have already determined they would tax such cloud services as a sale or license of software; information or data processing software; or a digital product or service.

New York, Colorado, Pennsylvania and Utah are all examples of States that have enacted rulings whereby remote access to software via the cloud is taxable if the software is used by in-State customer; Missouri and Tennessee also extend their tax laws to cloud services that are hosted out of State.

But some of those rulings have been challenged before, and a successful challenge can seemingly depend on how a state defines a cloud service and the level of the stack that offering sits in.

In April this year for instance the New York State Department of Taxation (which does tax some cloud services) released an advisory on a case where a company provided infrastructure-as-a-service to a business.  The Department found that the service provided by the cloud company is not taxable because it was used by one of the provider’s customers to run their own software application (advertising software).

“In purchasing an instance, a customer is provided with an operating system that is necessary for the instance to interact with Petitioner’s server network. The operating system represents prewritten software. The customer uses the operating system to perform certain administrative functions, such as to download an application, delete an application, or search for a file,” the advisory opinion reads.

“By granting the right to use the third-party operating system, Petitioner is transferring the right to use prewritten computer software within the meaning of § 526.7(e)(4) of the Sales Tax Regulations. However, a customer does not subscribe to Petitioner’s Cloud Computing product in order to use the operating system. Rather, it subscribes to the product in order to run an application of its choosing using Petitioner’s computing power. This makes Petitioner’s Cloud Computing product different from those products where the vendor’s transfer of the right to use prewritten software to the customer is what the customer primarily wants from the vendor.”

The opinion also concludes APIs do not constitute a taxable pre-written software good.

“It is not clear whether the Chicago tax decision will have an effect on cloud computing services in general.  For several years, States have examined the different categories of services and have opted, or not, to classify the service as taxable,” Gilbert explained.

But she reaffirmed that States will likely continue looking at cloud services for extra revenue, and that consumers shouldn’t write-off potential unintended consequences of taxing one class of cloud services or another – mainly, more taxation.

Can the cloud save Hollywood?

The film and TV industry is warming to cloud

The film and TV industry is warming to cloud

You don’t have to watch the latest ‘Avengers’ film to get the sense the storage and computational requirements of film and television production are continuing their steady increase. But Guillaume Aubichon, chief technology officer of post-production and visual effects firm DigitalFilm Tree (DFT) says production and post-production outfits may find use in the latest and greatest in open source cloud technologies to help plug the growing gap between technical needs and capabilities – and unlock new possibilities for the medium in the process.

Since its founding in 2000, DFT has done post-production work for a number of motion pictures as well as television shows airing on some of the largest networks in America including ABC, TNT and TBS. And Aubichon says that like many in the industry DFT’s embrace of cloud came about because the company was trying to address a number of pain points.

“The first and the most pressing pain point in the entertainment industry right now is storage – inexpensive, commodity storage that is also internet ready. With 4K becoming more prominent we have some projects that generate about 12TB of content a day,” he says. “The others are cost and flexibility.”

This article appeared in the March/April issue of the BCN Magazine. Click here to download your copy today.

Aubichon explains three big trends are converging in the entertainment and media industry right now that are getting stakeholders from production to distribution interested in cloud.

4K broadcast, a massive step up from High– Definition in terms of the resources required for rendering, transmission and storage, is becoming more prominent.

Next, IP broadcasters are supplanting traditional broadcasters – Netflix, Amazon or Hulu are taking the place of CBS, ABC, and slowly displacing the traditional content distribution model.

And, films are no longer exclusively filmed in the Los Angeles area – with preferential tax regimes and other cost-based incentives driving production of English-speaking motion pictures outward into Canada, the UK, Central Europe and parts of New Zealand and Australia.

“With production and notably post-production costs increasing – both in terms of dollars and time – creatives want to be able to make more decisions in real time, or as close to real time as possible, about how a shot will look,” he says.

Can Cloud Save Hollywood?

DFT runs a hybrid cloud architecture based on OpenStack and depending on the project can link up to other private OpenStack clouds as well as OpenStack-based public cloud platforms. For instance, in doing some of the post-production work for Spike Jonze’s HER the company used a combination of Rackspace’s public cloud and its own private cloud instances, including Swift for object storage as well as a video review and approval application and virtual file system application – enabling creatives to review and approve shots quickly.

The company runs most of its application landscape off a combination of Linux and Microsoft virtualised environments, but is also a heavy user of Linux containers – which has benefits as a transmission format and also offers some added flexibility, like the ability run simple compute processes directly within a storage node.

Processes like video and audio transcoding are a perfect fit for containers because they don’t necessarily warrant an entire virtual machine, and because the compute and storage can be kept so close to one another.

Aubichon: 'My goal is to help make the media and entertainment industry avoid what the music industry did'

Aubichon: ‘My goal is to help make the media and entertainment industry avoid what the music industry did’

“Any TV show or film production and post-production process involves multiple vendors. For instance, on both Mistresses and Perception there was an outside visual effects facility involved as well. So instead of having to take the shots, pull it off an LTO tape, put it on a drive, and send it over to the visual effects company, they can send us a request and we can send them an authorised link that connects back to our Swift object storage, which allows them to pull whatever file we authorise. So there’s a tremendous amount of efficiency gained,” he explains.

For an industry just starting to come out of physical transmission, that kind of workflow can bring tremendous benefits to a project. Although much of the post-production work for film and television still happens in LA an increasing number of shows aren’t shot there; DFT for instance is currently working on shows shot in Vancouver, Toronto, and Virginia. So what the company does is run an instance of OpenStack on-site where the shooting occurs and feed the raw camera footage into an object storage instance, which is then container-sunk back to Los Angeles.

“We’ve even been toying with the idea of pushing raw camera files into OpenStack instances, and have those instances transcode those files into an H.265 resolution that could theoretically be pushed over a mobile data connection back to the editor in Los Angeles. The editor could then start cutting in proxies, and 12 to 18 hours later, when the two OpenStack instances have then sunk that material, you can then merge the data to the higher resolution version,” he says.

“We get these kinds of requests often, like when a director is shooting on location and he’s getting really nervous that his editor isn’t seeing the material before he has to move on from the location and finish shooting.”

So for DFT, he says, cloud is solving a transport issue, and a storage issue. “What we’re trying to push into now is solving the compute issue. Ideally we’d like to push all of this content to one single place, have this close to the compute and then all of your manipulation just happens via an automated process in the cloud or via VDI. That’s where we really see this going.”

The other element here, and one that’s undoubtedly sitting heavily on the minds of the film industry in recent months more than ever, is the security issue. Aubichon says that because the information, where it’s stored and how secure that information is, changes over the lifecycle of a project, a hybrid cloud model – or connectable cloud platforms with varying degrees of exposure – is required to support them. That’s where features like federated identity, which in OpenStack is still quite nascent, comes into play. It offers a mechanism for linking clouds, granting and authenticating user identity quickly (and taking access away equally fast), and leaves a trail revealing who touches what content.

“You need to be able to migrate authentication and data from a very closed instance out to something more open, and eventually out to public,” he says, adding that he has spent many of the past few years trying to convince the industry to eliminate any distinction between public and private clouds.

“In an industry that’s so paranoid about security, I’ve been trying to say ‘well, if you run an OpenStack instance in Rackspace, that’s really a private instance; they’re a trusted provider, that’s a private instance.’ To me, it’s just about how many people need to touch that material. If you have a huge amount of material then you’re naturally going to move to a public cloud vendor, but just because you’re on a public cloud vendor doesn’t mean that your instance is public.”

“I spend a lot of time just convincing the entertainment industry that this isn’t banking,” he adds. “They are slowly starting to come around; but it takes time.”

It All Comes Back To Data

Aubichon says the company is looking at ways to add value beyond simply cost and time reduction, with data and metadata aggregation figuring front and centre in that pursuit. The company did a proof of concept for Cougar Town where it showed how people watching the show on their iPads could interact with that content – a “second screen” interactive experience of sorts, but on the same viewing platform.

“Maybe a viewer likes the shirt one of the actresses is wearing on the show – they can click on it, and the Amazon or Target website comes up,” he says, adding that it could be a big source of revenue for online commerce channels as well as the networks. “This kind of stuff has been talked about for a while, but metadata aggregation and the process of dynamically seeking correlations in the data, where there have always been bottlenecks, has matured to the point where we can prove to studios they can aggregate all of this information without incurring extra costs on the production side. It’s going to take a while until it is fully mature, but it’s definitely coming.”

This kind of service assumes there exists loads of metadata on what’s happening in a shot (or the ability to dynamically detect and translate that into metadata) and, critically, the ability to detect correlations in data that are tagged differently.

The company runs a big MongoDB backend but has added capabilities from an open source project called Karma, which is an ontology mapping service that originally came out of museums. It’s a method of taking two MySQL databases and presenting to users correlations in data that are tagged differently.

DFT took that and married it with the text search function in MongoDB, a NoSQL paltform, which basically allows it to push unstructured data into the system and find correlations there (the company plans to seed this capability back into the open source MongoDB community).

“Ultimately we can use all of this metadata to create efficiencies in the post-production process, and help generate revenue for stakeholders, which is fairly compelling,” Aubichon says. “My goal is to help make the media and entertainment industry avoid what the music industry did, and to become a more unified industry through software, through everyone contributing. The more information is shared, the more money is made, and everyone is happy. That’s something that philosophically, in the entertainment industry, is only now starting to come to fruition.”

It would seem open source cloud technologies like OpenStack as well as innovations in the Linux kernel, which helped birth Docker and similar containerisation technologies, are also playing a leading role in bringing this kind of change about.

Netflix Open Sources its Eureka Load Balancing Tool for AWS

Netflix has moved its Eureka mid-tier load-balancing tool, formerly known as the Netflix Discovery Service, to open source.Eureka Architecture Diagram

From the Netflix announcement of the move:

Eureka is a REST based service that is primarily used in the AWS cloud for locating services for the purpose of load balancing and failover of middle-tier servers. We call this service, the Eureka Server. Eureka also comes with a java-based client component, the Eureka Client, which makes interactions with the service much easier. The client also has a built-in load balancer that does basic round-robin load balancing. At Netflix, a much more sophisticated load balancer wraps Eureka to provide weighted load balancing based on several factors like traffic, resource usage, error conditions etc to provide superior resiliency. We have previously referred to Eureka as the Netflix discovery service.

Netflix Chaos Monkeys: “If You Love Something Set It Free”

Image representing Netflix as depicted in Crun...

Netflix has released its Chaos Monkey AWS “test-by-failure” tool as open source for all to use. The tool seeks to improve the resilience if the Netflix AWS cloud by forcing failure.

“We have found that the best defense against major unexpected failures is to fail often. By frequently causing failures, we force our services to be built in a way that is more resilient. We are excited to make a long-awaited announcement today that will help others who embrace this approach.

We have written about our Simian Army in the past and we are now proud to announce that the source code for the founding member of the Simian Army, Chaos Monkey, is available to the community. Do you think your applications can handle a troop of mischievous monkeys loose in your infrastructure? Now you can find out.”