Dropbox has announced plans to deploy shingled magnetic recording (SMR) technology on a massive scale in a bid to transform its in-house cloud infrastructure.
The file hosting platform said deploying SMR drives on its custom-built Magic Pocket infrastructure at exabyte scale will increase storage density, reduce its data centre footprint and lead to significant cost savings, without sacrificing performance.
Dropbox says it is the first company to deploy SMR hard drive technology on such a scale.
“Creating our own storage infrastructure was a huge technological challenge, but it’s already paid dividends for our customers and our business,” said Quentin Clark, Dropbox’s senior vice president of engineering, product, and design.
“As more teams adopt Dropbox, SMR technology will help us scale our infrastructure by providing greater flexibility, efficiency, and cost savings. We’re also excited to make this technology open-source so other companies can benefit from it.”
SMR, a hard drive technology that allows tracks on circular disks to be layered on top of one another, will be deployed on a quarter of the Magic Pocket infrastructure by 2019, according to Dropbox, with plans to open source the test software created in this process underway in the coming months.
Magic Pocket is the name of Dropbox’s custom-built infrastructure project that was rolled out after the file sharing company decided to migrate away from Amazon Web Services (AWS). The company initially built a prototype as a proof of concept in 2013, before managing to serve 90% of its data from in-house infrastructure in October 2015.
In what Dropbox describes as a “significant undertaking”, SMR technology was chosen for its ability to expand disk capacity from 8TB to 14TB while maintaining performance and reliability. Drives were sourced from third-parties, before the company designed a bespoke hardware ecosystem around it, also creating new software to ensure compatibility with Magic Pocket architecture in the process.
“SMR HDDs offer greater bit density and better cost structure ($/GB), decreasing the total cost of ownership on denser hardware,” the Magic Pocket and hardware engineering teams explained. “Our goal is to build the highest density Storage servers, and SMR currently provides the highest capacity, ahead of the traditional storage alternative, PMR.
“This new storage design now gives us the ability to work with future iterations of disk technologies. In the very immediate future we plan to focus on density designs and more efficient ways to handle large traffic volumes.
“With the total number of drives pushing the physical limit of this form factor our designs have to take into consideration potential failures from having that much data on a system while improving the efficacy of compute on the system.”
Towards the end of the year, the file hosting service says its infrastructure will span 29 facilities across 12 countries, with Dropbox projecting huge cost-saving and increased storage density benefits if SMR deployment is deemed a success.