The amount of data processed in the world doubles every three years and a global commitment to open source technology is the way to handle this growth.
An open technology approach fosters innovation through massive community involvement and impedes expensive vendor lock-in. This benefits buyers as markets remain more competitive. In doing so, open standards and technologies also allow for market hypergrowth, and this is the key to handling the growth of data.
A doubling every three years means we’ll be grappling with a full Yottabyte of data in the year 2040. That’s one billion petabytes, an amount of data that, similar to pondering geologic time, I can understand in the abstract but not truly grasp.
Meanwhile, the nature of this data—which can truly be called Big Data in today’s age of the Zettabyte—is transforming from a jet plane model to a chewing gum model.
By this I mean Big Data in its original conception 20 years ago referred to a small number of massive files, the type found in meteorology and nuclear-bomb building. Tomorrow’s Big Data will largely be a product of billions of sensors, transmitting less than 10K at a time. Rather than thinking about a few 747s, we’ll be thinking about billions of pieces of chewing gum.
Already There
We’re already in such a Little Big Data era, with stuff like Hadoop and NoSQL databases equipped to handle the onslaught of data volume, variety, and–because of much this data’s real-time nature—velocity.
But we’ve only just begun. Innovation must continue apace, in hardware even more so than software. Today’s technology already requires more almost 3% of the world’s electricity grid to power its data centers—exponential increases in data processing simply will not be met by the global electricity grid in the absence of vast new hardware efficiencies.
The OPG
Thus I’m involved with something called the Open Performance Grid, or OPG. Announced in San Francisco in August 2015, the Open Performance Grid measures openness, performance, and leadership of hardware, software, and designs for modern data centers.
The OPG is a community effort with input from technology users and buyers, analysts and researchers, and vendors who wish to compare their own self-assessments with what the community is saying.
Sample measures of openness, beyond simple open-source availability, include the presence, size, and activity of a community and foundation for a particular technology. Market share, benchmark performance, and what we call the Innovation Curve are also part of the mix.
Software categories include operating systems, virtualization, containers, PaaS, IaaS services and stacks, monitoring/analytics, management consoles, software-defined storage, SDN, SDDC. For hardware, we’re looking at chips, boards, subsystems, and even overall data center designs.
The challenge of meeting the astounding growth in data is enormous. The Open Performance Grid is a way to encourage and enable the technology provider, development, and user communities to meet the challenge. Contact me on Twitter to learn more.