How important is Big Data? How important is structured data? To what extent will they be perceived as the same thing?
My own personal education in big data started a decade or so ago, when I interviewed Paul Saffo of the Instiute of the Future about sensor-based computing. He described a day when ubiquitous sensors would provide amounts of unstructured, critical data into large grid computing systems.
Soon after that, Greg Papadapoulous, a CTO at Sun Microsystems, told me about “spray painting” tiny sensors on walls and anywhere else where the information they gathered could be useful. This was the first I’d heard of real nanotechnology. It sounded like “static telemetry” for lack of a better term.
Concerns about Big Brother were missing from these conversations. This is usually the case, as inventors tend to invent without considering the consequences of having their great new stuff falling into the wrong hands. Or maybe they do consider it, and come to the conclusion that there will always be bad people in the world, but that nothing would get invented if their presence made one stop inventing.
Today, the sensors live among us. They’ve also taken a form in the world of software, as data collectors and Web stalkers, er, trackers. The latter are the true viruses of the cyberworld, it seems.
But are we getting any smarter, or just learning more and more about less and less? Real telemetry, combined with GPS, provides great benefits to the transportation and agriculture industries. RFID data and tracking continues to improve supply chains for even the most worthless crap you want to buy.
More important to our industry, the dawn of massive, unstructured databases seems to pose a clear challenge to traditional enterprise IT departments who’ve built duchies out of large amounts of structured corporate data. It’s also a threat to some of their traditional vendors.
Who doesn’t like traditional Big Data, which focuses on simulating nuclear bombs rather than blasting off real ones, trying to improve weather forecasts, trying to create earthquake forecasts, and tracking epidemics? And who’s not in favor of good air traffic control, high corn yields, and cheap party favors?
But some of this new stuff – the kind that sits on my shoulder online, or watches whether I try to sneak a smoke on a city street somehwere, or once showed the world my car sitting in my driveway – I find obnoxious.