As I’ve recently written, people are throwing around the term Big Data as if they’re speaking of duct tape\u2014you can use it here, you can use it there, you can use it anywhere. <\/p>\n
This promiscuous use of the term has led to some amusing \u201cfindings\u201d in today’s headlines\u2014eg, \u201c60% of companies expect to invest in Big Data dramatically within three years\u201d while at the same time \u201c85% of Fortune 500 companies unable to leverage Big Data for competitive advantage.\u201d<\/p>\n
The latter quote is from Gartner, so is easily discounted\u2014besides, there should be an automatic penalty of having to watch The View on a continuous loop for a year whenever one utters the phrase \u201cleverage for competitive advantage.\u201d The first of the two quotes is one of those wildly general, unfocused nonsense predictions that mean nothing. The use of the word \u201cexpect\u201d makes it even less convincing.<\/p>\n
So how do we restore some order here? Perhaps a little checklist can help. <\/p>\n
Are you considering Big Data for a project, an initiative, or a makeover?
\nAre you with a Fortune 500 or similarly large company?
\nIs your data new and unstructured? How much do you know about Hadoop?
\nIs your data traditional and structured? How much of it do you have?
\nAre you allowed to go outside the company for public-cloud instances? (See New York Times and its creation of 11 million pdfs during one large public-cloud session.)
\nHow many of your current vendors are touting cloud computing? How many are touting Big Data? How many of them did so at your prompting, rather than theirs?
\nWhat’s your timeframe?
\nWhat’s your budget for using new resources, even if they’re rented, cloud resources?
\nWhy are you considering Big Data in the first place? Have you seen an answer to a longstanding problem, or are you just reading too many articles about it?<\/p>\n
I think it’s fair to say Big Data can run anywhere from a gigabyte on up, depending on the size of your organization, your problem, and your previous ability to solve your problem. This statement may sound horrifying, as Big Data has until recently been the province of people who work with hundreds of terabytes and dozens of petabytes. <\/p>\n
Facebook claims 100 petabytes (ie, 100 million gigabytes) in its Hadoop-driven repository\u2014the mind boggles at the uselessness of that information for anything other than hawking targeted potpourri. You will likely have much less data, but perhaps at least as serious a use for it.<\/p>\n
In any case, I never trust predictions of \u201c60% of this\u201d or \u201c85% of that\u201d or whatnot. We are all as unique as the challenges we face. We can perhaps validate our concerns by reading of other people’s problems and solutions, but can only learn about working this stuff best by doing it on our own, whether we’re part of the 60 percent, the 85 percent, the 99 percent, or the 1 percent.<\/p>\n