I realized after my post on big data last week that I probably needed to take a step backwards and define "big data" in the first place. What are the characteristics of big data?
- At a minimum, it's a big data problem when the size of the data itself is part of the problem (Mike Loukides' definition), with potentially petabytes or exabytes of data to process. Raw transaction data over a long enough period of time can scale to this size.
- Frequently, the structure of the data is part of the problem as well. Unstructured data processes require technologies different from the relational database technologies we've been accustomed to working with in the past. The number of data sources and potential need to infer relationships among them also can come into play. Sentiment analysis leverages unstructured social media commentary, for example.