Big Data: Game On

Most weather forecasts institutions run weather models that deal with a lot of data. But they are not really good at predicting the weather. For all the hardware power, the sophistication of the models and increased use of more and different data, forecasting the weather has not improved a lot. The atmosphere is such a highly complex environment that it is seemingly impossible to generate reliable and accurate predictions more than three days out. The financial world is dealing with the same problem. It has borrowed models, technologies and methods of physical sciences to get a better grip on financial trends and buying/selling behavior. But for all the alleged smartness of traders and their models, the results are often under par when comparing with market averages and also frequently below the results of apes (Gorilla Jacko) and clowns (Frenky and Milko).

While meteorologist and traders are very intelligent people they deliver suboptimal results. Clearly, for understanding complex relationships intelligence, our current data and data-models don’t cut it. One needs more data and better models. But with the help of more data and new data-analytics, unexpected patterns may emerge that could significantly improve forecasts. The promise of big data and big analytics can be a game changer in more than one way. In our CIO roadmap we alluded to this phenomenon as an important market trend in 2012.

The acceptance that there is value in data is not new but the scale at which we are generating, collecting and storing data over the last ten years is unprecedented. The speed at which the digital universe is expanding is still increasing. Structured and – increasingly – unstructured data from social media, sensor data, machine generated data, spatial data, meta data; there is hardly any data that is not stored. But what is more is the realization that we are now able to tap into the value of big data.

Interestingly enough big data is no longer the sole domain of telecom operators, banks and retailers. While traditional data warehousing and data-marts and their associated olap tools were all about analyzing structured data, big things are now happening with semi-structured and unstructured data. Crunching structured transaction data is just one side of the intelligence equation. Finding the best customers, the best selling products, the right time etc is the result of combining transaction data and context or interaction data; the latter being derived from a cocktail of spatial, sensor, machine and social data and meta-data. As a result Big Data is a key concept in the strategic vision of companies in other sectors as well.

Read the rest of the blog at the website of The METISfiles