Using Big Data To Predict The Future
One of the best speakers at last month’s SXSW in Austin was Becky Wang. Becky is the Head of Analytical Strategy at Drog5 in NYC. She has an interesting background involving financial services, film production and new media marketing. She presented a compelling presentation on the power “big data” (i.e. any data created by an action – and we’re creating, storing and analyzing more and more of it every day) is going to have in predicting what’s going to happen in the future.
We wanted to share some of Becky’s insights since, as the saying goes, that’s where we’re going to spend our time (and in our case investment dollars as well):
- Use of data from the Internet is still evolving in ways we can’t fully appreciate (yet) – We’re all familiar with Web 1.0 (i.e. static electronic pages we read that were similar to what newspapers and magazines provided). Then Web 2.0 came along which allowed two-way communication with millions of users (hello, Facebook!). Web 3.0 (aka the semantic web) is happening right now. It’s when machines encode all the data on the web and derive meaning from it. This is the focus of Netbase (www.netbase.com), one of Thomvest’s portfolio companies.
What comes next is…wait for it: Web 4.0! This evolution of the Internet has the ability to use billions of different sources of data to predict what will happen next. In other words, sentiment precedes outcome. For example, some start-ups are experimenting using Twitter to predict what the stock market is going to do next (http://bit.ly/GQftJd). A team of California researchers recently concluded this approach was 11 percent more accurate than other computer models. There are mutual funds being creating using just such an approach (and based on how some mutual funds have done over the years, it’s hard to imagine they could do much worse).
This isn’t a new concept – financial analysts have been trying to do this for decades using competitive analysis and sharing information (“buy on the rumor, sell on the news”). What’s different now is machines can help them do this faster and better than anyone could imagine 5 years ago.
- “Big Data” is being used in new and compelling ways by business and agencies –To quote The New York Times when talking about Big Data, “Big analysis….(is) looking at information in novel ways to find new patterns for prediction.” So once you’ve got all your Big Data, what do you do with it? Here are three different ways to get something valuable out of it:
- Modeling Using Algorithms – This is the one all the engineers love at first sight and is called the “Karman Filter” (http://bit.ly/TQiMg). An example of this would be what the military developed to predict where a missile would be in the very near future (i.e. minutes) using real-time data in order to shoot it down (one of Ronald Regan’s favorite ideas for his Star Wars defense initiative).
- Application of Neural Networks – If you’ve ever rented a film from Netflix, you’ve already seen this in action. Ever wonder how they know what movie you’re going to want to rent next? Netflix uses individual data from millions of movie lovers to detect patterns on what you would like to see next. If you don’t think that’s a big deal in terms of what Netflix was willing to pay to improve it by just 10%, check out the story about the $1 million prize awarded to a team of super geeks at The Netflix Challenge (http://bit.ly/BNvzQ).
- Prediction Models – This approach (similar to how the stock market works) takes all available data and uses it to value something. It’s not just for trading stocks or bonds. There’s a site called “Hollywood Stock Exchange” (www.hsx.com) that uses this type of modeling to allow users to buy and sell virtual shares of celebrities and movies with a currency called The Hollywood Dollar®. Companies such as Ford are using this type of modeling to determine what consumers want in their cars.
Of course, there are skeptics to the whole concept of using Big Data to predict the future. As Daniel Rasmus declared in a recent edition of Fast Company, “Big Data can’t make you smart, pretty or rich.” While it’s true that all models have limitations and predictions can sometimes become self-fulfilling, my personal prediction is these types of models are going to become more important to all of us in the future.
If you’d like to see more of Becky’s presentation from SXSW, check it out at http://slidesha.re/wdOAWU.
