Big Data Could Be a Big Problem

Think about this: NASA knows where Saturn will be in 5,000 years while traders do not know what the price of a stock will be in 24 hours.

The problem of prediction “is not that we are universally good or bad at it, but rather that we are bad at distinguishing predictions that we can make reliably from those that we can’t,” says Duncan Watts, author of Everything is Obvious. So while NASA can count on physical laws remaining stable, traders must navigate the highly unstable financial markets.

The overlooked error is that more information is better in simple and stable domains (physics) while in complex and unstable domains (finance but not weather), more information can be toxic. As Watts puts it, “there is a difference between being uncertain about the future and the future itself being uncertain. The former is really just a lack of information—something we don’t know—whereas the latter implies that the information is, in principle, unknowable.”

We embrace big data but forget that as information increases the number of false hypotheses and incorrect conclusions drawn will increase as well.

Comments are closed.