The idea of big data goes something like this: In a world of ever-increasing digital connectivity, ever larger mountains of data are produced by our cellphones, computers, digital cameras, RFID readers, smart meters and GPS devices. The huge quantity of data becomes unwieldy and difficult for companies and governments to manage and understand.
“My smartphone produces a huge amount of data, my car produces ridiculous amounts of really valuable data, my house is throwing off data, everything is making data,” said Erik Swan, 47, co-founder of Splunk, a San Francisco-based start-up whose software indexes vast quantities of machine-generated data into searchable links. Companies search those links, as one searches Google, to analyze customer behavior in real time.