What’s the Big Data Idea?
Big Data Solving Big Problems
Ready or not, Big Data is more and more a part of our lives.
It’s significance has not hit the mainstream yet.
Learning what Big Data is and does is vitally important to decisions for
individuals,
communities
and the world.
Big Data already saturates everyone’s lives, whether they know it or not.
Data used to be the punch cards entered into the room-sized computer (singular) or perhaps a set of countable items to statistically manipulated. Tangible and discrete, pre-Big Data was a museum exhibit, easy for controlled, deliberate viewing and bounded by solid walls and consistent calculations.
Today Big Data is your every move. Big Data is continuous, always-on flow. Big Data knows what you do, where you go, what you buy, what you use, who you know, who knows you … and a whole lot more through a complex network of inputs and sensors that grows exponentially every day.
Short of catastrophic power collapse, Big Data cannot be turned off either. Wondrous, awesome and intimidating, Big Data is our lives and our future.
Big Problems need Big Data
Big Data’s capability has only come into play within the past ten years. Accordingly, its nascent tools are just evolving. Now it is being captured in ways and means that even the most creative imaginations didn’t predict.
Data storage used to be limited in a small data world but now it expands moment by moment into the 3V space (volume, velocity and variety). Storage is just the first tenet. Next the power to manipulate the data is keeping pace. The typical work environment still relies heavily on spreadsheets and traditional data bases; this is changing. This reflects the huge demand signal for data scientists and data analysts (and data engineers and …) who are needed for manipulation of Big Data. Finally, the data can be articulated in ways that only Big Data can express. Data expression and visualization are necessary to communicate emerges from the Big Data.
Big Data is complex – not complicated. It’s chaotic. Each of us carries around genetic predispositions and social networks which are either enhanced or deferred with the daily choices that compound into life time patterns and twist with chance events. The constant interplay accumulates vertices until a tipping point is reached.
We have always lived in a world of Big Problems – cancer, disease, hunger, poverty, human trafficking, poaching, natural disasters, piracy, terrorism. With the interconnectedness of the world increasing, Big Data is capturing the flow that previously has not been possible. Within the minutiae of internal and external influences the breakthroughs dwell.
Signal in the Noise
In WWII, air navigation and bombing instruments were mechanical devices that oddly worked better while airborne. Problems with the equipment on the ground were not the same on the ground and vice versa. What they discovered was the vibrations of the aircraft itself while in the air “smoothed” the discrete mechanical churn. To simulate this enhancement, small motors were installed, which was called dithering.
Our world of small data has been those discrete motions giving us a certain limited capability to solve problems and make things happen. Big Data is the continuous flow that is poignantly likened to noise. For those that understand the true character of noise, it is not disruptive annoyance that interrupts the signal. Those that understand noise know that without noise the signal is not possible. By adding noise in specific ways we can discover phenomena previously undetectable. Mysterious and exotic patterns exist in the noise.
That noise is where we will find the cure to cancer. That noise is where we will find poachers and pirates and human traffickers. That noise is where we will find ways to support wildlife conservation that preserves cultural practices and livelihoods.