2018 Guide to Big Data (5 Easy Concepts you need to know today)

Big Data innovations continue to drive business intelligence and integrate into everyday life. Whether you are an experienced data scientist or an aspiring one, whether you are in big business or a one-man shop, whether you are worried about your weight or what your government is doing – Big Data is a part of everyone’s future.

Big Data made a Big Difference in the biggest story of 2016 – the US Presidential election. Although President Trump had pooh-poohed the impact of Big Data during his initial campaign, he rallied a last minute expert team just months before the polls that just may have made the difference.

Using sophisticated analytics and digital targeting, President Trump’s technology strategy collected characteristics from online and offline sources to find potential voters. With over 4,000 finely tuned messages, a specific one was placed after assessing the potential voter’s Facebook, Pandora and snapchat activity. Virtual grassroots at its finest.

Bringing Big Data to the people.  So, what is Big Data and what concepts do you need to know right now?


What is the big deal about Big Data?

Big Data is the collective term for the accumulation, processing and utilization of lots and lots (and lots) of data.  Big Data is huge quantities of data – Volume. Big Data is an array of types of data, from an equally diverse set of sources – Variety.  Big Data is collection and interpretation at ever-faster rates – Velocity. These are the “3 Vs” often referred to in discussion of Big Data.

Although humans have been collecting information about what they do and create since the beginning of recorded history arguably somewhere in the Roman Empire, Big Data is the relatively recent capability to capture and process such significantly larger and more robust data sets. Although computers began the data accumulation in the 1950s-70s, the phenomena of Big Data evolved as recently as 2001 when the term was coined by analyst Doug Laney.  What makes the BIG in Big Data is the exponential increase in the 3 Vs discussed. Here’s a couple of examples.

The Big Picture

When the Sloan Digital Sky Survey began in 2000, its telescope in New Mexico collected more information in the first few weeks than had been amassed in the entire history of astronomy.  By 2010, there was over 140 Terabytes of information. That amount of information can now be collected every 5 days.

When scientists first decoded the human genome in 2003, it took them a decade of intensive work to sequence the three billion base pairs. Now a single facility could sequence that much DNA in a day. The cost of that processing went from $40 million to $5000.

What data you can store and process on your phone today in 24 hours has probably more capability than all computer processing up through the 1970s. In 2005, a cell phone – without even a camera – had more processing power than NASA’s mission control during the Apollo flights that put men on the moon.

To understand why you need to know about Big Data, let’s start with The Fab Five.

 

 

#1 Not a Fad

In the past decade plus years, the 3 Vs of Big Data – Volume, Velocity and Variety has gotten a lot of attention from techies, industry and the public. There’s even been a fourth (Veracity) and Fifth V (Value) to further explore its opportunities and challenges.  Like any popular uprising, the hype or substance of Big Data (depending on how you look at it) reached a certain level of attention before the naysayers began to cast the first predictions of if being a passing fad.

To some, Big Data melts into a crucible of technology slugs and ingots that are pedestrian and passing. But it’s not. The volume, velocity and variety of data available today, versus last year or ten years ago are not about to peak. Following the Second Law of Thermodynamics, its disorder only increases.

https://michaelhanley.ie/elearningcurve/learning-curves-workplace-environment/

Big Data is still in the flat slope climbing the learning curve of what Big Data is and isn’t or what it can and cannot do. Utilizing its capability has considerable challenges, ranging from how it is initial collected to how to get to its mined “gold” – prediction. The philosophic trellis supporting Big Data is complexity and chaotic systems. It’s tricky stuff that the best experts are still beginning to explore.

It’s all emerging technology with all the nubile stumbling of a toddler.  As its potential is only unfolding, the impact of Big Data is less like a popular novel and more like the Gutenberg bible. The bell can’t be unrung; it is here to stay.

Business uses it. Government uses it. Non-government organizations and non-state actors – both beneficent and malevolent (terrorist) – use it. And you use it too.

 

 

#2 You’re Wearing It

Wearables continue to infiltrate everyday life. Right now, the obvious example is your mobile phone. Somewhere in 2014, the number of cell phone subscriptions rose to equal the world population. (Land lines in the US never made that ratio, peaking way back in 2000.)

Cell phones provide you with more and more capability that is also your identity. It’s not just contacts and email connectivity. It’s not just communication. It has your banking information. It has your pics and music and social media, all brimming over with the 3 Vs of data. It entertains you and provides you with convenience. Some argue it is also security. It tells you where you are as well, and as it captures everywhere you have been.

Wearables have become increasingly popular with connecting into more robust medical applications – blood content, vital signs, respiration. Shoes have been designed to give directions to the blind. Socks can charge batteries with walking. These may seem like cool or awkward technologies but their implementation will break barriers in ways that aren’t obvious to the casual technology observer.

Wearables aren’t just for humans either. Wildlife is tracked for numbers and habits. Domestic animals also wear their own version of biometric sensors. The data analysis is used to optimize breeding and feeding practices. Even a honey bee can be fitted out for tracking movement for scientific experiment. These are data points that have been available in small portions before, but as the cost has gone downward, the capacity of data to be analyzed has gone up. Before it was a few discrete points; now it is a flow with more robust and significant and actionable outcomes.

Wearables are moving into more platforms and becoming more ubiquitous. They can be literally woven into fabric and painted or embedded into the skin. The Big Data doesn’t stop capturing your life though with wearables. It keeps going.

#3 It’s All Around

Wearables are just a subset of the propagation of sensors embedded in every aspect of life. Sensors will continue to combine with increased ability to interact and utilize that information. This – the Internet of Things (IoT) – started as a cool idea, but you can bet it already has effect in your life. You are always “on”.

Mobile phones and wearables are examples already provided, but there are others you already know. A suite of home monitoring products on the market provide remote control and observation to check on your electricity usage, environmental status, fire protection, doors locked. You can add monitoring to your car as well, and new models are incorporating more and more sensors that analyze its operation, alerting the driver to hazardous operating conditions and providing maintenance observations.

The Internet of Things (IoT) monitors crop growth. It’s used to drive building space utilization and builds maintenance plans for that building. Big Data and the IoT predict the weather and provide direction for recovery efforts when weather goes awry. The IoT is tolls tags in your car that don’t impede traffic and intelligent labels in your clothing that provide wardrobe inventory analysis and suggestions.

As the Internet itself is the eruption of software – bits and bytes that have become the blood of life, the Internet of Things (IoT) is essentially the physical hardware that we touch and manipulate connecting to the data flow. The embedded technologies weaving together your daily life are becoming more robust, providing an increase in productivity, an increase in relevance, and increase in well-being.

Consumers and society want this capability and they are willing to sacrifice at least some privacy and security for the perceived benefits. See Who’s Betting On the IoT.

#4 It’s Your Business

Big business has been the early adopter of Big Data and it touches all aspects of business – product/service development, manufacturing, operations, distribution, marketing, sales. More importantly, Big Data affects the most important function of business – the bottom line. Big business has had the deep pockets to explore the emerging technology, recognizing the not only the potential return on investment but also the danger of competitive advantage. As Big Data expands, the cost of entry is decreasing as the availability of resources extends to smaller businesses and individuals.

At last year’s (2016) Paris Air Show for example, Bombardier showcased its C Series jetliner that carries Pratt & Whitney’s Geared Turbo Fan (GTF) engine, which is fitted with 5,000 sensors that generate up to 10 GB of data per second. A single twin-engine aircraft with an average 12-hr. flight-time can produce up to 844 TB of data. In comparison, at the end of 2014, it was estimated that Facebook accumulated around 600 TB of data per day; but with an orderbook of more than 7,000 GTF engines, Pratt could potentially download zeta bytes of data once all their engines are in the field. It seems therefore, that the data generated by the aerospace industry alone could soon surpass the magnitude of the consumer Internet.  

http://aviationweek.com/connected-aerospace/internet-aircraft-things-industry-set-be-transformed

We live in a world of increasing choices. The Mad Men marketing schema are iconic caricatures of what capability has begun and will continue to evolve. Your computer already learns from your search history what products and services you are at just thinking about purchasing. That’s a linear example. You search; the sites you visit take the information from your activity to pitch you products and services you are more likely to want. In a way, it’s annoying. In a way, it is convenient.

Big Data will make the message more compelling and more satisfying as it is derived from multivariate activity that accumulates from the 3 Vs. It’s going to start passing products and services you didn’t’ know you need (or want.)

“A lot of times, people don’t know what they want until you show it to them.” – Steve Jobs

Big Data marketing will know your transaction history, your lifestyle patterns and deviations, and fashion a very, very personal sales message to you (whether you like it or not).

 

#5 Your Tax Dollars at Work

Governments are getting into Big Data, not so much by leaps and bounds, but more by specific experiments. The United States uses Big Data in several agencies. Fraud, default and illegal activities can be detected or even predicted by observing the huge volumes of data available from agencies that use a huge volume of transactional data, like the Social Security Administration, the Federal Housing Authority and the Securities Exchange Commission. In the interest of public health, the Food and Drug Administration and Department of Health and Human Services utilize Big Data for better decision-making on the impact of individual lifestyle choices.

The Department of Homeland Security is another obvious player, utilizing the 3Vs of data available from not just federal sources, but state and local law enforcement entities. In the aftermath of the Boston Marathon bombing, over 480,000 images were ingested for investigation. Cross pollination of NASA and the US Forest System Big Data resources coordinated to better predict weather patterns affecting ground and space events.

The next wave of Big Data in government goes even further. It’s a bit more “out there,” and it is a little scary. China citizens have stopped using wallets and instead use their phones for all transactions.  At first it was simple and convenient for buying groceries or renting a bike, but it has evolved into personal credit and social monitoring. Big Data or Big Brother, only the Chinese government algorithms know.

 

Greater Good

The 2018 Guide to Big Data has the 5 things to know about Big Data; it’s not just big business, although that group will continue to invest for both ROI and competitive advantage. Big Data also isn’t just about lifestyle choices. Wearables and the Internet of Things are building a Big Data trellis that grows the fruit of your life. Businesses that utilize Big Data will nurture that fruit, providing the tools and subsistence to grow the optimal grape.

Big Data is also about a bigger picture too. Ill intent will continue to undermine the soil and bind the vines. The bad guys aren’t going away; they will continue to find new ways to steal, or worse.

Big Data can do really great things. It is used for disaster search and rescue as well as damage assessment. It’s used for wildlife assessment.  It brings together the people throughout the world who want to help.

Is Big Data a silver bullet or final solution? No. Big Data is only just beginning. Is all the technology in place? No. But we did see Big Data turn the tide of the US Presidential race.  What will happen in 2018??

Stay tuned.

Who is Betting $25 Million on the IoT? Who will be left behind?

Marc Andreessen is betting $25million

on the Internet of Things (IoT) becoming a big thing. His famous ability to forecast the evolution of emerging technology weaves a picture of sensors embedded in walls and doors and doorknobs. It makes today’s humble offerings – the likes of remote operation of your house security and appliances or fitness bands that connect to a mobile device and the cloud – seem kinda silly.

Marc Andreessen stretches out that belief a little further with his opinion that cell phones will become obsolete. In his perspective a single point of contact such as a cell phone is a limiting vision. He conceptualizes that screens will pervade many more surfaces so much that having to carry your own electronic portal seems a bit ridiculous. That’s an intriguing perspective from an expert on the subject. Goodness knows that a home phone or a pay phone seems quite antiquated today; even paying for a phone call is becoming upended. It’s becoming possible to speak to anyone in the world with an Internet connection and a choice of free apps.

So what happens to the countries …or continents … that are amassing mobile service by leaps and bounds but still lagging far behind? Are they going to be left even further behind …again? Will the technology gap drop those that can’t afford the latest and greatest?

Mobile Phone Adaption

Africa for example doesn’t have as many mobile devices per capita as the rest of the world, but they do use mobile phones in the din of the more remarkable remote locations. Their mobile technology isn’t so much smart as functionally adaptive, performing the most cursory and vital functions, such as currency, in more creative ways than more technologically adapted regions.

So how could it be possible this makes Africa a better candidate to take a technology leap than further advanced countries? Let’s looks at that.

In 2008 Intel reported on their self-developed “Technology Metabolism Index” – a map of what countries adapt technology relative to economics. What they found was that those that “have” don’t necessarily beat out the “have nots” to adopting new technology. The United States had a relatively low index in consideration of the greater resources, access and availability of new technology.

tmi_2007_global_map

Some experts worry that a technology gap threatens to leave behind vast swaths of the world population too poor to afford new innovations, but Nafus work could be seen as evidence that more complex dynamics are at play than just disposable income levels. http://www.wired.com/2008/06/intel-anthropol/

Intel’s Dawn Nafus, a Cambridge PhD, conducted the study in order to better determine the potential of emerging markets versus mature markets. Her team found a couple of surprises. The first is a bit counter-intuitive.

Population size is actually a constraint on technology adoption, just the sheer number of connections between people seems to slow adoption.

The research also supported that foreign direct investment, or how much money foreign firms pour into a country’s economy, can actually constrain how fast technology is adopted. Again, it’s not what you’d think, when more money usually portends resources and capabilities.

Didn’t See It Coming

So let’s look at another technology adaptor story – Colombia. For years, Colombia was known for illegal drug activity that held it hostage. By investing in technology growth its economy has tripled from what it was 10 years ago and its middle class has increased 50%. The country’s remarkable make over has attracted US businesses, increasing import from the US to Colombia by nearly 400% since 2003. Google, Facebook and Microsoft have established presence in country, and of course, Starbucks is there as well to fuel the creativity.

150312172901-colombia-latina-america-780x439

The advances are attributed in great part to fostering entrepreneurship in tech hubs such Ruta N in Medellin and Bogoda’s HubBOG, a 5-year-old tech campus. These hubs are a common theme worldwide.

Another reason entrepreneurs may want to take a strong look at Colombia is because it’s coming online quickly. The Colombia government is working to bring 63 percent of its population online by 2018, according to a report by Colombia Reports. It also boasts 69 percent smartphone adoption, according to the GSMA “Mobile Economy” report for Latin America. http://www.cnbc.com/2015/05/07/three-growing-start-up-cities-in-south-america.html

Big Data Africa Opportunity

According to the TMI index map, a case could be made for South America, Africa, Eastern Europe or the Far East to make Big Data a turn-around industry for any of these regions or countries within.

The availability of Big Data is an inevitable global factor that grows exponentially continuously. Because it is ubiquitous and open source, in Africa or any of the other emerging regions, Big Data is a tremendous opportunity.

Big Data is already deployed in multiple directions: competitive industry, government interest and non-government action. Where there is a will, there is a Big Data way. Training and deployment of all levels of Big Data tools are ubiquitous, given connectivity. Big Data has a relatively low cost of entry as well.

Innovation Hubs in Africa

Africa already has a healthy network of these centers. The number of technology centers in Africa has increased dramatically over the past couple of years. More than half of African countries have a technology hub and leader South Africa has nineteen. According to World Bank, more than half of African nations have at least one tech hub, totaling around 90. In 2014, investment in tech hubs has gone from $12 to $26 million.

Africa Mobile Adaption

A wide variety of participants are needed to make Big Data work. The Big Data industry has an extreme shortage of data scientists and technicians. Even the occupation “data scientist” is actually not as straight forward as it would seem. This is opportunity to find a niche or create a new one.

Not Another NGO

The volume, velocity and variety of data accumulating are a worldwide phenomenon. Perhaps it is relative size, but every person in every country is creating more and more streams of information. In the same thread, every business creates and has access to tremendous amounts of data, but who uses it varies greatly. It’s not a necessity now for entity differentiation, but it is rapidly becoming so.

Those that don’t adapt to utilizing Big Data will get left behind – corporation or country.  We are in a global economy ecology as well.  In a post 9/11 world, we can’t leave others behind and hope for the best.  We’ve got to create the rising tide to lift all boats.  Bring on the Big Data.