Love, life and work – 3 ordinary examples of the extraordinary capability of Big Data

“How to save a life?” The Fray makes an excellent song from an age old question. Big Data and Cloudera’s Doug Cutting answer with 3 scenarios that impact how we live now: love, life and work.

LOVE. Historically, love has been dependent on locality, like high school sweethearts, office romance, or the rarest possibility of the blind date set up. Online dating changed all that and Big Data love is the drug that makes the digital connection around the world possible.

LIFE. The medical industry is one of the largest producers of information – previously all that data was dumped. Even the simplest efforts to recover and utilize literally vital data has tremendous effect.

WORK. Big Data isn’t all high tech. Maybe it’s not as sexy as love and saving lives, but Big Data can improve work production from information we already collect, but previously couldn’t use.


GMS attends Big Data Tech Con in Boston courtesy of The Sky INTL

Big Data is a digital Wild Wild West of opportunity.   A gamut of players sit at an open-source poker table holding the data cards (your information) from transactions, social web, telemetric/sensors, archives, and web media.   Although US corporate participation today is found mostly in the mega businesses, garage start-ups are prevalent to making the Big Data products and services that change paradigms in the Information Age.

Perhaps someday there will be a new sheriff in town, but for now, few standard practices exist, even in regard to delicate conversation of data privacy.  Big Data can be as simple as an app product that makes your life simpler.  Big Data apps help you make decisions faster by understanding who you are and what you do from your data exhaust.  Big Data also has grand possibilities.  In the collective intelligence of crowds, world peace is possible.   Global Maritime Sensing is one such example.

But what is Big Data?  Is it the same as Web 2.0?  Is it Web 3.0? Is it a new way to refer to the Information Age?  Is it a concept or a practice or an application?  Big Data is a paradigm shift evolving in several layers.  In the next posts, we will step through those layers to get a better understanding of what Big Data is and what the paradigm shifts are that will continue to affect the world in expanding circles


Kapow graphic on Big Data sources


Global Maritime Sensing Evolves into Big Data

Many great inspirations take years to develop in concepts and more time yet again to become  physical manifestation.  The Global Maritime Sensing concept has been met with positive if not raving reviews by US and international navy partners.  The implementation has still been challenged  as in even the best of financial times, there are but limited resources.  These are definitely austere times and thus more creativity is required.

Regardless of budgetary limitations, GMS has always focused on the extraction of value and from the meaning world around us as individuals and not as institutions.  As time continues to take us by leaps in bounds through emerging technologic miracles, the promise, the capability, of Big Data is now the foundation for GMS implementation.

Big Data is most definitely a paradigm shift, and one in which those that don’t get it will be left behind with failures.  Those that do see its potential are still grappling with its power, much as we all do with the digital world.  Although it would be good to give a couple posts relating the general fit of GMS and Big Data, I’m  instead going to jump into my upcoming attendance at the Big Data Tech Conference in Boston, 30 MAR – 2 APR.  As I go to the tutorials and classes on broad Big Data implementation and specific Big Data tools, I will use the GMS lens as subject for the infinite opportunities of Big Data and the action plan for successful deployment of GMS.

Quantitative & Qualitative Collide to Solve MDA

Solutions to complex problems such as Maritime Domain Awareness are best approached with a combination of creativity and quantification.  Harvard Business Review’s article Learning to Live with Complexity (SEP 2011) suggests that in attaining better risk mitigation of complex systems in the business venue, four elements make sense.  These include storytelling and triangulation – which is a powerful combination of what they refer to as combining “soft” and “hard” approaches.

Storytelling and counter-factuals are visualizing potential shock events and sharing anecdotes of “near-misses” respectively.  This aspect of mitigation is relevant to capturing the risk that is “unlikely but potentially catastrophic.”  This dream weaving and arm chair quarterbacking is creative and interpretive.

Triangulation then reigns in the infinite array of possibilities that can be woven.  Triangulation asserts varying methodologies and perspectives through technical or algorithmic lenses.

This risk mitigation concept marries well with game theory and its application to military strategy.  The numerous elements at play in a given situation are mapped out and given weighted probabilities.  Decision making is then aided with rigorous calculations to provide substance to consequence engagement.  

To that extent another methodology derived in 2007 that could be utilized is computer generated fiction.  Computer software is fed scenario elements and probabilities.  The fiction novel then crafts original stories based on computerized representations of emotions and tensions between characters.

The premise then in looking for solutions to Maritime Domain Awareness is creating the elements that derive the “characters” and “emotions” of the maritime domain and then quantitatively weighing them.  This  which is within the cognitive ability of the individual players and then applying these types of algorithms to triangulate the unlimited potential into strategic plans or comprehensive courses of action.

To further address the challenge of identifying the “dark” vessels of Maritime Domain Awareness then lies with citizen science and crowd sourcing to feed random and specific data points to these lines of possibility.  The data points then twist the algorithms to fit real time events.

Because MDA is not an isolated issues but one shared with the world’s maritime forces, using the ISS as the forum for storytelling and counter-factual collection is an opportunity integral to solving the “dark vessel” gaps.

Global Information Systems Socialization at the International Seapower Symposium

Complicated or Complex?
Maritime Domain Awareness is a complex system, not merely a complicated one. Although the electrical grid for the New York City metropolis area is vast and intricate, it’s workings are none the less only complicated because it consists of a basic network on/off switches. The linear relationship makes it easy to diagnose past incidents and anticipate potential problems. The massive blackout of 1977 could be traced to a series of lightning strikes that triggered unintended switches in the power grid algorithms with catastrophic consequences.

Complexity is different because the elements are not independent and the relationships are nonlinear. Accordingly, MDA, as a complex system, cannot be predicted through probabilistic methods such as extrapolating averages and interpretting median deviations. A complex system also overwhelms the cognitive ability of a decision maker to see the emerging patterns or significant elements. In the case of MDA, that cognitive ability currently encompasses the collection capabilities of several institutions and all nations and navies.

Signal in the Noise
Like most intelligence collection, complexity research is also about finding the anomalies and outliers that have significance – of finding signal within the noise. Data mining of complex systems such as MDA involves three buckets of information: lagging, current, and leading. Lagging is information such as ships’ movements; current is ships’ positions. For MDA this data has tremendous gaps. Traditional AIS tracking fails to capture “dark vessels,” those that are defiantly misleading and those that are too small for AIS. Collection, fusion and analysis of information on these dark vessels are the higher priorities of Maritime Domain Awareness.

Finally, the golden egg of the system is leading information, which is best described by former US Coast Guard commandant ADM Allen’s comment that what you need for piracy is a pirate and a pirate-able ship. Harvard Business Review’s suggestion for risk mitigation of complex systems includes storytelling and counterfactuals in conjunction with triangulation of those creative outcomes.

As in any counterterrorism textbook, capturing the near misses is as important as the responses to real events. Arm chair quarterback analysis of past events and hypothesizing even grand scenarios is much more productive now than ever before with the aid of information systems such as Global Information Systems (GIS).

Global Information Systems
Utilizing Global Information Systems is the key to answering the MDA complex system risk mitigation, especially dark vessels. Using the open source data bases and combining deep intelligence is the first step in the process. Next is sharing the practices, near misses, and possible scenarios that currently reside within silos within the various United States’ government and non-government organizations. Finally, the results are triangulated by assessment from various angles that provide unique perspective that collectively minimize the risk terrain. A more rich and comprehensive solution set would be possible in expanding the cognitive efforts of the world’s navies at such forums as the International Seapower Symposium.

Why GIS?
Utilizing techniques such as creative storytelling in conjunction with quantitative chop has not been used extensively until recently. Global Information Systems are an open source, intuitive capability that captures real time data and overlays it geographically. As such it is perfect for such sophisticated analysis of complex systems of significant emphasis in MDA.

Why the International Seapower Symposium?
The International Seapower Symposium is a biennial event that collects the world’s navies to socialize and share the collective issues of the sea. By far the most prevalent issue is Maritime Domain Awareness. The capabilities of GIS were just budding two years ago for the last meeting, and now the digital power curve for this technology is rising exponentially. This critical technology needs to be presented and socialized at this forum. Because of the scale of its simplicity to sophistication as well as its open source basis, anyone can use it. That also means the “dark vessel” owners. The ISS is the time and place to ensure gaps in MDA are illuminated and leveraged.  The ISS is the opportunity to socialize current practices and draft scenarios for testing.  [polldaddy poll=5462876]  Now is the time.

What is GIS?

A geographic information system (GIS) is a system used to describe and characterize the earth and other geographies for the purpose of visualizing and analyzing geographically referenced information.

Many have characterized GIS as one of the most powerful of all information technologies because it focuses on integrating knowledge from multiple sources (for example, as layers within a map) and creates a crosscutting environment for collaboration. In addition, GIS is attractive to most people who encounter it because it is both intuitive and cognitive. It combines a powerful visualization environment—using maps to communicate and visualize—with a strong analytic and modeling framework that is rooted in the science of geography.

This combination has resulted in a technology that is science based, trusted, and easily communicated using maps and other geographic views.

Dangers of Geotagging

The critical question is whether photos already on the web and for future concept of operations are “readable” for the geotagging information. The answer is arguably yes and no. The use of geotagging for social networking sites suggests a plethora of data available for the sole or enhanced purpose of advertising or sharing location and yet there is opportunity for misdirection. Wikipedia provides an interesting footnote.

Dangers of geotagging
Following a scientific study[11] and several demonstrative websites,[12][13] a discussion on the privacy implications of geotagging has raised public attention.[14][15][16] In particular, the automatic embedding of geotags in pictures taken with smartphones is often ignored by cell-phone users. As a result, people are often not aware that the photos they publish on the Internet have been geotagged. Many celebrities reportedly gave away their home location without knowing it. According to the study, a significant number of for-sale advertisements on Craigslist, that were otherwise anonymized, contained geotags, thereby revealing the location of high-valued goods—sometimes in combination with clear hints to the absence of the offerer at certain times. Publishing photos and other media tagged with exact geolocation on the Internet allows random people to track an individual’s location and correlate it with other information. Therefore, criminals could find out when homes are empty because their inhabitants posted geotagged and timestamped information both about their home address and their vacation residence. These dangers can be avoided by removing geotags with a metadata removal tool for photos before publishing them on the Internet. (

In regard to the unintended consequences geotagging has listed in Wikipedi for Craigslist and celebrities, it is easy to recognize ill intentions are already using the geotag capability. The intent of GMS is to leverage this vulnerability against such efforts in the Maritime Domain. Hopefully, US forces are preventing the exploitation of this gap against the US and its allies.

US Land Data Captured via Nat’l Spacial Data Infrastructure

In the United States the National Spacial Data Infrastructure (NSDI) was created to aggregate the multiple resources already at work capturing geospacial information.  This effort includes data share with government, corporate, and academics.  The governing body, the Federal Geographic Data Committee recognizes the plethora of sources available from which to create more robust maps of information and seeks to leverage the possibilities through the NSDI.

Global Maritime Sensing intends to this same premise with a larger scope – the world’s water area – and to include as many nations as possible to provide the same depth of coverage throughout the world.  The intent is very pointed for GMS – greater Maritime Domain Awareness – and thus greater prospects for safe and peaceful sea operations.

A Solution for increased Maritime Domain Awareness – Global Maritime Sensing

Global Maritime Sensing (GMS) is a digital data capture of maritime traffic via use of location technology and citizen science.  Locative, or location, technology utilizes the plethora of geo-spacial tools available in combination with mobile platforms such as smart phones.  With smart phones now having the capability not only of time and location, but also speed, acceleration and direction, a new “participatory sensing” has evolved.

The participatory sensing is likened to “citizen science.”  Citizen science recruits any and all participants, regardless of expertise, to record observations toward a project. GMS employs all mariners to be “citizen scientists”  and to take photos for geo-tagging.  Geo-tagging is taking photos and uploading them the web for placement on a map.  The GMS map accumulates these data points and then evaluates tthe trends and overlays the maps with intelligence.