Big Data

What a great topic of science fiction from my childhood with the menacing H.A.L. 9000 killing the crew of the space ship in 2001: A Space Odyssey. In fiction, Artificial Intelligence (AI) is a computer or robot thinking for itself and typically trying to destroy all humankind. While wonderful stories, AI is the ability for computers to process data, analyze trends, and make predictive to accurate communications or actions through the processing and analysis of massive amounts of data. AI and machine learning (ML) surround us daily: from Alexa or Siri learning our voice patterns, to Google Maps or Waze predictively choosing the best route home, to spam filters, and even to the emergence of self-driving cars such as the Tesla (and George Jetson thought he had a monopoly). Gone is the villainous killing-machines, and the days of time-savings, convenience, and great accuracy are growing exponentially via AI.

Many business applications already make use of AI in their work. But, think of a platform (computer, mobile device, tablet, voice-activated service, etc.) that assimilated all your household calendars for each person, had stock off all the food in your home, trended preferences over time, and had the family’s health data and dietary restrictions stored and cross referenced. What if it was then tied into a recipe service, a super market delivery service, and of course your financial budget data for meals. Every Saturday you would receive the “menus” for each meal, skipping of course that night you are dining out for Jamie’s birthday. With a click of a button, mouse, or a verbal “make it so number one,” the groceries would arrive the next day with quick meals of PB&J before the ball game, and health/time/cost appropriate foods for the next week.

One that would rock for me would be the next time I pull out my mobile device to pay for something at Best Buy (a severe vice of my tech obsession), it forecasts the financial impact of the purchase by processing my buying trends, upcoming birthdays, holidays, travel, predictive medical expenses, and a myriad of other data. I hear via my Bluetooth earbud, “Tim, you may want to reconsider this purchase as your son has applied to only private colleges and your savings will need to increase dramatically.” Whether I listen or not is another story better suited to the American Journal of Psychology.

As our minds race thinking of the limitless applications of AI—as do the minds of this generation’s entrepreneurs—there are a few critical constraints to keep in mind:

Privacy – the amount and varied data sources would each have a scary amount of data on us. Is that an invasion of privacy? Can this be regulated? How amazingly valuable would that data be to marketers (ask Google and Facebook)?
Profitability – no technology or technological service comes without someone financially gaining on the efforts to make it work and take it to market. Just because I would enjoy a piece of AI that told me when my camera needed to be serviced based on the model, number of pictures, and the environments where I took pictures, there may not be a big enough market to warrant the data collection and analytical programming necessary to make it happen.
Personal organization and documentation – there must be data to analyze before it can be linked to other data sources and processed appropriately. Further, that data must be accessible, and it must be “good data.” Thus, my written list of things to do today that is always 4 times more than I can accomplish will not due. In fact, Dr. Melissa Gratias, a Productivity Psychologist, shared that “too many people spend as much time trying to figure out what they are supposed to do instead of actually doing the work. They either do not have a system to effectively and realistically track tasks, or they have too many different systems that conflict with each other.” Not only is that a powerful message to get organized, it also means my grand AI project is not going to happen until I become organized!

Personal Artificial Intelligence is the ultimate high-tech hack. Big business, big data, and Wall Street are all relying on it. It is coming, and I am ready…well mostly for the cool stuff and maybe not quite as much as the stuff I will have to do to make it viable for me.

It sounds like a drive through meal, why should I care?

If you join the ranks of those entering the world of the technology religion…er, I mean industry, or technological exploration and invention, or are just a hobbyist there seem to be two prerequisites:

You must love, love, love, love, and dream in acronyms
If you name something, make it so generic as to leave it completely open to any conversation, application or ability.

“Big Data” falls into that latter to the proportion of the “smart phone” (what the heck does that mean???). This inability of the technology industry to intuitively name anything that self-describes in this case is tragic because of the wide-ranging implications and potentialities of Big Data. In fairness, none of us were English majors! What is Big Data in simple terms? Here is a non-PhD explanation of Big Data and why it is truly one of the last exciting frontiers, albeit not in a completely physical sense.

In our infancy of exploration of the scientific method, business, fitness, marketing, and almost any arena big or small, we started with siloed data. In other words, a business had accounting numbers; marketing had demographic data; and science had measurements beyond my comprehension. The goal was simple, to bring this information specific to a process or entity and make sense of it to make decisions, assumptions, valuations, or whatever was the pertinent decision point. We humans, in our inherent need to organize things, created data organized in columns and rows which brought the data together into “information” which we in a moment of utter lack of a comprehensive vocabulary, called reports. By sorting the data in in linear and even multi-dimensional formats, these reports enabled measurement of profitability, scalability, probability, etc.… Today this is still the widest used decision-making technique from household budgets to probabilities of orbital change affecting global temperature.

After years, actually decades, of analyzing reports to assist in decision-making someone asserted that this process of looking at formatted data that gave insightful information and then making decision missed a quintessential question: “what if?” After all, that leaves a great deal to interpretation and “gut feel.” By tracing both good and bad causal effects across information, modern computing could provide models by which we could look for predictive trends. In essence, by breaking down the silos of data, and joining them with other silos, technology could produce more accurate, and more actionable, information. In addition, models could then be manipulated to see where and how it impacted business, health, molecular generation, oil prices…you get the idea. The advent of relational databases made this sort of data storage, configuration, indexing and results available. Hence, the oxymoron term “business intelligence” (BI) was born because technology could now relate data to other data, with very minute variables in common.

The scientific industries and big business fueled the growth of what is now a several billion dollar a year business of BI. Technological advances with computing power and data storage and retrieval continued to make this type of analysis more accessible. Once the professional industries (consultants) joined in with their specific theories and understandings, beautiful modeling techniques such as Key Performance Indicators (KPI’s), benchmarks, acceptable tolerances, sprouted up across every industry. In the simplest forms, many organizations use Excel with pivot tables to accomplish this task at a high level. Yet even in 2016, only a small portion of private organizations take full advantage of these capabilities. The major hurdles, which are slowly coming down, are the cost to deploy technologically, the expertise to ensure the models are actually sound (and not biased to the desired results), and the cost to adapt to each individual data set and configuration of decision trees which is a costly venture in consulting fees alone. Frankly, it has become a battle between the cost savings of proven models versus the innovation of out of the box thinking that brings an entire new set of variables into the mix. Undoubtedly, BI is today one of the most powerful analysis tools available, and it is just now starting to be deployed outside fo the fortune 5,000 and academia.

So, what is left? And, why Big Data? We broke down the data silos very intentionally, joining data with commonalities so technology could give us some predictive results on the question of “what if.” What is next, and more appropriately, why is it big? Simply, Big Data can look at tremendously vast data sets across seemingly unrelated data sets—more than the Library of Congress to the third power of ten hypothetically. This is only possible because computing power has exponentially risen and the cost of storage diminished. There are challenges as well, mind you: cooperation, collaboration, anonymity to name a few. A final frontier for technologists—programmers, analysts, subject-matter experts, and the experiences of past data joining—is coming together to create programs that systematically look for the “joins” instead of the answer in a universe of unlimited data. In short, we are tasking the massive computing power to actually find the relevant causal effects, no matter how distant they may be. Tell me: 42 year old humans living between the 40-43^rd parallel of the earth with chromosome 19 at 1,450 genes…what are material circumstances, outcomes, effects, trends, etc.… Except in this case, subject matter experts are only giving potential variables and the computing power takes over the rest, looking for trends, and other material variables. Imagine finding the correlation between potable water elements of a region and the effects on scientific exploration from those hailing from that region. The trends and how things may relate are limitless.

This is important because it does not give us all the answers. Rather, we are now finding and identifying connections between things we would have never imagined. None of this answers “why am I here.” Nor, will it create immediate change or cures. What it will do is expand our thinking—computers are just machines keep in mind…Artificial Intelligence (AI) is another giant technological leap. Big Data can deliver significant insights into the relevance and relation of organisms, people, ideas, environments, outcomes, tendencies to each other. Those results will lead to more “ologies” than we already know, and certainly to a new era of thought, philosophy, understanding, and connection.

In short to me, the only way to truly describe Big Data’s potential is taking a giant step towards technological wisdom. Out of that will sprout the next generation of entrepreneurial ventures and innovations that touch every aspect of our lives.

Here are a few interesting resources:

Big Data

Personal Artificial Intelligence Nexus (PAIN)

Big Data – The Final Frontier