Everybody Needs (or Wants) More Data: But How Much is Too Much?

Data. It is simultaneously the best and the worst thing about being in technology. To many people it is personal, and to many technology professionals it's the bane of their existence. To me, it is the gateway to an ever-expanding knowledge base, just waiting to be tapped.

We all know that every modern business, organization, and government agency stores data. But what is this data? It's worth thinking about why data is stored, as well as what kinds of data are stored. What factors — including government regulations — limit data storage? Finally, how do businesses, organizations, and government agencies determine what to keep (and for how long) and what to dump (and when to dump it)?

Massive amounts of data

The data needs of every business and government agency vary. I currently work for a company that has data back to 1977. My personal take on this is positive. I believe that data is an amazing thing and, since our business is real-estate, you can see trends dating back to 1977. We currently have about 10 terabytes of such information stored online.

One of my previous companies had 4 petabytes of data because they had to store X-ray data for 25 years. The made munitions and the X-rays for the outside casings were critical to detect defects and provide proof of safe manufacture government clients. A petabyte of data is equivalent to about 10 railroad cars of paper — so it's a lot of data.

(The next step up is an exabyte, the equivalent of a million trillion ordinary bytes. It's been said that 5 exabytes could hold every word man has even spoken.)

Some corporations are obligated, sometimes by law, to store the data. Others are just packrats. I heard a great quote once: "You can store and report anything, but you can't digest everything." Indeed, there has to be a happy medium between accumulation and analysis. I like to call that target — which everyone should aim at, but not everyone will hit — "actionable data."

Groups that gather information, to whatever ultimate end, are told time and time again that data is a prized commodity. It is very easy to have a wealth of, and yet a paucity of insight, if information isn't harvested and analyzed efficiently. This has become increasingly important as the Internet of Things (IoT) connects more and more devices, churning out even more data.

The Internet of Things (IoT), of course, refers to a system of interrelated, internet-connected objects that can collect and transfer data over a wireless network without human intervention. The personal or business possibilities may be endless — but time and effort are finite. It's important to pick and choose, and whatever gets picked or chosen must be actionable.

Mining, or Panning for Gold?

Mining so-called Big Data for actionable insight is a major challenge for most corporations. Searching a piece at a time would be like panning for gold: a long and often thankless task with little or no reward, and thrilling discoveries lying few and far between.

Ultimately, it's important to decide what data to keep and what data to discard. This means having the right data scientists and tools in place to make effective decisions while rapidly processing huge amounts of data. And yet there are still organizations that spend more on collecting data than analyzing it. This is an easy trap to fall into

Many organizations are aware of both the strategic importance of Big Data and the hurdles to be overcome in analyzing it. According to a recent survey, 31 percent of senior executives say the timeliness of data in their organization is poor, while 25 percent admit their teams lack the skills or expertise to make greater use of data.

Additionally, 61 percent of executives acknowledge that their organizations should rely on data-driven analysis more, and intuition less. At the same time, they don't see their organizations as highly data-driven, leaving them open to being overtaken by competitors.

Drowning in Data

The lesson here for business is that having too much data can become a fatal flaw. This is especially true because we now live in the Big Data era where everything is being tracked, recorded, and analyzed with machine learning.

Can you take action on the data set? If not, then why do you have it? Have you ever acted on it before? If not, then why are you collecting it? Is it possible to collect a different subset of information or data that would be more relevant? Answering these questions helps organizations ensure that they are not wasting space or other people's time.

Data has become an asset. Entire business exist based on Big Data, advanced analytics, and pure data science. These organizations know what to do with their data. Companies that don't should partner with someone who does

Otherwise, it is possible you can overwhelm and drown your business and your leadership team with too much data. The leadership team should see the data and should be making decisions from it. Just remember (and beware of) the dangers of the siren song that data produces simply because it exists.

Rather than helping you make decisions or reduce risk, too much data can actually slow you down to the point where you can become paralyzed. Studies have shown that reliable decisions can be made based on just 75 percent of available information.

The goal should always be to move faster than the market, not simply to accumulate more data than everyone other corporation out there. The trap that I see so many managers fall into is that they struggle to find the line between having the right amount of data to be able to decide.

A Targeted Approach

I have worked for a lot of bosses at different companies who become obsessed with accumulating more and more data to make the perfect decision. Instead, they wind up never making any decision at all. Irrelevant data clogs system and costs tons of cash. Analysis paralysis that last weeks or even months can end up letting the market beat you to whatever opportunity might have existed.

In addition to slowing down decision-making, collecting data just to collect it carries a heavy organizational cost. Employees could be doing more valuable work. Resources devoted to physical storage, cloud storage, and data processing are just a few costs of collecting mass amounts of data.

Go ahead and feed your computers as much information and data as they can handle. Let the machine process all day long.  But ultimately data has to drive decisions and decisions must result in action

It's important to not lose sight of the goal. Data should always provide a direct benefit and organizations should always carefully consider how much data is truly needed to achieve that benefit. In the end, actionable data is what counts and an organization's ability to see what is and is not actionable is paramount to success.