John Q. Todd

Sr. Business Consultant/Product Researcher

Total Resource Management (TRM), Inc. 

We know what a data warehouse is. Data cubes are not new. Data lakes are a little new. Now we have data lake houses. No kidding.

I look at the sets of data available to us more like a forest. Lots of different kinds of trees, shrubs, bugs on the ground, spiders on the branches, birds, bats, and of course a few cute animals and a few scary ones. Sunlight streaming through the leaves, and rain drops coming from above. Rivers running through it, and yes, maybe even a lake or two. Moss, dead fall branches, sticker bushes, and rotting vegetation. Yeah, that’s our data alright. Visions of lakes and the houses around them are too tranquil.

No matter the metaphor that you wish to apply to your data situation, the reality is the sets of data we have access to (or perhaps better said, those that we know about) vary as much as the trees of the earth do. Tall, short, wide, thin, leafy, not, etc. Our data is stored in spreadsheets, huge databases, and servers often hidden under a desk in the production area. One could even argue that our mobile devices are stores of data that might be of use to us.

What to do? Don’t consolidate!

Your first instinct is more to merge all your data into a single store. This was a fine idea many years ago but can be wasted effort given the tools we have today. Certainly, if you truly have redundant or duplicate sets of data, you might consider doing a little consolidation, but don’t go overboard. In many cases it might be best to simply leave the source of data in place, just access it differently.

A good first step however is to capture “all” of the sources of data you are aware of and indicate what they do for you. This can be an eye-opening activity. I challenge you to find a purpose for every single spreadsheet stored out on your network and shared storage locations. What was needed 3 years ago is a puzzle as to why it exists today.

Remember, you are not consolidating, you are exploring and taking notes.

Purpose, purpose, purpose

Now that you know what you have, take a hard look at why it exists. Many years ago, I was one of many sources of reports and analyses that mission management used to make go/no-go decisions. While deemed important at the time, we would spend weeks preparing for a 10 second statement, “Function X is green. Any questions?” Critical stuff for sure, but there had to be a better way to form our information and be able to visualize it vs. having to recreate the wheel every time.

Your data collections must have a purpose. If data is being created (or reports and analyses for that matter) but has no purpose other than for the boss to spend 2 seconds looking at it each week, then you need to question its purpose. I am not saying toss it away or even stop collecting it. Rather just knowing what its purpose is can be gold when it comes to decision making. Maybe a data set that you are not sure of its purpose should be expanded, making it useful to other parts of the organization.

For those data sets with a clear purpose and upon which the organization relies heavily for decision making, be sure they are safe. If the years of business knowledge and history are stored on George’s desktop, well, you are headed for trouble.

But what you might not know is…

Your list of data sets/source might be a bit overwhelming now, so let’s add to the stress: What don’t you know? Are there sources of data that might be heavily relied upon (or just collected because someone was told to 3 years ago) that you are not aware of? Oh, the things you will find!

Ask your team to have them show you the source(s) of the reports and other information they provide to you on a regular basis. Some may be the sources you already have because of your investigation. Others may be squirreled away and guarded with razor-sharp job security teeth. Be gracious and involve your team with your hike through “their” forest.

But what do you need?

So, you have arrived at your “as found,” state. Time for a break from your hike. Consider now the decisions you make and perhaps those you think you should be making.

Do you have access to the sources of data you need to make “good,” decisions? Is there data you feel you are missing that could augment your decision-making process. Maybe you have this sinking feeling that there is a set of decisions you should be making, but you don’t know if supporting data exists, or if it does, you might be afraid to ask?

I was involved in a management directed study to compare the hours reported by field teams on work orders vs. what was being entered into the timecard system. Talk about a political hot potato that was. Where is the data? Who “owns,” it? Can I have access to it? What concerns does everyone have? Study results were clear, made people uncomfortable, and in the end work order time reporting improved dramatically. Success!

Bringing all this together

Back to the comment about not consolidating. Business intelligence solutions have been around for quite some time. What is new about them is they have evolved considerably and now include artificial intelligence features that make their use even more powerful. There are many examples, but IBM Cognos and Microsoft Power BI are two I run into all the time. Now with the advent of IBM watsonx and its suite of functions, exploring data has become fun again.

Here is a thought: If you are a regular consumer of tabular reports, you are only getting a single view of the underlying data. In fact, the report might be filtering out some of the data, which might be very insightful. Typical static/tabular reports are a very narrow view of the data and over time they can become quite irrelevant.

BI tools used to be the forte’ of data nerds and geeks. Second only to Developers, the Data Scientists and Analysts of the world loved their algorithms and occasional coding opportunities well into the evenings. However, BI tools have made concerted efforts to deliver features for users who might not work with data all day long. N code queries, self-help dashboards, built-in statistical models and the like all bring these tools out of the IT shop and into the hands of those working the production floor.

Wrap up

Healthy forests are a result of good management. Doing some trimming and thinning on a regular basis is always a good idea. But, having a bigger picture view of what is there, and how it is used is where the gold for decision making is. Take a Friday afternoon to do some research on tools such as IBM Cognos or even Microsoft Power BI, and see how they might help you better navigate on your hike(s) through your data.

TRM / IDCON have worked with clients of all sizes and shapes to capture the right data, do analyses, and then make good decisions based upon it. We have used many, many tools to accomplish our clients’ goals in this area, so we feel uniquely qualified to be of assistance. How can we help you?