You can improve the effectiveness of your data analytics investments. Here’s a good way to start.

“Half the money I spend on advertising is wasted; the trouble is, I don’t know which half.”

You’ve probably heard that saying by department-store magnate John Wanamaker (1838-1922).Given that data analytics initiatives are facing approximately a 60% failure rate, those in corporate leadership who are allocating resources to mining Big Data, becoming data-driven organizations (I still roll my eyes at that bumper sticker phrase), or investing in the new, new, new economy of “performing on analytics” are effectively asking the same question, whether they know it or not. For every $5 spent on new data analytics initiatives, $3 will be wasted. And if you’re the spender, you won’t know which $3 until it’s too late. Didn’t you think things were supposed to be more spectacular than this? Why is this situation the case? Why doesn’t the data analytics revolution – whose exponents seem to suggest that data analytics will relentlessly solve all our problems – solve its own problem of value delivery? Several culprits likely contribute to the current reality, for example…

• The talent pool is inexperienced, undersupplied to the call of the market, and overpriced. A lot of people are getting certified as data scientists through a range of boot camps of varying quality, but they still lack the critical business acumen to design solutions for effective usability by decision makers.
• The actual data quality used in data analytics continues to defy standards of high quality, including having access to data that is relevant to the problem at hand.
• The solution approaches are over-engineered to the problem at hand. Recurrent and convolutional neural networks are cool, but not every problem needs to be addressed with one.
• Project mismanagement: schedule and budget overruns, and deliverables failing to meet specifications.

I suspect that more mundane explanations exist. But I do not believe that the reasons for failure are that data analytics cannot deliver value. We’ll get to that latter point in a moment. The problem as I see it is that a data analytics initiative is just like any other initiative or project, and all other kinds of projects are known to fail at similar rates (see this summary of the Chaos Report) and for repeatedly known categories of reasons. According to the research by Paul C. Nutt, most projects fail before execution begins in the problem solving and planning phases of management. Among the worst things that decision-makers can do to ensure that they waste good money is to solve irrelevant problems for the wrong reasons (A close second is to throw good money after bad in an attempt to recover sunk costs once execution begins).

Guess what? It gets worse.

A year ago, Gartner estimated that 60% of big data projects fail. As bad as that sounds, the reality is actually worse. According to Gartner analyst Nick Heudecker‏ this week, Gartner was ‘too conservative’ with its 60% estimate. The real failure rate? ‘[C]loser to 85 percent.’ In other words, abandon hope all ye who enter here, especially because ‘[T]he problem isn’t technology,’ Heudecker said. ‘It’s you.’

Gartner says that one of the main reasons big data projects have failed to come together is a lack of proper management, combined with lots of trial and error from companies that just throw systems together willy-nilly.

So, really, for every $5 spent on new data analytics initiatives, $4.25 will be wasted.

The problem isn’t data science, per se. It’s the way we choose which initiatives to do and why, followed by how. And so, just like for any other project, the biggest reason that data analytics initiatives fail is that many problems applied to data analytics are irrelevant to the business. Instead of framing a data analytics project in terms of business case value, very often these projects are just interesting science fair projects. We’re all for fairness and science, but nothing represents a bigger waste of time and money than irrelevant projects.

So, how can you know for sure before you spend any money on any data analytics initiative whether you will waste more than half your money or not? You can’t. But, you can get better at making better bets. Consider the following example.

A few years ago, a business unit leader within a development group requested some help justifying an increase of the group’s budget to accommodate a raft of really interesting new ideas. The suggested portfolio contained 122 new projects. But my Spidey sense began to tingle. In my nearly 20 years of experience doing decision and risk analysis, I have rarely seen an initial portfolio of initiatives stand completely justified. For every incremental dollar expended to acquire value, a diminishing return effect occurs that eventually makes a few projects economically unjustifiable. And so it was with this case.

To address the issue, we first framed every initiative in terms of the objectives of the company each should achieve and identified at least a notional way to know that each was supporting the achievement of those objectives. We then devised a way to measure the value of each initiative using “small data” (most of which was derived by the help of subject matter expert guidance and assessment) and probabilistic reasoning using simulation, a kind of virtual analysis of variance. And what did we find? Well, approximately 50% of the projects generated 98% of the portfolio’s value, and that 50% suggested a required budget of only 15% more than the previous year’s budget.

Fig. 1 – The cumulative incremental cost of each project ordered by mean return ($Value/$Investment) yields a declining marginal return. Not all projects are feasible, and they should be pruned from consideration when their marginal contribution of mean $Value ≤ 0.

We pruned off the 50% that remained beyond the “rational” budget (i.e., original budget plus the incremental 15%), many of which were actually just science fair projects. I recall that one engineer explained to me that if he solved the heat transfer problem he was so concerned about, he would gain some fame in the mechanical engineering world – he’d get the blue ribbon. Unfortunately, that project didn’t appear to get out of the red. Fortunately, we caught it before it became a part of the development plan. We saw which dollars most likely would have been wasted before they even got a chance to be wasted. For the initiatives that consumed the largest amount of budget, we used Value of Information and Value of Control to find ways to tweak their value even more, adding more slope to the marginal contribution of value of each initiative. For the next four years thereafter, we calibrated the portfolio budget using the same approach, and each year the project portfolio marginal contribution curve increased its slope and its total contribution with fewer projects and with about the same budget. The group simply got better at identifying potentially valuable projects before we evaluated them economically, most likely because they developed a deeper understanding of the features of initiatives that supported organizational objectives and why.

Fig. 2 – Over time, with each planning period, you will learn how to recognize better and better things to do with your budget using a portfolio analysis approach to guide the selection and prioritization of data analytics initiatives.

The analytics maturity evolution model tells us that the most sophisticated organizations achieve upright status when they embrace prescriptive analytics; i.e., those analytics that tell you what you ought to do beyond the basics of merely describing what the state of nature is (descriptive), why the state of nature presents the way it does (diagnostic), and what the future state of nature will be (predictive). However, that organizational evolutionary process takes too long and it leaves behind a bloody trail of extinct dollars and projects. What we need is a kind of black monolith that tells us what kinds of data initiatives we should be doing and why—a meta-prescriptive system, if you will. That guidance system is the application of portfolio analysis, a selection process that serves as the unifying glue for business plans that helps to remove the pain of strategic misalignment and bias.

Fig. 3 – Evolution gets us there, maybe, but it’s brutish and bloody. It would be nice to have some meta-prescriptive guidance.

If you are a leader in an organization wanting to capture the value of the new revolution of data science, and you care that your resources are appropriately allocated to achieve success, you will want to avoid the failure of relevance. One of the key contributors to doing that is to implement a risk-based project portfolio analysis process for your data analytics initiatives, one which specifically helps you understand…john

• The reason an analytics initiative should be assumed;
• What the potential value and risks to your organization might be for each;
• How to mitigate those risks and amplify the benefits to maximize the likelihood of success;
• How to prioritize and optimize the portfolio of all your analytics projects so that your capital is used in the most efficient way.