How to Become Data Driven

🧠 Free Recall ❓ Quiz 🔍 Cues 🔮 Simulation 🥷 ShadowBox 📐 Encode Cards

Donald Wheeler’s Understanding Variation opens with a cute story. It goes something like this:
Many years ago, David Chambers found the following time series on the wall of the office of the president of a shoe company. Here was a simple and powerful presentation of data in context. The caption on the vertical axis was “Daily Percentage of Defective Pairs.”

Intrigued, David asked the president why he had the graph on the wall. The president condescendingly replied that he had the chart on the wall so he could tell how the plant was doing.
David immediately responded with “Tell me how you’re doing.”
Evidently no one had the temerity to ask the president this, because he paused, looked at the chart on the wall, and then said, “Well, some days are better than others!”
My bet is that you’ve probably experienced something similar. How many times have you opened a tool like Google Analytics, looked at the number of website visits, and said to yourself “oh I guess some weeks are better than others?” How many times have you then turned to the overall trend of visitors, say over a period of months, just to reassure yourself that the number is tracking upwards and that all is good with the world?
Many times, I’d bet.
The honest truth is that if you are like me, you probably never know what to do with this information. You don’t know what drives the number up some months, or what to do when the number goes down in December, or what to think when visits have dipped for three weeks in a row. This is, bar none, the most common experience people have with data. And if you’re like most people — myself included — you’ve probably never been taught how to interpret things differently.
My hope is that we’ll be able to change that with this essay.
Listening to the Voice of the Process
In our previous piece we took a look at Goodhart’s Law, and examined how Donald Wheeler’s Understanding Variation contains a method for dealing with the Law in various organisational contexts. In the middle of that piece, I quoted the following bit from Wheeler (all emphasis mine):
Before you can improve any system you must listen to the voice of the system (the Voice of the Process). Then you must understand how the inputs affect the outputs of the system. Finally, you must be able to change the inputs (and possibly the system) in order to achieve the desired results. This will require sustained effort, constancy of purpose, and an environment where continual improvement is the operating philosophy.

Comparing numbers to specifications will not lead to the improvement of the process. Specifications are the Voice of the Customer, not the Voice of the Process. The specification approach does not reveal any insights into how the process works.

So if you only compare the data to the specifications, then you will be unable to improve the system, and will therefore be left with only the last two ways of meeting your goal (i.e. distorting the system, or distorting the data). When a current value is compared to an arbitrary numerical target (... it) will always create a temptation to make the data look favourable. And distortion is always easier than working to improve the system.
At the time, I said that Wheeler was trying to make a point about the impossibility of improving a process if you focused solely on stated goals. His name for goals (or other kinds of specifications such as, for instance, a budget) was the ‘Voice of the Customer’; his name for observing the process itself was “listening to the ‘Voice of the Process’”. I explained that Wheeler was arguing for listening to the Voice of the Process — which is really a counter-intuitive notion, since our natural inclination is to focus on targets. But hidden in that passage lies the solution to our current problem.

It turns out that Wheeler meant something very specific when he said “you must listen to the Voice of the Process.” He was talking about a particular approach to working with data — one that generalises rather broadly to all forms of business activities. That approach goes something like this:
Nearly any chart that you look at is the result of some ‘process’: a website visitors graph in Google Analytics, for instance, is the output of some combination of traffic sources bringing readers to your website, which is in turn the result of a complex mix of your marketing activities, your search engine site performance, some amount of link sharing on social media, and perhaps the number of people who decided to search for keywords that you ranked for in that specific month.
The process control approach to data is to go, “ok, a time series of visitors is a report on the performance of that process in a given month. So we need to ask, like, three obvious questions, right?”
Those three obvious questions:
Have you successfully changed the process? In the specific case of website visitors: are your marketing activities working? Have they resulted in permanent change in the underlying complex mix of sources, such that you see a permanent increase in your visitor numbers?
Has something else changed in the underlying process? For instance, a major celebrity might have linked to your website, causing a massive short-term spike in visitors, followed by a permanent boost in traffic. Conversely, your second largest source of traffic might have changed its outbound link policies, reducing its total contribution to your visitor numbers. Both are things that you might want to know about, and preferably sooner rather than later.
Finally, and perhaps most importantly: are the current changes in your visitor numbers totally normal, and therefore just routine variation?  In other words, should you just ignore the changes in visitor numbers this month? There is, after all, nothing quite as depressing as being forced to write a made up report about why marketing qualified leads (MQLs) is down for the current month, when it could just as easily be explained by routine seasonal variation.
If you boil these three questions down to their core, what you end up with is a radical, almost laughably simple idea: telling the difference between routine versus exceptional variation is the beginning of becoming data driven. Statistical process control practitioners would even go so far as to say that it is impossible to be properly data driven if you do not fully understand variation. And I will admit to having great difficulty with this idea — the very notion that understanding variation lies at the heart of data-driven decision making was such an alien notion, such a radically dumb, simple idea, that it took me awhile to grasp its importance (“Surely this entire book can’t be about variation right? Right? Right? Wrong.”) Perhaps this shouldn’t be so surprising: you cannot investigate changes in your underlying process if you do not know which data points signal real change. Worse, you will waste huge amounts of time if you investigate every instance of ‘oh no, number go down’.

Every instance of exceptional variation is an opportunity for process improvement
So let me attempt to drive this point home by restating this idea, albeit more strongly. The field of statistical process control (SPC) was founded on the ideas of American statistician Walter Shewhart, and spread through the work of W. Edwards Deming during and immediately after World War II. Deming liked to say that every instance of exceptional variation is an opportunity for process improvement. If you see positive exceptional variation (a massive visitor spike, for instance), you’ll want to investigate because such a spike will tell you something insightful about the underlying process; if you see negative exceptional variation (a sudden dip in visitor numbers, say), you’ll want to start digging into the causes that produced that dip, hopefully before it becomes too large of a problem.
But there are actually many more implications when you start to see variation as ‘routine vs exceptional’, and we’ll get into those in a bit.
Process Behaviour Charts
The chosen approach that statistical process control uses to accomplish this is to create something called a process control chart. Wheeler prefers the term ‘process behaviour chart’, since you’re reading time series charts to figure out the nature of some underlying process; that process doesn’t necessarily have to be under your control. For the rest of this article I will adopt Wheeler’s terminology, since he picks his words carefully.
A process behaviour chart looks like this:

A commonly-used name for this two-chart pair is the ‘XmR chart’ — so named for the two charts it contains: an ‘X-chart’ (where X is the variable, or business metric being observed), and a ‘Moving Range’ chart, which shows the ‘moving ranges’, which are the differences between the successive values in the X-chart. All moving range values are, by definition, positive — we’re interested in the magnitude of differences, not whether the numbers go up or down.
The basic idea behind process behaviour charts is simple: on the X-chart and the moving range chart we plot a few lines that represent ‘limits’:
On the X-chart we plot an Upper Natural Process Limit and a Lower Natural Process Limit
On the moving range chart we plot an Upper Range Limit.
If values on either of the charts fall outside the range, this signals to you that you’re seeing exceptional variation, and you should investigate! The XmR charts are thus incredibly simple to use: a point outside the limits is a signal; process variation between the limits represent routine variation and are (usually — not always!) regarded as noise.
How do you calculate those limits? I’ll give you the formulas in a second, but I don’t want to bog us down with the math; instead, I’ll link out to a bunch of resources at the end of this piece that gets into greater detail. For what it’s worth, the math involved is simple high school arithmetic, but there are a few common errors you’ll need to be aware of; covering all of these caveats will take up too much time. I want to focus this piece on giving you the intuition behind this kind of thinking instead.
To calculate the process and range limits, you do the following:
Plot a time series for the metric (X) you’re interested in.
Calculate and then plot the moving range for that same metric.
Calculate the average for the business metric and plot the average as a straight line through the time series.
Calculate the average for the moving range and plot the average as a straight line through the moving range time series.
You’re going to end up with two charts that look like this:

To calculate the Upper Natural Process Limit: multiply the average moving range by a scaling factor 2.66 and add the product to the average X. UNPL = Average X + 2.66 Average mR
To calculate the Lower Natural Process Limit: subtract the 2.66 x mR product from the average X. LNPL = Average X - 2.66 Average mR
Finally, calculate the Upper Range Limit (the upper limit for the moving range) by multiplying the average moving range by the scaling factor 3.269: URL = 3.269 x Average mR
You’ll end up with your final process behaviour chart, like so:

Is it really that simple? Yes, it’s that simple. Where do those scaling constants come from? How did SPC practitioners come up with this approach? How do we know that it works? Again, I shall focus on intuitions first; I’ll link to answers in the appendix.
Process Behaviour Charts are an entry point into a different way of thinking about business operations
It’s worth stating here that process behaviour charts are decades old at this point, have been proven in all sorts of situations (and the situations where they don’t apply have mostly been worked out) — they are not a game changing tool in and of itself. It’s more valuable to think of them as an entry point into a different way of thinking about business operations. Again, the purpose of plotting the charts is to differentiate between routine and exceptional variation in your processes.
The ability to tell the difference between both types of variation leads us to several interesting follow-on ideas. For instance, one major implication of this worldview is that we may now categorise business processes:
A business process is said to be ‘predictable’ if it displays no exceptional variation, just routine variation.
A business process is said to be ‘unpredictable’ if it displays both exceptional variation and routine variation.
Why is this important?
One of the foundational beliefs of SPC is that management is prediction: you cannot run your business well if you cannot predict the outcomes of your actions. (To be clear, when SPC practitioners say this, they mean ‘predictable within limits’ — the world is fundamentally unpredictable, but the act of setting targets and moving towards goals requires you to at least be able to predict the effects of your actions on business processes.)
Put differently, if you want to improve your business, you really need to remove exceptional variation from your business processes, as far as is feasibly possible. An unpredictable business process cannot be managed, since you’ll always be stymied by unexpected events and systemic problems that you do not understand.
This flows into a second implication, as useful as the first:
A business process that is predictable (that is, only shows routine variation), cannot be easily improved upon. It is already running optimally. This means that if you want to improve this type of process, you must make a fundamental process change — you’ll need to rethink your entire approach, buy some new machinery, adopt new technology, etc.
A business process that is unpredictable (that is, shows exceptional variation) can be easily improved upon — in the sense that your first step is obvious: remove the sources of exceptional variation! You shouldn't attempt to modify the process to hit your targets until you do so.
To simplify further, SPC tells us that we should:
Reengineer or completely rethink the good processes
And tweak the bad processes.
This sounds well and good, but contrast the SPC approach with the following, more common sense one, which stems from the same folk knowledge as ‘if it ain’t broke, don’t fix it’ and ‘first, do no harm’ and ‘the squeaky wheel gets the grease’:
If it’s working well, tweak it.
If it’s working badly, reengineer the process.
And perhaps this describes you as well. How many times have you taken a look at some marketing campaign or internal process and said to your team: “oh, this process is doing badly, we need to RETHINK EVERYTHING.” Or, “hmm, we seem to be getting a 15% conversion rate on this marketing channel, which is AMAZING; our tweaks for the past quarter all seem to result in marginal gains, I guess that’s just the best we can do? What shall we do next?” And maybe there really are only marginal gains for that finely tuned, very successful marketing channel of yours, but SPC rebuts the common sense approach and alerts you to the fact that you need to fundamentally rework your approach if you want to turn good processes into great ones.
Process behaviour charts are also useful for identifying if the actions you’ve taken have actually started influencing the underlying process. Understanding Variation notes that while process limits are obviously going to be more accurate the more data points you have, you really only need as few as five to six data points to start constructing a process behaviour chart, which then means that you may observe, reasonably quickly, if process variation has changed as a result of your efforts. One example from the book looks like this:

Which is the result of a manager repeatedly making changes to some process, and replotting the XmR chart after every change to see if the limits have permanently shifted. Hopefully this gives you an idea of how one might use XmR charts in the context of process improvement.
A Worked Example
Let’s go through one worked example from Understanding Variation, just so you understand how this approach might be used in practice. Like all examples in the book, this is taken from a real world case, with names changed and the details modified, for obvious reasons.
A large manufacturing facility produces a management report each month. The report for the month of July looks something like this:

At the start of each month, the facility’s managers would sit down and review the figures from the prior month’s output. Before your eyes glaze over, I’d like to direct your attention to the ‘% Diff’ column, in addition to the ‘% Diff from July Last Year’ column. When presented with such tabular information, one’s first instinct is to scan the numbers for the biggest changes, and then ask “wait, why was this change so large?”
In this particular example, the numbers that leapt out at the managers were:
First-time approvals, which is down 23%
In-process inventory, which is up a whopping 42%!
On-time closings of accounts, which is down 21.8%
In a typical meeting, the managers would question the respective leaders of these three departments, demanding to know why the percentage changes have occurred. In-process inventory is particularly worrying, since ‘work-in-progress’ is a bad sign for any production process — high in-process inventory means there are bottlenecks in the manufacturing process, causing inventory to pile up between processing steps.
However, a different picture emerges when the production manager requests for three years worth of in-process inventory data and plots out a process behaviour chart:

It turns out that this 42% change is perfectly within the Natural Process Limits! This means that there is nothing to see here — a 42% change is within routine variation.
At this point you might say: “Holy smokes, in-process inventory is something that goes from 8.3 to 31.6 (in 100s of lbs) — which is an extremely large variance! Surely we can do better than that!”
But that is almost certainly not possible — at least not without significantly rethinking the underlying process. Wheeler writes (all bold emphasis mine):
The Natural Process Limits are the Voice of the underlying process. They define what the process will deliver as long as it continues to operate as consistently as possible.

When a process is operated predictably it is also operating as consistently as possible. The process doesn’t really care whether or not you like the Natural Process Limits, and it certainly does not know what the specifications may be (specifications should be thought of as the Voice of the Customer, which is distinctly different from the Voice of the Process).

Therefore, if the manager issued a decree that the in-process inventory should not vary by more than +-20% from its average value, what would happen? A +-20% variation from a value of 20.0 is 16.0 to 24.0. Is this process going to operate within these limits? The process behaviour chart says that it has not done so in the past, and it should not be expected to do so in the future — at least not without some fundamental change in the underlying process. Thus, such a decree will simply encourage the workers to distort the system or to distort the data. Such decrees, by themselves, do nothing to change or improve the system.
This particular example ties together four lessons:
Listening to the Voice of the Process means understanding what the process’s current variation actually looks like. Only then can you start thinking about improving the process.
The line “the Natural Process Limits (...) define what the process will deliver as long as it continues to operate as consistently as possible” gets at an important truth about the SPC approach: that it only works for processes that are performed with consistency. If your business activity is ad-hoc, and varies from case to case (such as, say, startup sales in early markets, when you’re still trying to figure things out), then creating an XmR chart will not give you any insight into your process performance, much less how to improve it.
If the process only shows routine variation (like this one), you need to fundamentally rethink the process in order to change its variation or performance. (We’ve already covered this above, but I just wanted to highlight this here.)
Percentage metrics are dangerous!
That fourth lesson is an interesting one. Wheeler explains that percentage metrics, while commonly used, can be misleading for three reasons:
The size of the percent difference will partially depend upon the magnitude of the base number — a ten unit change from 300 to 310, for instance, is a 3.3% change. We all know that percentages show the relative size of a change rather than the actual amount of change. This in turn means that comparing one percentage change with another is often not a reliable way of finding the interesting parts of the data. You can easily imagine this: say, a large absolute change in a financial metric you care about may be more important than a large percentage change in a more trivial operational metric — but the large absolute change in that financial metric shows up as a tiny percentage difference due to the large base number. Since people naturally pay more attention to large % changes in the monthly report format, you’re likely to miss the significance of the absolute change.
Second, comparing percentage changes across metrics assumes that all lines should show the same amount of relative variation month-to-month. But this cannot be the case! We know that every time series will have its own variation, since it depends on the differing nature of the underlying processes. So putting up a list of % changes will ensure that some lines (the ones with large relative changes) will receive more attention than they deserve while others will receive less attention than they deserve.
Finally, comparing percentage differences with last year’s value is problematic because a large percent difference may be due to an unusual value in the past rather than an unusual value in the present. Who isn’t to say that the large yearly % change you’re staring at was due to a machine blowing up in July last year?
We’ve seen some of these issues with in-process inventory. And now we’ll see a little more.
It turns out that the July monthly report hosts a bigger problem, albeit one that is hidden within an innocent-looking figure:

On the face of it, on-time shipment looks like a perfectly good metric. In July, 91.0% of shipments were shipped on time. This is 0.3% below the historic average, and 0.9% below the value for last July. The year-to-date value is also decent: 90.8% shipped on time, which is 0.6% below the average and 0.3% below last year’s value.
But let’s take a look at an XmR chart for this process. The transportation manager asks for some historical data:

And then plots the process behaviour chart:

She is then shocked to learn that — once routine month-to-month variation has been taken into account, there is still too much variation in this time series to be due to chance alone. There are, in fact, six values below the Natural Process Limits. The manager should have looked for an explanation of why the percentage of on-time shipments dropped in those months.
Wheeler writes:
The six values outside the Natural Process Limits are exceptional. The process is trying to tell you that it has a problem. Something was happening in the Summer of Year One, in March of Year Two, and in April and May of Year Three. This something could reoccur, and it could be worse next time.

- You have already missed three opportunities to improve this process.

- The process has already done all that it can do to alert you to the presence of this problem.

- How many more signals are you going to miss before this problem causes you to lose a large account?

- If you concentrate on the percent differences in the monthly report, you are not likely to ever be aware of this problem until it is already too late.

Process behaviour charts are the way to listen to your process. When you listen to the Voice of the Process as revealed by process behaviour charts, you can often detect signals that you would otherwise miss.
Wrapping Up
This essay has been surprisingly difficult to write, because the ideas in them seem so stupid, so simple, that they can’t possibly be profound. Even process behaviour charts, when explained properly (as I hope I’ve done above!) don’t seem particularly ground breaking!
But I really think that ’understanding variation is the beginning of becoming data driven’ is a profound observation. It seems a little bizarre that I’ve never come across this idea before, given how fundamental it is to the decades-old body of work that is Statistical Process Control. It is, I think, profound and obvious in the same way that literacy is profound and obvious — profound because a whole universe of knowledge is unlocked once you learn how to read, but obvious because you quickly take it for granted afterwards.
Data literacy in the context of business begins with variation, and this is profound, but also obvious once you learn it. And then, afterwards, when you look at time series charts in your business, alongside your colleagues, you’ll soon notice who understands variation, and who doesn’t, and those who don’t will stick out like illiterate sore thumbs.
But who would’ve guessed?
Appendix
If you’d like an in-depth tutorial on creating XmR charts, refer to this Deming Alliance piece titled Process Behaviour Charts: An Introduction. The article covers the step-by-step process of creating your own charts, and walks through a few edge cases. In general, the edge cases boil down to: you should create separate XmR charts for phenomena that are logically separated. If you’ve made a process change, for instance, you’ll want to plot a new XmR chart after the change; if you’re measuring seasonal data (say, shopping mall footfall, which is higher during weekends), you’ll want to separate out the weekends out into its own XmR graph, and analyse variation separately from data collected during the rest of the week.
Where do these scaling factors come from? The basic idea behind the Natural Process Limits is that they are roughly three sigma around the centre line (the average), and the scaling factors help you approximate three sigma. Wheeler was apparently responsible for the current method of using one set of scaling constants; in the Deming Alliance piece linked above, the authors write:
It is true that process behaviour charts used to be the complicated part of continual improvement. At the time Deming was developing his approach to management, one had to get to grips with about half a dozen different types of charts. Over the last couple of decades however, thanks to the contribution of Donald J Wheeler PhD, the statistical side of continual improvement has become a lot more accessible to everyone. Donald Wheeler specialises in business and management applications of statistics. He has proven that one type of process behaviour chart — the one that we will outline in this paper — does the job in most situations.
Wheeler himself explains the scaling constants in a 2010 paper. He writes:
The three-sigma limits are sufficiently conservative to work with any type of data and still filter out virtually all of the routine variation, thereby minimizing false alarms while allowing you to detect those signals that are of economic importance.
And there are, apparently, five main uses of process behaviour charts:
As a report card.
As feedback for process improvement.
As feedback for short term, simple experiments conducted on a process.
As extended monitoring — that is, you use several different process behaviour charts over a period of time to discover which metric provides the best predictor of product or process performance. (Note the similarities here with Amazon’s WBR approach — as I’ve alluded to elsewhere, Amazon has taken a process control approach throughout their entire business).
As an input to continuous improvement. This is the typical end state, or result, of using process behaviour charts, but Wheeler notes that it is not typically possible when XmR charts are first introduced to an org. Rather, XmR charts are usually the way companies get to continuous improvement; usage of the charts over time usually results in the spread of a certain type of thinking throughout the organisation — and it is this thinking that makes continuous improvement possible.

Unstarred

Dunk (exclude from study)

How to Become Data Driven

Review Questions

Todos

Sensemaker