Measuring Training Effectiveness (ROI)
Most training is measured by accident - attendance, smile sheets, hours delivered. Here is how to prove what learning is actually worth, and why the conservative number wins.
The number that never gets asked for
Walk into almost any learning function and ask what last quarter's flagship program delivered. You will get activity: seats filled, completion rates, an average satisfaction score of 4.3 out of 5. What you will rarely get is the one thing a sponsor actually bought - evidence that the business would be measurably worse off had the program never run. That gap between what we report and what we are asked to prove is the L&D credibility problem, and it is almost entirely self-inflicted.
The fix is not more dashboards or fancier statistics. It is a change in when we decide what to measure, and a willingness to publish numbers that are smaller than we would like.
Vanity metrics measure the wrong thing
Attendance, completion, hours, happiness - these are vanity metrics: easy to collect, reliably flattering, and useless for any decision. None is worthless; a course nobody finishes has a real problem. But every one of them sits at the bottom of the value chain. They measure activity, not outcome. When activity is the only evidence you can produce, you are asking sponsors to take the value on faith. Sooner or later they stop, and the budget conversation gets harder every year.
Effectiveness is not one thing. It is a ladder of increasingly demanding claims: they liked it, they learned it, they use it, it moved the business, it was worth the money. Each rung is harder to prove and more valuable to know. The cardinal sin is to gather evidence on the first rung and imply the fifth - to run a satisfaction survey and let people infer business impact. An honest claim names its rung. "92 percent passed the competency check" is a Learning claim, not a Results claim, and blurring the two is precisely what erodes trust.
The four levels, and the fifth
Donald Kirkpatrick gave us the durable map in 1959: Reaction, Learning, Behavior, Results. Reaction asks whether learners found the session relevant and intend to apply it. Learning asks whether they actually acquired the skill - best shown as a gain from a pre-test to a post-test, not a flattering final score. Behavior asks whether they do things differently back at work weeks later, which needs delayed, observed evidence and almost always depends on managers. Results connect that changed behavior to the KPI a sponsor cares about: defects, sales, attrition, safety.
The pattern is unmistakable. As you climb, evidence gets harder and slower to collect but more valuable to the organization. That is why most programs are measured only at Level 1 - it is the cheapest - and why that is exactly the wrong place to stop.
Jack Phillips added the level finance actually wants: Level 5, ROI. Level 4 tells you the metric moved. Level 5 tells you how much of the movement was the training and whether the value beat the cost. It forces two problems most evaluations duck. First, isolation: a sales figure can rise from a new product, a price change, or a better economy, so you must credit only training's share - ideally with a control group, otherwise a trend line, a forecast, or, weakest of all, discounted expert estimates. Second, money conversion against a fully loaded cost that includes participant salaries and the evaluation itself. Anything you cannot credibly convert to money you do not force into the ratio; you report it as an intangible benefit instead.
Why the smaller number wins
Here is the counterintuitive heart of the discipline. In a typical worked case, a 75,000 gross improvement becomes a 42,000 adjusted benefit once you credit only training's 70 percent share and discount for 80 percent estimator confidence - a 40 percent ROI rather than a headline that would have looked twice as good. That shrinkage is not a weakness of the method; it is the method working.
A skeptic who recomputes your numbers should find them cautious, not generous. A defensible 40 percent survives the CFO's scrutiny; an indefensible 300 percent collapses the moment someone asks what else changed that quarter - and takes your credibility with it. The Phillips guiding principle is blunt: when in doubt, choose the most conservative estimate, and always isolate before you convert. The report that volunteers its own limitations is the one that gets believed.
Decide before you build
Almost every failed evaluation fails for the same reason: by the time someone asks "did it work?", the baseline is gone, the control group was never held out, and the business KPI was never agreed. Measurement is a design activity, not a post-mortem. The Kirkpatricks now insist the model only works run backward - start from the Level 4 result you want, ask what behavior would move it, then what learning enables that behavior, then what experience makes people willing to learn. Capture the baseline before launch, pair every lagging result with a leading behavior indicator you can read in week two, and write a one-page data collection plan on the day you charter the program. Do that, and the evaluation falls out of the design almost for free.
Measure to the decision, then stop
None of this argues for measuring everything to the fifth level. Over-measurement has real costs - survey fatigue, wasted analyst time - and buys no extra credibility. Evaluate every program at Levels 1 and 2, the strategically important ones at Level 3, and reserve the expensive Level 4 and 5 work for the handful of programs that are costly, controversial, or that a sponsor has openly questioned. When speed matters more than precision, Brinkerhoff's Success Case Method delivers a vivid value story and a precise fix list in days by studying the best and worst cases rather than the average.
Whatever the level, end every study with a decision - scale it, fix it, or stop it - and feed the lesson back into the next program. Do that consistently and the measurement stops being a ritual. It becomes the thing that turns L&D from a cost centre whose value is taken on faith into an investment whose returns a board can read. The goal was never the perfect number. It was a claim a CFO can take upstairs without flinching.
No comments:
Post a Comment