Metrics as "Observables" and the Clairvoyant Test
To the extent possible, metrics should be observables; that is, characteristics of projects or project outcomes that can be observed and measured in the real world. Since estimating project value requires forecasting the future, metrics don't, obviously, all have to be things we can observe today. Metrics can, for example, include a projected future state of some observable, for example, an improvement in a reliability-of-service statistic important to customer satisfaction.
A useful device for checking whether a metric is observable is the so-called "clairvoyant test" devised by my college mentor, Professor Ron Howard. Before accepting what appears to be a good metric, consider whether a clairvoyant could give an unequivocal value for that metric given that a project decision is made in a specific way. Oftentimes, the clairvoyant test points out inexactness of what initially appears to be a well-defined metric. For example, "customer satisfaction" doesn't pass the clairvoyant test. However, "percent reduction in recorded customer complaints" and "company ranking in the next industry customer satisfaction survey" are metrics that do pass the test.
Metrics that don't pass the clairvoyant test are vague. They create inconsistency and imprecision when used for estimating. More importantly, if the metrics are not observables, they cannot be monitored so that actual values can be compared against estimates. Observable metrics allow project proponents to be held accountable for achieving estimates submitted as part of project proposals, which is essential for minimizing biases and gaming in project forecasts.
|
Finding the Right Metrics
