The danger of choosing the wrong metric : The VA Scandal

The Veteran’s affair scandal has been newsworthy lately.  The facts about the VA scandal will be forthcoming in August, but David Brooks made some smart inferences back on May 16th on NPR’s Week In Politics:

BROOKS: Yeah, he’s (Shinkseki) in hot water. He’s been there since the beginning. So I don’t know if I’d necessarily want to bet on him. But, you know, I do have some sympathy for the VA. It’s obviously not a good thing to doctor and cook the books, but you – there is a certain fundamental reality here, which is the number of primary care visits over the last three years at this place rose 50 percent. The number of primary care physicians rose nine percent.
And so there’s just a backlog, and if you put a sort of standard in place that you have to see everybody in 14 days but you don’t provide enough physicians to actually do that, well, people are going to start cheating. And so there is a more fundamental problem here than just the cheating.

An administrative failure was made by mandating patients be seen within 14 days but not providing the staffing capabilities to do so.  The rule designed to promote a high level of care had ‘unintended consequences.’  However, I do have some sympathy for an institution which depends on procurement from congress for funding in a political process where funds can be yanked, redistributed, or earmarked based on political priorities.

More concerning, multiple centers may have been complicit with the impossibility of fulfilling the mandate, and whistleblowers were actively retaliated against.

I need to disclaim here that I both trained and worked at the VA as a physician.  I have tremendous respect for the veterans who seek care there, and I had great pride working there and in being in a place to give service to these men and women who gave service to us.  The level of care in the VA system is generally thought to be good, by myself and others.

As I’ve written before in The Measure is the Metric and Productivity in Medicine – what’s real and what’s fake?, the selection of metrics is important because those metrics will be followed by the organization, particularly if performance evaluations and bonuses are tied to the metrics.  Ben Horowitz, partner at Andreessen Horowitz, astutely notes the following from his experience as CEO at Opsware and an employee at HP (1):

At a basic level, metrics are incentives.  By measuring quality, features, and schedule and discussing them at every staff meeting, my people focused intensely on those metrics to the exclusion of other goals.  The metrics did not describe the real goals and I distracted the team as a result.

And if he didn’t get the point across clearly enough (2):

Some things that you will want to encourage will be quantifiable, and some will not.  If you report on the quantitative goals and ignore the qualitative onces, you won’t get the qualitative goals, which may be the most important ones.  Management purely by numbers is sort of like painting by numbers – it’s strictly for amateurs.
At HP, the company wanted high earnings now and in the future.  By focusing entirely on the numbers, HP got them now by sacrificing the future…
By managing the organization as though it were a black box, some divisions at HP optimized the present at the expense of their downstream competitiveness.  The company rewarded managers for achieving short-term objectives in a manner that was bad for the company.  It would have been better to take into account the white box.  The white box goes beyond the numbers and gets into how the organization produced the numbers.  It penalizes managers who sacrifice the future for the short-term and rewards those who invest in the future even if that investment cannot be easily measured.

I’ll have to wait until the official report on the VA scandal is released before commenting on why the failure occurred.  However, it does seem to me as a case of failure of the black box, as Ben Horowitz explained so adeptly.  His writing is recommended.

1.  Ben Horowitz, The Hard Thing about Hard Things, HarperCollins 2014, p.132

2. IBID p.132-133