Police Explain? How targets don’t work
Last week Australian Senator Pauline Hanson suggested solving the growing cane toad population in Queensland with a Cash-for-cane toad program involving the unemployed.
For those who don’t know Pauline Hanson it’s probably one of her least offensive suggestions, but amazingly shows how people in power don’t understand human behaviour.
I don’t even need to talk theoretically. If you go far back enough history will teach you. In 1902 Vietnam’s sewer system was overrun with rats. The solution from the French governor? A rat rebate. To encourage people to kill the rats a fee was paid for every rat’s tail submitted. As you can imagine, an entrepreneurial population soon started breeding rats for their tails and the problem magnified.
Probably much scarier when it comes to targets and numbers is the news that Victoria Police falsified 258,000 breath tests. They faked quarter of a million negative test results. They did this so that positive results were no more than a target level of 0.5% of all tests. As the actual numbers of drink driving cases lower, it’s harder to hit each new lower target, but it’s easier to falsify the data.
This reminds me of Goodhart’s law.
Goodhart suggests;
“When a measure becomes a target,
it ceases to be a good measure.”
But it also reminds me of a recent episode of Reply All
The episode was all about Compstat. A system used in New York, designed to measure types of crime and their rates. It was the brainchild of Jack Maple, an eccentric underground cop (literally he patrolled the subway). His decade of observing crime on the subway led him to suggest that NY wasn’t the victim of a crime wave, but just a few perps who traveled the subway committing multiple crimes. The data he collected highlighted that crime rates matched the train routes and times. This allowed his team to predict where crimes would happen and enabled his team to set traps. The subway crime rate plummeted and his methods were deemed so successful he was made deputy police commissioner and implemented a data-based system across NY.
He used data to identify where crime was occurring and the types. He held sessions with police chiefs from each district to find out why certain crimes weren’t being investigated and these terrifying sessions were feared by all the Chiefs. So eventually over time and subsequent decades, when they couldn’t squeeze the crime rate down any further, they started to cheat. Rape started being reported as assault, assault as affray, and sometimes they may discourage the report altogether. Pressure was on every year to lower the crime rate. Each department was responsible for gathering the data so they had an incentive to alter it. At the same time, New York Mayor Rudi Gulianni had his broken window policy. He felt that an increase in arrests was a good thing. It meant every small crime was being dealt with, which in the long term prevented people from progressing to more serious ones. This was in contrast to Jack Maple who believed if you reduce the crime rate by arresting the right people then overall arrests won’t go up as you’ve literally arrested the problem. He felt crime wasn’t committed by lots of individuals, there weren’t lots of bad people in the city, just a few bad people doing a lot. Targetting the actual damaging crime was what solved the problem, not the small ones.
But Gulliani got his way as “tough on crime” is an easier story to sell and politically looked better. So arrests were added to key measures and therefore had to go up each year to show an improvement. Tricky if people aren’t committing a crime, so all cops ended up with quotas to arrest more and more, whether it’s a serious crime or just a minor offence, like jaywalking, or littering.
What everyone hadn’t really appreciated except Jack Maple was that the stats weren’t used to measure, they were used to highlight. They were used as an insight. They showed where to look. But the crime might be committed by lots or just a few. The situation was more nuanced than just numbers. But that requires work, investigation, skill in storytelling and understanding of motivations. Things often associated with soft approaches to crime. Whereas chasing numbers each year shows a strong approach to crime.
Jack Maple’s creation mutated into a machine that forced NY police officers to arrest more and more people to increasingly show they were being tough on crime. Even if that meant arresting a 16-year-old girl for blocking the sidewalk.
So where does this story bring us?
As a child growing up in the 80’s UK I remember the government’s solution to the rising unemployment rate was to redefine the term unemployed. Out of work for just a month didn’t count, they now measured how many jobs you looked for and how many interviews you attended, not enough meant you weren’t actually unemployed. The aim was to reduce the total, not to solve the problem.
VW recently got caught building detection evading technology to avoid emissions tests into their cars. Their flagship diesel‘s’ were a wonder of efficiency. How could they get the level of performance they suggested and be so clean with their emissions? Easy, they rigged their cars to know when they’re being emissions tested and alter their exhaust output.
There are countless other measures where the statistic is used as the primary goal rather than to help drive insight.
It reminds me of a project I worked on for a financial services client. After reviewing their application process we made a heap of recommendations. However, the larger suggestions meant changing their entire process. This was rejected as they had no way to identify what individual item made the business improvement. They weren’t disputing that it would probably improve the business, just that their business targets were based on measuring where the improvements came from. So the target became what they focused on, not what was better for the business. It made me realise that data was being used the wrong way, people were using it as their goal rather than their guide. Data needs to be used to highlight where to focus your attention.
So my message to all senior product and marketing managers is this;
Data isn’t your master it’s your servant and as David Ogilvy once said,
“Too many people use data as a drunk uses a lamppost,
for support rather than for illumination”.
