More thoughts about more indices

CIPE have a piece on the growing number of indices. It is an interesting take on the issue but I disagree with their analysis, not because they say anything wrong so much as they miss what I think is the key point about indices. I had a go at previewing a paper I’m currently working on in a blog post here, so that is probably the best place to look for my thoughts. CIPE’s take is that more measures are good as long as they are accurate. Their beef is:

… one major weakness of many of today’s indicators and indexes, especially those that measure policies and institutions, is that a law on paper may be miles away from the reality of enforcement on the ground (as CIPE and its partners recently found in Kenya, for example) . Conditions may also differ radically from one region to another within a single country — Lima may be full of successful small and medium enterprises, but in rural areas of Peru attitudes towards entrepreneurship are still weak.

I disagree. The problem with the indices that I’ve looked at isn’t that they are right in general but wrong in specifics, or that they lack detail. Instead I think indices are very sensitive to a host of decisions that are ‘back end’ and the uninformed reader underestimates the effects that simple decisions have. An index is deceptively simple to explain, and it implies that this lack of complexity translates into a lack of sensitivity to different choices. From what I’ve seen they appear more whimsical and sensitive than regression techniques. They might be more dangerous as their simplicity can give the reader a false confidence.

Obviously, if the indices aren’t robust to simple changes in measurement technique, then they’re not giving a lot of information. And that means that the instruments on the pilots dashboard (to use CIPEs metaphor) aren’t working very well.

On the false simplicity of Simple Methods

Writer’s block  was one reason I started this blog. I started writing a paper that I had spent so long doing the data analysis for that I felt out of practice at writing. It is a piece about the current spate of rankings in development, a topic I’ve written about before. The piece basically looks at the actual measures the indices are based on and shows that there is a large amount of whimsy in these ‘simple’ methods. What’s more that whimsy leads to large differences in final conclusions. The take away is basically that these simple methods rely on the researcher’s judgement just as strongly as more complicated methods, but give a false sense of security.

Well, I came across an article comparing the ad-hoc methods of Bill James to more complicated techniques in the field of baseball. The question is whether one type of pitcher is better than another in hot/cold weather. Bill James is the father of the types of statistical analysis that are covered in the moneyball book and film. He paired 30 pitchers of the two types that had the same win-loss record, compared them in the cold weather month of April and overturned conventional wisdom, as was his wont. The author then favourably compares this method to an imaginary alternative regression. He argues that simple methods are better. I argue the exact opposite of this view in the new paper (which isn’t ready yet) in the field of development economics. Others seem to think that you have two choices. Choice 1 is simple, easy-to-understand, intuitive stats that give you the right answer but aren’t fancy. Choice 2 is easy-to-misunderstand, complicated and beloved of professionals who want to be employed to do this kind of work. From the article:

 

“You don’t have to trust Bill about it. Well, you have to trust that he aggregated the data properly, and didn’t cheat in what pitchers he chose. But you don’t have to trust Bill’s judgment, or Bill’s knowledge of baseball, or Bill’s interpretation.”

 

This is wrong. You do have to trust Bill, his judgement, knowledge of baseball and interpretation in either method. Yes, one method is easier to explain and can be written up in popular science books. However, there are many sources of whimsy, and if these choices lead to different outcomes they are important. To keep the baseball metaphor, Bill James chose one season, a group of 30 and one month. He equally could have chosen one of 30 other tests that are as easy to explain, not all of which will give the same answer. My basic argument is this: simple methods are easier to understand, but are too easy to trust. At least ‘complicated’ methods inspire some hesitancy on the part of the reader. Bill James was a good man to trust from the sounds of things, but I’m sure you could do other plausible tests of the hypothesis and find the opposite conclusion. The use of simple methods doesn’t mean that the reader doesn’t need to be skeptical of the results, but it often does. Which can make them more dangerous.