On the false simplicity of Simple Methods

Writer’s block  was one reason I started this blog. I started writing a paper that I had spent so long doing the data analysis for that I felt out of practice at writing. It is a piece about the current spate of rankings in development, a topic I’ve written about before. The piece basically looks at the actual measures the indices are based on and shows that there is a large amount of whimsy in these ‘simple’ methods. What’s more that whimsy leads to large differences in final conclusions. The take away is basically that these simple methods rely on the researcher’s judgement just as strongly as more complicated methods, but give a false sense of security.

Well, I came across an article comparing the ad-hoc methods of Bill James to more complicated techniques in the field of baseball. The question is whether one type of pitcher is better than another in hot/cold weather. Bill James is the father of the types of statistical analysis that are covered in the moneyball book and film. He paired 30 pitchers of the two types that had the same win-loss record, compared them in the cold weather month of April and overturned conventional wisdom, as was his wont. The author then favourably compares this method to an imaginary alternative regression. He argues that simple methods are better. I argue the exact opposite of this view in the new paper (which isn’t ready yet) in the field of development economics. Others seem to think that you have two choices. Choice 1 is simple, easy-to-understand, intuitive stats that give you the right answer but aren’t fancy. Choice 2 is easy-to-misunderstand, complicated and beloved of professionals who want to be employed to do this kind of work. From the article:

 

“You don’t have to trust Bill about it. Well, you have to trust that he aggregated the data properly, and didn’t cheat in what pitchers he chose. But you don’t have to trust Bill’s judgment, or Bill’s knowledge of baseball, or Bill’s interpretation.”

 

This is wrong. You do have to trust Bill, his judgement, knowledge of baseball and interpretation in either method. Yes, one method is easier to explain and can be written up in popular science books. However, there are many sources of whimsy, and if these choices lead to different outcomes they are important. To keep the baseball metaphor, Bill James chose one season, a group of 30 and one month. He equally could have chosen one of 30 other tests that are as easy to explain, not all of which will give the same answer. My basic argument is this: simple methods are easier to understand, but are too easy to trust. At least ‘complicated’ methods inspire some hesitancy on the part of the reader. Bill James was a good man to trust from the sounds of things, but I’m sure you could do other plausible tests of the hypothesis and find the opposite conclusion. The use of simple methods doesn’t mean that the reader doesn’t need to be skeptical of the results, but it often does. Which can make them more dangerous.