Who will publish boring results?

There was a great little discussion on radio 4′s today program about the British Medical Journal’s editorial about the publication of positive results and quiet death of negative or null results. It was a little simplistic in its conclusion, implying that publishing everything on the internet will mean there is no publication bias. I think the vast majority of work doesn’t get read anyway, and this kind of transparency only works if there is an unlimited attention span which is able to sort through the results.

Bad Science is a book about this kind of stuff, and as RCTs become more popular this book should get more attention in the development/economics community. It deals withthe problems within the ‘gold standard’, publication bias, media faults and flaws in scientific research. It is, bizarrely, only £1.99 on the kindle (UK) at the minute. But not for long apparently.

Note: See also this BMJ article about publication bias.  

Transparent about a Transparency Debate

On Tuesday a post about transparency went live. Owen Barder was not impressed. He said (on twitter: conversation between @aidwriting and @owenbarder) that “I agree that transparency is not sufficient; but I think it is very unfair (and unhelpful) to suggest anyone thinks otherwise.” (Ironically, this was in a private message in a debate about transparency.) Also, he accused me of making things up. Which I find odd for a number of reasons. Basically, there are two points to my fairly simple post. Point 1: Transparency is not an end in itself. Point 2: Some people seem to forget this. This is bad, we should use data better.

Owen obviously agrees with me on point 1. He has written very eloquently on the subject, and has made this point himself, for example :

Transparency by itself does not lead to more accountability, less waste, or better coordination. That happens when people are able to use the information

So, he disagrees on point 2.  ”I profoundly disagree. I don’t think any advocates assume it is an end in itself. Give us some credit. Do you have an example?” I stated that Easterly and Williamson’s recent paper that ranks donors includes a measure of transparency. E&W say one donor is better than another in part because one donor replied to their emails/put info on its website, and the other didn’t. The QuODA rankings do a similar thing. Their indicators of transparency directly influence their judgements of the quality of aid. In my view, this logically means that you are saying transparency is good in itself. A measure of transparency helps determine a measure of aid quality. Surely that means you think that transparency determines aid quality, no? Owen Barder disagrees, claiming a distinction between sufficient and necessary. Hmm. I’m not convinced.

So, what are my thoughts? First, I don’t take kindly to people saying I made things up. Especially when, as far as I can see, I followed basic logic. The blog post is actually very kind about transparency, and admitted that I might be being a grump. Second, we disagree on whether allowing a ranking of donor/aid quality to be partly determined by a measure of transparency means you think transparency is a determinant of aid quality. I don’t feel the force of Owen’s argument here, which always makes me think I’ve missed something, but right now I just don’t get it. Third, if it is unkind and unfair to imply that anyone might not know that transparency is not an end in and of itself, why would one feel the need to write the above quote?

Let us end on a positive note. The main point I wanted to make in the previous post was this: lets make sure that we use data better. For me that means moving away from rankings, but hopefully some good research will get done using all kinds of approaches.

The Promise of Transparency

The fourth High Level forum on aid effectiveness, Busan, will talk a lot about transparency. There is a lot of optimism for what transparency can offer, and it fits within a general context of more transparency. I am thinking of wikileaks, but also the UKs growing commitment to open data in all walks of government and the MPs expenses scandal in the UK. The idea is very much that obtaining information leads to better public scrutiny, which leads to better policy.

There are some clear  ’wins’ that can be chalked up in transparencies favour. The Extractive Industries Transparency Initiative seems like a good case study. By getting information on how natural resource contracts work they are pushing the equilibrium in the favour of developing countries. Perhaps an under-appreciated aspect is how this works through simple information sharing between developing countries. This makes the most of technical expertise and learns sensible lessons about past behaviour.

There is also a part of me that looks at the prospect of new data with a great deal of enthusiasm. That is because I am a development economist, who typically uses data to answer interesting questions. The better the data available, the easier my job. A friend was working on interest rates as the finanical crisis of 2008 happened. While she was obviously not happy about a global recession, her professional mind was excited by the turn of events. Being excited by a global financial crisis is the economist’s lot I suppose. The development community will generally push to get more information than the equilibrium between costs and benefit suggests, because intellectual curiosity means data is interesting to us, even if not useful for others.

While the role of transparency in the UK is quite positive, I am not yet absolutely convinced about the role of transparency in development. Especially given the opportunity cost of time, money and political will. Why? Two reasons. First, there a lots of examples of donors failing to carry out promises where transparency hasn’t made a difference. To take one example, the failure of donors to honour the aid commitments they made in Gleneagles (independent ’09, oxfam ’11 and guardian ’11). There are other things we already know: the USA tie too much aid (ironically, we know this because unlike other donors they don’t report how much they tie), all donors spread their aid too thinly and there is too much volitility in the aid system. There are so many different promises and so little attention that I don’t have much hope in the ability to build and sustain public pressure on all of the important fronts.

Second, I am often less than impressed with how the data is used. The popular method right now is an index, which typically collates a load of data on a number of topics and gives a nice clean ranking of donors. I dealt with one type of ranking in a paper with Ed Anderson (pdf working paper version, and see posts on this), but the bottom line is we didn’t much like the assumptions underlying the measures. My point here is similar to the crime map in the UK: crime was shown over the UK and anomalies lead to some very odd conclusions. If the only consequence of the new data is that we get a wave of news stories with simple observations devoid of analysis, transparency hasn’t really helped. Basically I am worried that the data may produce more heat than light.

Neither of these reasons are fatal or inevitable. Maybe I am just being a grump. One of the joys of making data available is that people find interesting ways to use the data that would never have been envisaged beforehand. I hope the promise of transparency is fulfilled, but that fulfilment will not be automatic.