In science we trust

24 Mar 2014

The Cambridge Science Festival is an amazing event. Over a fortnight there are hundreds of events on all aspects of science. Many of them have very catchy titles and appear extremely interesting and it’s clear that quite a few are aimed at children, including NIAB's own stand in the Plant Science Marquee talking about roots and shoots (see picture)! What a shame that I might stand out if I try one of these events.NIAB at the Cambridge Science Festival 2014

Last Saturday I attended a panel discussion (for adults) on the publication and communication of science. I was the oldest person in the 200 plus audience! The peer review system, where the paper is checked before publication by two or three other scientists, has been the established method for around 300 years for trying to ensure the integrity of science. Generally it’s stood the test of time well but it’s certainly not perfect. For the author it can lead to frustration, particular when the reviewers, usually anonymous, provide conflicting advice and comments. Then there are the re-writes to endure.

However, once through this process, the author can proudly say that he or she has a peer reviewed paper. There’s a hierarchy of journals in which to publish and scientists aim to submit papers to so-called high impact publications that only include the best of science. Down the pecking order are numerous journals that will have peer reviewed papers but there is a suspicion that some appoint reviewers who are less than forensic and who share the same views as the author.

A few years ago, a peer reviewed journal published a paper saying that organic agriculture produced similar or higher yields than conventional agriculture and could easily feed the world. It claimed to be a review of all relevant papers where organic and conventional systems were compared. The organic movement was triumphant. It must be true, they said, because it is in a peer reviewed paper.

The conclusions of this paper were a shock to anyone who had experience of the relative production levels of the two systems and prompted an American researcher to check the sources of information used in this review. His findings were shocking.

Over 100 of the papers included in the review described only a comparison between different conventional systems and not a comparison between organic and conventional systems. Also organic yields were misreported and false comparisons were made to unrepresentatively low conventional yields. Where reasonable organic yields were achieved they were counted 2, 3, even 5 times by citing different papers that referenced the same data. Finally, favourable and unverifiable “studies” from biased sources were given equal weight to rigorous studies. I must admit that when I read this critique, my faith in the peer review system plummeted.

There has also been a recent paper suggesting that GM ‘causes cancer’ and this has had to be withdrawn because of weaknesses in the peer review system of the journal.

Trust in science is absolutely vital.  The panel discussion I attended pointed towards a way that increases trust. Not surprisingly, it involves the dramatic innovation that has occurred since the peer review system started 300 years ago - the Internet.

The panel predicted that science will slowly move to a system something approximately along the lines of the following: the draft paper is submitted to a journal and reviewers, perhaps not anonymously this time, are appointed; and then the draft paper, together with the reviewers’ comments, will then appear on the journal’s website plus the data generated in the project, for anyone to review and comment on. 

This whole process could take a year before a final paper script is agreed.  Such an approach would result in the non-publication of the examples that I have previously quoted.

The same approach should now be contemplated by Defra and the UK levy organisations. Recently I have aired my misgivings on the conclusions of the Defra project NT26 where the efficacy of ammonium nitrate and urea were compared (‘Up in the air’).

I feel that a more open discussion should have taken place before the final conclusions were widely promulgated. The primary question should have been why the project concluded that the nitrogen in urea was on average 20% less efficiently taken up by the crop than that in ammonium nitrate, and yet a major review of UK trials’ data, in a peer reviewed journal in the 1990s, suggested that the difference was 3.5%. I also have concerns with the lack of open debate about some of the conclusions in some levy-funded reports.

I think that the era of improved electronic communications will mean that the trend towards more openness and greater transparency will prove unstoppable and should lead us to say with more confidence that in science we trust.