 Professor Andy Pitman |
When to peer-review a cab driverBy Andy Pitman* An Internet blogger has just come up with an idea for an operation that he says will cure cancer. Not only that, he has recommended a fantastic taxi driver to perform it. A newspaper columnist has picked up the story and reckons it's the best way to go. He writes convincingly to his legions of readers telling them that medical science got it wrong and there is no one better than this taxi driver to perform the difficult operation. He tells his readers that years of driving cabs and talking to his passengers - some of them doctors - has made the taxi driver the pre-eminent expert in cancer treatments. Meanwhile, 97% of medical scientists and surgeons are warning that the operation is likely to kill the patient. These medical experts refer to a massive archive of peer-reviewed research to support their case. The cab driver has never published a paper that has been reviewed by medical scientists but that doesn't matter, according to the columnist, because medical scientists can’t be trusted. They are all on a government-funded gravy train, which has a vested political interest in maintaining the health system. Clearly, he says, their point of view must be biased. So, which would you trust to deal with your cancer, the taxi driver who confidently believes they have the cure to cancer or the medical experts who have decades of peer-reviewed research backing them? Well, to decide you kind of need to understand the peer-review system. Why do scientists hold the peer review system as the Gold Standard? The peer-review system is the single best way known to test, challenge and clarify research. It is a little like Winston Churchill’s statement that “democracy is the worst form of government, except for all those other forms that have been tried from time to time”. Scientific peer review might be the worse way of doing science, except for all other ways that have ever been tried. In science, there is one measure of quality scientists agree on. Is the paper published in a peer-reviewed journal and, if so, how good is the journal? Journals like Nature and Science are commonly seen as best, but in a specific discipline other journals are also seen as world-class, in part because it is hard to get a paper published in them. Lets look at the process. A scientist develops an idea and a way to test that idea. They work on this for some time (commonly years). Once they think its finished they usually get their mates to read it. This is usually an informal processes but groups like CSIRO require “internal peer review” before a paper is submitted to a journal. Why? Well, the next stage is like running naked across the Melbourne Cricket Ground on Boxing Day. You would kind of want to be able to withstand the scrutiny. The next stage for a scientist’s paper is to submit it to a journal. An Editor selects two or three independent scientists as reviewers – this process is anonymous. Editors send papers to experts in the field who are not linked to the author or to the institution from which the paper came. These reviewers have the capacity to say what they think without fear or favour since they do not have to identify themselves to you. There is an aspect of ego on display here. If I submit a paper, then I think it’s scientifically right, clearly written, important etc. If you, as a reviewer, can show me where I have messed up, reached wrong conclusions etc there is something a bit satisfying for you in pointing this out. Scientists are ultra competitive – there is no silver or bronze in science, only gold. To come second is to be second rate. Lets assume, however, that I have written the paper well, and conducted the science in a way you cannot fault and lets assume the two other reviewers agree. This is actually rather rare by the way - usually the author has to revise the paper several times to reach the required levels of quality. But if you can satisfy the reviewers and the Editor your paper will be accepted and published. Champagne, chocs, perhaps a press release, pats on the back. I start dreaming of promotion, or that new research grant that I might now hope to win. However, that is not the end of it. Lets assume my paper is exciting. I have, perhaps, argued that some other group wrong or I believe that I have discovered something new. The more exciting the results the more I will be attacked. Those I believe I have proven wrong will attack to show that I have made a mistake. Others will think “that seems odd” and they will attack. All of those attacks will be attempted through the peer-reviewed literature. At every stage they will try to prove me wrong. ...the next stage is like running naked across the Melbourne Cricket Ground on Boxing Day. You would kind of want to be able to withstand the scrutiny.
If my science survives these attacks I will be delighted. However, it is not uncommon for others to show I was wrong. There is no disgrace in this – indeed, this is the very point of publishing in the peer-reviewed literature, to see if others with different approaches can prove me wrong. Science advances slowly and conservatively in this way – lots of people doing new science and then lots of people seeing if they can reproduce my results. Step by step, by step. There is a caveat to this process. If I am proved wrong because new data became available, that is okay. But if I am proved wrong because I have messed up more than a couple of times then my career will take a nose dive. I will become as employable as a soccer forward that never scores a goal. This means I have to be very careful in what I submit to journals to be published. So, peer review is a two-stage process. First, get the paper published. Second, can it withstand attacks. Getting a paper published does not mean the science is right – it only becomes scientifically legitimate when it is both published in the peer-reviewed literature and has withstood some years of attack. Examples such as evolution, plate tectonics and climate change science are examples of science that have withstood anything thrown at them. The only attacks that dispute the status of these areas of science are found on blogs and web sites. Without a peer-review process they have no legitimacy whatsoever. Of course, the science of evolution, plate tectonics and climate change science do advance. Specific details are improved and new details are discovered. If someone discovers a new detail, this does not make the whole theory wrong. However, if someone could publish one single paper that did prove Evolution, Plate Tectonics or Climate Change Science wrong – and that proof withstood subsequent scrutiny - they would win international glory. Around 21,000 journals from many disciplines use a peer review system. In fact there is no science journal that does not. Peer-review is not a convenience or an option – it is the basic foundation of all science. In the area of climate science, which is focussed on the basic science of global warming, there are now literally hundreds of papers that have tested the theories and failed to find significant fault with them. These papers confirm the basic science around global warming. There is not one single paper that has withstood subsequent analysis that proves global warming wrong. Not one. This is why climate scientists say, “the science is in". Any scientist who could write a paper that proved global warming wrong would over turn a theory as certain as Evolution. They would win a Nobel Prize, international glory, unlimited research funding, a position at the best University in the world. They would be scientifically immortal. So, why doesn’t some scientist write this paper? Because they can’t. No one can find a problem with the science. Peer-review is the bedrock of high-quality science. Nearly all of the important scientific discoveries in the 20th-century that we today take for granted came out of a peer review process. Our 21st Century lifestyle is built on these discoveries. By attacking the peer review system to dispute climate science, sceptics not only undermine science they, literally, stand in the way of progress. So, here is my test to help you determine the reliability of the information you are getting. It doesn’t matter if it is a blog, newspaper article or a report on television or the radio – the test holds good for all. First, find out if the report is talking about something that has been published in the peer-reviewed literature? No? Then it is not scientifically legitimate (though it might become so) and you can discount it and move on. If the source of the information does not tell you it has been published in peer-reviewed literature you can generally assume that it has not been. Second, is it published in a high quality peer-reviewed journal? How can you tell? Well, take a look at the ERA Journal Rankings Access website - the Australian Government’s ranking of journals. A* and A-rated journals are the elite. This does not mean B-rated journals are rubbish but A* and A-rated journals are commonly better. If the paper is in a journal that is not listed this website, you can discount it and move on. Third, has the paper withstood scrutiny yet? Have others tested it, analysed it? No? Then it might be exciting and seem important but it is not yet possible to determine if it will survive. If a paper is peer reviewed in a legitimate journal and it has been published for (say) five years, and other papers have not proven it wrong, then you can begin to rely on it. You can rely on that paper because full-time scientists are constantly investing time in reading all the literature, checking the results, seeing if someone else has proven it right or wrong and testing the paper’s results themselves if they have any doubts. This is what the scientists do full time. It is our job. The peer review system is incredibly robust, allowing scientists to challenge each other’s research in a civilised and completely transparent manner. With such a strong foundation underpinning their work, scientists in even the most controversial fields can be trusted. However, should you remain doubtful about the conclusions, the only other option is to invest your own time in looking at peer-reviewed science or carry out your own research which can then be published in peer-reviewed papers. In that way, you can still form your own views based on the science and nothing but the science. * Professor Andy Pitman is Director of the ARC Centre of Excellence for Climate System Science. |
Latest news
Latest blog entries
|
|
 |
 |
 |
 |
|
 |
Site map - Privacy statement - Copyright and Disclaimer -
Australian Research Council's Centre of Excellence for Climate System Science |