Coffee!! Is peer review the wheel of life or the Wheel of Fortune?

Bit of both. Neuroskeptic offers:

In the spirit of the 9 Circles of Scientific Hell, and inspired by the evidence showing that scientific peer reviewers agree only slightly more often than they would by chance, here's a handy tool for randomly generating your review.

How's this one:

4. Cite Me, Me, Me!: The problem with this paper is that it doesn't reference the right previous work... yours. Unless the authors change it to cite everything you've written in the past 10 years, they can get lost. If they do, the paper will be immediately accepted - to reject it would harm your citation count.

Some readers may wish to try it on one of their work in progress or on this week's  grocery flyer.

Hat tip: Stephanie West Allen at Brains on Purpose

More peer review stories:

Everyone hates the blogosphere and loves peer review, right, but ...

Early coffee: Traction, retraction, and self-plagiarism (when scientists retread what they should retire)

Peer review: have we run out of polish for the iron rice bowls?

Peer review: have we run out of polish for the iron rice bowls?

At Slate, Daniel Engber offers another slam at peer review:

When journal editors are asked about these ideas, they often quote Winston Churchill's line, "Democracy is the worst form of government except all those other forms that have been tried from time to time." Or rather, they quote other journal editors quoting that line. But it's a poor analogy, since few alternatives to peer review have been tried in modern times. And democracy isn't really a good description of peer review, either. Sure, peer review allows scientists to participate in a system of self-governance. But wouldn't BMJ's policy of open review or Ginsparg's proposal for Web-published preprints be far more democratic?

So far, though, the Churchill quoters are winning.

You know, "The worst system , except for all the others." The trouble is, any system can exhaust the benefits for which it was brought in- in this case, to cope with the flood of post-World War II science efforts. In my own view, it has become the same sort of drag on fresh thinking as reliance on Aristotle was in the early modern period of science.

If the object is to do good science while pleasing all possible reviewers, and the gist of the paper is an idea that disconfirms their theories, one may have to downplay findings, quit the field, or go nuts. Michael Behe is a rare example of someone who stood up to all the garbage, just to make a simple point or two about the shortcomings of Darwin's Rice Bowl.

Other peer review stories:

Peer review: How much more believable than fortunetelling?

"Peer review, mere review, and smear review"

"Peer review: Life, death, and the British Medical Journal"

Science: A year-end wad of fraud, falsified data, and other award-winning tenure strategies ...

Peer review: What if your peers would have to be otherconspiracy theorists? (No, really!)

Peer review: Gold standard or gold in "them thar hills"

Peer review, mere review, and smear review

Andrew Sibley here discusses a thoughtful article by Fred Pearce in the Guardian (02 February 2010) on the climate change scandal, an article which had also been mentioned to me by a kind reader recently. The article takes a critical look at peer review, a well-justified critical look in my view.

I have written about the problem with peer review here, and would recommend Frank Tipler's paper on the subject.

The basic problem is that the peer review process, intended to enforce quality, can end up enforcing mere orthodoxy or, worse, mediocrity. Or worst of all, as in the now-famous climategate e-mails, it can lead to a classic "bunker" mentality.

I would be inclined to treat all science-based dissent as legitimate. The mere fact that some scientists cannot replicate others' work or support their conclusions is not evidence of incompetence or dishonesty. It may lead to useful corrections or valuable new information.

Of course, if someone claims that climate change is caused by space aliens, an evil plot by a minority group, or proof that Jesus is coming again soon, I would say, please, this is not science. Science is about evidence from nature.

I was trying to remember recently what peer review reminded me of, and then I suddenly remembered:

For some time, politicians in my country have tried to prevent interest groups from publishing the opinions of politicians about controversial issues during an election. All the parties voted for that. Of course they voted for it! They piously informed the public that their policy prevented wealthy special interests from hijacking the election.*

But that was nonsense. What the policy really did was guarantee that politicians could keep off the table issues that no party wanted to tackle, even though much of the electorate wanted the politicians to tackle them.

Peer review can function the same way. It can simply prevent the publication of problematic data that the current establishment in science does not want to tackle.

*This problem of wealthy special interests could be dealt with simply by requiring any participant to identify the funders within the ad itself. If it turned out to be Microsoft or Ford or McDonald's, well, anyone smart enough to find their way to the polling station without falling down a hole somewhere would consider the possible motives.

Peer review: Life, death, and the British Medical Journal

This controversy erupted over estimates of war deaths since World War 2 (1939-1945:)

Researchers from Canada, the UK and Sweden have slammed the influential British Medical Journal (BMJ) for publishing an error-filled study on global war deaths, refusing an equivalent rebuttal article and having a flawed peer-review process.

Apparently, the contested article took issue with the fact that Oslo's International Peace Research Institute data show that global war deaths "declined by more than 90 per cent between 1946 and 2002."

"This is not some trivial academic disagreement," says Andrew Mack, director of the Simon Fraser University-based Human Security Report Project (HSRP), which p published a detailed critique of the BMJ's claims in the December issue of the Journal of Conflict Resolution (JCR).

"Accurate statistics on the health impacts of war are critically important not just for researchers but also for humanitarian organizations whose assistance programs save millions of lives around the world."

The BMJ doesn't deny the problem:

"But the BMJ is well aware that its peer review process is flawed," says Spagat. "A recent study, whose authors include the journal's current editor, revealed that, on average, only a third of the 'major errors' deliberately inserted in a BMJ article were picked up by reviewers."

In what other line of work would such incompetence be accepted? Would you like your electrician to achieve only this level of competence? He only "gets" one third of the electrical safety hazards in your home?

And remember, if you live in the UK, your taxes pay for these scholars to "do their thing."

Adds Mack: "There appears to be no way of effectively rebutting BMJ articles that contain unwarranted -- and damaging -- critiques of the work of other scholars.

A couple of years back, I wrote on the problem of peer review: Often, it is simply the way establishment hacks prevent competition from new information and new interpretations.

Re war deaths, two notes:

- It would hardly be surprising if deaths in battle declined steeply, post World War II, because battlefield medicine has greatly improved. Indeed, it was improving during the war itself (1939-1945), and some sources credit the Allies' victory in part to discovery of penicillin, which restored personnel who might otherwise have been disabled or dead. Plasma, anti-malarials, and other drugs also received a huge boost due to the War.

- Modern warfare increasingly targets civilians. It could be 9-11. Or 7-11. Or it could be someone's granny, shopping at a Halal meat market in Iraq, to prepare a family celebration. When the conflict is between a trained terrorist and your granny, you should expect low "battlefield" casualties. That is not a battlefield, after all; it is a monstrous crime scene.

- Still, it ought to be possible to maintain another point of view, with solid references. That's one thing peer review should enable, but it is increasingly obvious that peer reviewers do not want to bother.

Anyway, the intelligent design controversy is hardly the only area where peer review can merely maintain a convenient consensus - or tweak beards safely in a politically correct way.

Find out why there is an intelligent design controversy:

Peer review: What if your peers would have to be other conspiracy theorists?

A couple of years ago, I wrote an underappreciated article on the growing scandal of peer review. Now, according to historian Max Holland, Harvard University's Bellknap Press seems to be getting in on the act of publishing peer reviewed conspiracy literature:

The scholarly sheen given The Road to Dallas is not wholly unprecedented, to be sure. A few other fallacious books about the assassination have received the academic stamp of approval.[3] Still, that begs the question: How did The Road to Dallas ever survive the gauntlet of peer-review at the august Harvard University Press, which is part of an industry that likes to think of itself as “a bulwark against the confusions of error and unsupported opinion, of ideology masquerading as fact, magic as science, and prejudice as theory.”[4]

Neither Kaiser nor HUP was willing to answer questions about the editorial vetting The Road to Dallas underwent. “Any such questions would have to be addressed to the Press,” Kaiser responded when asked, and HUP refused to disclose information about its editorial process in general, or as it pertained to Kaiser’s book specifically.[5] Indeed, insofar as HUP is concerned, the rotating membership on its Board of Syndics—or what other university presses call their editorial or publications committee, and readily make public—is top secret.

Holland engagingly quotes:

As Thomas Jones, a contributing editor at the London Review of Books observed in 2005, "The first rule when concocting a conspiracy theory is not to make any claims that can be proved not to be true . . . . A decent conspiracy theory is made up of hard facts; the invention lies in drawing the connections."

Regular readers of this space will know that I regard all conspiracy theory as presumptive evidence that the author is a crank. Most people simply cannot keep a secret whose disclosure would enhance their social importance. And most of what goes on in the world is a mystery only to those who do not pay close attention.

Hat tip Five Feet of Fury

Peer review: A recipe for squelching science?

A friend draws my attention to this choice quotation from an article in Current Biology, Volume 17, Issue 15, Pages R583-R585 by P. Lawrence, in which he quotes the answer the hero of Leo Szilard's 1948 story "The Mark Gable Foundation" provided when a wealthy entrepreneur asked him how he could slow science down:

You could set up a foundation with an annual endowment of thirty million dollars. Research workers in need of funds could apply for grants, if they could make a convincing case. Have ten committees, each composed of twelve scientists, appointed to pass on these applications. Take the most active scientists out of the laboratory and make them members of these committees. ...First of all, the best scientists would be removed from their laboratories and kept busy on committees passing on applications for funds. Secondly the scientific workers in need of funds would concentrate on problems which were considered promising and were pretty certain to lead to publishable results. ...By going after the obvious, pretty soon science would dry out. Science would become something like a parlor game. ...There would be fashions. Those who followed the fashions would get grants. Those who wouldn't would not.

Why does this sound like the peer review problem I reported on last fall?

Peer-reviewed literature that supports intelligent design?: Here is a list from a friend

Recently, on a Vancouver radio program, I said that I was aware of nine such papers (in response to a claim that there weren't any), but a friend has sent me a list of 12. Earlier, I had written that they were all peer-reviewed, but apparently not all are, so I am asking for clarification. Maybe my original count will turn out to have been correct:

- Stephen Meyer, "The Origin of Biological Information and the Higher Taxonomic Categories" Proceedings of the Biological Society of Washington 117(2004):213-239.

- Lönnig, W.-E. Dynamic genomes, morphological stasis and the origin of irreducible complexity, Dynamical Genetics, Pp. 101-119.

- Jonathan Wells, "Do Centrioles Generate a Polar Ejection Force? Rivista di Biologia/Biology Forum 98 (2005): 37-62.

- Scott Minnich and Stephen C. Meyer, "Genetic Analysis of Coordinate Flagellar and Type III Regulatory Circuits," Proceedings of the Second International Conference on Design & Nature, Rhodes Greece, edited by M.W. Collins and C.A. Brebbia (WIT Press, 2004).

- William A. Dembski, The Design Inference: Eliminating Chance through Small Probabilities (Cambridge: Cambridge University Press, 1998).

- M.J. Behe and D.W. Snoke, "Simulating Evolution by Gene Duplication of Protein Features That Require Multiple Amino Acid Residues," Protein Science, 13 (2004): 2651-2664.

- W.-E. Lönnig & H. Saedler, "Chromosome Rearrangements and Transposable Elements," Annual Review of Genetics, 36 (2002): 389-410.

- D.K.Y. Chiu & T.H. Lui, "Integrated Use of Multiple Interdependent Patterns for Biomolecular Sequence Analysis," International Journal of Fuzzy Systems, 4(3) (September 2002): 766-775.

- Ø. A. Voie, "Biological function and the genetic code are interdependent," Chaos, Solitons and Fractals, 2006, Vol 28(4), 1000-1004.

- John A. Davison, "A Prescribed Evolutionary Hypothesis," Rivista di Biologia/Biology Forum 98 (2005): 155-166.

- Granville Sewell, "A Mathematician's View of Evolution," The Mathematical Intelligencer, Vol 22 (4) (2000).

So he keeps better records than me. So? I wonder if all these guys can be retroactively flunked.

If you like this blog, check out my book on the intelligent design controversy, By Design or by Chance?. You can read excerpts as well.

Peer review: My recent piece sure garnered plenty of comments

The piece I put up the day before yesterday on peer review has garnered a lot of comment. The riff piece at Evolution News and Views probably did too.

Clearly, I was on to something.

There have been some thoughtful reactions, some establishment huffing, and some bizarre reactions. These stand out:

- many people treated the problem as one I had invented, when in reality it only came to my attention because I had lots of stories in an inbox slot featuring high-level dissatisfaction and broadly based new systems . I let a number of stories on peer review get away before I realized that this was definitely slow-cycle news - perfect for a backgrounder.

- some assumed that I had written the piece to highlight the ID guys' troubles, but their troubles are really only a small piece of a bigger problem, and rated only a mention in the article. What I said was "- given the woeful state of peer review - papers that support or undermine ID hypotheses would probably be neither better nor worse recommended if they were never peer reviewed, just published, amid cheers and catcalls."

(Don't forget that, just as ID guys' papers may be unfairly held back, Darwinists' papers may be unfairly advanced, with glaring errors for the ID guys to pick apart. Both outcomes undermine the quality of the system as a SYSTEM.)

- Many people had a hard time grasping a fact that seems simple to me: There is no conflict between the inevitable limitations of a system and the fact/possibility that it is not working. Indeed, that is the precise circumstance that drives the demand for a new system. When I am told that the peer reviewers cannot possibly examine the data in detail due to time constraints, I agree that that is true. But in fields where data rules, I am hardly surprised that new systems are sought where eager grad students may be more than willing to spend the time in order to make their name.

My own view is that the system will become more open and dynamic whether anyone likes it or not because that is simply how things go these days.

If you like this blog, check out my book on the intelligent design controversy, By Design or by Chance?. You can read excerpts as well.

Peer Review: Gold standard or gold in them thar hills?

I've been busy writing, and now with radio, so I haven't been blogging for a couple of days, but here is a piece I just put up elsewhere, in five parts, on peer review. Back to blogging soon.

Introduction

Part One: If peer review always worked before, why doesn't it work now?

Part Two: How bad can it get? Pretty bad.

Part Three: How the system is slowly becoming more open and dynamic, whether anyone wants it to or not

Part Four:How will we know if a more open system works better?