According to an article in The Chronicle of Higher Education (30 July 2008), a study by Valen E. Johnson, a biostatistician at U Texas, scientists’ biases mar the purported objectivity of the grant review process.
The Chronicle notes:
Mr. Johnson found that the biases of those few actual readers can influence the votes of other members. If the readers tended to give scores more favorable than those of the other reviewers, then the proposals they read were more likely to receive grants, he reported. Correcting for those individual tendencies would change one-fifth to one-fourth of the decisions about which proposals receive financing.
The influence of bias was most severe, Mr. Johnson said, for proposals that received scores near the line separating those that received funds from those that did not.
He suggested that for proposals near that border, reviewers should take into account the dollar amount requested, and reject the more expensive ones. “If they did that,” he said, “they could fund more proposals than they do now.” Such a step would also encourage researchers to request less money, he argued, thereby increasing further the number of grants the NIH could provide.
The Chronicle article NIH’s Grant-Awards System Is Prone to Bias, Report Says by Lila Guterman is available on line to subscribers. Valen E. Johnson’s article Statistical analysis of the National Institutes of Health peer review system in PNAS is also available on line to subscribers.