Society for
Epidemiologic Research (SER) Presidential Addresses
SER President
Highlights Problems With NIH Peer Review and Offers Suggestions for
Change
How many epidemiologists does it
take to change a light bulb? The answer is five. One to install the
bulb and four to critique the methods.
This was the joke told by SER
President Leslie Bernstein in her opening remarks to
the audience gathered in Edmonton, Alberta for the 30th annual Society
for Epidemiologic Research meeting in mid-June. She may have been
poking fun at epidemiologists but she had a more serious point in
mind—namely that too much of a good thing (i.e. criticism), or at
least too much of a good thing done poorly, can cause very serious
consequences for epidemiologists.
In wondering what topic to
select for her SER address, Bernstein was influenced by the comments
of National Breast Cancer Coalition members who asked her, “why are
epidemiologists so incredibly critical of each other?” According to
these members, many of whom are laypersons, epidemiologists are much
more rigid and likely to be unsupportive of their colleagues’ ideas
than are scientists in other fields.
The epidemiology profession’s
proclivity for fault finding is a cause for concern, according to
Bernstein, because not all the reviews epidemiologists receive are
sound. “We can all cite examples of reviews we consider to be
unscientific, that have misunderstood our intent, and contain
criticisms that have little bearing on our ability to carry out our
proposed study successfully,” said Bernstein. Based on these
observations and experiences, and because of its importance to
professional success, Bernstein chose to focus her presidential
remarks on the status of peer review at NIH.
Problems with Peer
Review
Bernstein identified several
problems with peer review in NIH study sections as it is currently
practiced. The first is the high volume of work imposed on reviewers
that causes fatigue over time and makes reviewing a “wearisome chore.”
Also, many section reviewers do not read all of the numerous grant
proposals submitted to them and rely on the opinions of the primary
and secondary reviewers to score their proposals. Thus, not all the
scores from a review panel are independent measures of the quality of
the proposal.
Second, applicants cannot
anticipate who the reviewer will be and there is “the luck of the
draw,” according to Bernstein. The interests of the reviewer play a
critical role in the reviewer’s level of interest and enthusiasm for a
proposal. Also, personal feelings can come into play and whether or
not a reviewer likes or dislikes the researcher can “affect the tone”
of the review, says Bernstein. Some reviewers may be competitors of
the researchers and that creates a conflict of interest situation. How
much an individual proposal is affected by these vagaries is the “luck
of the draw” problem.
Third, there is a lack of
accountability for peer reviewers. “As reviewers of epidemiology
proposals, we often focus excess weight on issues of statistical
power, minor logistical issues, minor problems that may be encountered
in achieving secondary aims,” says Bernstein. “How do we hold
reviewers accountable for such actions?”
Solutions
To help correct these failings
in the peer review system, Bernstein urged her colleagues to serve on
peer review panels even if tired. It could be seen as jury duty or
made a precondition for receipt of a grant, she said. Also, giving
feedback to NIH is potentially useful, and she got the ball rolling by
offering the following suggestions for improvement:
1) Lessen the reviewer workload
by composing “Study Section Pools” of regular members
2) Require that reviews be
submitted in advance and “review the reviews” to assure high quality
3) Facilitate interaction
between the reviewer and the applicant, possibly by submitting the
review in advance to the applicant and allowing for the opportunity to
rebut the review
4) Develop an oversight
mechanism to review study section membership, appropriateness of
reviewers, and reviewer behavior within the review process.
In addition to suggestions for
NIH, Bernstein offered these suggestions to other epidemiologists:
1) Insist that we only review
grants for which we are appropriate reviewers
2) Be aware that every comment
we make at a review can affect the priority score
3) Prioritize our comments so
that other members of a study section will understand that the minor
issues are not fatal ones
4) Use the review process to
make helpful suggestions for improving the study while being careful
not to redesign the study
5) Train our graduate students
and post-docs to ensure that they understand the process, can compete,
and that they know how to review
Dr. Bernstein is professor of
preventive medicine, senior associate dean for faculty affairs, and
scientific director of the Cancer Surveillance Program at the
University of Southern California School of Medicine in Los Angeles.
Published July 1997 v
Postscript 2000
In my SER
presidential remarks made in 1997, I addressed the crises
epidemiologists were facing as subjects of the peer review process and
as peer reviewers implementing the process. Many of the problems I
described still exist, including our extremely critical stance
relative to our colleagues’ proposals. In addition, we now have to
determine whether an epidemiologic study is “innovative.” In our field
this could be considered an oxymoron. Further, we now face a growing
problem with “conflict of interest,” a result of our need to conduct
collaborative studies, particularly in molecular and genetic
epidemiology, to ensure sufficient statistical power. Those of us who
collaborate widely and who serve on numerous advisory committees are
unable to serve as reviewers when it might be perceived that a
conflict of interest exists. Perhaps we need a less stringent
definition of such conflicts. In addition, I believe that we still
need a workable oversight mechanism that reviews study section
membership, the appropriateness of reviewers and the reviews
themselves. On the plus side, over the past three years, we have seen
more experienced researchers providing service on peer review panels
and renewed interest in the etiology and prevention of diseases and
health conditions.
Leslie Bernstein
|
