The Perils of Reviewing Peer Review

100percentAs a recovering federal employee, I recognize that one of the biggest challenges the government faces in funding science is that of being truly Darwinian in a rapidly evolving scientific environment. Engineering natural selection doesn’t come easily to government funding agencies, which are often playing with one hand behind the back.

I served in the government at NIH as a senior scientific executive. Now as Executive Director of ASCB, an independent scientific society, I must look at things from the outside while still remaining an integral member of the scientific community. From this vantage point, I can recognize how even experienced scientists can fail to understand how the wonderful and dedicated NIH employees must struggle to do the right thing. Evaluating proposals is like dancing on a floor overgrown with rules and red tape. It often seems more like a game of Twister than a waltz.

Perhaps one of the biggest challenges facing NIH is making the peer review system serve today’s science—not yesterday’s. For NIH, the charge is to deliver better health outcomes in 5, 10, and 20 years from today. So I applaud NIH efforts to continually review peer review.

The peer review system faces challenges at multiple levels. Perhaps the greatest is to enlist a steady supply of high-quality reviewers in the study sections. NIH needs the leaders of every field who can focus on the substance of the grants, without tinkering with minutia, and be able to craft helpful reviews so applicants can learn from the process and if possible improve future proposals. There is no simple solution here because the best reviewers are usually the best scientists, the best speakers, the best mentors, and the busiest people. Thus they are often unavailable for study sections. Currently NIH is receiving roughly 80,000 grant proposals a year, so finding qualified and objective reviewers is a never-ending hide and seek exercise.

Recently, the NIH announced an initiative that will take a fresh look at its peer review system from a slightly different angle. NIH is asking itself how internal review groups (IRGs) and study sections could be thematically organized. In particular, NIH wants to analyze input and output metrics for each study section. In other words NIH is interested in measuring productivity, although NIH seems very careful not to use this word. On the input side of the equation, the idea would be to see if the number of new applications and new awards reviewed by different study sections, controlled for field size, is balanced across study sections. On the output side, NIH wants to use various bibliometric analyses that rely mostly on citations to analyze the performance of each study section normalized to its field of science.

Thematic update is a question worth addressing. It is essential that the structure of the various IRGs stays relevant to the demands of ever-changing science. The first study section in NIH history was on syphilis but, clearly, structuring a study section today dedicated to syphilis would be less than optimal. Ensuring that study sections represent a good cross-section (no pun!) of current biomedical science is particularly important because of the NIH system of relative ranking (by percentiles ) across all IRGs in the current review round and in the two preceding rounds. If the IRG structure is thematically obsolete, the risk is that funding applications would not be of the highest novelty and quality for evolving science.

But here comes the thorniest problem—how. How do you evaluate scientific productivity? What are the best metrics? And what are the appropriate controls? The issue is particularly complex, and I am firmly convinced that there is no silver bullet, no single metric or single summary statistic that can adequately express the value and the relevance of a particular publication or set of publications. While I could imagine that such a summary statistic would be extremely helpful, it simply doesn’t exist. Science is way too complex and nuanced to be represented by a single metric. Ultimately, it is an imperative to read what is in a paper and judge the content of that contribution, regardless of the number of citations or other metrics that it has received.

Nobel laureate Renato Dulbecco once told me that, if his pioneering research in cancer were to be judged by the impact factors of the journals where it was published, not only would he never have gotten the Nobel Prize, but he probably would not have even been tenured. Last week, I was talking to a bright postdoc, who does research on extracellular matrix, who told me with great pride that he had just had two papers accepted in one day. I asked, “What were the papers about?” His response was almost automatic. He said one was in the journal of such-and-such and the other in the journal of this-and-that! I said, “Hold it. I asked you what, not where!”

Even our best and brightest are victims of this pervasive culture in science that focuses on where, rather than what and why. ASCB has expressed these sentiments on many occasions. Most recently, ASCB has been a leader in the so-called DORA insurrection against the misuse of impact factors, which is a particularly flawed metric. But most of all, ASCB wants to change the culture, help scientists focus on the important what, rather than where, while moving away from one single treacherous, flawed algorithm that runs our scientific lives by making them more miserable than they need to be.

But back to the NIH reviewing the peer review system. I applaud NIH’s effort to make sure that IRGs and study sections are aligned with present and emerging scientific areas. However, I hope NIH will stay away from inflaming a culture already overly influenced by treacherous metrics. Instead, NIH should lead the way by saying to the community that content matters, not a single metric, nor a single algorithm. NIH should demonstrate that in re-aligning its peer review system that what, not where is what matters.

Stefano Bertuzzi

Dr. Stefano Bertuzzi is the Executive Director of the American Society for Cell Biology. In this position he is responsible, with the ASCB Board, for strategic planning and all operations at the Society to serve the needs of its ~9,000 members and to promote the field of cellular biology and basic science.

Email This email address is being protected from spambots. You need JavaScript enabled to view it.

Most Recent Articles *

To read the full article, click on the READ MORE button below.

  • 1
  • 2
  • 3
  • 4
Prev Next
ASCB Kaluza Prize Reflects ASCB’s Concern that Young Scientists Are Now a Vulnerable Population

ASCB Kaluza Prize Reflects ASCB’s Concern that Young Scientists Are Now a Vulnerable Population

The ASCB Kaluza Prize recognizes significant research achievement by early career scientists who still pursue excellence amidst economic uncertainties and political shortsightedness. ASCB Photo.In the...

11-20-2013

Read more
Round Trip from Stockholm to ASCB

Round Trip from Stockholm to ASCB

With Stockholm still in the future, Randy Schekman (third from left) with fellow 2013 Nobel winner, Jim Rothman (far right), shared the ASCB's highest scientific honor, the E. B. Wilson Medal, with St...

11-06-2013

Read more
Colliding Worlds—A Rare Visit to the CERN Collider Gives a Biologist New Hope

Colliding Worlds—A Rare Visit to the CERN Collider Gives a Biologist New Hope

Nearly 170 meters beneath CERN, ASCB Executive Director Stefano Bertuzzi gets a rare look at the Large Hadron Collider. Here he stands in front of the detector. Photo Credit: ASCBLast week, I was invi...

10-23-2013

Read more
Watching Train Wrecks

Watching Train Wrecks

It’s full steam ahead for disaster or will the brakes stop the runaway fiscal crisis train in time? Photo by John FleischmanAlong with the future of U.S. research science, the train wreck metaphor has...

10-16-2013

Read more
UPDATED-NIH Furloughs to Widen—From Slowdown to Shutdown, U.S. Science Takes a Hit

UPDATED-NIH Furloughs to Widen—From Slowdown to Shutdown, U.S. Science Takes a Hit

The USDA is closed and no U.S. orders for international fruit flies can be processed until the government is back in business.Photo Credit: André KarwathUUPDATED—The "Activation Energy" blog has learn...

10-09-2013

Read more
Shutting Down Basic Science—Some Thoughts at Midnight (no, this is not a fairy tale)

Shutting Down Basic Science—Some Thoughts at Midnight (no, this is not a fairy tale)

Last night at midnight, the government ran out of money and shut down its main operations. Sadly, it left me wondering if perhaps our democracy ran out of steam as well. I've tried to explain what is ...

10-01-2013

Read more
New Partners and New Ventures for ASCB in Science Education Reform

New Partners and New Ventures for ASCB in Science Education Reform

In a 2012 Science editorial, Bruce Alberts, a former ASCB president who was then the journal's editor, urged professional societies to team up in leading innovation in science education. ASC...

09-25-2013

Read more
The Scholarly Paper That No One Will Want to Read Is Being Written in Congress

The Scholarly Paper That No One Will Want to Read Is Being Written in Congress

Spelling It Out: Spending on basic research has increased our longevity and the quality of that longer life. Photo Credit: John FleischmanThis week, a paper in the American Journal of Public Health, a...

09-18-2013

Read more
A Scientist-Senator with a Seat — and a Vote — for Life

A Scientist-Senator with a Seat — and a Vote — for Life

I recently spent some time on a family vacation in Italy. We have a tradition in our family; at the dinner table, each of us has to say the best and the worst thing that happened to him or her during ...

09-10-2013

Read more
Showing the Faces of Science

Showing the Faces of Science

In 2011, the Nobel Foundation awarded its prize of prizes to cell biologist, immunologist, and longtime ASCB member Ralph Steinman. When the foundation tried calling to deliver the good news, no one p...

09-05-2013

Read more
Lessons from a Society of Societies

Lessons from a Society of Societies

This week, ASCB went to ASAE, or rather five ASCB staffers and yours truly attended a meeting of the ASAE in Atlanta. ASCB, I trust you know, stands for the American Society for Cell Biology. The ASAE...

08-07-2013

Read more
Why Congress Should Be Worried About the Endoplasmic Reticulum

Why Congress Should Be Worried About the Endoplasmic Reticulum

Morphogenesis in biology always fascinated me. As a developmental neurobiologist at the bench, I studied how homeobox genes patterned the neural epithelium, the retina in particular, to understand the...

07-30-2013

Read more