Journal Impact Factors: No Shortcuts for Research Assessment

A special “Activation Energy” blog by Stefano Bertuzzi and David G. Drubin

Stefano Bertuzzi is Executive Director of the American Society for Cell Biology. David G. Drubin is Editor-in-Chief of the ASCB journal Molecular Biology of the Cell.

impactImpact is not easily predicted in real world science yet researchers are increasingly judged
by a deeply flawed metric which measures where their articles are published not on their content.

Funding agencies, journal editors, and hiring and promotion committees expend large amounts of time and resources deciding how to allocate precious funds, what science to publish, and which scientists deserve a job or promotion because of their scientific contributions. Now imagine an automated information system that can make this process much more efficient. Every academic researcher in the world is ranked based on a productivity index. Let us call it the Metric for Evaluation of Scientific Scholarship (MESS). This system works with an algorithm that tabulates the number of grants awarded to a researcher, the award amount for each grant, the number of publications authored, and the number of times those publications were downloaded and cited. In addition, the MESS would include a variable reflecting the prestige of the journals in which a researcher’s work was published. The MESS would be adopted to rank all researchers in the world, providing the basis for hiring, funding, and promotion decisions. And no need for a Nobel Prize selection committee, either! We have the MESS. The MESS algorithm would be undisclosed proprietary information, but the MESS rankings would be available to anyone willing to pay a fee for access to this means of ranking researchers.

Such a frightening system might have a few minor flaws, like favoring senior investigators and researchers in fields with more funding and more publications and replacing the quest for deeper understanding with a quest for a higher MESS. As absurd as this imaginary system might seem, sadly, the research community often relies on a system that is almost as absurd as this hypothetical one, and researchers have only themselves to blame for this unfortunate state of affairs.

The Journal Impact Factor (JIF), developed to help librarians make subscription decisions, has de facto been repurposed by researchers, journals, administrators, and funding and hiring com­mittees as a proxy for the quality and importance of research publications. The result of this shortcut is that researchers are judged by where their articles are published rather than by the content of their publications. This is fundamentally wrong.

To address the issue, a group that includes representatives from many leading scientific journals, funding agencies, and research institutions across the globe has released the San Francisco Dec­laration on Research Assessment (DORA), which has been posted on the website of the American Society for Cell Biology. This document is a call for reform of how research outputs are assessed. Anyone can sign the document, if he or she wants to support this cause.

There are many reasons why shortcuts to research assessment don’t work. One reason is that outputs and outcomes from researchers are varied. These include the following, in addition to publications: data sets, new methods, reagents, and computer programs, scientists trained, contributions to society, and influence on public policy, better health, a cleaner environment, and more efficient utilization of energy resources. Assessment of the value of any of these outputs and outcomes requires an appreciation for context and history and often can only truly be achieved retrospectively.

Although the JIF is the metric that is most often misused to quantitatively assess research outputs, many other metrics, based on different assumptions and algorithms, have been introduced over the years. The drafters and signers of the San Francisco Declaration believe that assessing a research publication requires actually reading it and understanding its content. Metrics, particularly article-specific metrics, may augment such an assessment by providing a numerical gauge of how well an article has been received and has influenced subsequent work, but in such cases a whole array of metrics, not just one numerical ranking, should be used.

Misuse of journal metrics has harmful consequences for scientists and science. Many scientists feel pressure to publish in journals with the highest JIFs. In embracing JIFs as meaningful tools for research assessment, scientists are in effect handing over research assessment to journal editors. Journal editors do their best to select good work for publication and to choose papers with the widest appeal to journal readers, but their decisions often amount to informed guesses. The value of an article cannot really be determined until after it has been published. Moreover, when a publication is evaluated according to the JIF of the journal in which it appeared, it is really being evaluated not on its own merits but on the number of citations to all of the other articles that happen to be published in that journal. But of course it is perfectly possible for very important (and highly cited) work to be published in a low-JIF journal or for work that ultimately turns out to be uninteresting or even wrong to be published in a high-JIF journal. Does the scientific community really want its work to be evaluated by the company it keeps rather than on its own merits?

The San Francisco Declaration is a call for scientists to take control of research assessment. The Declaration encourages researchers serving on funding, hiring, and promotion committees to judge research articles based on their content, not on where they are published, and even less on the JIFs of the journals in which they are published. In addition, researchers are encouraged to look beyond metrics when choosing where to submit a manuscript for publication. Different journals fill different niches. Some journals are highly specialized, and some target general audiences. Some journals mainly publish short articles, and some publish long ones. The edi­torial board of each journal covers certain areas better than others. We look forward to establishing a scientific culture in which authors have the incentive to target their manuscripts to a journal based on their intended audience, best fit for format, and scope, not on the JIF of that journal.

This editorial appears in the May 15, 2013, issue of Molecular Biology of the Cell (Mol Biol Cell 24, 1505–1506) under license from the authors. It is reprinted here by permission. DOI:10.1091/mbc.E13-04-0193

Address correspondence to: Stefano Bertuzzi This email address is being protected from spambots. You need JavaScript enabled to view it. and David G. Drubin This email address is being protected from spambots. You need JavaScript enabled to view it. .

Stefano Bertuzzi

Dr. Stefano Bertuzzi is the Executive Director of the American Society for Cell Biology. In this position he is responsible, with the ASCB Board, for strategic planning and all operations at the Society to serve the needs of its ~9,000 members and to promote the field of cellular biology and basic science.

Email This email address is being protected from spambots. You need JavaScript enabled to view it.

Most Recent Articles *

To read the full article, click on the READ MORE button below.

  • 1
  • 2
  • 3
  • 4
Prev Next
ASCB Kaluza Prize Reflects ASCB’s Concern that Young Scientists Are Now a Vulnerable Population

ASCB Kaluza Prize Reflects ASCB’s Concern that Young Scientists Are Now a Vulnerable Population

The ASCB Kaluza Prize recognizes significant research achievement by early career scientists who still pursue excellence amidst economic uncertainties and political shortsightedness. ASCB Photo.In the...

11-20-2013

Read more
Round Trip from Stockholm to ASCB

Round Trip from Stockholm to ASCB

With Stockholm still in the future, Randy Schekman (third from left) with fellow 2013 Nobel winner, Jim Rothman (far right), shared the ASCB's highest scientific honor, the E. B. Wilson Medal, with St...

11-06-2013

Read more
Colliding Worlds—A Rare Visit to the CERN Collider Gives a Biologist New Hope

Colliding Worlds—A Rare Visit to the CERN Collider Gives a Biologist New Hope

Nearly 170 meters beneath CERN, ASCB Executive Director Stefano Bertuzzi gets a rare look at the Large Hadron Collider. Here he stands in front of the detector. Photo Credit: ASCBLast week, I was invi...

10-23-2013

Read more
Watching Train Wrecks

Watching Train Wrecks

It’s full steam ahead for disaster or will the brakes stop the runaway fiscal crisis train in time? Photo by John FleischmanAlong with the future of U.S. research science, the train wreck metaphor has...

10-16-2013

Read more
UPDATED-NIH Furloughs to Widen—From Slowdown to Shutdown, U.S. Science Takes a Hit

UPDATED-NIH Furloughs to Widen—From Slowdown to Shutdown, U.S. Science Takes a Hit

The USDA is closed and no U.S. orders for international fruit flies can be processed until the government is back in business.Photo Credit: André KarwathUUPDATED—The "Activation Energy" blog has learn...

10-09-2013

Read more
Shutting Down Basic Science—Some Thoughts at Midnight (no, this is not a fairy tale)

Shutting Down Basic Science—Some Thoughts at Midnight (no, this is not a fairy tale)

Last night at midnight, the government ran out of money and shut down its main operations. Sadly, it left me wondering if perhaps our democracy ran out of steam as well. I've tried to explain what is ...

10-01-2013

Read more
New Partners and New Ventures for ASCB in Science Education Reform

New Partners and New Ventures for ASCB in Science Education Reform

In a 2012 Science editorial, Bruce Alberts, a former ASCB president who was then the journal's editor, urged professional societies to team up in leading innovation in science education. ASC...

09-25-2013

Read more
The Scholarly Paper That No One Will Want to Read Is Being Written in Congress

The Scholarly Paper That No One Will Want to Read Is Being Written in Congress

Spelling It Out: Spending on basic research has increased our longevity and the quality of that longer life. Photo Credit: John FleischmanThis week, a paper in the American Journal of Public Health, a...

09-18-2013

Read more
A Scientist-Senator with a Seat — and a Vote — for Life

A Scientist-Senator with a Seat — and a Vote — for Life

I recently spent some time on a family vacation in Italy. We have a tradition in our family; at the dinner table, each of us has to say the best and the worst thing that happened to him or her during ...

09-10-2013

Read more
Showing the Faces of Science

Showing the Faces of Science

In 2011, the Nobel Foundation awarded its prize of prizes to cell biologist, immunologist, and longtime ASCB member Ralph Steinman. When the foundation tried calling to deliver the good news, no one p...

09-05-2013

Read more
Lessons from a Society of Societies

Lessons from a Society of Societies

This week, ASCB went to ASAE, or rather five ASCB staffers and yours truly attended a meeting of the ASAE in Atlanta. ASCB, I trust you know, stands for the American Society for Cell Biology. The ASAE...

08-07-2013

Read more
Why Congress Should Be Worried About the Endoplasmic Reticulum

Why Congress Should Be Worried About the Endoplasmic Reticulum

Morphogenesis in biology always fascinated me. As a developmental neurobiologist at the bench, I studied how homeobox genes patterned the neural epithelium, the retina in particular, to understand the...

07-30-2013

Read more