How Peer Review Works: Science’s Quality Control System

Before a scientific finding becomes part of the published record, it must survive scrutiny from other experts in the field. This process, peer review, is the cornerstone of scientific quality control. It is also widely misunderstood, frequently criticized, and currently being reimagined. This is how peer review works.

Understanding how peer review actually works, and where it fails, is essential for anyone who wants to read science intelligently.

What Peer Review Is (and Isn’t)

Peer review is the process by which a manuscript submitted to a scientific journal is evaluated by other scientists before publication. These evaluators, the peers, are researchers with relevant expertise who assess the work for scientific soundness, methodological rigor, novelty, and appropriate framing. Journals select reviewers from their networks, often using databases like Publons or Web of Science, and authors may suggest or exclude candidates to avoid conflicts of interest. As the demand for reviewers grows, many journals increasingly invite postdocs and senior PhD students who possess the necessary expertise, even though they may not yet hold permanent positions.

What peer review is not:

• A guarantee that a published finding is correct
• A replication or independent verification of the results
• A thorough statistical audit
• A check for fraud or data fabrication

Peer review is a filter, not a seal of truth. It catches many problems, obvious errors, flawed methodology, overclaiming, missing controls, but it is imperfect, and some of its most important limitations are structural.

How Peer Review Works: The Typical Process

1. Submission and Editorial Assessment

An author submits a manuscript to a journal. An editor, who may be a professional editor or an active scientist, performs an initial assessment. Many manuscripts are rejected at this stage without going to peer review if they are out of scope, clearly below the journal’s standards, or obviously flawed.

2. Reviewer Selection

For manuscripts that pass initial screening, the editor identifies and invites reviewers. Typically 2–3 reviewers are recruited, selected for their expertise in the relevant subfield. Reviewers are volunteers; they receive no payment, though some journals offer small honoraria or subscription access.

Finding willing reviewers has become increasingly difficult. The number of papers submitted has grown dramatically, while the pool of available reviewers has not kept pace.

3. The Review

Reviewers evaluate the manuscript, typically over several weeks, and submit written reports assessing:

• The scientific validity of the claims
• The appropriateness of the methods
• The quality of the data and statistical analysis
• The placement of the work in context of the existing literature
• The clarity of writing and presentation
• Whether the conclusions are supported by the evidence

Reviewers recommend one of several outcomes: accept, minor revisions, major revisions, or reject.

4. Author Revision

Most papers are not accepted outright. Authors receive reviewer comments and revise their manuscript in response, submitting a revised version along with a point-by-point response explaining how each comment was addressed. The editor then decides whether to accept the revised manuscript, send it for another round of review, or reject it.

This process often goes through multiple rounds, taking months to over a year. Some papers go through three or four rounds of revision before acceptance or rejection.

5. Publication

Accepted manuscripts are edited, formatted, assigned a DOI (Digital Object Identifier), and published: first online (ahead of print) and then in the journal’s issue. At that point, the work enters the scientific record.

Types of Peer Review

Single-Blind

The most common traditional model. Reviewers know who the authors are, but authors don’t know who reviewed their work. This allows reviewers to give honest assessments without risk of retaliation from authors, but leaves reviewers’ identities hidden. Single-blind review has been shown to harbor biases, for example, against women, early-career researchers, or authors from non-English-speaking countries, since reviewer anonymity can allow unconscious prejudice to go unchecked.

Double-Blind

Both authors’ and reviewers’ identities are concealed from each other. Intended to reduce bias: reviewers can’t be influenced by author prestige or institutional affiliation, and authors are equally protected. Evidence on whether double-blind review reduces bias is mixed; author identities can often be guessed from the writing style, citations, or topic.

Open Peer Review

Increasingly common, especially in newer journals. Reviews and reviewers’ identities are published alongside the paper. Proponents argue this increases accountability and quality, reviewers are less likely to write lazy or dismissive reviews if their names are attached. Critics worry it may reduce bluntness.

Post-Publication Peer Review

Some models move review after publication. Papers are posted publicly (often on preprint servers like arXiv or bioRxiv), and the community reviews them informally through comments, responses, and citations. More formal post-publication platforms like PubPeer allow structured critique of published work. This model has gained prominence in physics and mathematics, where preprints are the primary mode of communication.

What Peer Review Catches, and What It Misses

What It Catches

Peer reviewers are effective at:

• Identifying obvious errors in reasoning or methodology
• Flagging underpowered studies
• Noticing when conclusions overreach the evidence
• Pointing out important prior work that was ignored
• Requiring clearer presentation of methods

What It Misses

Statistical errors: Studies show that reviewers frequently miss statistical mistakes. A 2002 study in BMJ found that reviewers detected only 30% of deliberately introduced statistical errors in manuscripts. This is partly why some journals now use statistical checklists, but human oversight remains limited.

Data fabrication and fraud: Peer review almost never catches fraud. Reviewers typically don’t have access to raw data, can’t verify that experiments were actually performed, and aren’t tasked with checking for fabrication. The Diederik Stapel and Hwang Woo-suk fraud cases, both massive, sailed through peer review. Consider a vivid example: one reviewer later recounted flagging suspiciously pristine Western blot images in a paper, but doubts were dismissed; only years later, after a formal investigation, were multiple papers retracted for fabricated data. Such failures highlight why many scientists argue that peer review cannot serve as a fraud detection system.

Selective reporting and p-hacking: If a paper reports only the positive results from a larger set of analyses, reviewers may not notice. Unless the analysis plan was pre-registered, reviewers have no way to know what analyses were run but not reported.

Replication failures: Peer review doesn’t test whether results will replicate. The experiments described in the manuscript are reviewed for plausibility and methodology, but whether the results would hold up if repeated is not assessed.

The Time and Labor Problem

The peer review system depends on scientists donating their time voluntarily. A tenured professor might review 10–20 papers per year, spending hours on each. Journals publish tens of thousands of papers annually. The math only works because so many people participate.

But the system is under increasing strain. The volume of papers has exploded: roughly 3 million scientific papers are published annually according to the STM Global Brief (2021). Many reviewers report being overwhelmed by review requests and declining more than they accept. Review turnaround times have lengthened.

Meanwhile, reviewers receive no formal career credit for reviewing. It doesn’t appear on CVs in a way that counts toward tenure or promotion. The incentive to do reviews is largely altruistic: a contribution to the scientific community. As pressure to publish increases and academics’ time becomes more constrained, that altruism is tested.

Predatory Journals and Fake Peer Review

Not all journals conduct genuine peer review. Predatory journals: exploitative publishers that collect article processing fees while providing little or no real editorial service – have proliferated dramatically. They claim to conduct peer review but often accept virtually anything submitted, sometimes within days.

A series of sting operations has exposed the scale of the problem. Journalist John Bohannon submitted a deliberately flawed paper about cancer-fighting compounds to 304 open-access journals in 2013. 157 journals accepted it, most of which claimed to conduct peer review.

More recently, investigations have uncovered fake peer review rings, groups of researchers who create fake reviewer identities or collude to write favorable reviews for each other. Several hundred papers have been retracted after such rings were discovered.

Preprints: Bypassing Peer Review (Productively)

In physics and mathematics, it has been standard practice for decades to post papers to the arXiv preprint server before or during peer review. Other fields have adopted similar practices: bioRxiv for biology, medRxiv for medicine, SSRN for social sciences.

Preprints allow results to circulate and be evaluated by the community immediately, without waiting months for the formal review process. During the COVID-19 pandemic, preprints became crucial for the rapid dissemination of research, but also caused problems when journalists reported on unreviewed preprints as if they were established findings.

The preprint culture represents a partial decoupling of dissemination from certification. The paper goes out immediately; the formal peer review process continues in parallel, serving as an eventual quality certification rather than a prerequisite for sharing.

Reforms and the Future of Peer Review

The scientific community has been actively experimenting with peer review reforms:

How to Evaluate a Peer-Reviewed Paper as a Non-Scientist

You don’t need a PhD to read a scientific paper critically. If you encounter a study making headlines, ask a few simple questions:

• Who funded it? Look at the conflict-of-interest statement at the end. Industry-funded studies are not automatically flawed, but they are more likely to report favorable results.
• Is it a single study or part of a larger body of evidence? One peer-reviewed paper is a claim, not a conclusion. Check Google Scholar for subsequent papers that cite the study: do they confirm or challenge its findings?
• What did the journal actually review? A paper that reports a correlation may be methodologically sound, but correlation is not causation. Reviewers check methods, not interpretations.
• Has it been corrected or retracted? Check sites like Retraction Watch or the journal’s own notice for errata, corrections, or retractions.
• What does the community say? Science is a conversation. If a finding is truly revolutionary, it will be discussed and debated. See what other experts are saying in op-eds, blog posts, or on social media platforms like Bluesky or Mastodon.

This approach helps you avoid over-interpreting a single study while still benefiting from the quality control peer review provides.

What Peer Review Really Means for Science Communication

The most common misuse of peer review in public discourse is treating it as a binary: peer-reviewed (trustworthy) vs. not peer-reviewed (untrust-worthy).

The reality is more nuanced. A peer-reviewed paper in a top journal is not infallible; many peer-reviewed papers fail to replicate. A preprint is not automatically unreliable; it may have received more scrutiny from the scientific community than a formal peer-reviewed paper ever will.

What peer review offers is a baseline of quality control: expert assessment that the work is not obviously wrong, the methods are not obviously inappropriate, and the conclusions are not wildly unsupported. It is a necessary but insufficient condition for reliable scientific knowledge.

The more important filter is the accumulation of evidence over time. A single peer-reviewed study is a claim. Multiple independent studies with consistent results are evidence. A systematic review and meta-analysis is the strongest form of evidence. Scientific consensus, like the broad acceptance of evolution by natural selection or the Big Bang theory, represents the collective judgment of a field that has examined evidence from many angles.

Peer review is the beginning of that process, not the end.

One Big Idea to Remember: Peer review is a beginning, not an end, and cumulative evidence is what you should trust.

Sources

• Tennant, J.P. et al. (2017). The academic, economic and societal impacts of Open Access. F1000Research, 5, 632.
• Smith, R. (2006). Peer review: a flawed process at the heart of science and journals. Journal of the Royal Society of Medicine, 99(4), 178–182.
• Bohannon, J. (2013). Who’s afraid of peer review? Science, 342(6154), 60–65.
• Registered Reports. (2023). What are Registered Reports?. Center for Open Science.
• Nature Editorial Board. (2022). Peer review is still the best way to ensure quality science. Nature, 603, 206.

Further reading: Scholarly peer review on Wikipedia