Seven steps for critically analysing research papers (original) (raw)
Deconstructing papers systematically is an essential skill that allows you to critique authors’ arguments, verify their data and form your own conclusions.Credit: fizkes / Shutterstock
Reading a primary-research paper can feel like trying to decipher an ancient text, or at least it has in my career. From a foundation in biomedical science and medicine, I am now a trainee in paediatrics working towards a PhD in pathology at the University of Cambridge, UK. This work involves using systems biology to investigate mechanisms of therapy resistance in some cancers in children. As I build towards my goal of becoming a physician-scientist, I am continually reading massive amounts of complex, dense literature.
Although there is no universal framework for pulling apart a dense primary paper, having a consistent method is crucial. I now apply the same foundational technique whether I am reading an article for my own work or conducting formal peer review for a journal. When supervising medical students, I explain that you cannot simply read a paper from start to finish like a novel, regardless of what side of the publication process you are on. You need a more systematic approach if you want to deconstruct the authors’ arguments, verify their data and form your own conclusions.
How reliable is this research? Tool flags papers discussed on PubPeer
To help to bridge the gap between passively reading literature and actively analysing it, I’ve developed a seven-step framework, split into three phases. I usually record my thoughts with pen and paper in a laboratory notebook, but including the same information in the relevant section of reference management software also works.
Phase 1: the aerial view
Before you get bogged down in the minutiae of P values and complex assays, you need to understand the landscape of the paper. This phase is about gathering context.
Step 1: get a broad overview. Resist the urge to dive straight into dense methodology. Instead, scan the abstract and review the figures to get a high-level understanding of how the research was conducted. For example, is this a longitudinal cohort study, a CRISPR knockout screen or a computational model? Getting a broad sense of the tools and scope early on will anchor your understanding when you read the rest of the paper.
Step 2: identify the core research question. Every good piece of research hinges on a single, definable question. Although the title and abstract offer hints, the true research question is usually found at the end of the introduction. Find out what specific hypothesis is being tested and make a note of it. As you read through the rest of the study, refer back to this to see whether the authors stayed on track.
Step 3: map the existing knowledge gap. Research doesn’t happen in a vacuum; it builds on previous work. A critical reader should understand why the research is being conducted at that moment. Read the introduction to see how the authors frame the current state of the field. I try to get some questions clear in my head before reading on: what is already known about this topic? What piece of the puzzle is this paper trying to fill? Why does filling this gap matter?
Phase 2: the interrogation
This is when I roll up my sleeves and get into the details. Now that you understand what the authors are trying to achieve, you must evaluate how well they did it.
Step 4: assess the methodology. This is often the most challenging yet most crucial step of critical appraisal. You must compare the core research question from step 2 against the paper’s methodology and determine how appropriate the experimental design was.
• Did the authors use the right tools for the job? For example, using a highly adapted, immortalized cell line to study a specific metabolic pathway might yield vastly different results compared with using primary patient-derived cells or whole model organisms.
• Is the sample size large enough to provide adequate statistical power? Check the methods section for a power analysis. This calculation uses the expected effect size and a significance level (usually α = 0.05) to determine the minimum sample size required to avoid a type two error (false negative). If a study is underpowered, the authors might conclude that a treatment or intervention has no effect simply because there weren’t enough data or they might report a significant effect that is actually a statistical fluke.
• Did they use appropriate positive and negative controls? For instance, if establishing whether a compound kills cancer cells, a positive control (such as a known, highly toxic chemotherapy) proves that their assay correctly measures cell death. A negative control (such as treating the cells with only the solvent that the drug was dissolved in) ensures that the solvent itself isn’t responsible for the cell death.
• Are the raw data sets (such as sequencing reads or patient metadata) publicly accessible in repositories such as NCBI’s Gene Expression Omnibus or the European Nucleotide Archive? Providing raw data allows independent researchers to verify the findings through reanalysis, which is a cornerstone of the scientific method and guards against accidental errors or data manipulation. If the data are “available on request”, it often signals a barrier to peer verification.
• Have the authors provided the computational pipelines or scripts used to generate their models? For example, a study involving complex bioinformatics should ideally link to a GitHub repository or a Docker container containing the exact R or Python scripts used. This allows other researchers to replicate the exact digital workflow to see if they achieve the same results.
Step 5: reach your own conclusion. A common mistake is reading the authors’ discussion section before looking at the data. The discussion is the section in which the authors interpret their data and, because they’ve written a paper about the results, you can be fairly confident that they’ve reached their own hypothesis. Instead, skip the discussion and go straight to the results section and figures. Look at the graphs, read the tables and check the error bars. Ask yourself: “What do these data show?” Formulate your own conclusion solely on the basis of the numbers and visual data presented. Make a note of that conclusion so you can compare it with the authors’.
How to spot suspicious papers: a sleuthing guide for scientists