Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation (original) (raw)

  1. Research
  2. Preferred reporting...
  3. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation

Research Methods & Reporting BMJ 2015;349 doi: https://doi.org/10.1136/bmj.g7647 (Published 02 January 2015) Cite this as: BMJ 2015;349:g7647

Loading

  1. Larissa Shamseer1,
  2. David Moher1,
  3. Mike Clarke2,
  4. Davina Ghersi3,
  5. Alessandro Liberati (deceased)4,
  6. Mark Petticrew5,
  7. Paul Shekelle6,
  8. Lesley A Stewart7
  9. the PRISMA-P Group
  10. 1Ottawa Hospital Research Institute and University of Ottawa, Canada
  11. 2Queen’s University Belfast, Ireland
  12. 3National Health and Medical Research Council, Australia
  13. 4University of Modena, Italy
  14. 5London School of Hygiene and Tropical Medicine, UK
  15. 6Southern California Evidence-based Practice Center, USA
  16. 7Centre for Reviews and Dissemination, University of York, UK
  17. Correspondence to: L Shamseer lshamseer{at}ohri.ca

Abstract

Protocols of systematic reviews and meta-analyses allow for planning and documentation of review methods, act as a guard against arbitrary decision making during review conduct, enable readers to assess for the presence of selective reporting against completed reviews, and, when made publicly available, reduce duplication of efforts and potentially prompt collaboration. Evidence documenting the existence of selective reporting and excessive duplication of reviews on the same or similar topics is accumulating and many calls have been made in support of the documentation and public availability of review protocols. Several efforts have emerged in recent years to rectify these problems, including development of an international register for prospective reviews (PROSPERO) and launch of the first open access journal dedicated to the exclusive publication of systematic review products, including protocols (BioMed Central’s Systematic Reviews). Furthering these efforts and building on the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analyses) guidelines, an international group of experts has created a guideline to improve the transparency, accuracy, completeness, and frequency of documented systematic review and meta-analysis protocols—PRISMA-P (for protocols) 2015. The PRISMA-P checklist contains 17 items considered to be essential and minimum components of a systematic review or meta-analysis protocol.

This PRISMA-P 2015 Explanation and Elaboration paper provides readers with a full understanding of and evidence about the necessity of each item as well as a model example from an existing published protocol. This paper should be read together with the PRISMA-P 2015 statement. Systematic review authors and assessors are strongly encouraged to make use of PRISMA-P when drafting and appraising review protocols.

Introduction

Systematic reviews hold a unique place in healthcare. They help form the basis for developing practice guidelines and they provide information on gaps in knowledge, thus informing future research efforts. This information is relevant to stakeholders across the health system. The rigour and trustworthiness of systematic reviews is, in large part, based on the a priori planning and documentation of a methodical approach to conduct (that is, a protocol).

A systematic review protocol is important for several reasons: (1) it allows systematic reviewers to plan carefully and thereby anticipate potential problems; (2) it allows reviewers to explicitly document what is planned before they start their review, enabling others to compare the protocol and the completed review (that is, to identify selective reporting), to replicate review methods if desired, and to judge the validity of planned methods; (3) it prevents arbitrary decision making with respect to inclusion criteria and extraction of data; and (4) it may reduce duplication of efforts and enhance collaboration, when available.

Various international organizations such as the Cochrane and Campbell Collaborations and the Agency for Healthcare Research and Quality (AHRQ) regularly require and publish protocols. However, outside of such organizations, few protocols are published in traditional journals and most reports of completed reviews (89%) do not mention working from a protocol1 (2014 update under way). Many experts have called for improved documentation and availability of review protocols. In response, experts (some of whom are authors on this document) launched an international, prospective register for systematic review protocols (PROSPERO, www.crd.york.ac.uk/prospero/) through the Centre for Reviews and Dissemination at the University of York (UK) in February 2011, in which more than 5000 systematic review protocols from 69 countries have been registered as of December 2014. In February 2012, the first open access journal to exclusively publish systematic review products including protocols (BioMed Central’s Systematic Reviews) was launched, in which 142 protocols have been published (June 2014). Outside of select systematic review organizations, little to no general guidance exists for preparing review protocols.

Selective reporting

Arguably one of the most important functions of systematic review protocols is their role as a documentation of planned review methods, outcomes, and analyses that can be compared with completed reviews to detect whether unintended and undocumented changes were made. Bias related to selective reporting of outcomes (that is, when reporting is related to the statistical significance or direction of effect estimate) is a problem in clinical research. This is a well documented phenomenon in clinical trials,2 3 4 5 6 7 and similar findings are starting to emerge for systematic reviews (see item 13 for full discussion).8 9 10 When reviewers selectively choose which information to include in a report based on the direction and significance of findings, they risk biasing the evidence base on which healthcare decisions and policies are made.

Further to recent efforts to increase the documentation and availability of review protocols, the next logical step is the development of a set of standards that should be included in a review protocol. A well described protocol may facilitate and enhance the detection of undocumented changes to review methodology; it also may allow readers to gauge the potential impact of such changes as well as selective reporting of information on review findings.

To that end, a reporting guideline for systematic review protocols, an extension of the PRISMA (Preferred Items for Reporting Systematic Reviews and Meta-analyses) statement has been developed for protocols (PRISMA-P) and is described in detail in this paper.

Scope of PRISMA-P

PRISMA-P is intended to guide the development of protocols of systematic reviews and meta-analyses evaluating therapeutic efficacy. Even for systematic reviews that are not evaluating efficacy, authors are encouraged to use PRISMA-P because of the lack of existing protocol guidance overall. For the purpose of this guidance, we define a protocol, broadly, as a document written before the start of a systematic review describing the rationale and intended purpose of the review, and the planned methodological and analytical approach (see box 1 for comprehensive definitions).

Box 1: PRISMA-P terminology

PRISMA-P is meant to be used primarily by authors preparing systematic review protocols for publication, public consumption, or otherwise. It is also intended for those commissioning and potentially funding reviews as a guide for applicants on what should they should include in their review protocols, and as a tool for peer reviewers to gauge whether a protocol contains essential details. PRISMA-P will also be helpful for journal editors and peer reviewers gauging the adequacy of review protocols for publication. A list of stakeholders to whom we believe PRISMA-P will be useful along with proposed benefits for each group is provided in table 1⇓.

Table 1

Proposed stakeholders, actions, and potential benefits for supporting adherence to PRISMA-P

Development of PRISMA-P

The PRISMA-P checklist is based on elements from the PROSPERO register,11 the PRISMA checklist,12 SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) checklist items,13 and Standard 2.6 from the Institute of Medicine’s Standards for Systematic Reviews.14 A detailed description of the steps undertaken during PRISMA-P development can be found in the PRISMA-P Statement paper.15 The process follows general recommendations of the EQUATOR (Enhancing the Quality and Transparency of health Research) Network on how to develop a reporting guideline, of which one fundamental part is a consensus process.16 An in-person consensus meeting of international experts was held in June 2011 in Rockville, MD, USA, to develop and refine PRISMA-P checklist items. All related guidance documents have undergone iterative revision within the PRISMA-P Group listed at the end of this document; members of the PRISMA-P Group contributed to the writing and identifying relevant examples in this document.

PRISMA-P checklist

The final PRISMA-P checklist contains 17 numbered items (26 sub-items) that should be described, at minimum, in protocols of systematic reviews and meta-analyses (table 2⇓). The checklist is divided into three main sections: administrative information, introduction, and methods. Readers familiar with PRISMA will observe that wording of the PRISMA-P checklists has, where possible, been harmonized with PRISMA checklist items, at least 13 of which are overlapping with PRISMA-P. We anticipate this will aid authors in transitioning their systematic review protocols prepared in accordance with PRISMA-P into full text, PRISMA-compliant, systematic review reports.

Table 2

PRISMA-P (preferred reporting items for systematic review and meta-analysis protocols) 2015 checklist: recommended items to address in a systematic review protocol

PRISMA-P Elaboration and Explanation

The format of this document follows that of previously established reporting guidelines such as the PRISMA Explanation and Elaboration document17; it aims to provide readers with comprehensive explanations and evidence based rationales for each checklist item. Examples of good reporting for each checklist item have been identified from existing systematic review and meta-analysis protocols and are provided throughout this document to enhance reader understanding of items.

Although PRISMA-P focuses on a minimal list of items to consider when preparing a systematic review protocol, we have indicated instances where additional information may be desirable to improve transparency of the planned review process. The recommendations within PRISMA-P may require more words or space than authors are accustomed to. Providing detailed descriptions for some protocol elements (such as item 8, eligibility criteria; item 13, outcomes and prioritisation) will facilitate transparency and future reproducibility, and allow authors to shorten their methods section in a completed systematic review report, if desired, by providing a brief summary of the methods and referring readers to the completed protocol or PROSPERO record. We believe that providing in depth descriptions of planned methodological details for systematic reviews is in line with emerging journal policies aimed at facilitating reproducibility.18

Checklist items are numbered as we envision them appearing in a protocol, and reporting them in this sequential order is a suggestion that may facilitate reader comprehension. Authors should amend the order of appearance of checklist items if they deem it to be necessary. Most important is that authors describe each PRISMA-P item somewhere in their protocol.

One point to note is that, while the development of a protocol abstract is not a listed requirement on the PRISMA-P checklist, authors are urged to consult the PRISMA extension for reporting conference and journal abstracts if so desired.19 The examples and explanations for each checklist item follow; citations contained within examples have been removed to avoid potential confusion with citations in this article.

Section 1: Administrative information

Title

Item 1a: Identification. Identify the report as a protocol of a systematic review

Example

“Postoperative outcomes following preoperative inspiratory muscle training in patients undergoing open cardiothoracic or upper abdominal surgery: protocol for a systematic review”20

Explanation

The knowledge in systematic reviews can be harnessed only if readers can easily identify them. Data indicate that systematic reviews are not always described as such in either the title or abstract; only 50% of systematic reviews included in a November 2004 sample used the terms “systematic review” or “meta-analysis” in their title or abstract.1 Similar results have been reported elsewhere.21 When this happens, reviews and meta-analyses may not be indexed in databases appropriately and risk not being found by potential users. This can lead to wasted efforts by systematic reviewers when knowledge they produce cannot be identified, one consequence of which may be unnecessary duplication of efforts by future reviewers.

Authors should title their report as a protocol of a systematic review and planned meta-analysis (the latter, only if known at the protocol stage). The term protocol indicates the existence of a plan for an upcoming, ongoing, or existing systematic review. Identification as a protocol may reduce unnecessary redundancy of systematic review efforts22 and may also be helpful for readers seeking assistance in the design of future reviews. Although sensitive search strategies have been developed to identify systematic reviews,23 inclusion of the terms systematic review or, if a meta-analysis is planned, meta-analysis in the title of a protocol may improve identification and retrieval.

We advise authors to use informative titles that make key information easily accessible to readers. Ideally, a title reflecting the PICO approach (participants, interventions, comparators, and outcomes) as well as time frame, setting, and study design, if desired (see Item 7), will provide readers with key information about the scope of the planned review.

Item 1b: Update. If the protocol is for an update of a previous systematic review, identify as such

Example

“The association between proximity to animal-feeding operations and community health: a protocol for updating a systematic review”24

Explanation

As explained in item 1a, authors can help to ensure awareness of the existence of a systematic review and review protocol by indicating this information in their title. Similar transparency will help readers identify whether the protocol in question is for conducting a new systematic review or an update of an existing one; ideally, this information should be reported within the title. Updates and, sometimes, expansions of an existing systematic review allow for the consideration of new evidence to bring previously published systematic reviews up to date.25 Updating systematic reviews and identifying methods and signals for when to do so are increasingly being studied,26 27 28 29 30 given that out of date systematic review evidence can be harmful,31 particularly when updates yield changes in the direction of effect of one or more outcomes. Although systematic review updates are not always published as full length articles, they warrant an independent publication, the title of which should reflect its purpose.

Registration

Item 2. If registered, provide the name of the registry (such as PROSPERO) and registration number

Example

“In accordance with the guidelines, our systematic review protocol was registered with the International Prospective Register of Systematic Reviews (PROSPERO) on 11 July 2011 and was last updated on 19 January, 2012 (registration number CRD42011001410).”32

Explanation

Registration of systematic review protocol details is now recognized as desirable in order to promote and maintain transparency in the systematic review process, to assist in minimizing the risk of bias(es), and help to reduce unnecessary duplication of reviews.33 At the time of publication, only one registry for prospective systematic review registration exists—the PROSPERO register (www.crd.york.ac.uk/prospero/). The PROSPERO register provides review authors with the opportunity to freely register reviews evaluating interventions and strategies to prevent, diagnose, treat, and monitor conditions for which there is a health related outcome.34 35 Since October 2013, key details from new protocols published in the Cochrane Library have been automatically added to PROSPERO on a daily basis. Future plans for PROSPERO include broadening inclusion to all systematic reviews with a health related outcome in the broadest sense (such as reviews of risk factors and genetic associations).

PROSPERO contains 22 mandatory items and 18 optional fields to capture key review attributes. However, it does not capture all information that should be included in a review protocol and does not preclude documentation and publication of a full review protocol. For easy transition from a registry entry into a full review protocol, many PRISMA-P items are based on PROSPERO items.

As with the preparation of a review protocol, the process of review registration forces authors to think through review methods and hopefully avoid future changes which may be associated with reporting biases. Furthermore, the registry entry itself provides readers with a reference to compare against complete reviews, in the absence of an available protocol, to examine for reporting biases. Logically, the planning, conduct, and reporting of reviews should involve efforts to help detect and minimize such bias.10 36 Registration helps by prospectively recording key features of the planned review when the protocol has been finalized but before any eligibility screening has started, and making this information available publically and freely. This information provides those contemplating commissioning or undertaking a review to identify whether a relevant review is already planned or underway, if not completed. This should help avoid unplanned duplication, ensuring efficient use of resources and offering potential for future collaboration.37 38 Of 73 randomly selected systematic reviews of randomised trials published in 2010, 49 (67%) had at least one overlapping meta-analysis that did not represent an update (that is, same comparison, type of population or indication, and outcome).37 This signals a potentially large degree of wasted efforts.

Details and justification of any changes or amendments (see Item 4) made during the review process should be added to the registration record and reported in the final systematic review results report. By registering this information, the opportunity for post hoc manipulation and potential consequent bias are likely minimized. The public record allows comparison of published review results with what was planned so that readers can judge whether any discrepancies are likely to have introduced bias.

Registration information is increasingly being asked for by a number of journals as part of their submission process.33 39 40 Once reviews are registered on PROSPERO, authors receive a unique identification number that authors should report in a review protocol, and in all publications arising from a review (that is, the protocol and completed review); doing so ensures that they can easily and confidently be identified as related.

Authors

Item 3a: Contact information. Provide name, institutional affiliation, and email address of all protocol authors; provide physical mailing address of corresponding author

Example

“*Corresponding author: Frances C Hillier frances.hillier@durham.ac.uk

Author Affiliations

Explanation

Individuals who have made substantive intellectual contributions to the development of the systematic review protocol should provide their names, affiliations, and contact information even if the protocol is not published or intended to be published. Together with contributorship (Item 3b), this information can help identify competing interests and ghost authorship42 and enhance the recognition and accountability of protocol authors and transparency of the review.43 Although ghost authorship itself may not necessarily contribute to scientific bias, it may reflect the undisclosed shaping role played by companies or other groups with vested interests in the design or reporting of a study.42 44 45 46

In some instances, because of the nature of a relationship with a funder or sensitivity of the potential data, reviewers may not wish to have their names on a protocol before the systematic review is completed. In these instances, reviewers should provide contact information for the sponsor (host institution or funder) or for an individual assigned to deal with reader queries.

Item 3b: Contributions. Describe contributions of protocol authors and identify the guarantor of the review

Example

“DF is the guarantor. JE, RR and DM drafted the manuscript. All authors contributed to the development of the selection criteria, the risk of bias assessment strategy and data extraction criteria. SB developed the search strategy. RR provided statistical expertise. DF provided expertise on venous thromboembolism. SJ contributed to the section on health economics. All authors read, provided feedback and approved the final manuscript.”47

Explanation

Some journals urge that published articles include descriptions of the contributions of each named author.43 48 Likewise, in review protocols, together with names and contact information, the role(s) of each author should be clearly described. In biomedical publishing, journals require authors to have contributed to an article in at least the following ways: (1) contributed substantially to the conception and design of the study, the acquisition of data, or the analysis and interpretation; (2) drafted or provided critical revision of the article; and (3) provided final approval of the version to be published.49

The guarantor of a research article is the author who assumes the overall responsibility for the scientific integrity of the work as a whole and should be identified as such.46 49 The term corresponding author typically represents the notion of “guarantor,” and is also used to indicate which co-author is responsible for pre- and post-acceptance communication with the publishing journal and for taking queries to all other co-authors. A guarantor should be able to answer queries about the order of authors on the manuscript and about the research itself.49 The guarantor is often listed as either the first named or most senior (often last) author.

Amendments

Item 4 If the report represents an amendment of a previously completed or published protocol, identify as such and indicate what changes were made; otherwise state plan for documenting important protocol amendments

Example 1

“In the event of protocol amendments, the date of each amendment will be accompanied by a description of the change and the rationale.”50

Example 2

“If we need to amend this protocol, we will give the date of each amendment, describe the change and give the rationale in this section. Changes will not be incorporated into the protocol.”51

Explanation

Systematic review protocols are typically iterative documents; modifications to protocols before and during the review process are to be expected. Systematic reviewers should give careful consideration to a review’s methodological and analytical approach early on to avoid unnecessary changes after protocol development. A study of trials funded by pharmaceutical companies indicate that at least a third of amendments made to original trial protocols could have been prevented if key issues were given more consideration during protocol development52; this is likely true for systematic reviews as well. A 2002 study of 66 Cochrane reviews found that 91% of completed reviews had major changes from the protocol.36 More recently, at least 20% of Cochrane reviews have been found to make post-protocol modifications to review outcomes (that is, addition, removal, or reprioritization), many of which are based on significance of the outcome in the completed review. Making changes to review outcomes, after knowledge of findings from included studies can introduce bias into the review process, mislead readers and possibly affect patient care. Cochrane reviews have since evolved to provide a dedicated section in which authors should report any changes made from the documented protocol.53 Likewise, inclusion of a table summarizing protocol amendments is a mandatory requirement for reviews produced by AHRQ’s Effective Health Care Program (table 3⇓). The PROSPERO register also allows for and tracks amendments of registered protocols.

Although many amendments do not introduce bias, changes from earlier protocol versions or from the registry entry should be transparently identified as such in each documented version of the protocol so that, at minimum, readers can evaluate the potential for bias. For protocols in which no amendments have yet been made, authors should include a description of the process for dealing with and documenting future amendments (that is, who will ultimately be responsible for approving, documenting, and implementing them). An updated protocol should be identified with a new version number and a list of specific amendments that were made to the previous version (see table 3⇑).

Support

Item 5a: Sources. Indicate sources of financial or other support for the review

Example

“This systematic review is funded by the Institute for Neurosciences, Mental Health and Addition, Canadian Institutes of Health Research (funding reference number KSD-115551; Effectiveness of the Screening, Brief Intervention and Referral to Treatment (SBIRT) Model for Reducing Illicit Drug Use: A Systematic Review).”54

Explanation

An updated Cochrane review indicates that drug trials funded by the pharmaceutical industry report significantly greater benefits, fewer harms, and more favourable overall conclusions than those with non-industry funding.55 56 This issue, termed sponsorship bias, has been characterized less frequently in systematic reviews and meta-analyses. Of note, since 2004 the Cochrane Collaboration has prohibited industry support for its reviews.57 One study indicates that conclusions from company supported reviews (2003, issue 1) recommended a drug not recommended in a matching, non-industry funded Cochrane review, despite both reviews having similar treatment effects; Cochrane reviews also had greater methodological transparency.58 Another study of 124 meta-analyses found that meta-analyses with financial ties to one pharmaceutical company (n=49) were associated with more favourable conclusions, yet not more favourable results, than those with other financial ties.59 Another study failed to replicate these findings, but it did find that industry supported meta-analyses have worse methodological quality than meta-analyses supported by non-profit organizations or unsupported meta-analyses.60

Review authors should disclose sources of financial and non-financial support for their review, if known at the protocol stage. If a review is not funded at the time the protocol is first registered and made available, the proposed sources of support should be listed and updated once funding is confirmed. Along with Item 5c (role of funder or sponsor), this information will help readers assess whether any competing interests or potential influences are present. As an example, the evaluation of sugar sweetened beverages and weight gain has recently received much attention for their purported association with negative health outcomes. A systematic review of reviews of sugar sweetened beverages and weight gain found that reviews identified as being affiliated with or supported by the food industry were five times more likely to report no positive, significant association with weight gain than non-industry affiliated reviews.61 This finding highlights a need for authors to disclose their affiliations and sources of funding. Inclusion of the “financial conflicts of interest checklist 2010” with a protocol is recommended to help readers identify potential conflicts to be aware of; many journals have already instituted its use.62

Non-financial sources of support that should be disclosed may include the provision of services by an institution or funder, an information specialist who will help to obtain articles, access to a commercial database not otherwise available to reviewers, or in-kind use of software to manage or analyze review data.

Item 5b: Sponsor. Provide name of the review funder and/or sponsor

Example 1

“The Chartered Society of Physiotherapy Charitable Trust funded this research.”63

Example 2

“The Laboratory of Research and Clinical Applications in Ophthalmology (Aristotle University of Thessaloniki) is the Sponsor, meaning that it has overall control of the data. No funding has been received for this study.”64

Explanation

The term “sponsor” is most often associated with clinical trials in reference to the individual, company, institution, or organization assuming overall responsibility for the initiation and management of the trial.65 However, because systematic reviews are often commissioned and funded by large agencies or companies, it is important for protocol authors to name both the sponsor and funder (Item 5a) in the review protocol, if applicable. The sponsor may not necessarily refer to the main funder if, for instance, a funder provides monies to a third party (sponsor) to carry out the research. This may happen, for example, if a company provides funds to a university researcher, whereby the university would become the sponsor of the review. Where relevant, the sponsor should be named in a review protocol.

Item 5c: Role of sponsor and/or funder. Describe roles of funder(s), sponsor(s), and/or institution(s), if any, in developing the protocol

Example

“The Nova Scotia Health Research Foundation (NSHRF) is funding the Chronic LBP IPD Meta-analysis project. This funding will support the collection of the individual participant data by the original investigators, data management and analyses. The NSHRF is not involved in any other aspect of the project, such as the design of the project’s protocol and analysis plan, the collection and analyses. The funder will have no input on the interpretation or publication of the study results.”66

Explanation

When the sponsor or funder (sometimes the same entity) with competing interests has a substantial role in the planning, conduct, or dissemination of a systematic review, there is potential for bias if authors do not manage the interests of all parties appropriately. Although both industry and non-industry reviews are subject to potential bias(es), published reports of reviews with commercial sponsorship tend to describe lower quality methods and more favourable conclusions.58 59 60 67 Examples exist of unfavourable reviews being suppressed by commercial sponsors.68 69

To provide full transparency into the potential relevance of competing interests, review protocols should explicitly describe the roles (if any) of the sponsor and funders in protocol development, review conduct, data analysis and interpretation, and dissemination of the final report. It is important to specify who will make the final decision about these elements of the systematic review, particularly if disagreements arise. Any restrictions on disseminating the final report of the review should also be documented.

Section 2: Introduction

Rationale

Item 6. Describe the rationale for the review in the context of what is already known

Example

[Review title: Trends in child and adolescent obesity prevalence according to socioeconomic position: protocol for a systematic review]

“It is well recognised that childhood obesity is a significant public health issue, with adverse physical and psychological effects that persist beyond childhood into the adult years. After decades of rapid increase, it appears that childhood obesity prevalence in developed countries is starting to plateau. Reviews of international evidence have shown that the prevalence of obesity in children and adolescents is stabilising in countries including Australia, Japan, France, the UK and US. However, evidence also suggests that such progress may not have been shared among children across all socioeconomic groups.

An international systematic review published in 2010 examined obesity prevalence trends and reported levelling off of the obesity epidemic in recent years. Heterogeneity in obesity trends were reported across socioeconomic strata, with levelling of obesity prevalence less apparent for more disadvantaged socioeconomic groups. However, the authors noted that trends by socioeconomic strata were only explored in a small number of their included studies. Individual studies reporting the impact of socioeconomic position (SEP) on obesity prevalence provided mixed results. Studies from Australia and England reported socioeconomic differences in obesity trends among children and adolescents, while evidence from France did not show a difference. With a specific focus on SEP and childhood obesity, this review will capture additional data, including papers published since 2010, to allow greater understanding of trends in the prevalence of obesity by SEP.

Further investigation is warranted, particularly because of the existing excess burden of obesity in children in a lower SEP. Given the health risks associated with excess weight, and the observed socioeconomic patterning in chronic diseases, if trends in obesity prevalence are not improving at the same rate across socioeconomic groups, this will likely lead to further inequalities across a range of health and wellbeing outcomes. Understanding the differences between subgroups of the population is critical to ensuring policy makers can make informed decisions as to where preventive efforts should be focused. This is particularly important in light of evidence that demonstrates differential effectiveness of a number of obesity prevention interventions according to SEP.”70

Explanation

Readers need to understand the rationale behind the decision to perform the systematic review and what the results may add to what is already known. Authors should explain the impetus for the systematic review (such as to support clinical guideline development, to address uncertainty or variation in practice in approaches to a specific clinical problem, to support policy development, to provide a more precise estimate of effect, to update a previous review) and briefly summarize how the review builds on and could add to prior knowledge. In the case of a protocol to update an existing review, authors should cite the previous or original review and, in the methods section, point out any planned modifications from the original review in the protocol for the update,71 perhaps with a section heading “updated methods.” Where possible, the primary audience for the review and the review perspective (that is, patient or clinician decision making, public health, health policy) should be clear.

Ideally, the rationale section should set the context for both the protocol as well as the systematic review. Background detail on the clinical condition should be sufficient to help the reader establish the overall significance of the proposed systematic review for developing new knowledge of interest and to help clarify key decisions or processes undertaken in the research protocol. These might include the specific focus of the population, intervention, comparator(s), and outcome (with emphasis on specific outcomes), settings, study designs, and time frames. As well, the means by which key perspectives represented in the review were obtained (that is, patient or other stakeholder engagement) should be described.

Objectives

Item 7. Provide an explicit statement of the question(s) the review will address with reference to participants, interventions, comparators, and outcomes (PICO)

Example 1

“The aim of this systematic review is to evaluate the effectiveness and harms of perioperative pregabalin in the management of postoperative pain for the diverse patients undergoing various surgical procedures. To this end, the proposed systematic review will answer the following questions:

1. When compared with standard multimodal analgesia, what are the comparative effectiveness and harms of the co-administration of pregabalin in the perioperative pain management of adult patients?

2. Is there a definitive opioid-sparing advantage of pregabalin (for example, lower risk of nausea, vomiting, somnolence, opioid use, and other opioid-related side effects) when used for perioperative pain management in adults?

3. For questions 1 and 2 above, what clinical and study methodological characteristics explain the heterogeneity in results?”72

Example 2

“The objectives of our study are to systematically review the literature for qualitative evidence that explores the factors that influence the decision of individuals aged 50 years or over at average risk for CRC to participate in CRC screening, and how those factors vary by sex, ethnicity and SES. Our secondary aim will be to generate a framework to better understand the perceived benefits and barriers that affect individual decision-making.”73

Explanation

Among the most crucial pieces of information to include in a review protocol are the question(s) the reviewers plan to investigate, or simply, the review’s objectives. Along with the review’s rationale (Item 6), this information provides the reader with context and understanding for why the review is being carried out and what the reviewers hope to achieve. Several key components, namely the planned population, intervention, comparator, and outcome (that is, PICO elements) at minimum should form the basis for developing a specific, well designed review question. Additional elements such as setting, study design, and time frame (that is, length of follow-up) may also be included in the review question, but if not, should certainly appear in the review’s eligibility criteria (Item 8). Guidance is available to help researchers develop a research question.74 75 Reviews may focus on one PICO element more than others given the planned scope of the review; authors should clearly state this emphasis in the protocol.

Section 3: Methods

Eligibility criteria

Item 8. Specify the study characteristics (such as PICO, study design, setting, time frame) and report characteristics (such as years considered, language, publication status) to be used as criteria for eligibility for the review

Example:

“Eligibility criteria

“Studies will be selected according to the criteria outlined below.

Study designs

We will include randomized controlled trials (RCTs), including cluster RCTs, controlled (non-randomized) clinical trials (CCTs) or cluster trials, interrupted time series (ITS) studies with at least three data points before and after the intervention, controlled before-after (CBA) studies, prospective and retrospective comparative cohort studies, and case-control or nested case-control studies. Cluster randomized, cluster non-randomized, or CBA studies will be included only if there are at least two intervention sites and two control sites. We will exclude cross-sectional studies, case series, and case reports.

Participants

We will include studies examining the general adult human population or healthy adult humans (18 years or older). We will also include studies on people who are overweight or obese, but will otherwise exclude studies of populations restricted to specific diseases, conditions, or metabolic disorders. We will include studies addressing both adults and children if data provided for adults are reported separately.

Interventions

Of interest are interventions addressing SSB consumption, taking a broad perspective. In addition to direct consumption studies, we would consider interventions that influence consumption, such as those addressing the level of access to SSBs (e.g. university/college policy) and educational interventions addressing consumption as relevant. Non-specific or multi-faceted behavioural, educational, or policy interventions may also be included subject to the level of evidence that exists for the aforementioned interventions/exposures. We will also consider other types of interventions on a case by case basis, subject to what exists in the literature.

In terms of defining an SSB, we view them as akin to a complex intervention because they are composed of several parts. For example, in addition to sugar, some beverages contain caffeine and the by-products of caramel colouring (2-methylimidazole, 4-methylimidazole), which may contribute independently to adverse health outcomes. The scope of the review, therefore, warrants an examination of SSB consumption as a whole, rather than the specific constituents as exposure variables. Otherwise, such evaluations would have necessarily required the inclusion of studies addressing those constituents and in foods and drinks other than SSBs.

We will use the Centers for Disease Control and Prevention (CDC) definition of SSB for drinks that should be included. According to the CDC, SSBs contain added caloric sweeteners, which would include natural sweeteners such as honey and concentrated fruit juice. We have developed a classification scheme based on the CDC definition for use during the review (see classification scheme for SSBs below). For beverages such as coffee, tea, and homemade lemonade, studies will be included in the review if they explicitly state that sugar was added. We will exclude artificially sweetened (e.g. with aspartame or sucralose) beverages, alcoholic beverages, and 100% fruit or vegetable juices as exposures/interventions.

We will classify SSBs described in studies according to the following broad categories:

Comparators

Given the broad perspective for interventions of interest, several comparisons will be relevant to include. Some may be more likely to come from observational designs and others from experimental studies.

Direct consumption studies:

Interventions that influence consumption:

For comparator groups 2 and 3, we anticipate that volume will be the most feasible to analyse; however, we will extract all measures in which consumption is reported (e.g. volume, caloric intake from sugar) in studies to see what analysis is possible.

For feasibility, category 6 comparisons (non-specific, multi-faceted interventions) will be coded at title/abstract screening and not put through to full text screening. If sparse evidence exists in the other potential comparison types, we will revisit eligibility for comparison 6.

Outcomes

Endpoints important for decision making are of primary interest. If reported on, these will be analysed and graded. If a given clinical endpoint is not reported on, we will analyse and grade their relevant surrogate outcome(s).

As some outcomes may be reported as a composite measure, we will extract all composite and individual outcomes as reported in the studies.

Outcomes will be collected as reported, with the exception of quality of life, which will be collected only if assessed with generic (not disease specific), validated tools. Due to possible variation in disease definitions over time, we will extract definitions of outcomes as reported in individual studies. We will extract outcomes in all data forms (e.g. dichotomous, continuous) as reported in the included studies.

Timing

Studies will be selected for inclusion based on the length of follow-up of outcomes. The following will be used as a guide for all study designs:

Setting

There will be no restrictions by type of setting.

Language

We will include articles reported in the English and French languages. A list of possibly relevant titles in other languages will be provided as an appendix.”76

Explanation

The requirement and ability to pre-specify eligibility criteria (sometimes denoted inclusion or exclusion criteria) that reviewers will use to identify relevant studies for inclusion is a defining feature of a systematic review.77 Making this information available to readers of protocols, as in completed reviews, is essential in appraising the validity, applicability, and comprehensiveness of a review.74 Thus, authors should provide an unambiguous description of planned eligibility criteria for the impending review; such descriptions are a fundamental component upon which later stages of the review process are conducted. For instance, eligibility criteria often influence the terminology used to develop the search strategy and work to prevent the introduction of bias into the study selection process of a systematic review.

As in PRISMA, there are two general categories of eligibility criteria: study characteristics and report characteristics.17 Authors should describe both. As in the example above, authors can anticipate that these details will require substantial space in the methods section of a review protocol while at the same time facilitating review transparency and future reproducibility.

Study eligibility criteria are the typical PICO elements that form the basis of clinical questions. These include populations, interventions, comparators, outcomes, time frames for follow-up, settings in which the interventions are delivered, and study designs of interest; they also can include other study specific elements, such as specifying a minimum length of follow-up or a minimum sample size for certain types of studies. Authors should state whether they will exclude studies because the studies do not include (or report) specific outcomes; doing so will help readers ascertain whether the eventual review may be biased as a consequence of selective reporting.4

Review eligibility criteria are likely to include geographical location, languages of publication, publication status (such as inclusion of unpublished material or abstracts), and years of publication. Inclusion or not of literature in multiple languages,78 79 unpublished data, or older data can influence the effect estimates in meta-analyses.80 81 If it is planned to filter out (via search filter, see Item 10) or exclude specific types of records (such as commentaries, letters, editorials, etc) during screening, this should be stated.

Information sources

Item 9. Describe all intended information sources (such as electronic databases, contact with study authors, trial registers or other grey literature sources) with planned dates of coverage

Example

“Literature search strategies will be developed using medical subject headings (MeSH) and text words related to influenza vaccination. We will search MEDLINE (OVID interface, 1948 onwards), EMBASE (OVID interface, 1980 onwards), and the Cochrane Central Register of Controlled Trials (Wiley interface, current issue). The electronic database search will be supplemented by searching for trial protocols through metaRegister (http://www.controlled-trials.com/mrct/). The literature search will be limited to the English language and human subjects.

To ensure literature saturation, we will scan the reference lists of included studies or relevant reviews identified through the search. We will also search the authors’ personal files to make sure that all relevant material has been captured. Finally, we will circulate a bibliography of the included articles to the systematic review team, as well as to influenza experts identified by the team.”82

Explanation

A systematic review search typically includes a variety of information sources including electronic bibliographic databases (such as Medline, Embase), reference lists, contact with authors of included studies, study registries, and grey literature. Most biomedical topics will include a Medline search, plus additional electronic databases. Searching additional electronic databases helps ensure more complete coverage of the topic by accounting for variability between the indexing in each database. In situations in which identifying all relevant studies through hand searching and database searching is difficult, if any other searching, such as reference lists, is planned to supplement searching, authors should report this.83 Documentation of the planned information sources should include the name of each source, the date range that was searched (that is, start and end dates, and, for electronic database searches, the search platform or provider such, as Ovid or PubMed). This information will be important to the person developing and conducting the search if an update to the review is carried out. Authors should also report who developed and carried out the search.83 84

The Cochrane Collaboration,85 AHRQ’s Effective Health Care Program,86 and the Institute of Medicine (Standard 3.1),14 among others, offer guidance on developing a rigorous systematic review search strategy. If these sources are used, authors should report this information.

Search strategy

Item 10. Present draft of search strategy to be used for at least one electronic database, including planned limits, such that it could be repeated

Example

“Both qualitative and quantitative studies will be sought. No study design, date or language limits will be imposed on the search, although only studies in languages other than English that can be translated adequately using Google translate1 will be included, due to resource limits. Medline, EMBASE, PsycINFO, and the CENTRAL trials registry of the Cochrane Collaboration will be searched. The specific search strategies will be created by a Health Sciences Librarian with expertise in systematic review searching. The MEDLINE strategy will be developed with input from the project team, then peer reviewed by a second librarian, not otherwise associated with the project, using the PRESS standard.2 A draft MEDLINE search strategy is included in Appendix 1. After the MEDLINE strategy is finalized, it will be adapted to the syntax and subject headings of the other databases.

As well, the International Clinical Trials Registry Platform Search Portal and ClinicalTrials.gov will be searched for ongoing or recently completed trials, and PROSPERO will be searched for ongoing or recently completed systematic reviews. As relevant studies are identified, reviewers will check for additional relevant cited and citing articles.

“The search will be updated toward the end of the review, after being validated to ensure that the MEDLINE strategy retrieves a high proportion of eligible studies found through any means but indexed in MEDLINE.

…

Appendix 1

Draft MEDLINE search - Ovid interface

Explanation

The comprehensiveness and completeness of a literature search is extremely important in systematic reviews. High quality searches of information resources are essential components in the efforts toward accuracy and completeness of the evidence base.88

At a minimum, authors should provide the transcript of a draft search strategy for one major database (such as Medline) for each search question (if different searches were run for each question). In the documented strategy, it should be evident which indexing terms reviewers selected and what limits (such as language and date restrictions) were (or will be) applied to the search. If authors plan to use any search filters, information about their validity and performance metrics should be provided. Authors should also describe the planned search strategy approach for other databases, including planned modifications to indexing terms, free text terms, and limits, which may vary across databases.

If limits were used to restrict the search to particular study type (that is, trials, human, or clinical studies) or date range, authors should report what these were and how they were achieved. Simply stating, for example, that all publications in the form of letters will be excluded from the search can be problematic given that the publication of randomised trials as “letters to the editor,” is a documented problem,89 and authors may be intending to make an exception for such reports. Authors should report the logical construction of text used to create such limits within the draft search strategy (such as “NOT (letter.pt NOT randomized controlled trial.pt”).90 Doing so can help readers assess the appropriateness of intended limits within a search strategy.

Most searches have constraints—for example, relating to limited time or financial resources, inaccessible or inadequately indexed reports and databases, unavailability of experts with particular language or database searching skills, or review questions for which pertinent evidence is not easy to find. Authors should be straightforward in describing their search constraints.17

Authors should also report the approach that was or will be taken in the development of a search strategy, including qualifications of the searcher (such as a health information specialist with systematic review experience), planned databases to be searched (see Item 9), limits to be imposed (to demonstrate alignment with review eligibility criteria), and whether the search was or will be peer reviewed and by whom.91 Having a search strategy peer reviewed may help to increase its comprehensiveness or decrease yield where search terminology is unnecessarily broad.

The draft search strategy can be presented in the body of the text or as a table. If the protocol is being published in a journal, the journal may advise on this issue (that is, in their instructions to authors). If space is a concern, authors should ask the editor whether it can be included it as a web based appendix or whether an electronic link to where it can be found can be provided in the manuscript.

Providing details of the planned search strategy will allow readers of systematic review protocols to appraise and avoid potential duplication of efforts, as well as possibly enhance the development of their own searches. Including at least one main search strategy can also specifically facilitate updating.

Study records

Item 11a: Data management. Describe the mechanism(s) that will be used to manage records and data throughout the review

Example

“Literature search results will be uploaded to Distiller Systematic Review (DSR) Software, an Internet based software program that facilitates collaboration among reviewers during the study selection process. The team will develop and test screening questions and forms for level 1 and 2 assessments based on the inclusion and exclusion criteria. Citation abstracts and full text articles will be uploaded with screening questions to DSR. Prior to the formal screening process, a calibration exercise will be undertaken to pilot and refine the screening questions. Further, we will provide training to new members of the review team not familiar with the DSR software and the content area prior to the start of the review.”54

Explanation

Systematic review data management software is becoming increasingly common. Examples of web based software are Distiller SR and Eppi-Reviewer. These web based software management programs are helpful in managing small or large scale datasets by allowing importation of citations and PDFs to be screened and included. They may reduce data entry errors during the data extraction process by allowing direct entry into pre-created data extraction forms and export of data directly into statistical analysis software. They may also facilitate the creation of a PRISMA flow diagram once the screening process is completed. Whether use of such software is planned to manage records in the review should be described in the protocol. Several other tools may be used during the review process to de-duplicate references (such as reference management software) and to extract or manage data (such as electronic software).92 Reviewers using more traditional forms of data management should also describe their process. Whatever process is used, it should be described in sufficient detail so that interested readers can replicate the process.

Some studies are published more than once. Duplicate publications may be difficult to ascertain, and their inclusion may introduce bias.93 94 We ask authors to describe any steps they are proposing to use to avoid double counting and to piece together data from multiple reports of the same study (such as juxtaposing author names, treatment comparisons, sample sizes, or outcomes). We also recommend that authors indicate whether all reports on a study were considered, as inconsistencies may reveal important limitations. For example, a review of multiple publications of drug trials showed that reported study characteristics may differ from report to report, including the description of the design, number of patients analyzed, chosen significance level, and outcomes.95 See Item 12 (data items) for more information.

Item 11b: Selection process. State the process that will be used for selecting studies (such as two independent reviewers) through each phase of the review (screening, eligibility, and inclusion in meta-analysis)

Example

“The review authors will independently screen the titles and abstracts yielded by the search against the inclusion criteria. We will obtain full reports for all titles that appear to meet the inclusion criteria or where there is any uncertainty. Review author pairs will then screen the full text reports and decide whether these meet the inclusion criteria. We will seek additional information from study authors where necessary to resolve questions about eligibility. We will resolve disagreement through discussion. We will record the reasons for excluding trials. Neither of the review authors will be blind to the journal titles or to the study authors or institutions.”96

Explanation

Reviewers will often identify a large number of studies from electronic database searches, and then use pre-defined eligibility criteria (Item 8) to determine which records are relevant and should be included in the review. There is currently no agreed process for how studies should be selected for inclusion in a systematic review. For example, it is unclear whether all records identified by the search should be initially screened for potential inclusion by two independent reviewers, or if only those noted as excluded by one reviewer should be. Protocol authors should therefore describe their specific approach for identifying potentially eligible records (that is, by title and abstract screening) and for selecting studies for final inclusion (that is, by full text screening). Typical methodology for study selection is aimed at enhancing objectivity and preventing mistakes. Often, screening is carried out in duplicate by independent reviewers at each stage of the review to reduce the possibility of excluding relevant reports.97 The benefit may be greatest for topics where selection or rejection of an article requires difficult judgments.98

Authors should report whether one or several persons will be involved in each stage of screening and name those who will be involved, if known. If independent screening is planned, authors should describe the process for dealing with discrepancies (such as third party arbitration or contacting authors of original studies) and whether inter-rater agreement will be calculated.

Item 11c: Data collection process. Describe planned method of extracting data from reports (such as piloting forms, done independently, in duplicate), any processes for obtaining and confirming data from investigators

Example

“Using standardized forms … and a detailed instruction manual that will be used to inform specific tailoring of an online data abstraction program (DistillerSR), ten teams of reviewers will extract data independently and in duplicate from each eligible study. To ensure consistency across reviewers, we will conduct calibration exercises before starting the review. Data abstracted will include demographic information, methodology, intervention details, and all reported patient-important outcomes. Reviewers will resolve disagreements by discussion, and one of two arbitrators (JWB or GHG) will adjudicate unresolved disagreements. We will contact study authors to resolve any uncertainties.”99

Explanation

Reviewers should plan and document the approach they plan to use to extract data from included studies in the review along with which data items (Item 12) and types of data. Data extraction forms should be developed a priori and included in the published or otherwise available review protocol as an appendix or as online supplementary materials.

As with screening, data extraction is often carried out in duplicate by independent reviewers or by one reviewer with verification by another in order to reduce bias and reduce errors in data extraction. The planned approach for resolving discrepancies should be stated. Although single data extraction has not been shown to substantially affect treatment effect estimates, reviewers should explicitly indicate whether single extraction will be employed to allow reviewers and readers to be more mindful of the possibility for errors in the completed review.100

Data extraction can be complicated, especially with more complex topics, and level of reviewer experience has not been shown to affect extraction error rates.101 102 As such, additional strategies planned to reduce errors, such as training of reviewers and piloting of extraction forms should be described. In addition, if reviewers plan to make use of data extraction techniques to obtain outcome data not reported in a usable format, such as translating graphically presented data into a usable (that is, numeric) format,103 they should plan for this during the protocol stage and report details of proposed software and its sensitivity and specificity.

If an individual patient data (IPD) meta-analysis is planned, authors should also tell readers when and how they sought individual patient data from the original researchers.104 Data extraction for IPD reviews will often involve collection and scrutiny of detailed raw databases; authors should describe their planned approach clearly. The description might include how they attempted to contact researchers, what they asked for (that is, using a reply form with pre-specified data items), and their plan if they are unable to obtain all requested information. For IPD meta-analyses or otherwise, reviewers should also state whether they intend to confirm the accuracy of the extracted information to be included in their review with original researchers, for example, by sending them a copy of the draft review when available.105

Data in primary studies may not always be presented in a format that is useful to systematic reviewers. Contacting authors for missing information about treatments, for example, has been shown to improve the completeness of treatment descriptions by at least 27%.106 Ideally, authors of primary studies should be urged to report all aspects of their studies more clearly.107 However, in the absence of complete descriptions of treatments, outcomes, effect estimates, or other important information, reviewers may consider asking authors for this information. Whether reviewers plan to contact authors of included studies and how this will be done (such as a maximum of three email attempts) to obtain missing information should be documented in the protocol.

Knowledge of duplicate, overlapping, or companion studies (that is, multiple reports of a single study) may come to light only during the data extraction process.94 The inclusion of data from multiple reports as separate studies may lead to biased treatment effects93 and should be anticipated by reviewers. Methods for identifying and dealing with multiple reports of a single study have been described.108 109 Authors should present the algorithm they will follow to select data from overlapping reports and the planned approach for solving logical inconsistencies across reports.

Data items

Item 12. List and define all variables for which data will be sought (such as PICO items, funding sources) and any pre-planned data assumptions and simplifications

Example 1

“We will extract the generic and the trade name of the experimental intervention, the type of control used, dosage, frequency and duration of treatment, patient characteristics (average age, gender, mean duration of symptoms, type of joints affected), type of pain or function related outcome extracted, trial design, trial size, duration of follow-up, type and source of financial support and publication status from trial reports. For non-pharmacological interventions, we will extract type, modes of application and intensity, if appropriate. When necessary, means and measures of dispersion will be approximated from figures in the reports. For cross-over trials, we will extract data from the first period only because of possible carry-over effects. Whenever possible, we will use results from an intention to treat analysis. If effect sizes cannot be calculated, we will contact the authors for additional data.”110

Example 2 (data simplifications)

“It is possible that individual studies may consist of multiple treatment groups, such as different types of depression interventions or different doses of medication. In order to avoid the possibility of introducing bias caused by multiple statistical comparisons with one control group, we will combine the groups from multiple arm studies into a single group.”111

Explanation

Readers need to know what information review authors plan to obtain from the included studies. Data items and pre-specified time points are essential to document in a review protocol because this information allows readers to refer back to the protocol when the review is complete to determine whether changes occurred. Extraction forms should include definitions of variables, with particular details about the planned outcomes, and their measurement duration and frequency (Item 13).

The selective reporting of information in reviews is a documented concern.8 36 Providing readers with the opportunity to identify and make their own judgments about selective reporting is crucial.112 If the review is limited to reporting only those variables that were obtained, rather than those that were deemed important a priori but could not be obtained, bias might be introduced and the reader might be misled. In protocol amendments and completed reviews, authors should clearly outline whether any data items were added after the protocol was developed or after the review began and give the reasons why. Such variables might include aspects of treatments or outcomes identified as important because they recur during the review process (such as important outcome measures that the reviewers initially overlooked). A more complete discussion of selective outcome reporting in systematic reviews and related bias is found in Item 13.

Authors should describe assumptions they intend to make if they encounter missing or unclear information and explain how they plan to deal with such data or lack thereof, in addition to contacting authors (Item 11c). For example, in studies of women aged 50 or older it may be reasonable to assume that none was pregnant even if this is not reported. Ideally, authors should anticipate as many uncertainties as possible before they arise and have a documented, agreed approach for dealing with such data. Likewise, review authors might make assumptions about the route of administration of drugs assessed. However, a more prudent approach is required when dealing with qualitative information. For example, the upper age limit for “children” can vary from 15 years to 21 years, or the level of severity of an outcome (such as an adverse effect) might be poorly described in primary research and mean very different things to different researchers at different times and for different patients.

If simplifications such as combining treatment arms (for multiple treatment trials) or using first period data for cross over trials are planned, these should be described.

Outcomes and prioritisation

Item 13. List and define all outcomes for which data will be sought, including prioritisation of main and additional outcomes, with rationale

Example

“Primary outcomes

“The primary outcome will be the number of patients who responded to treatment, defined as a reduction of at least 50% on the Hamilton Depression Rating Scale (HAM-D), the Montgomery-Asberg Depression Rating Scale (MADRS) or any other depression scale, or ‘much or very much improved’ (score 1 or 2) on the Clinical Global Impression (CGI) Improvement Scale. All response rates will be calculated from the total number of randomised patients. Where more than one criterion is provided, we will use the HAM-D for judging the response and then follow the sequence described above. Despite the problems surrounding scale-derived response cutoffs, dichotomous outcomes can be understood more intuitively by clinicians than the mean values of rating scales and are therefore preferred.

When studies report response rates at various time points of the trial, we have decided a priori to subdivide the treatment indices as follows.

The acute phase treatment response, that is between six and 12 weeks, was our primary outcome of interest.

Secondary outcomes

1. The number of participants in remission, as defined by either: (a) at 7 or less on the 17-item HAM-D and at 8 or less for all the other longer versions of HAM-D; (b) at 10 or less on the MADRS; (c) ‘not ill or borderline mentally ill’ (score 1 or 2) on the CGI-Severity; or (d) other criteria as defined by the trial authors. All remission rates will be calculated out of the total number of randomised patients. Where two or more scales are provided, we prefer the first criteria for judging remission.

‘Remission’ is a state of relative absence of symptoms. This outcome adds to the primary outcome ‘response’ to treatment. The disadvantage of ‘remission’ is that its frequency depends on the initial severity of the participants. If they were only relatively mildly ill, many will be classified as in remission while only few will be in the case of high average severity at baseline. Therefore, studies and meta-analyses usually apply response and not remission as the primary outcome.

2. Change scores from baseline or endpoint score at the time point in question (early response, acute phase response, or follow-up response as defined above) on the HAM-D or MADRS, or any other validated depression scale. The results of mean values of depression rating scales can be more sensitive than dichotomous response data. Therefore, they should also be presented even though their interpretation is less intuitive than with dichotomous response data. Change data will be preferred to endpoint data but both will have to be presented separately because we will use the standardised mean difference as an effect size measure for which pooling of endpoint and change data is not appropriate. We prefer change scores to endpoint scores because they, to a certain extent, take into account small baseline imbalances.

3. Social adjustment, social functioning including the Global Assessment of Function scores.

4. Health-related quality of life as measured by validated disease specific and generic scales such as the Short Form (SF)-36 or the Health of the Nation Outcome Scales (HoNOS).

5. Various reasons for dropping out of the studies:

6. Death:

7. Side-effects:

We anticipate including the following main outcomes in a summary of findings table using GRADEpro: response to treatment, acceptability of treatment (dropout due to any reason), quality of life, death due to suicide and overall tolerability (dropout due to adverse events).”113

Explanation

Systematic reviews must include a description of all outcomes (endpoints) of interest,74 and by extension the same applies to protocols. Systematic reviews that aim to inform decision making should summarize both benefits and harms of interventions,114 and specifying what those are during the planning phases of a review is, at minimum, a reminder or a commitment to do so. Review protocols should distinguish between which outcomes are considered the main outcome(s), also known as primary outcome(s), of a review and those that are additional (secondary) outcomes; these may differ from the prioritisation assigned to outcomes in primary studies.

Listing all outcomes for which data will be sought in a review and providing sufficient details and definitions are essential in a review protocol. Some outcomes may warrant additional details in their definitions such as distinctions between surrogate versus clinical, composite versus non-composite, and objective measurement versus subjective assessment. If, for example, a surrogate outcome is specified in lieu of a clinical outcome, a rationale as to why this was done and how the surrogate outcome is an indicator (associated) of a clinically important outcome should be stated. Consider, for example, a systematic review that focuses primarily on whether continuous positive airway pressure treatment reduces symptoms of somnolence and fatigue in patients with obstructive sleep apnoea (an abnormality of breathing patterns during sleep). The outcomes of interest should include instruments measuring symptoms (such as the Epworth Sleepiness Scale)115 but not necessarily neurophysiological signals such as the frequency of apnoeas (no breathing) or hypopnoeas (reduced breathing), muscle tone, and heart rate variability, which are commonly reported but do not correlate well with symptoms.116 Authors should do sufficient investigation during the planning stage to ensure that selected outcomes are relevant. Given increasing efforts to involve patients in the selection and assessment of outcomes,117 reviewers should indicate whether planned outcomes are patient centred, and further, whether they are patient reported, and how such outcomes will be treated.118

The reporting of composite outcomes within a completed systematic review has been found to be variable across the abstract, methods, and results sections of the report.119 Because the various components of a composite outcome have the potential to be combined in different ways, yielding differences in the direction, strength, and significance of an outcome, it is essential in a review protocol to state and define each component of a composite outcome explicitly, and, further, state how components within a composite outcome will be analysed, whether independently, all together, or in specific combinations (Item 15b).

Meta-analyses within systematic reviews are often limited by information available in included study reports. As such, discrete descriptions of the endpoints are not always possible at the protocol stage. The minimum and often only information one can practically specify is a broad description of the “outcome concept”—for example, what is the effect of an intervention on “survival or mortality.” Such a description is too generic, and authors will need to refine it when they conduct their systematic review. Examples of more refined descriptions are “mortality at 12 months” or “mortality at 5 years” (for example, as odds ratios from cross tabulated counts of deaths at these follow-up durations) and “survival” (typically hazard ratios from time-to-event analyses). Reviewers should state their plans to refine outcome definitions based on definitions used in included studies.

Careful consideration of outcomes during the planning stages of a review can also improve efficiency in the review process. For example, if authors make a decision to add an outcome(s) at some point during data extraction, they will need to revisit all included papers to extract the additional information; this is a waste of reviewers’ time. Minimizing such back and forth economizes time and resources and reduces the likelihood of mistakes.

The main outcome(s) of a review should be distinguished from additional outcomes and specific definitions of each should be provided. The scientific question or the decisional problem that motivates the systematic review typically dictates the main outcome(s) of interest. Thus for systematic reviews that aim to inform healthcare decisions or policy, the main outcomes are likely to be patient relevant outcomes (such as risk of stroke) or validated surrogate outcomes (for example, change in cholesterol levels is a valid surrogate for the risk of cardiovascular events for statin based interventions). In contrast, systematic reviews that aim to summarize the state of the science in the pathophysiology of a disease might appropriately choose biochemical or other measurements as main outcomes. All other outcomes are considered additional and are reviewed to provide complementary information and for completeness.

Listing and defining outcomes in a review protocol, as well as the prioritization of each as a main or additional outcome, will facilitate the ability of future readers of completed reviews to investigate selective reporting. Selective reporting of outcomes—that is, the addition, removal, or change in the priority of review outcomes between the protocol, methods section, and results of a review—is well recognized.10 120 A 2010 study comparing Cochrane protocols with the completed reviews found that 22% of Cochrane reviews had a discrepancy in at least one outcome measure compared with their protocols, at least 75% of which were attributable to changes in the primary outcome, some after knowledge of review findings.10 This is described as outcome reporting bias and occurs when the reporting of an outcome is associated with its significance. Whether in a completed review, outcomes are prioritized as main or additional should not be dependent on their prioritization or statistical significance in included studies.

Readers will note that the contents of this item are overlapping with Item 8 (eligibility criteria). Given the importance of outcomes in the review process, issues in the selection of relevant outcomes, and their potential to be manipulated during the review process, we felt that an item specifically dedicated to the reporting of outcomes would greatly facilitate complete and transparent reporting around this item. Readers should also note that complete definition and description of planned review outcomes, as proposed above, will occupy substantial space in a review protocol.

Risk of bias individual studies

Item 14. Describe anticipated methods for assessing risk of bias of individual studies, including whether this will be done at the outcome or study level, or both; state how this information will be used in data synthesis

Example 1

“To facilitate the assessment of possible risk of bias for each study, we will collect information using the Cochrane Collaboration tool for assessing the risk of bias (Table 8.5.a in the Cochrane Handbook for Systematic Reviews of Interventions), which covers: sequence generation, allocation concealment, blinding, incomplete outcome data (e.g. dropouts and withdrawals) and selective outcome reporting. For each domain in the tool, we will describe the procedures undertaken for each study, including verbatim quotes. A judgement as to the possible risk of bias on each of the six domains will be made from the extracted information, rated as ‘high risk’ or ‘low risk’. If there is insufficient detail reported in the study we will judge the risk of bias as ‘unclear’ and the original study investigators will be contacted for more information. These judgements will be made independently by two review authors based on the criteria for judging the risk of bias (Table 8.5.c in the Cochrane Handbook Higgins 2011). Disagreements will be resolved first by discussion and then by consulting a third author for arbitration. We will compute graphic representations of potential bias within and across studies using RevMan 5.1 (Review Manager 5.1). We will consider each item in the risk of bias assessment independently without an attempt to collate and assign an overall score.”121

Example 2

“Included non-randomised studies may or may not have a comparison group. To assess the risk of bias within included … studies, the methodological quality of potential studies will be assessed by using the Newcastle-Ottawa scale (NOS) for assessing the quality of non-randomised studies in meta-analyses. The NOS for case-control and cohort studies will be adapted (Table 1) to meet the specific needs of this systematic review. The cohort scale will be modified for use in case series. Using the NOS, studies will be awarded a maximum of nine points on items related to the selection of the study groups, the comparability of the groups, and the ascertainment of outcome of interest. Using this modified score, case series will be eligible for a maximum of six points. This will be undertaken by two separate reviewers. Where there is disagreement, a third reviewer will be used as an arbitrator.”122

Explanation

An assessment of the risk of bias (or “quality”) of studies included in a review is an important component of any well planned or conducted systematic review. Such an assessment contributes to the evaluation of the overall strength of evidence of the review (Item 17). Established methods for assessing risk of bias in reviews have been documented.123 124 Descriptions of the planned approach to assessing risk of bias should include the constructs being assessed and a definition for each, reviewer judgment options (high, low, unclear), the number of assessors, experience of assessors (training, piloting, previous risk of bias assessment experience), as well as method(s) of assessment (independent or in duplicate).125 Whether reviewers are going to be blinded to studies should also be reported,126 127 as well as whether agreement between reviewers will be evaluated and, if so, how.

Details of planned methods to summarise risk of bias assessments across studies or outcomes should be provided. Although authors may spend a large proportion of time assessing risk of bias in included studies, they are often silent on how the results might influence their review findings.128 129 Thus, we encourage reviewers to think about this at the development stage and document their plans in the protocol. Authors should also describe how risk of bias assessments will be incorporated into data synthesis (that is, subgroup or sensitivity analyses) and their potential influence on findings of the review (Item 15c)129 in the protocol.

The likelihood that the treatment effect reported in a systematic review represents the true effect depends on the validity of the included studies, namely, the internal validity. Certain methodological characteristics of primary studies may be associated with their resulting effect sizes.129 130 131 For example, trials describing inadequate methods of allocation concealment or with unclear concealment exaggerate treatment effects on average compared with trials reporting adequately concealed allocation132 Therefore, authors should not only describe risk of bias methods and constructs to be assessed for each included study, but also describe how results of the assessment contribute to the overall findings of the review.128 Additionally, authors should provide a rationale if they do not intend to assess risk of bias.

Many methods exist to assess the overall risk of bias in included studies, including scales, checklists, and individual components.133 134 As summarized in the PRISMA elaboration document,17 scales that numerically summarize multiple components into a single number are misleading and unhelpful.135 Rather, authors should specify the methodological components that they plan to assess and how they plan to assess said components. Common markers of validity for randomised trials, in the Cochrane Risk of Bias tool,123 include appropriate generation of random allocation sequence136; concealment of the allocation sequence132; blinding of participants, healthcare providers, data collectors, and outcome adjudicators137 138; and proportion of patients lost to follow-up.139 Reviewers may also anticipate assessing other items that do not necessarily indicate bias, such as the impact of early stopping of trials for benefit,140 141 industry sponsorship,55 142 single trial centres,143 and improper analyses or fabrication of primary study data.144 145 If authors plan such assessments they should explain this information in the protocol.

Authors should give careful consideration to assessments for reviews that expect to include non-parallel group randomised controlled trials and studies of non-randomised design, for which methodological standards are currently under development.146 The ultimate decision regarding which methodological features should be evaluated requires consideration of the strength of the empirical data, theoretical rationale, and the unique circumstances of the included studies within the context of the review question.

Data synthesis

Item 15a. Describe criteria under which study data will be quantitatively synthesised

Example 1

“If studies are sufficiently homogeneous in terms of design and comparator, we will conduct meta-analyses using a random-effects model.”121

Explanation

Diversity in study populations, interventions, outcomes, or trial conduct may mean that including some studies in a meta-analysis, or even conducting meta-analyses at all, will be impossible. Authors should describe, with reference to the PICO criteria, the conditions that should be present before they will proceed with statistical synthesis (Item 15b). Thus authors might consider whether to include trials with differing formulations or doses of the experimental treatment, studies using differing versions of a technology (such as a device), studies with different age profiles in the sample population, or studies with different follow-up times.

Item 15b. If data are appropriate for synthesis, describe planned summary measures, methods of handling data, and methods of combining data from studies, including any planned exploration of consistency (such as I2, Kendall’s τ)

Example

“Measures of treatment effect

Unit of analysis issues

The primary analysis will be per individual randomised; however, all included trials will be assessed in order to determine the unit of randomization and whether or not this unit of randomization is consistent with the unit of analysis. Special issues in the analysis of studies with non-standard design, like cluster randomised trials, cross-over trials, and studies with multiple treatment groups, will be addressed. For cluster randomised trials we will extract an interclass correlation co-efficient to modify the results according to the methods described in the Cochrane Handbook for Systematic Reviews of Interventions. For cross-over trials, a major concern is carry-over effect. We will only use the data from the first phase, guided by the Cochrane Heart Group. When a study has more than two treatment groups, we will present the additional treatment arms. Where the additional treatment arms are not relevant, they will not be taken into account. We will also acknowledge heterogeneity in the randomization unit and perform a sensitivity analysis.

Dealing with missing data

When there are missing data, we will attempt to contact the original authors of the study to obtain the relevant missing data. Important numerical data will be carefully evaluated. If missing data cannot be obtained, an imputation method will be used. We will use sensitivity analysis to assess the impact on the overall treatment effects of inclusion of trials which do not report an intention to treat analysis, have high rates of participant attrition, or with other missing data.

Assessment of heterogeneity

We will test the clinical heterogeneity by considering the variability in participant factors among trials (for example age) and trial factors (randomization concealment, blinding of outcome assessment, losses to follow-up, treatment type, co-interventions). Statistical heterogeneity will be tested using the Chi2 test (significance level: 0.1) and I2 statistic (0% to 40%: might not be important; 30% to 60%: may represent moderate heterogeneity; 50% to 90%: may represent substantial heterogeneity; 75% to 100%: considerable heterogeneity). If high levels of heterogeneity among the trials exist (I2 >=50% or P <0.1) the study design and characteristics in the included studies will be analysed. We will try to explain the source of heterogeneity by subgroup analysis or sensitivity analysis.

Data synthesis

Each outcome will be combined and calculated using the statistical software RevMan 5.1, according to the statistical guidelines referenced in the current version of the Cochrane Handbook for Systematic Reviews of Interventions. The Mantel-Haenszel method will be used for the fixed effect model if tests of heterogeneity are not significant. If statistical heterogeneity is observed (I2 >=50% or P <0.1), the random effects model will be chosen. If heterogeneity is substantial, we will not perform a meta-analysis; a narrative, qualitative summary will be done.”147

Explanation

When authors intend to perform meta-analyses, they should specify the effect measure (such as relative risk or mean difference) (Item 13) and the statistical method (such as inverse variance, DerSimonian-Laird, Mantel-Haenszel, Bayesian) to be used and whether they plan to apply a fixed or random effects approach.148 Although experts debate this topic, fixed effects meta-analyses have been shown to overestimate confidence in treatment effects; thus, reviewers may wish to use this approach conservatively.149 150 If estimates of heterogeneity are to be used to decide between fixed and random effects approaches, authors should state the threshold of heterogeneity required.151 If possible, authors should explain the reasons for these choices.

Reviewers should anticipate that data from included studies may not be in a suitable format for analysis or presentation in the review. For that reason, authors may need to take various steps to process the data, even if they do not plan meta-analyses. Authors should describe their plans for data processing, focusing on anticipated problems specific to their review. In trials with more than two intervention groups (for example, receiving similar but non-identical interventions), combining or splitting results across groups may be necessary.152 If individual patient data (IPD) meta-analyses are planned, reviewers should consult the (forthcoming) PRISMA extension for IPD meta-analyses.153

For analyses of dichotomous data (that is, event data), authors should consider how best to handle rare events or when events are absent from some studies. Outcomes reported as measurement scales (such as for depression) may use different scales in different studies; results may need to be adjusted so that all scales are aligned (for example, so that low values represent good health on all scales).

Reviewers should also anticipate that some desired data will not be reported in included studies at all. In particular, standard deviations and standard errors may have to be reconstructed from other statistics such as P values and t statistics154 155; occasionally they may be imputed from the standard deviations observed in other studies.156 157 In analyses of time-to-event data, reviewers should anticipate spending more time and caution during data extraction (for example, from Kaplan-Meier survival curves) and report how conversion to a consistent format is planned.158

Statistical combination of data from two or more separate studies in a meta-analysis may not always be necessary, feasible, or desirable. Regardless of the decision to combine individual study results, authors should report how they plan to evaluate between-study variability (heterogeneity or inconsistency), such as by using I2 or Cochran’s Q test. The consistency of results across studies may influence the decision whether to combine individual study data in a meta-analysis. If reviewers plan to use statistical estimates of consistency (such as I2 or Kendall’s τ) to determine whether to perform a meta-analysis, they should state this explicitly (Item 15a) and specify the required number.

Finally, the name (and version) of any software planned for completing meta-analyses should be reported.

Item 15c. Describe any proposed additional analyses (e.g., sensitivity or subgroup analyses, meta-regression)

Example

“Subgroup analysis and investigation of heterogeneity

Subgroup analyses will be used to explore possible sources of heterogeneity, based on the following.

Sensitivity analysis

Sensitivity analysis will be performed in order to explore the source of heterogeneity as follows.

Explanation

Investigating possible causes of between-study variability or exploring the robustness of meta-analyses by using subgroup analysis or meta-regression may be desirable. If authors plan such analyses, they should state this and specify the covariates anticipated for the analyses (such as disease type or severity, or treatment dose). For subgroup analyses, authors should describe how they will partition the covariate into subgroups (for example, what will constitute mild or severe disease, low or high treatment dose). Whether they plan a fixed or random effects approach and how they will evaluate residual heterogeneity should also be stated.

If any sensitivity analyses are intended—such as including or excluding small studies, studies with high risk of bias,159 industry funded studies, or outlier studies—authors should describe their plan for doing so.

Item 15d. If quantitative synthesis is not appropriate, describe the type of summary planned

Example

“A systematic narrative synthesis will be provided with information presented in the text and tables to summarise and explain the characteristics and findings of the included studies. The narrative synthesis will explore the relationship and findings both within and between the included studies, in line with the guidance from the Centre for Reviews and Dissemination.”160

Explanation

In nearly all cases, reviews will include a qualitative (narrative) synthesis or summary even if meta-analyses or other quantitative analyses have been done. If, in addressing items 15a, 15b, and 15c, authors have concluded that some or all of the expected data will not be suitable for combining quantitatively, they should explicitly say so in the protocol and provide the rationale for such decisions. Then for item 15d they should describe the way they propose to present results in narrative form.

Established methods for narrative syntheses are available.161 162 Authors should, to the extent possible at the protocol stage, highlight the order in which they will present information and what they will give in text or (only) in tables. They should describe what priority they will give to information about participant populations (such as overall patient groups before subgroups, subgroups defined by sociodemographics before those defined by coexisting conditions) and about interventions and comparisons of interventions (such as head to head trials before trials with placebo or usual care controls, ultimate health outcomes before intermediate outcomes, patient related outcomes before utilization outcomes, and so forth). For example, authors may say that they will present results in order by key question and, within key questions, in order of main then additional outcomes. In other cases, they might specify that results will be reported first by key questions but then by important comparisons and outcomes within comparisons.

In addition, authors should say whether they plan to report only on studies for which risk of bias was either low or moderate and omit studies with high risk of bias, or whether they expect to retain studies of any level of risk of bias in their analyses. They should note that levels of risk of bias for a given study may differ depending on the outcome of interest, so that some studies may be retained for certain key questions or outcomes but not for others. In some cases, authors might note that they will report on studies at high risk of bias only when they provide the available information or a critical outcome or population of interest.

Authors should describe how they plan to present information by type of study design (for example, report results only for randomised controlled trials, and then supplement the results with information drawn from non-randomised trials or non-experimental studies). In some cases authors may want to stratify how they present information based on key aspects of how studies were conducted (such as whether investigators, patients, and outcome assessors were all masked to intervention). If authors will focus on specific types of outcome measures, such as demonstrably reliable and valid instruments to measure depression or pain, they should report this information.

Regardless of how many quantitative analyses authors expect to present, they should indicate the extent to which they plan to use tables to summarize (a) the characteristics of studies (perhaps only those of low or moderate risk of bias) and (b) the principal comparisons or outcomes of concern.

In some cases, review authors may plan to do types of analyses other than meta-analyses. These may include cost of illness, cost of treatment, or cost effectiveness analyses, decision modelling analyses, or various types of subgroup analyses (independent of any required by a key question). In all these cases, authors should be as specific as possible about what they will attempt to do.

Meta-bias(es)

Item 16. Specify any planned assessment of meta-bias(es) (such as publication bias across studies, selective reporting within studies)

Example

“In order to determine whether reporting bias is present, we will determine whether the protocol of the RCT was published before recruitment of patients of the study was started. For studies published after July 1st 2005, we will screen the Clinical Trial Register at the International Clinical Trials Registry Platform of the World Health Organisation (http://apps.who.int/trialssearch). We will evaluate whether selective reporting of outcomes is present (outcome reporting bias). We will compare the fixed effect estimate against the random effects model to assess the possible presence of small sample bias in the published literature (i.e. in which the intervention effect is more beneficial in smaller studies). In the presence of small sample bias, the random effects estimate of the intervention is more beneficial than the fixed effect estimate. The potential for reporting bias will be further explored by funnel plots if ≥10 studies are available.”163

Explanation

Authors should pre-specify any methods used to explore the possibility that the data identified are biased due to non-study related processes.164 Such bias may result from non-publication of studies (publication or dissemination bias) and the reporting of a subset of measured outcomes and analyses within studies (outcome reporting bias) (see box 2).

Box 2: Meta-bias caused by selective publication of studies and selective reporting within studies

Systematic reviews aim to synthesise the results of all relevant studies. However, some studies may not be published, and a subset of outcomes and analyses may be incompletely, inadequately, or selectively reported in a published article, based on the results (such as statistical significance, magnitude, or direction of effect). The validity of systematic reviews may be threatened if the outcome data available to reviewers comprise a biased selection of all data that actually exists.181 182 Such biases are termed meta-biases, meaning that they occur independent of procedural problems during the conduct of a primary study as do typical methodological biases (such as inappropriate method of random sequence generation in randomized trials).164

_Publication or dissemination bias—_Several systematic reviews of empirical studies have found that clinical trials with statistically significant (P<0.05) or positive results are more likely to be published than those with non-significant or negative results.2 165 183 Investigators’ decisions not to submit papers with negative results for publication, rather than editors’ rejection of such papers, tend to be the main source of publication bias.184 However, the decision to write up a study for publication may be influenced by pressure from study sponsors and journal editor.185 Studies with statistically significant results also tend to be published earlier than studies with non-significant results.165 If studies are missing from a systematic review for these reasons, exaggerated results may be produced.

_Outcome reporting bias—_The selective reporting of outcomes due to their significance, magnitude, or direction is termed outcome reporting bias and has been widely documented across the trial literature.2 Outcomes specified in the protocol may be completely omitted from the published report. When an outcome is measured using multiple scales or at multiple time points, and analysed in various ways (such as intention-to-treat and per-protocol analysis, unadjusted and adjusted for covariates), the choice of which data to present may be influenced by the results. Non-significant results may be partially reported (such as reporting an effect estimate with no measure of variation), resulting in insufficient data to include in a meta-analysis. All of these examples of selectively reported outcome data in primary studies can bias (and sometimes, overestimate) the results of systematic reviews.2 7 186

Empirical evidence of selective outcome reporting bias in trials exists. A systematic review of 16 cohorts of clinical trials comparing outcomes reported in trial protocols with the published reports found that at least one primary outcome was omitted, introduced, or changed in 4-50% of reports.3 In a landmark study, Chan and colleagues found that statistically significant outcomes had higher odds of being fully reported in trial publications compared with non-significant outcomes for efficacy (pooled odds ratio 2.4 (95% confidence interval 1.4 to 4.0)) and safety (pooled odds ratio 4.7 (1.8 to 12)).164

Detecting or correcting for publication bias in a systematic review is difficult. The results of available studies may provide clues that some studies may be missing (such as when smaller studies have systematically different effect estimates than larger studies (“small study effects”)).165 Recommendations regarding appropriate graphical methods (such as funnel plots) and statistical methods (such as Egger’s test) to assess small study effects have been proposed.166 However, publication bias is only one of several possible explanations for small study effects, and the interpretation of such tests can be problematic.166 167 168 Authors should report their planned testing strategy to assess publication bias in detail. The risk of publication bias was formally assessed in only 21% of 100 intervention reviews published in 2006, and only 32% considered this type of bias.169 A review of antidepressant trials found that effect estimates of meta-analyses of only the published trials were 32% larger on average than effect estimates of meta-analyses including published and unpublished trials.170 The corresponding magnitude of publication bias in antipsychotic trials was smaller (8%).171

Several methods to detect selective outcome reporting exist. If a study protocol is available, reviewers can compare outcomes reported in the protocol and the published report.7 172 Comparing the outcomes reported in the methods and results sections of the published report is an option when a protocol is unavailable.173 For some trials, reviewers might assume that it is likely that an outcome was measured even if it was not reported, based on knowledge of the clinical area (such as when systolic, but not diastolic, blood pressure is reported).112 Authors may use the Outcome Reporting Bias in Trials (ORBIT) classification system.4 A sensitivity analysis to assess the impact of selective reporting on meta-analytic results may also be considered.174 In eight of 28 Cochrane reviews published in March 2010, authors did not assess outcome reporting bias; in 16 reviews, authors did assess this bias using the published report; and in the remaining reviews, trial protocols were used.175 In another study, after investigators applied sensitivity analyses to adjust for outcome reporting bias in 81 Cochrane reviews, the treatment effect estimate was reduced by 20% or more in 19 (23%) of the meta-analyses.4

Both publication bias and outcome reporting bias may affect meta-analyses, and the effect can be unpredictable. Adding unreported data from both published and unpublished drug trials to 41 meta-analyses caused 46% of the meta-analytic effect estimates to show lower efficacy of the drug, 7% to show identical efficacy, and 46% to show greater efficacy.176

Confidence in cumulative estimate

Item 17. Describe how the strength of the body of evidence will be assessed (such as GRADE)

Example

“The quality of evidence for all outcomes will be judged using the Grading of Recommendations Assessment, Development and Evaluation working group methodology. The quality of evidence will be assessed across the domains of risk of bias, consistency, directness, precision and publication bias. Additional domains may be considered where appropriate. Quality will be adjudicated as high (further research is very unlikely to change our confidence in the estimate of effect), moderate (further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate), low (further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate), or very low (very uncertain about the estimate of effect).”54

Explanation

Authors should describe which approach they plan on using to summarize the confidence they have in the resulting body of evidence, ideally using an established and validated approach. The description should include a plan for assessing the risk of bias across studies, inconsistency, imprecision, indirectness, publication bias, and factors that increase the confidence in an effect (such as large effects, dose effect relations, and issues around opposing bias and confounding not explaining an effect or lack thereof) for each outcome that is included in the PICO. The Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach is increasingly recommended.168

If no such assessments are planned, the authors should state this with a rationale for why not. Authors should describe whether and how they assess the directness related only to populations (including applicability) who are included in the evidence that is assessed (such as if they extrapolated and for what reasons), so that users of the systematic review can make these judgments later for other populations.177 178 Authors should specify whether the assessment of the strength of evidence will include studies that are excluded from meta-analysis (if applicable).

“Strength of evidence” and “quality of evidence” have been previously been used interchangeably.

Discussion

We hope this detailed explanatory paper will become a pedagogical document that the entire systematic review community can use. Similarly, we have strived to ensure that the paper is useful to authors seeking guidance in what to include in a protocol of their systematic review. We recommend that authors use this paper when seeking a more complete explanation of each item included in the PRISMA-P checklist. We developed this protocol extension to PRISMA in the hopes that it will improve the reporting of protocols and also simplify the process of reporting a protocol, and registering it with PROSPERO. The development of the PRISMA-P 2015 checklist borrowed heavily from the mandatory items included in PROSPERO. When authors register their protocol on PROSPERO, much of this information is the same as what is recommended when completely reporting a protocol using the PRISMA-P checklist.

Similarly, the intent of using PRISMA-P is to make reporting completed systematic reviews easier for authors. For example, once reviewers have described the methods in detail in their protocol, they may not need to repeat them when reporting the final systematic review results, particularly if there have been no protocol amendments. Providing explicit details about planned review methods in a protocol is essential for clarity, transparency, and future reproducibility, and is in line with emerging journal policies.18 Authors may also wish to develop a protocol to expand on information reported in PROSPERO. For journals that require a more detailed methods section in completed review articles, authors can easily cut and paste information already in their protocol, change the tense of the wording, and add any necessary documentation about protocol modifications or post-review changes where relevant (more likely in complex reviews such as network meta-analyses).

Protocols are important and provide readers with information about the rationale, question(s), and methods proposed by the systematic reviewers. They should always be made available in the public domain. However, for a variety of reasons, they are not always reported or published. Systematic reviewers may, for instance, be unsure of what information should be included in a review protocol—a problem PRISMA-P 2015 aims to solve.

We hope PRISMA-P will help increase the proportion of systematic review protocols being reported and published. Peer reviewers, editors, and other interested readers might also find protocols helpful in their assessment of completed reviews. Comparing protocols with completed reviews enables users to assess possible selective reporting and other possible deviations from the proposed systematic review plan. Investigators completing systematic reviews of systematic reviews (that is, overviews) might also find protocols useful for similar reasons.

We hope that journal editors will encourage authors submitting systematic review protocols for publication to comply with PRISMA-P. We hope funders and sponsors of systematic reviews will do likewise. We also invite readers to let us know what they think of PRISMA-P and ways we can improve it and keep it up to date.

Notes

Cite this as: BMJ 2015;350:g7647

Footnotes

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See:http://creativecommons.org/licenses/by-nc/4.0/

References

  1. ↵
    Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med2007;4:e78.
  2. ↵
    Dwan K, Gamble C, Williamson PR, Kirkham JJ. Systematic review of the empirical evidence of study publication bias and outcome reporting bias—An updated review. PloS One2013;8:e66844.
  3. ↵
    Dwan K, Altman DG, Cresswell L, Blundell M, Gamble CL, Williamson PR. Comparison of protocols and registry entries to published reports for randomised controlled trials. Cochrane Database Syst Rev2011;(1):MR000031.
  4. ↵
    Kirkham JJ, Dwan KM, Altman DG, Gamble C, Dodd S, Smyth R, et al. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ2010;340:c365.
  5. ↵
    Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan AW, Cronin E, et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One2008;3:e3081.
  6. ↵
    Chan AW, Krleza-Jeric K, Schmid I, Altman DG. Outcome reporting bias in randomized trials funded by the canadian institutes of health research. Can Med Assoc J2004;171:735.
  7. ↵
    Chan AW, Hróbjartsson A, Haahr MT, Gøtzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: Comparison of protocols to published articles. JAMA2004;291:2457-65.
  8. ↵
    Page MJ, McKenzie JE, Kirkham J, Dwan K, Kramer S, Green S, et al. Bias due to selective inclusion and reporting of outcomes and analyses in systematic reviews of randomised trials of healthcare interventions. Cochrane Library, 2014.
  9. ↵
    Page MJ, McKenzie JE, Forbes A. Many scenarios exist for selective inclusion and reporting of results in randomized trials and systematic reviews. J Clin Epidemiol2013;66:524-37.
  10. ↵
    Kirkham JJ, Altman DG, Williamson PR. Bias due to changes in specified outcomes during the systematic review process. PLoS One2010;5:e9810.
  11. ↵
    Booth A, Clarke M, Ghersi D, Moher D, Petticrew M, Stewart L. Establishing a minimum dataset for prospective registration of systematic reviews: An international consultation. PLoS One2011;6:e27319.
  12. ↵
    Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. BMJ2009;339:b2535.
  13. ↵
    Chan A, Tetzlaff JM, Gøtzsche PS, Altman DG, Mann H, Berlin JA, et al. SPIRIT 2013 explanation and elaboration: Guidance for protocols of clinical trials. BMJ2013;346:e7586.
  14. ↵
    Institute of Medicine. Finding what works in health care: standards for systematic reviews. National Academies Press, 2011.
  15. ↵
    Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev2015;4:1.
  16. ↵
    Moher D, Schulz KF, Simera I, Altman DG. Guidance for developers of health research reporting guidelines. PLoS Med2010;7:e1000217.
  17. ↵
    Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JP, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med2009;6:e1000100.
  18. ↵
    Enhancing reproducibility. Nature Methods2013;10:367.
  19. ↵
    Beller EM, Glasziou PP, Altman DG, Hopewell S, Bastian H, Chalmers I, et al. PRISMA for abstracts: reporting systematic reviews in journal and conference abstracts. PLoS Med2013;10:e1001419.
  20. ↵
    Mans CM, Reeve JC, Gasparini CA, Elkins MR. Postoperative outcomes following preoperative inspiratory muscle training in patients undergoing open cardiothoracic or upper abdominal surgery: protocol for a systematic review. Syst Rev2012;1:63.
  21. ↵
    Wen J, Ren Y, Wang L, Li Y, Liu Y, Zhou M, et al. The reporting quality of meta-analyses improves: a random sampling study. J Clin Epidemiol2008;61:770.
  22. ↵
    Moher D, Booth A, Stewart L. How to reduce unnecessary duplication: Use PROSPERO. BJOG 2014;121:784-6.
  23. ↵
    Montori VM, Wilczynski NL, Morgan D, Haynes RB. Optimal search strategies for retrieving systematic reviews from medline: analytical survey. BMJ 2005;330:68.
  24. ↵
    O’Connor AM, Auvermann BW, Higgins JP, Kirychuk SP, Sargeant JM, Von Essen SG, et al. The association between proximity to animal-feeding operations and community health: a protocol for updating a systematic review. Syst Rev2014;3:99.
  25. ↵
    Moher D, Tsertsvadze A. Systematic reviews: When is an update an update? Lancet2006;367:881-3.
  26. ↵
    Shojania KG, Sampson M, Ansari MT, Ji J, Doucette S, Moher D. How quickly do systematic reviews go out of date? A survival analysis. Ann Intern Med2007;147:224-33.
  27. ↵
    Moher D, Tsertsvadze A, Tricco AC, Eccles M, Grimshaw J, Sampson M, et al. A systematic review identified few methods and strategies describing when and how to update systematic reviews. J Clin Epidemiol2007;60:1095.
  28. ↵
    Garritty C, Tsertsvadze A, Tricco AC, Sampson M, Moher D. Updating systematic reviews: an international survey. PloS One2010;5:e9914.
  29. ↵
    Chung M, Newberry SJ, Ansari MT, Yu WW, Wu H, Lee J, et al. Two methods provide similar signals for the need to update systematic reviews. J Clin Epidemiol2012;65:660-8.
  30. ↵
    Ahmadzai N, Newberry SJ, Maglione MA, Tsertsvadze A, Ansari MT, Hempel S, et al. A surveillance system to assess the need for updating systematic reviews. Syst Rev2013;2:1-16.
  31. ↵
    Chalmers I, Enkin M, Keirse MJ. Preparing and updating systematic reviews of randomized controlled trials of health care. Milbank Q1993:411-37.
  32. ↵
    Cancelliere C, Cassidy JD, Côté P, Hincapié CA, Hartvigsen J, Carroll LJ, et al. Protocol for a systematic review of prognosis after mild traumatic brain injury: an update of the WHO collaborating centre task force findings. Syst Rev2012;1:17.
  33. ↵
    Stewart L, Moher D, Shekelle P. Why prospective registration of systematic reviews makes sense. Syst Rev2012;1:7.
  34. ↵
    Booth A, Clarke M, Ghersi D, Moher D, Petticrew M, Stewart L. An international registry of systematic-review protocols. Lancet2011;377:108-9.
  35. ↵
    Booth A, Clarke M, Dooley G, Ghersi D, Moher D, Petticrew M, et al. The nuts and bolts of PROSPERO: An international prospective register of systematic reviews. Syst Rev2012;1:2.
  36. ↵
    Silagy CA, Middleton P, Hopewell S. Publishing protocols of systematic reviews: comparing what was done to what was planned. JAMA2002;287:2831-4.
  37. ↵
    Siontis KC, Hernandez-Boussard T, Ioannidis JP. Overlapping meta-analyses on the same topic: survey of published studies. BMJ2013;347:f4501.
  38. ↵
    Moher D. The problem of duplicate systematic reviews. BMJ2013;347:f5040.
  39. ↵
    The PLoS ME. Best practice in systematic reviews: the importance of protocols and registration. PLoS Med2011;8:e1001009.
  40. ↵
    Chien PFW, Khan KS, Siassakos D. Registration of systematic reviews: PROSPERO. BJOG2012;119:903-5.
  41. ↵
    Bambra C, Hillier F, Moore H, Summerbell C. Tackling inequalities in obesity: a protocol for a systematic review of the effectiveness of public health interventions at reducing socioeconomic inequalities in obesity amongst children. Syst Rev2012;1:16.
  42. ↵
    Mowatt G, Shirran L, Grimshaw JM, Rennie D, Flanagin A, Yank V, et al. Prevalence of honorary and ghost authorship in cochrane reviews. JAMA2002;287:2769-71.
  43. ↵
    Rennie D, Yank V, Emanuel L. When authorship fails. JAMA1997;278:579-85.
  44. ↵
    Sismondo S. Ghost management: How much of the medical literature is shaped behind the scenes by the pharmaceutical industry? PLoS Med2007;4:e286.
  45. ↵
    Ross JS, Hill KP, Egilman DS, Krumholz HM. Guest authorship and ghostwriting in publications related to rofecoxib. JAMA2008;299:1800-12.
  46. ↵
    Smith R. Maintaining the integrity of the scientific record. BMJ2001;323:588.
  47. ↵
    Ensor J, Riley RD, Moore D, Bayliss S, Jowett S, Fitzmaurice DA. Protocol for a systematic review of prognostic models for the recurrence of venous thromboembolism (VTE) following treatment for a first unprovoked VTE. Syst Rev2013;2:91.
  48. ↵
    Smith R. Authorship is dying: long live contributorship. BMJ1997;315:696.
  49. ↵
    Uniform requirements for manuscripts submitted to biomedical journals: writing and editing for biomedical publication. 2010. www.ICMJE.org.
  50. ↵
  51. ↵
  52. ↵
    Getz KA, Zuckerman R, Cropp AB, Hindle AL, Krauss R, Kaitin KI. Measuring the incidence, causes, and repercussions of protocol amendments. Drug Inf J2011;45:265-75.
  53. ↵
    Higgins JPT, Green S, eds. Guide to the contents of a Cochrane protocol and review. In: The Cochrane handbook for systematic reviews of interventions. Wiley-Blackwell, 2011.
  54. ↵
    Young M, Stevens A, Porath-Waller A, Pirie T, Garritty C, Skidmore B, et al. Effectiveness of brief interventions as part of the screening, brief intervention and referral to treatment (SBIRT) model for reducing the non-medical use of psychoactive substances: a systematic review protocol. Syst Rev2012;1:22.
  55. ↵
    Lundh A, Sismondo S, Lexchin J, Busuioc OA, Bero L. Industry sponsorship and research outcome. Cochrane Database Syst Rev2012;(12):MR000033.
  56. ↵
    Bero L. Industry sponsorship and research outcome: a cochrane review. JAMA Intern Med2013;173:580-1.
  57. ↵
    Roseman M, Turner EH, Lexchin J, Coyne JC, Bero LA, Thombs BD. Reporting of conflicts of interest from drug trials in Cochrane reviews: cross sectional study. BMJ2012;345:e5155.
  58. ↵
    Jorgensen AW, Hilden J, Gotzsche PC. Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review. BMJ2006;333:782.
  59. ↵
    Yank V, Rennie D, Bero LA. Financial ties and concordance between results and conclusions in meta-analyses: retrospective cohort study. BMJ2007;335:1202-5.
  60. ↵
    Jorgensen AW, Maric KL, Tendal B, Faurschou A, Gotzsche PC. Industry-supported meta-analyses compared with meta-analyses with non-profit or no support: differences in methodological quality and conclusions. BMC Med Res Methodol2008;8:60.
  61. ↵
    Bes-Rastrollo M, Schulze MB, Ruiz-Canela M, Martinez-Gonzalez MA. Financial conflicts of interest and reporting bias regarding the association between sugar-sweetened beverages and weight gain: A systematic review of systematic reviews. PLoS Med2013;10:e1001578.
  62. ↵
    Rochon PA, Hoey J, Chan A, Ferris LE, Lexchin J, Kalkar SR, et al. Financial conflicts of interest checklist 2010 for clinical research studies. Open Med2010;4:e69.
  63. ↵
    Holden MA, Haywood KL, Potia TA, Gee M, McLean S. Recommendations for exercise adherence measures in musculoskeletal settings: a systematic review and consensus meeting (protocol). Syst Rev2014;3:1-6.
  64. ↵
  65. ↵
  66. ↵
    Hayden J, Cartwright J, Riley R, vanTulder M. Exercise therapy for chronic low back pain: Protocol for an individual participant data meta-analysis. Syst Rev2012;1:64.
  67. ↵
    Moseley AM, Elkins MR, Herbert RD, Maher CG, Sherrington C. Cochrane reviews used more rigorous methods than non-cochrane reviews: survey of systematic reviews in physiotherapy. J Clin Epidemiol2009;62:1021-30.
  68. ↵
    Psaty BM, Kronmal RA. Reporting mortality findings in trials of rofecoxib for alzheimer disease or cognitive impairment: A case study based on documents from rofecoxib litigation. JAMA2008;299:1813-7.
  69. ↵
    Cohen D. Rosiglitazone: What went wrong? BMJ2010;341:c4848.
  70. ↵
    Chung A, Backholer K, Wong E, Palermo C, Keating C, Peeters A. Trends in child and adolescent obesity prevalence according to socioeconomic position: Protocol for a systematic review. Syst Rev2014;3:1-4.
  71. ↵
    Moher D, Tsertsvadze A, Tricco AC, Eccles MG, J., Sampson M, Barrowman N. A systematic review identified few methods and strategies describing when and how to update systematic reviews. J Clin Epidemiol2007;60:1095.
  72. ↵
    Eipe N, Penning J, Ansari M, Yazdi F, Ahmadzai N. A protocol for a systematic review for perioperative pregabalin use. Syst Rev2012;1:40.
  73. ↵
    Honein-Abouhaidar GN, Kastner M, Vuong V, Perrier L, Rabeneck L, Tinmouth J, et al. Benefits and barriers to participation in colorectal cancer screening: A protocol for a systematic review and synthesis of qualitative studies. BMJ Open2014;4:e004508.
  74. ↵
    Counsell C. Formulating questions and locating primary studies for inclusion in systematic reviews. Ann Intern Med1997;127:380-7.
  75. ↵
    Whitlock EP, Lopez SA, Chang S, Helfand M, Eder M, Floyd N. AHRQ series paper 3: Identifying, selecting, and refining topics for comparative effectiveness systematic reviews: AHRQ and the effective health-care program. J Clin Epidemiol2010;63:491.
  76. ↵
    Hamel C, Stevens A, Singh K, Ansari MT, Myers E, Ziegler P, et al. Do sugar-sweetened beverages cause adverse health outcomes in adults? A systematic review protocol. Syst Rev2014;3:108.
  77. ↵
    Higgins JPT and Green S, eds. Cochrane handbook for systematic reviews of interventions. version 5.0.2. Cochrane Collaboration, 2009.
  78. ↵
    Guyatt GH, Oxman AD, Montori V, Vist G, Kunz R, Brozek J, et al. GRADE guidelines: 5. rating the quality of evidence-publication bias. J Clin Epidemiol2011;64:1277-82.
  79. ↵
    Adams D, Wu T, Yasui Y, Aung S, Vohra S. Systematic reviews of TCM trials: How does inclusion of Chinese trials affect outcome? J Evid Based Med2012;5:89-97.
  80. ↵
    Hopewell S, McDonald S, Clarke Mike J, Egger M. Grey literature in meta-analyses of randomized trials of health care interventions. John Wiley & Sons, 2007.
  81. ↵
    Van Enst WA, Scholten RJPM, Hooft L. Identification of additional trials in prospective trial registers for cochrane systematic reviews. PloS One2012;7:e42812.
  82. ↵
    Tricco A, Chit A, Hallett D, Soobiah C, Meier G, Chen M, et al. Effect of influenza vaccines against mismatched strains: A systematic review protocol. Syst Rev2012;1:35.
  83. ↵
    Horsley T, Dingwall O, Sampson M. Checking reference lists to find additional studies for systematic reviews. Cochrane Database Syst Rev2011;(8):MR000026.
  84. ↵
    Zhang L, Sampson M, McGowan J. Reporting of the role of the expert searcher in cochrane reviews. Evid Based Libr Inf Pract2006;1:3-16.
  85. ↵
    Lefebvre C, Manheimer E, Glanville J, Cochrane Information Retrieval Methods Group. Searching for studies. In: Higgins JPT, Green S, eds. Cochrane handbook for systematic reviews of interventions. 5.1.0 ed. Cochrane Collaboration, 2011.
  86. ↵
    Relevo R, Balshem H. Finding evidence for comparing medical interventions. In: Methods guide for effectiveness and comparative effectiveness reviews. AHRQ Publication No 10(12)-EHC063-EF. Agency for Healthcare Research and Quality, 2012.
  87. ↵
    Daboval T, Ward N, Sampson M, Kharrat A. Antenatal counseling about the perinatal care plan at the limit of viability: What do parents want? A systematic review protocol. (unpublished).
  88. ↵
    McGowan J, Sampson M, Lefebvre C. An evidence based checklist for the peer review of electronic search strategies (PRESS EBC). Evid Based Libr Inf Pract2010;5:149-54.
  89. ↵
    Deeks JJ, Altman DG. Inadequate reporting of controlled trials as short reports. Lancet1998;352:1908.
  90. ↵
    Iansavichene AE, Sampson M, McGowan J, Ajiferuke ISY. Should systematic reviewers search for randomized, controlled trials published as letters? Ann Intern Med2008;148:714-5.
  91. ↵
    Sampson M, McGowan J, Cogo E, Grimshaw J, Moher D, Lefebvre C. An evidence-based practice guideline for the peer review of electronic search strategies. J Clin Epidemiol2009;62:944-52.
  92. ↵
    Elamin MB, Flynn DN, Bassler D, Briel M, Alonso-Coello P, Karanicolas PJ, et al. Choice of data extraction tools for systematic reviews depends on resources and review complexity. J Clin Epidemiol2009;62:506-10.
  93. ↵
    Tramèr MR, Reynolds DJM, Moore RA, McQuay HJ. Impact of covert duplicate publication on meta-analysis: a case study. BMJ1997;315:635-40.
  94. ↵
    Von Elm E, Poglia G, Walder B, Tramer MR. Different patterns of duplicate publication. JAMA2004;291:974-80.
  95. ↵
    Gøtzsche P. Multiple publication of reports of drug trials. Eur J Clin Pharmacol1989;36:429-32.
  96. ↵
    Macdonald G, McCartan CJ. Centre-based early education interventions for improving school readiness. Cochrane Database Syst Rev2014;(1):CD010913.
  97. ↵
    Edwards P, Clarke M, DiGuiseppi C, Pratap S, Roberts I, Wentz R. Identification of randomized controlled trials in systematic reviews: Accuracy and reliability of screening records. Stat Med2002;21:1635-40.
  98. ↵
    Cooper H, Ribble RG. Influences on the outcome of literature searches for integrative research reviews. Sci Commun1989;10:179-201.
  99. ↵
    Busse JW, Ebrahim S, Connell G, Coomes EA, Bruno P, Malik K, et al. Systematic review and network meta-analysis of interventions for fibromyalgia: A protocol. Syst Rev2013;2:18.
  100. ↵
    Buscemi N, Hartling L, Vandermeer B, Tjosvold L, Klassen TP. Single data extraction generated more errors than double data extraction in systematic reviews. J Clin Epidemiol2006;59:697-703.
  101. ↵
    Horton J, Vandermeer B, Hartling L, Tjosvold L, Klassen TP, Buscemi N. Systematic review data extraction: Cross-sectional study showed that experience did not increase accuracy. J Clin Epidemiol2010;63:289-98.
  102. ↵
    Jones AP, Remmington T, Williamson PR, Ashby D, Smyth RL. High prevalence but low impact of data extraction and reporting errors were found in cochrane systematic reviews. J Clin Epidemiol2005;58:741-2.
  103. ↵
    Shadish WR, Brasil ICC, Illingworth DA, White KD, Galindo R, Nagler ED, et al. Using UnGraph to extract data from image files: Verification of reliability and validity. Behav Res Methods2009;41:177-83.
  104. ↵
    Stewart LA. Practical methodology of meta?analyses (overviews) using updated individual patient data. Stat Med2007;14:2057-79.
  105. ↵
    Clarke M, Hopewell S, Juszczak E, Eisinga A, Kjeldstrom M. Compression stockings for preventing deep vein thrombosis in airline passengers. Cochrane Database Syst Rev2006;(2):CD004002.
  106. ↵
    Glasziou P, Meats E, Heneghan C, Shepperd S. What is missing from descriptions of treatment in trials and reviews? BMJ2008;336:1472.
  107. ↵
    Schulz KF, Altman DG, Moher D. CONSORT 2010 statement: Updated guidelines for reporting parallel group randomised trials. BMC Med2010;8:18.
  108. ↵
    Higgins JPT, Deeks JJ. Selecting studies and collecting data. In: Higgins JPT, Green S, eds. The Cochrane handbook for systematic reviews of interventions. 5.1.0 ed. Cochrane Collaboration, 2011.
  109. ↵
    Errami M, Sun Z, Long TC, George AC, Garner HR. Deja vu: A database of highly similar citations in the scientific literature. Nucleic Acids Res2009;37(suppl 1):D921-4.
  110. ↵
    Reichenbach S, Rutjes AWS, Nüesch E, Trelle S, Jüni P. Arthroscopic lavage for osteoarthritis of the knee. Cochrane Library. 2008, doi:10.1002/14651858.CD007320.
  111. ↵
    Tully PJ, Baumeister H. Collaborative care for the treatment of comorbid depression and coronary heart disease: A systematic review and meta-analysis protocol. Syst Rev2014;3:127.
  112. ↵
    Dwan K, Gamble C, Kolamunnage-Dona R, Mohammed S, Powell C, Williamson P. Assessing the potential for outcome reporting bias in a review: A tutorial. Trials2010;11:52.
  113. ↵
    Leucht C, Huhn M, Leucht S. Amitriptyline versus placebo for major depressive disorder (protocol). Cochrane Database Syst Rev2011;(5):CD009138.
  114. ↵
    Helfand M, Balshem H. AHRQ series paper 2: Principles for developing guidance: AHRQ and the effective health-care program. J Clin Epidemiol2010;63:484.
  115. ↵
    Johns MW. A new method for measuring daytime sleepiness: The Epworth sleepiness scale. Sleep1991;14:540-5.
  116. ↵
    Johns MW. Reliability and factor analysis of the Epworth sleepiness scale. Sleep1992;15:376-81.
  117. ↵
    Selby JV, Beal AC, Frank L. The Patient-Centered Outcomes Research Institute (PCORI) national priorities for research and initial research agenda. JAMA2012;307:1583-4.
  118. ↵
    Methodology Committee of the Patient-Centered Outcomes Research Institute (PCORI). Methodological standards and patient-centeredness in comparative effectiveness research: The PCORI perspective. JAMA2012;307:1636-40.
  119. ↵
    Cordoba G, Schwartz L, Woloshin S, Bae H, Gotzsche PC. Definition, reporting, and interpretation of composite outcomes in clinical trials: systematic review. BMJ2010;341:c3920.
  120. ↵
    Page MJ, McKenzie JE, Forbes A. Many scenarios exist for selective inclusion and reporting of results in randomized trials and systematic reviews. J Clin Epidemiol2013;66:524-37.
  121. ↵
    Hosseini Araghi M, Chen YF, Jagielski A, Mannan Choudhury S, Banerjee D, Thomas GN, et al. Weight loss intervention through lifestyle modification or pharmacotherapy for obstructive sleep apnoea in adults (protocol). Cochrane Database Syst Rev2012;(12):CD010281.
  122. ↵
    Lawley CM, Lain SJ, Algert CS, Ford JB, Figtree GA, Roberts CL. Prosthetic heart valves in pregnancy: A systematic review and meta-analysis protocol. Syst Rev2014;3:8.
  123. ↵
    Higgins JPT, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ2011;343:d5928.
  124. ↵
    Viswanathan M, Ansari MT, Berkman ND, Chang S, Hartling L, McPheeters M, et al. Assessing the risk of bias of individual studies in systematic reviews of health care interventions. AHRQ Methods for Effective Health Care2012 Mar 08.
  125. ↵
    Hartling L, Hamm MP, Milne A, Vandermeer B, Santaguida PL, Ansari M, et al. Testing the risk of bias tool showed low reliability between individual reviewers and across consensus assessments of reviewer pairs. J Clin Epidemiol2013;66:973-81.
  126. ↵
    Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, et al. Assessing the quality of reports of randomized clinical trials: Is blinding necessary? Control Clin Trials1996;17:1-12.
  127. ↵
    Berlin JA. Does blinding of readers affect the results of meta-analyses? Lancet1997;350:185-6.
  128. ↵
    Moja LP, Telaro E, D’Amico R, Moschetti I, Coe L, Liberati A. Assessment of methodological quality of primary studies by systematic reviews: Results of the meta-quality cross sectional study. BMJ2005;330:1053.
  129. ↵
    Wood L, Egger M, Gluud LL, Schulz KF, Jüni P, Altman DG, et al. Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: Meta-epidemiological study. BMJ2008;336:601-5.
  130. ↵
    Savović J, Harris RJ, Wood L, Beynon R, Altman D, Als-Nielsen B, et al. Development of a combined database for meta-epidemiological research. Res Synth Methods2011;2:78.
  131. ↵
    Dechartres A, Boutron I, Trinquart L, Charles P, Ravaud P. Single-center trials show larger treatment effects than multicenter trials: Evidence from a meta-epidemiologic study. Ann Intern Med2011;155:39.
  132. ↵
    Odgaard-Jensen J, Vist G, Timmer A, Kunz R, Akl E, Schünemann H, et al. Randomised controlled trials as a safeguard against biased estimates of treatment effects. Cochrane Database Syst Rev2011;(4):MR000012.
  133. ↵
    Sanderson S, Tatt ID, Higgins JPT. Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: A systematic review and annotated bibliography. Int J Epidemiol2007;36:666-76.
  134. ↵
    Dechartres, A. Charles, P. Hopewell, S. Ravaud, P. Altman, D.G. Reviews assessing the quality or the reporting of randomized controlled trials are increasing over time but raised questions about how quality is assessed. J Clin Epidemiol2011;64:136-44.
  135. ↵
    Jüni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. JAMA1999;282:1054-60.
  136. ↵
    Ioannidis JPA, Haidich AB, Pappa M, Pantazis N, Kokori SI, Tektonidou MG, et al. Comparison of evidence of treatment effects in randomized and nonrandomized studies. JAMA2001;286:821-30.
  137. ↵
    Savovic J, Jones HE, Altman DG, Harris RJ, Juni P, Pildal J, et al. Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Ann Intern Med2012;157:429-38.
  138. ↵
    Hróbjartsson A, Thomsen ASS, Emanuelsson F, Tendal B, Hilden J, Boutron I, et al. Observer bias in randomised clinical trials with binary outcomes: Systematic review of trials with both blinded and non-blinded outcome assessors. BMJ2012;344:e1119.
  139. ↵
    Bell ML, Kenward MG, Fairclough DL, Horton NJ. Differential dropout and bias in randomised controlled trials: When it matters and when it may not. BMJ2013;346:e8668.
  140. ↵
    Guyatt GH, Briel M, Glasziou P, Bassler D, Montori VM. Problems of stopping trials early. BMJ2012;344:e3863.
  141. ↵
    Briel M, Bassler D, Wang AT, Guyatt GH, Montori VM. The dangers of stopping a trial too early. J Bone Joint Surg2012;94(suppl 1):56-60.
  142. ↵
    DeAngelis CD FP. Impugning the integrity of medical science: The adverse effects of industry influence. JAMA2008;299:1833-5.
  143. ↵
    Kahan BC, Morris TP. Analysis of multicentre trials with continuous outcomes: When and how should we account for centre effects? Stat Med2013;32:1136-49.
  144. ↵
    Shephard RJ. Ethics in exercise science research. Sports Med2002;32:169-83.
  145. ↵
    Fanelli D. How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLoS One2009;4:e5738.
  146. ↵
    Higgins J, Ramsay C, Reeves BC, Deeks JJ, Shea B, Valentine JC, et al. Issues relating to study design and risk of bias when including non-randomized studies in systematic reviews on the effects of interventions. Res Synth Methods2012.
  147. ↵
    Mao C, Yang ZY, Chung VCH, Qin Y, Tam W, Kwong JSW, et al. Tong-xin-luo capsule for patients with coronary heart disease after percutaneous coronary intervention. Cochrane Library2012.
  148. ↵
    Villar J, Mackey ME, Carroli G, Donner A. Meta-analyses in systematic reviews of randomized controlled trials in perinatal medicine: Comparison of fixed and random effects models. Stat Med2001;20:3635-47.
  149. ↵
    Hunter JE, Schmidt FL. Fixed effects vs. random effects meta-analysis models: implications for cumulative research knowledge. Int J Select Assess2002;8:275-92.
  150. ↵
    Schmidt FL, Oh IS, Hayes TL. Fixed- versus random-effects models in meta-analysis: model properties and an empirical comparison of differences in results. Br J Math Stat Psychol2009;62:97-128.
  151. ↵
    Higgins J, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ2003;327:557-60.
  152. ↵
    Higgins JPT, Deeks JJ, Altman DG. Special topics in statistics. In: Higgins JPT, Green S, eds. Cochrane handbook for systematic reviews of interventions. 5.1.0 ed. Cochrane Collaboration, 2011.
  153. ↵
    Stewart LA, Clarke M, Rovers M, Riley RD, Simmonds M, Stewart G, et al. PRISMA-IPD: an extension to the PRISMA statement guidance for reporting systematic review and meta-analysis of individual participant data. (submitted).
  154. ↵
    Thiessen PH, Barrowman N, Garg A. Imputing variance estimates do not alter the conclusions of a meta-analysis with continuous outcomes: A case study of changes in renal function after living kidney donation. J Clin Epidemiol2007;60:228.
  155. ↵
    Stevens JW. A note on dealing with missing standard errors in meta-analyses of continuous outcome measures in WinBUGS. Pharmaceut Stat2011;10:374-8.
  156. ↵
    Wiebe N, Vandermeer B, Platt RW, Klassen TP, Moher D, Barrowman NJ. A systematic review identifies a lack of standardization in methods for handling missing variance data. J Clin Epidemiol2006;59:342-53.
  157. ↵
    Furukawa TA, Barbui C, Cipriani A, Brambilla P, Watanabe N. Imputing missing standard deviations in meta-analyses can provide accurate results. J Clin Epidemiol2006;59:7.
  158. ↵
    Tierney JF, Stewart LA, Ghersi D, Burdett S, Sydes MR. Practical methods for incorporating summary time-to-event data into meta-analysis. Trials2007;8:16.
  159. ↵
    Wood L, Egger M, Gluud LL, Schulz KF, Jüni P, Altman DG, et al. Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ2008;336:601-5.
  160. ↵
    Whitehead PJ, Drummond AE, Walker MF, Parry RH. Interventions to reduce dependency in personal activities of daily living in community-dwelling adults who use homecare services: protocol for a systematic review. Syst Rev2013;2:1-7.
  161. ↵
    Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O, Peacock R. Storylines of research in diffusion of innovation: a meta-narrative approach to systematic review. Soc Sci Med2005;61:417-30.
  162. ↵
    Popay J, Roberts H, Sowden A, Petticrew M, Arai L, Rodgers M, et al. Guidance on the conduct of narrative synthesis in systematic reviews. a product from the ESRC methods programme. Version 1. ESRC, 2006.
  163. ↵
    Richards BL, Whittle SL, Buchbinder R. Neuromodulators for pain management in rheumatoid arthritis (protocol). Cochrane Database Syst Rev2011;(1):CD008921.
  164. ↵
    Goodman S, Dickersin K. Metabias: A challenge for comparative effectiveness research. Ann Intern Med2011;155:61-2.
  165. ↵
    Song F, Parekh S, Hooper L, Loke Y, Ryder J, Sutton A, et al. Dissemination and publication of research findings: an updated review of related biases. Prepress Projects, 2010.
  166. ↵
    Sterne JAC, Sutton AJ, Ioannidis JPA, Terrin N, Jones DR, Lau J, et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ2011;343:302.
  167. ↵
    Terrin N, Schmid CH, Lau J. In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias. J Clin Epidemiol2005;58:894-901.
  168. ↵
    Lau J, Ioannidis J, Terrin N, Schmid CH, Olkin I. The case of the misleading funnel plot. BMJ2006;333:597-600.
  169. ↵
    Parekh-Bhurke S, Kwok CS, Pang C, Hooper L, Loke YK, Ryder JJ, et al. Uptake of methods to deal with publication bias in systematic reviews has increased over time, but there is still much scope for improvement. J Clin Epidemiol2011;64:349-57.
  170. ↵
    Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R. Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med2008;358:252-60.
  171. ↵
    Turner EH, Knoepflmacher D, Shapley L. Publication bias in antipsychotic trials: An analysis of efficacy comparing the published literature to the us food and drug administration database. PLoS Med2012;9:e1001189.
  172. ↵
    Williamson P, Gamble C, Altman D, Hutton J. Outcome selection bias in meta-analysis. Stat Methods Med Res2005;14:515-24.
  173. ↵
    Chan AW, Altman DG. Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors. BMJ2005;330:753.
  174. ↵
    Williamson PR, Gamble C. Application and investigation of a bound for outcome reporting bias. Trials2007;8:9.
  175. ↵
    Tharyan P, Kirubakaran R, Jabez P. The use of trial protocols to assess risk of bias due to selective reporting in cochrane systematic reviews: a cross-sectional survey. 18th Cochrane Colloquium and 10th Campbell Colloquium; Oct 18-22; Keystone, Colorado; 2010.
  176. ↵
    Hart B, Lundh A, Bero L. Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses. BMJ2012;344:d7202.
  177. ↵
    Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol2011;64:383-94.
  178. ↵
    Guyatt G, Oxman AD, Sultan S, Brozek J, Glasziou P, Alonso-Coello P, et al. GRADE guidelines. 11: Making an overall rating of confidence in effect estimates for a single outcome and for all outcomes. J Clin Epidemiol2013;66:151-7.
  179. ↵
    Antman EM, Lau J, Kupelnick B, Mosteller F, Chalmers TC. A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts. treatments for myocardial infarction. JAMA1992;268:240-8.
  180. ↵
    Oxman AD, Guyatt GH. The science of reviewing research. Ann N Y Acad Sci1993;703:125.
  181. ↵
    Dickersin K. The existence of publication bias and risk factors for its occurrence. JAMA1990;263:1385-9.
  182. ↵
    Begg CB. Publication bias: A problem in interpreting medical data. J R Stat Soc A1988;151:419-63.
  183. ↵
    Hopewell S, Loudon K, Clarke MJ, Oxman AD, Dickersin K. Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database Syst Rev2009;(1):MR000006.
  184. ↵
    Dickersin K, Chalmers I. Recognizing, investigating and dealing with incomplete and biased reporting of clinical research: from Francis Bacon to the WHO. J R Soc Med2011;104:532-8.
  185. ↵
    Dickersin K. Publication bias: recognizing the problem, understanding its origins and scope, and preventing harm. In: Rothstein HR, Sutton AJ, Borenstein M, eds. Publication bias in meta-analysis—Prevention, assessment and adjustments. John Wiley & Sons, 2005:11-33.
  186. ↵
    Williamson PR, Gamble C, Altman DG, Hutton JL. Outcome selection bias in meta-analysis. Stat Methods Med Res2005;14:515-24.

View Abstract