Real-world evidence: How pragmatic are randomized controlled trials labeled as pragmatic? (original) (raw)

Abstract

Introduction

Pragmatic randomized controlled trials (RCTs) mimic usual clinical practice and they are critical to inform decision-making by patients, clinicians and policy-makers in real-world settings. Pragmatic RCTs assess effectiveness of available medicines, while explanatory RCTs assess efficacy of investigational medicines. Explanatory and pragmatic are the extremes of a continuum. This debate article seeks to evaluate and provide recommendation on how to characterize pragmatic RCTs in light of the current landscape of RCTs. It is supported by findings from a PubMed search conducted in August 2017, which retrieved 615 RCTs self-labeled in their titles as “pragmatic” or “naturalistic”. We focused on 89 of these trials that assessed medicines (drugs or biologics).

Discussion

36% of these 89 trials were placebo-controlled, performed before licensing of the medicine, or done in a single-center. In our opinion, such RCTs overtly deviate from usual care and pragmatism. It follows, that the use of the term ‘pragmatic’ to describe them, conveys a misleading message to patients and clinicians. Furthermore, many other trials among the 615 coined as ‘pragmatic’ and assessing other types of intervention are plausibly not very pragmatic; however, this is impossible for a reader to tell without access to the full protocol and insider knowledge of the trial conduct. The degree of pragmatism should be evaluated by the trial investigators themselves using the PRECIS-2 tool, a tool that comprises 9 domains, each scored from 1 (very explanatory) to 5 (very pragmatic).

Conclusions

To allow for a more appropriate characterization of the degree of pragmatism in clinical research, submissions of RCTs to funders, research ethics committees and to peer-reviewed journals should include a PRECIS-2 tool assessment done by the trial investigators. Clarity and accuracy on the extent to which a RCT is pragmatic will help understand how much it is relevant to real-world practice.

View this article's peer review reports

Similar content being viewed by others

Background

There is an increasing interest for obtaining evidence on the relative benefits and harms of interventions in real-world, every day circumstances. Real-world evidence can be obtained with observational data capturing routine care through comparative effectiveness research, such as, among others, large cohort studies, registry studies or retrospective studies on databases [[1](/article/10.1186/s12916-018-1038-2#ref-CR1 "Institute of Medicine. Learning what works: infrastructure required for comparative effectiveness research. Workshop summary. Washington, DC: The National Academies Press; 2011. https://www.nap.edu/catalog/12214/learning-what-works-infrastructure-required-for-comparative-effectiveness-research-workshop

                    ")\]; however, non-randomized information has major limitations and biases \[[2](#ref-CR2 "Hemkens LG, Contopoulos-Ioannidis DG, Ioannidis JP. Current use of routinely collected health data to complement randomized controlled trials: a meta-epidemiological survey. CMAJ Open. 2016;4:E132-40."),[3](#ref-CR3 "Hemkens LG, Contopoulos-Ioannidis DG, Ioannidis JP. Routinely collected data and comparative effectiveness evidence: promises and limitations. CMAJ. 2016;188:E158–64."),[4](/article/10.1186/s12916-018-1038-2#ref-CR4 "Hemkens LG, Contopoulos-Ioannidis DG, Ioannidis JP. Agreement of treatment effects for mortality from routinely collected data and subsequent randomized trials: meta-epidemiological survey. BMJ. 2016;352:i493.")\]. Because, pragmatic (or naturalistic) randomized controlled trials (RCTs) \[[5](#ref-CR5 "Ali J, Andrews JE, Somkin CP, Rabinovich CE. Harms, benefits, and the nature of interventions in pragmatic clinical trials. Clin Trials. 2015;12:467–75."),[6](#ref-CR6 "Ford I, Norrie J. Pragmatic trials. N Engl J Med. 2016;37:454–63."),[7](/article/10.1186/s12916-018-1038-2#ref-CR7 "Campbell MK, Weijer C, Goldstein CE, Edwards SJL. Do doctors have a duty to take part in pragmatic randomised trials? BMJ. 2017;357:j2817.")\] can remove the biases due to lack of randomization and hopefully still provide evidence that closely captures routine care, they have become very attractive. The term “pragmatic” for RCTs was introduced half a century ago \[[8](/article/10.1186/s12916-018-1038-2#ref-CR8 "Schwartz D, Lellouch J. Explanatory and pragmatic attitudes in therapeutical trials. J Chronic Dis. 1967;20:637–48.")\]. In contrast to “explanatory” RCTs that test hypotheses on whether the intervention causes an outcome of interest in ideal circumstances, “pragmatic” RCTs aim to provide information on the relative merits of real-world clinical alternatives in routine care. A critical aim of an explanatory RCT is to ensure internal validity (prevention of bias); conversely, a pragmatic RCT focuses on maximizing external validity (generalizability of the results to many real-world settings), but should try to preserve as much internal validity as possible.

When assessing new medicines (drugs or biologics) or new indications prior licensing, the typical RCT is highly explanatory (double-blind, placebo-controlled) [9]. Conversely, the typical paradigm of a real-world, comparative effectiveness medicines RCT is highly pragmatic, and compares the effectiveness of two commercially available medicines that are prescribed in routine care but have not been previously compared to each other [10]. Pragmatic medicine RCTs help to inform decisions by end users of information such as clinicians and patients, and by decision makers such as hospital, insurance and other policy makers. They are also used by industry in their price and reimbursement discussions with regulators. In an unsustainable health-cost environment, Health Technology Assessment agencies and Managed Care Organizations want to have real-world evidence on comparative effectiveness of available interventions in clinical practice to inform their decisions. To this end, pragmatic RCTs could play a critical role in defining which interventions should be recommended or prioritized.

Trials on regulated interventions (medicines, devices) have to be conducted following strict regulatory requirements that, almost invariably, prevent pre-approval trials to be pragmatic in nature. The need of medicines trials to be conducted following Good Clinical Practice guidelines implies that many trial features -from lengthy informed consent documents to providing the experimental medicine in non-commercial packages with the warning ‘investigational drug-for clinical trials use only’ - invariably disrupts any intention to mimic normal practice. Furthermore, complexity of phase 3 trials has increased over time, thus moving away from pragmatism: between 2001 and 2005 and 2010–2015, phase 3 trials have increased the mean number of planned visits, number of distinct procedures, and total number of procedures performed per trial by 27%, 59%, and 70%, respectively [11]. Conversely, a high degree of pragmatism is clearly an option for post-approval trials.

For trials on non-regulated interventions (e.g., surgery, physiotherapy, behavior), pragmatism is possible in both early and late development trials. However, it is very difficult for readers to appraise how close to usual clinical practice authors conducted their research in their own settings. To avoid speculating, we have based the argument of this paper on medicines trials only.

In this article we first set the scene by describing what are the features that define a pragmatic RCT and conducted a search on PubMed to know how prevalent is this type of RCTs. Second, we qualitatively discuss the retrieved papers assessing medicine pragmatic RCTs, and describe several trial features that prevent them from being labeled as pragmatic and propose how PRECIS-2 tool should be used by investigators to label trials as pragmatic in both submissions of protocols to research ethics committees and manuscripts to journals. Finally, we argue that both investigators and editors should use PRECIS-2 tool as a common, transparent and standardized method to appropriately label RCTs as pragmatic.

Discussion

Self-labeling randomized controlled trials as pragmatic

Currently, it is widely accepted that explanatory and pragmatic are the extremes of a continuum [12]. Many RCTs have both pragmatic and explanatory features. The issue is to know how pragmatic is a given trial to deserve to be named as such, especially when there are many investigators labeling their RCTs as pragmatic [[13](/article/10.1186/s12916-018-1038-2#ref-CR13 "Dal-Re R. Could phase 3 medicine trials be tagged as pragmatic? A case study: The Salford COPD trial. J Eval Clin Pract. 2017:1–4. https://doi.org/10.1111/jep.12796

               Accessed 8 December 2017"), [14](/article/10.1186/s12916-018-1038-2#ref-CR14 "Sox HC, Lewis RJ. Pragmatic trials. Practical answers to ‘real world’ questions. JAMA. 2016;316:1205–6.")\]. As appreciation of the value of real-world evidence becomes more widespread, labeling a RCT as “pragmatic” is almost a badge of honor. Thus, we suspect that many RCTs currently self-labeled as pragmatic are not necessarily pragmatic enough.

A genuinely pragmatic RCT should fulfill at least two fundamental features. First, its conduct should resemble usual clinical practice. Second, the results should be applicable to multiple other settings, not only the one where the trial was conducted. Consequently, in principle, pragmatic RCTs of medicines should assess already marketed medicines (rather than those still in clinical development before licensing) and should be done in several sites providing care to heterogeneous populations. Some investigators [[15](/article/10.1186/s12916-018-1038-2#ref-CR15 "Loudon K. Explanatory trials versus pragmatic trials. Rapid responses. 18 November 2014 http:// www.bmj.com/content/349/bmj.g6694/rapid-responses

                    ")\] have argued that blinding can offer safeguards on internal validity in trials that otherwise have pragmatic intentions. However, pragmatism is heavily compromised by blinding. When RCTs compare different medicines head-to-head, using multiple placebos for blinding is a substantial deviation from usual clinical practice. Taking two masked medicines, one active and one placebo, is a very different patient experience than having to take only one medicine. Likewise, RCTs comparing a single active medicine versus a single placebo can hardly be pragmatic. The patient’s uncertainty about whether he/she will receive the active medicine or not would affect its motivation to participate and may also affect the therapeutic response compared with real life. Furthermore, the patient instead of going to his/her usual pharmacy to acquire (with or without co-payment or full-payment) the drug, would typically go to a designated pharmacy where he/she will be given (free of charge) assigned packages of drug or placebo. All this could produce the Hawthorne effect in many participants. Pragmatic trials should avoid blinding with the exception of using blinded assessors of outcomes, whenever possible \[[16](/article/10.1186/s12916-018-1038-2#ref-CR16 "Welsing PM, Oude Rengerink K, Collier S, Eckert L, van Smeden M, Ciaglia A, et al. Series: Pragmatic trials and real world evidence: Paper 6. Outcome measures in the real world. J Clin Epidemiol. 2017;90:99–107.")\].

Illustration of the prevalence of pragmatic medicine randomized clinical trials

To assess the prevalence of pragmatic RCTs we conducted a PubMed search on August 8th, 2017 with the aim to capture articles using the terms ‘pragmatic’ or ‘naturalistic in their titles. Trials that only used the terms in their full text were not captured. We considered that trials that only used these terms in their full text are not profiting that prominently from their claims to pragmatism unlike trials using these terms in a highly visible manner in their titles. Abstracts were screened to identify medicines RCTs, i.e. trials assessing a medicine (drug, biologic) in at least one arm of the trial. We then examined the full text when any relevant information was not available in the abstract. Articles reporting results and those describing the protocol or design of the RCT were included. When having one or more articles for one RCT (e.g., one describing the protocol and another describing the results; or one describing the results and another conducting an economic analysis) we have counted only one trial. We found 615 RCTs, from 1977 to 2017, self-labeled as pragmatic in their titles. They have recently increased geometrically with 58% (354/615) of articles published in the last 3 years and a half (2014–2017). Only 89 (16%) of the 615 trials were on medicines. Five of these 89 trials addressed investigational medicines before licensing. Another 16 trials were single-center. Four trials used multiple placebos for blinding and another 14 trials used a single placebo. Overall, 32 of the 89 RCTs (36%) were pre-licensing, single-center, or placebo-controlled: in these circumstances, they cannot be appropriately termed ‘pragmatic’.

Standardizing appropriate use of the term pragmatic: the PRECIS-2 tool

Jarow et al. [17] have recently described the FDA approach to real-world data and its use in regulatory decision making. Real-world evidence is defined by the data source and degree of pragmatism. The data source should be routine clinical care while the study design and conduct should have a high degree of pragmatism. Medicine RCTs can provide real-world evidence, if their execution reflect use in clinical practice [17]. We are dealing with a continuum and there are tools available to help investigators to assess the degree of pragmatism of their RCT [12, 18, 19]. The PRECIS-2 [12] is the most widely-known tool.

As of August 2017, the PRECIS-2 website [[20](/article/10.1186/s12916-018-1038-2#ref-CR20 "PRECIS-2 https://www.precis-2.org/

               Accessed 8 December 2017.")\] had 349 users, with an increased traffic due to the new NIH website devoted to pragmatic RCTs \[[21](/article/10.1186/s12916-018-1038-2#ref-CR21 "NIH. Rethinking Clinical Trials: A Living Textbook of Pragmatic Clinical Trials 
                http://www.rethinkingclinicaltrials.org
                
              / Accessed 8 December 2017.")\] (K. Loudon, personal communication). The PRECIS-2 tool comprises 9 scored domains (Table [1](/article/10.1186/s12916-018-1038-2#Tab1)) addressing the most important RCT features. The scoring should be done prospectively, i.e. before a trial starts \[[12](/article/10.1186/s12916-018-1038-2#ref-CR12 "Loudon K, Treweek S, Sullivan F, Donnan P, Thorpe KE, Zwarenstein M. The PRECIS‐2 tool: designing trials that are fit for purpose. BMJ. 2015;350:h2147.")\]. However, the PRECIS-2 tool can also retrospectively assess the degree of pragmatism of a RCT, after its conduct \[[22](/article/10.1186/s12916-018-1038-2#ref-CR22 "Johnson KE, Neta G, Dember LM, Coronado GD, Suls J, Chambers DA, et al. Use of PRECIS ratings in the National Institutes of Health (NIH) Health Care Systems Research Collaboratory. Trials. 2016;17:32."), [23](/article/10.1186/s12916-018-1038-2#ref-CR23 "Luoma KA, Leavitt IM, Marrs JC, Nederveld AL, Regensteiner JG, Dunn AL, et al. How can clinical practices pragmatically increase physical activity for patients with type 2 diabetes? A systematic review. Transl Behav Med. 2017;7:751–72.")\]. For an outside reader of the trial report, this can be done reliably only if detailed and accurate information is published on all of the 9 domains. Unfortunately, sufficient details are almost never available. Interestingly, it has been shown that scores in the 9 domains of the PRECIS-2 tool could diverge between the theory (what investigators intended at the protocol development stage) and the practice (what actually happened once the RCT was implemented) \[[22](/article/10.1186/s12916-018-1038-2#ref-CR22 "Johnson KE, Neta G, Dember LM, Coronado GD, Suls J, Chambers DA, et al. Use of PRECIS ratings in the National Institutes of Health (NIH) Health Care Systems Research Collaboratory. Trials. 2016;17:32.")\]. Protocol or logistic modifications during the conduct of a trial can change its degree of pragmatism.

Table 1 PRECIS-2 tool nine domains and scoring method [12]

Full size table

Clinical trials features that prevent trials from being pragmatic

As mentioned above, more than one in three (36%) trials on medicines that were labeled as pragmatic in their titles were placebo-controlled, pre-licensing, or single-center.

As clarified in the CONSORT extension for pragmatic trials [24], it is practically impossible for a pragmatic trial to be blinded: “Belief (or disbelief) in the intervention, extra enthusiasm and effort (or less), and optimism (or pessimism) in the self-assessment of outcomes may thus add to (or detract from) the effects of an intervention”- components which are part of the treatment effect in usual care. Blinding disrupts these components that differentiate effectiveness (the goal of pragmatic trials) from efficacy (the goal of explanatory trials). For placebo-controlled RCTs the ‘recruitment’, ‘flexibility-delivery’ and ‘flexibility-adherence’ domain scores in the PRECIS-2 tool (Table 1) may be 1 or close to the explanatory extreme. Use of placebos is an obvious deviation from the real world and only a few patients would choose to be recruited in a trial with such an artificial treatment experience where they don’t know what they are receiving for treatment. However, this has not prevented investigators labeling placebo-controlled trials as pragmatic [25,26,27].

Similarly, RCTs on medicines before they are licensed or assessing a new indication or dosage form could hardly be pragmatic, since they have to comply with clinical trials regulations that have no resemblance to their subsequent application in routine care. Such feature would affect the ‘recruitment’, ‘organisation’, ‘flexibility: delivery’, ‘flexibility: adherence’ and ‘follow-up’ PRECIS-2 domains resulting into scores of 1 or close to the explanatory extreme. However, both private [[13](/article/10.1186/s12916-018-1038-2#ref-CR13 "Dal-Re R. Could phase 3 medicine trials be tagged as pragmatic? A case study: The Salford COPD trial. J Eval Clin Pract. 2017:1–4. https://doi.org/10.1111/jep.12796

               Accessed 8 December 2017")\] and public \[[28](/article/10.1186/s12916-018-1038-2#ref-CR28 "Currow D, Watts GJ, Johnson M, McDonald CF, Miners JO, Somogyi AA, et al. A pragmatic, phase III, multisite, double-blind, placebo-controlled, parallel-arm, dose increment randomised trial of regular, low-dose extended-release morphine for chronic breathlessness: Breathlessness, Exertion And Morphine Sulfate (BEAMS) study protocol. BMJ Open. 2017;7:e018100.")\] sponsors are expanding the use of the term pragmatic to include RCTs conducted before licensing with open-label \[[13](/article/10.1186/s12916-018-1038-2#ref-CR13 "Dal-Re R. Could phase 3 medicine trials be tagged as pragmatic? A case study: The Salford COPD trial. J Eval Clin Pract. 2017:1–4. 
                https://doi.org/10.1111/jep.12796
                
               Accessed 8 December 2017")\] and even double-blind, placebo-controlled designs \[[28](/article/10.1186/s12916-018-1038-2#ref-CR28 "Currow D, Watts GJ, Johnson M, McDonald CF, Miners JO, Somogyi AA, et al. A pragmatic, phase III, multisite, double-blind, placebo-controlled, parallel-arm, dose increment randomised trial of regular, low-dose extended-release morphine for chronic breathlessness: Breathlessness, Exertion And Morphine Sulfate (BEAMS) study protocol. BMJ Open. 2017;7:e018100.")\].

Finally, acknowledging that this is a debatable point, single-center RCTs could almost never be pragmatic. It is almost impossible to have much certainty that the results obtained in one site are generalizable to other centers and settings; the ‘setting’ domain score will be 1 or close to the explanatory extreme. Yet, single-center RCTs are self-labeled and published as being pragmatic [29, 30].

Trials with features that defy pragmatism have been labeled as pragmatic in all types of journal, including major general medical journals such as BMJ [26, 31] and Annals of Internal Medicine [25]. These cases exemplify how the use of the term “pragmatic” needs better standardization.

As mentioned above, we focus here on pragmatic trials on medicines only. However, similar considerations can apply to trials of other types of intervention. The mere requirement to participate in a controlled clinical experiment already poses distance from everyday life experiences of interventions such as, for instance, cognitive behavior, diet, exercise or acupuncture. Some of these trials are probably not genuinely pragmatic, but it is difficult to judge without in-depth knowledge of their exact conduct and context. However, many trials self-labeled as pragmatic and conducted with interventions such as the ones mentioned above could have been truly pragmatic as opposed to the ones conducted with regulated interventions (i.e., medicines and devices). There are fewer impediments to achieve high degree of pragmatism for trials of non-regulated interventions and these trials can closely mimic real-world approaches to recruitment, flexibility (adherence), follow-up and other important aspects of pragmatism without any regulatory obligations.

Publication of protocols and results of pragmatic randomized clinical trials

Ideally, the scores of all 9 domains of a pragmatic RCT should be on or close to the pragmatic extreme (scores ≥4), in order for this trial to be labeled as pragmatic. However, it is suggested that trials with scores ≥4 in 4–5 domains could be labeled as pragmatic provided the scores of the remaining domains are 3. Several trials fulfill this requirement. Some examples [32,33,34] of retrospectively assessed RCTs by authors are posted on the PRECIS-2 homepage [[20](/article/10.1186/s12916-018-1038-2#ref-CR20 "PRECIS-2 https://www.precis-2.org/

               Accessed 8 December 2017.")\]. Conversely, in the PRECIS-2 homepage \[[20](/article/10.1186/s12916-018-1038-2#ref-CR20 "PRECIS-2 
                https://www.precis-2.org/
                
               Accessed 8 December 2017.")\], there are also trials with several domain scores close to the pragmatic extreme, but also one or more domain scores close to the explanatory extreme \[[35](#ref-CR35 "Bots AF. Kastelein JJ; Discovery Netherlands Investigators. Achieving lipid goals in real life: the Dutch DISCOVERY study. Int J Clin Pract. 2005;59:1387–94."),[36](#ref-CR36 "Topol EJ, Moliterno DJ, Herrmann HC, Powers ER, Grines CL, Cohen DJ, et al. Comparison of two platelet glycoprotein IIb/IIIa inhibitors, tirofiban and abciximab, for the prevention of ischemic events with percutaneous coronary revascularization. N Engl J Med. 2001;344:1888–94."),[37](/article/10.1186/s12916-018-1038-2#ref-CR37 "Piller LB, Baraniuk S, Simpson LM, Cushman WC, Massie BM, Einhorn PT, et al. Long-term follow-up of participants with heart failure in the antihypertensive and lipid-lowering treatment to prevent heart attack trial (ALLHAT). Circulation. 2011;124:1811–8.")\]. Calling these latter trials ‘pragmatic’ can be misleading. However, there is one type of exception: highly pragmatic trials where the intervention is how care is organized (and hence the ‘organisation’ domain score will be in the explanatory extreme \[[12](/article/10.1186/s12916-018-1038-2#ref-CR12 "Loudon K, Treweek S, Sullivan F, Donnan P, Thorpe KE, Zwarenstein M. The PRECIS‐2 tool: designing trials that are fit for purpose. BMJ. 2015;350:h2147.")\]), could be labeled as pragmatic since this is just an explicit feature of the intervention.

The assessment of pragmatism by PRECIS-2 tool (i.e., the score obtained in each of the 9 domains and reasons supporting those scores) should be publicly disclosed. When submitting the manuscript of an RCT labeled as pragmatic, authors should submit their PRECIS-2 tool assessment as supplemental information, allowing for reviewers and editors to appraise the degree of pragmatism of the RCT. The assessment should be honest enough to provide information not commonly reported in manuscripts, such as, for instance, the nature and extent of participants’ information sheet. This type of information should be provided to support adjudicated scores. Otherwise, currently a reader cannot reliably appraise a trial on the degree of pragmatism without insider information.

The final PRECIS-2 tool assessment agreed between authors and journal editor should be published to inform readers of the reasons supporting the use of ‘pragmatic’ to describe the RCT and, consequently, whether the trial is gathering real-world evidence. If there are no sound reasons to label the RCT as pragmatic, the authors should avoid using the terms real-world evidence, effectiveness and usual clinical practice when referring to the design, conduct and results obtained.

We also propose a wider adoption of the PRECIS-2 tool at the protocol development and grant application stages. The scores of the 9 domains (and the reasons supporting them) should be included as part of the trial protocol to inform research ethics committees on the appropriateness of using the term ‘pragmatic’. Similarly, it should be included in the published protocols of trials that claim to be pragmatic. There are already several examples of published trial protocols that include the scores of the 9 domains [38,39,40]. Finally, the 9 domain scores should be included on clinical trials registries –such as, for instance, clinicaltrials.gov– as part of the trial description.

Conclusions

Given the increasing value of real-world evidence, it is important to use the term “pragmatic” judiciously. As mentioned above, a homepage is available with information on how to use the PRECIS-2 tool, a toolkit, examples, podcasts and webinars [[20](/article/10.1186/s12916-018-1038-2#ref-CR20 "PRECIS-2 https://www.precis-2.org/

               Accessed 8 December 2017.")\]. Although the PRECIS-2 tool has shown to have good interrater reliability and reasonable discriminant validity \[[41](/article/10.1186/s12916-018-1038-2#ref-CR41 "Loudon K, Zwarenstein M, Sullivan FM, Donnan PT, Gágyor I, HJSM H, et al. The PRECIS-2 tool has good interrater reliability and modest discriminant validity. J Clin Epidemiol. 2017:113–21.")\], its use is not free of issues \[[42](/article/10.1186/s12916-018-1038-2#ref-CR42 "Dekkers OM, Bossuyt PM, Vandenbroucke JP. How trial results are intended to be used: is PRECIS-2 a step forward? J Clin Epidemiol. 2017;84:25–6.")\]. We should expect that future enhancements on this tool should be provided mainly by PRECIS-2 authors. Nevertheless, until a better tool is developed, its use in the editorial process should help investigators to better label RCTs as pragmatic. In the current situation, non-standardized subjective and non-transparent assessments can result in incorrectly labelling many RCTs as pragmatic. We may also introduce the PRECIS-2 tool scores in the CONSORT guidelines \[[24](/article/10.1186/s12916-018-1038-2#ref-CR24 "Zwarenstein M, Treweek S, Gagnier JJ, Altman DG, Tunis S, Haynes B, et al. Improving the reporting of pragmatic trials: an extension of the CONSORT statement. BMJ. 2008;337:a2390.")\].

We do believe that journals, research ethics committees, Health Technology Assessment agencies, regulators, and funders may help ensure that the use of the term ‘pragmatic’ actually conveys the correct message of describing a RCT or a protocol thereof with a high degree of pragmatism. Appropriate use of the term will allow us to know how much real-world evidence we really have available from RCTs.

Abbreviations

RCT (s):

randomized controlled trial (s)

References

  1. Institute of Medicine. Learning what works: infrastructure required for comparative effectiveness research. Workshop summary. Washington, DC: The National Academies Press; 2011. https://www.nap.edu/catalog/12214/learning-what-works-infrastructure-required-for-comparative-effectiveness-research-workshop
    Google Scholar
  2. Hemkens LG, Contopoulos-Ioannidis DG, Ioannidis JP. Current use of routinely collected health data to complement randomized controlled trials: a meta-epidemiological survey. CMAJ Open. 2016;4:E132-40.
    Article PubMed Google Scholar
  3. Hemkens LG, Contopoulos-Ioannidis DG, Ioannidis JP. Routinely collected data and comparative effectiveness evidence: promises and limitations. CMAJ. 2016;188:E158–64.
    Article PubMed PubMed Central Google Scholar
  4. Hemkens LG, Contopoulos-Ioannidis DG, Ioannidis JP. Agreement of treatment effects for mortality from routinely collected data and subsequent randomized trials: meta-epidemiological survey. BMJ. 2016;352:i493.
    Article PubMed PubMed Central Google Scholar
  5. Ali J, Andrews JE, Somkin CP, Rabinovich CE. Harms, benefits, and the nature of interventions in pragmatic clinical trials. Clin Trials. 2015;12:467–75.
    Article PubMed PubMed Central Google Scholar
  6. Ford I, Norrie J. Pragmatic trials. N Engl J Med. 2016;37:454–63.
    Article Google Scholar
  7. Campbell MK, Weijer C, Goldstein CE, Edwards SJL. Do doctors have a duty to take part in pragmatic randomised trials? BMJ. 2017;357:j2817.
    Article PubMed Google Scholar
  8. Schwartz D, Lellouch J. Explanatory and pragmatic attitudes in therapeutical trials. J Chronic Dis. 1967;20:637–48.
    Article CAS PubMed Google Scholar
  9. Nash P, Kirkham B, Okada M, Rahman P, Combe B, Burmester GR, et al. Ixekizumab for the treatment of patients with active psoriatic arthritis and an inadequate response to tumour necrosis factor inhibitors: results from the 24-week randomised, double-blind, placebo-controlled period of the SPIRIT-P2 phase 3 trial. Lancet. 2017;389:2317–27.
    Article CAS PubMed Google Scholar
  10. Staa TP, Goldacre B, Gulliford M, Cassell J, Pirmohamed M, Taweel A, et al. Pragmatic randomised trials using routine electronic health records: putting them to the test. BMJ. 2012;344:e55.
    Article PubMed PubMed Central Google Scholar
  11. Getz KA, Campo RA. Trends in clinical trial design complexity. Nat Rev Drug Discov. 2017;16:307.
    Article CAS PubMed Google Scholar
  12. Loudon K, Treweek S, Sullivan F, Donnan P, Thorpe KE, Zwarenstein M. The PRECIS‐2 tool: designing trials that are fit for purpose. BMJ. 2015;350:h2147.
    Article PubMed Google Scholar
  13. Dal-Re R. Could phase 3 medicine trials be tagged as pragmatic? A case study: The Salford COPD trial. J Eval Clin Pract. 2017:1–4. https://doi.org/10.1111/jep.12796 Accessed 8 December 2017
  14. Sox HC, Lewis RJ. Pragmatic trials. Practical answers to ‘real world’ questions. JAMA. 2016;316:1205–6.
    Article PubMed Google Scholar
  15. Loudon K. Explanatory trials versus pragmatic trials. Rapid responses. 18 November 2014 http://www.bmj.com/content/349/bmj.g6694/rapid-responses
  16. Welsing PM, Oude Rengerink K, Collier S, Eckert L, van Smeden M, Ciaglia A, et al. Series: Pragmatic trials and real world evidence: Paper 6. Outcome measures in the real world. J Clin Epidemiol. 2017;90:99–107.
    Article PubMed Google Scholar
  17. Jarow JP, LaVange L, Woodcock J. Multidimensional evidence generation and FDA decision making. Defining and using ‘real-world’ data. JAMA. 2017;318:703–4.
    Article PubMed Google Scholar
  18. Tosh G, Soares-Weiser K, Adams CE. Pragmatic vs explanatory trials: the Pragmascope tool to help measure differences in protocols of mental health randomized controlled trials. Dialogues Clin Neurosci. 2011;13:209–15.
    PubMed PubMed Central Google Scholar
  19. Alphs LD, Bossie CA. ASPECT-R—A Tool to Rate the Pragmatic and Explanatory Characteristics of a Clinical Trial Design. Innov Clin Neurosci. 2016;13:15–26.
    PubMed PubMed Central Google Scholar
  20. PRECIS-2 https://www.precis-2.org/ Accessed 8 December 2017.
  21. NIH. Rethinking Clinical Trials: A Living Textbook of Pragmatic Clinical Trials http://www.rethinkingclinicaltrials.org/ Accessed 8 December 2017.
  22. Johnson KE, Neta G, Dember LM, Coronado GD, Suls J, Chambers DA, et al. Use of PRECIS ratings in the National Institutes of Health (NIH) Health Care Systems Research Collaboratory. Trials. 2016;17:32.
    Article PubMed PubMed Central Google Scholar
  23. Luoma KA, Leavitt IM, Marrs JC, Nederveld AL, Regensteiner JG, Dunn AL, et al. How can clinical practices pragmatically increase physical activity for patients with type 2 diabetes? A systematic review. Transl Behav Med. 2017;7:751–72.
    Article PubMed PubMed Central Google Scholar
  24. Zwarenstein M, Treweek S, Gagnier JJ, Altman DG, Tunis S, Haynes B, et al. Improving the reporting of pragmatic trials: an extension of the CONSORT statement. BMJ. 2008;337:a2390.
    Article PubMed PubMed Central Google Scholar
  25. Rainer TH, Cheng CH, Janssens HJ, Man CY, Tam LS, Choi YF, et al. Oral Prednisolone in the Treatment of Acute Gout: A Pragmatic, Multicenter, Double-Blind. Randomized Trial Ann Intern Med. 2016;164:464–71.
    Article PubMed Google Scholar
  26. Aveyard P, Johnson C, Fillingham S, Parsons A, Murphy M. Nortriptyline plus nicotine replacement versus placebo plus nicotine replacement for smoking cessation: pragmatic randomised controlled trial. BMJ. 2008;336:1223–7.
    Article PubMed PubMed Central Google Scholar
  27. van Nieuwkoop C, van der Starre WE, Stalenhoef JE, van Aartrijk AM, van der Reijden TJ, Vollaard AM, et al. Treatment duration of febrile urinary tract infection: a pragmatic randomized, double-blind, placebo-controlled non-inferiority trial in men and women. BMC Med. 2017;15:70.
    Article PubMed PubMed Central Google Scholar
  28. Currow D, Watts GJ, Johnson M, McDonald CF, Miners JO, Somogyi AA, et al. A pragmatic, phase III, multisite, double-blind, placebo-controlled, parallel-arm, dose increment randomised trial of regular, low-dose extended-release morphine for chronic breathlessness: Breathlessness, Exertion And Morphine Sulfate (BEAMS) study protocol. BMJ Open. 2017;7:e018100.
    Article PubMed PubMed Central Google Scholar
  29. McIlroy D, Murphy D, Kasza J, Bhatia D, Wutzlhofer L, Marasco S. Effects of restricting perioperative use of intravenous chloride on kidney injury in patients undergoing cardiac surgery: the LICRA pragmatic controlled clinical trial. Intensive Care Med. 2017;43:795–806.
    Article CAS PubMed Google Scholar
  30. Laiz A, Malouf J, Marin A, Longobardi V, de Caso J, Farrerons J, Casademont J. Impact of 3-Monthly Vitamin D Supplementation Plus Exercise on Survival after Surgery for Osteoporotic Hip Fracture in Adult Patients over 50 Years: A Pragmatic Randomized, Partially Blinded. Controlled Trial J Nutr Health Aging. 2017;21:413–20.
    Article CAS PubMed Google Scholar
  31. Freedman SB, Parkin PC, Willan AR, Schuh S. Rapid versus standard intravenous rehydration in paediatric gastroenteritis: pragmatic blinded randomised clinical trial. BMJ. 2011;343:d6976.
    Article PubMed PubMed Central Google Scholar
  32. Bleidorn J, Gágyor I, Kochen MM, Wegscheider K, Hummers-Pradier E. Symptomatic treatment (ibuprofen) or antibiotics (ciprofloxacin) for uncomplicated urinary tract infection?—results of a randomized controlled pilot trial. BMC Med. 2010;8:30.
    Article PubMed PubMed Central Google Scholar
  33. Bulpitt CJ, Beckett NS, Cooke J, Dumitrascu DL, Gil-Extremera B, Nachev C, Nunes M, Peters R, Staessen JA, Thijs L; Hypertension in the Very Elderly Trial Working Group.Results of the pilot study for the Hypertension in the Very Elderly Trial. J Hypertens 2003;21:2409–2417.
  34. The Dutch TIA Trial Study Group. Trial of secondary prevention with atenolol after transient ischemic attack or nondisabling ischemic stroke. Stroke. 1993;24:543–8.
    Article Google Scholar
  35. Bots AF. Kastelein JJ; Discovery Netherlands Investigators. Achieving lipid goals in real life: the Dutch DISCOVERY study. Int J Clin Pract. 2005;59:1387–94.
    Article CAS PubMed Google Scholar
  36. Topol EJ, Moliterno DJ, Herrmann HC, Powers ER, Grines CL, Cohen DJ, et al. Comparison of two platelet glycoprotein IIb/IIIa inhibitors, tirofiban and abciximab, for the prevention of ischemic events with percutaneous coronary revascularization. N Engl J Med. 2001;344:1888–94.
    Article CAS PubMed Google Scholar
  37. Piller LB, Baraniuk S, Simpson LM, Cushman WC, Massie BM, Einhorn PT, et al. Long-term follow-up of participants with heart failure in the antihypertensive and lipid-lowering treatment to prevent heart attack trial (ALLHAT). Circulation. 2011;124:1811–8.
    Article CAS PubMed PubMed Central Google Scholar
  38. Bjerre E, Bruun DM, Tolver A, Brasso K, Krustrup P, Johansen C. Effectiveness of community-based football compared to usual care in men with prostate cancer: Protocol for a randomised, controlled, parallel group, multicenter superiority trial (The FC Prostate Community Trial). BMC Cancer. 2016;16:767.
    Article PubMed PubMed Central Google Scholar
  39. Wildes TS, Winter AC, Maybrier HR, Mickle AM, Lenze EJ. Stark S Protocol for the Electroencephalography Guidance of Anesthesia to Alleviate Geriatric Syndromes (ENGAGES) study: a pragmatic, randomised clinical trial. BMJ Open. 2016;6:e011505.
    Article CAS PubMed PubMed Central Google Scholar
  40. Haramiova Z, Stasko M, Hulin M, Tesar T, Kuzelova M, Morisky DM. The effectiveness of daily SMS reminders in pharmaceutical care of older adults on improving patients' adherence to antihypertensive medication (SPPA): study protocol for a randomized controlled trial. Trials. 2017;18:33.
    Article Google Scholar
  41. Loudon K, Zwarenstein M, Sullivan FM, Donnan PT, Gágyor I, HJSM H, et al. The PRECIS-2 tool has good interrater reliability and modest discriminant validity. J Clin Epidemiol. 2017:113–21.
  42. Dekkers OM, Bossuyt PM, Vandenbroucke JP. How trial results are intended to be used: is PRECIS-2 a step forward? J Clin Epidemiol. 2017;84:25–6.
    Article PubMed Google Scholar

Download references

Acknowledgements

Not applicable.

Funding

METRICS is supported by a grant from the Laura and John Arnold Foundation. The work of JPA Ioannidis is supported by an unrestricted gift from Sue and Bob O’Donnell. The authors assume full responsibility for the accuracy and completeness of the ideas presented.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Author information

Authors and Affiliations

  1. Epidemiology Unit, Health Research Institute-Fundación Jiménez Díaz University Hospital, Universidad Autónoma de Madrid, E-28040, Madrid, Spain
    Rafael Dal-Ré
  2. Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, CA, 94305, USA
    Perrine Janiaud
  3. Departments of Medicine, Health Research and Policy, Biomedical Data Science, Statistics, and Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, CA, 94305, USA
    John P. A. Ioannidis

Authors

  1. Rafael Dal-Ré
    You can also search for this author inPubMed Google Scholar
  2. Perrine Janiaud
    You can also search for this author inPubMed Google Scholar
  3. John P. A. Ioannidis
    You can also search for this author inPubMed Google Scholar

Contributions

This work started in a series of conversations between RDR and JPAI. RDR and PJ extracted the information from the PubMed search. RDR wrote the first draft. PJ and JPAI suggested further changes for important intellectual content. All authors approved the final version of the manuscript and are accountable for all aspects included in it.

Corresponding author

Correspondence toJohn P. A. Ioannidis.

Ethics declarations

Not applicable.

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article

Dal-Ré, R., Janiaud, P. & Ioannidis, J.P.A. Real-world evidence: How pragmatic are randomized controlled trials labeled as pragmatic?.BMC Med 16, 49 (2018). https://doi.org/10.1186/s12916-018-1038-2

Download citation

Keywords