Use of qualitative methods alongside randomised controlled trials of complex healthcare interventions: methodological study (original) (raw)

  1. Research
  2. Use of qualitative...
  3. Use of qualitative methods alongside randomised controlled trials of complex healthcare interventions: methodological study

Research BMJ 2009;339 doi: https://doi.org/10.1136/bmj.b3496 (Published 10 September 2009) Cite this as: BMJ 2009;339:b3496

Loading

  1. Simon Lewin, senior lecturer12,
  2. Claire Glenton, senior researcher3,
  3. Andrew D Oxman, senior researcher3
  4. 1Department of Public Health and Policy, London School of Hygiene and Tropical Medicine, London WC1E 7HT
  5. 2Health Systems Research Unit, Medical Research Council of South Africa
  6. 3Norwegian Knowledge Centre for the Health Services, Oslo, Norway
  7. Correspondence: S Lewin simon.lewin{at}nokc.no

Abstract

Objective To examine the use of qualitative approaches alongside randomised trials of complex healthcare interventions.

Design Review of randomised controlled trials of interventions to change professional practice or the organisation of care.

Data sources Systematic sample of 100 trials published in English from the register of the Cochrane Effective Practice and Organisation of Care Review Group.

Methods Published and unpublished qualitative studies linked to the randomised controlled trials were identified through database searches and contact with authors. Data were extracted from each study by two reviewers using a standard form. We extracted data describing the randomised controlled trials and qualitative studies, the quality of these studies, and how, if at all, the qualitative and quantitative findings were combined. A narrative synthesis of the findings was done.

Results 30 of the 100 trials had associated qualitative work and 19 of these were published studies. 14 qualitative studies were done before the trial, nine during the trial, and four after the trial. 13 studies reported an explicit theoretical basis and 11 specified their methodological approach. Approaches to sampling and data analysis were poorly described. For most cases (n=20) we found no indication of integration of qualitative and quantitative findings at the level of either analysis or interpretation. The quality of the qualitative studies was highly variable.

Conclusions Qualitative studies alongside randomised controlled trials remain uncommon, even where relatively complex interventions are being evaluated. Most of the qualitative studies were carried out before or during the trials with few studies used to explain trial results. The findings of the qualitative studies seemed to be poorly integrated with those of the trials and often had major methodological shortcomings.

Introduction

Randomised controlled trials are used widely for showing causal relations in health and social care because their study design is the only one that is able to control for unknown or unmeasured confounders. Randomised controlled trials are sometimes used to evaluate “complex” interventions—that is, those “made up of various interconnecting parts”1 that act both “independently and inter-dependently.”2 3 Qualitative approaches can contribute in several ways to the development and evaluation of both complex and other health interventions (box 1). Consequently, increasing numbers of randomised controlled trials of such interventions include qualitative components4 5 and interest in this approach is growing. The use of multiple, integrated approaches may be particularly useful in the evaluation of the effects of complex health and social care interventions as these involve social or behavioural processes that are difficult to explore or capture using quantitative methods alone.1

Box 1 Ways in which qualitative methods can be used alongside randomised controlled trials

Before a trial
During a trial
After a trial

The need for methodological research on the ways in which qualitative approaches should be used in randomised controlled trials has been discussed widely.6 7 8 9 10 11 However, to our knowledge no studies have attempted to examine systematically current practice on the use of qualitative approaches in randomised controlled trials of complex healthcare interventions and how they could be used to improve the usefulness and policy relevance of the findings of a trial. By complex interventions we mean those including at least some of the following characteristics: several elements that “may act both independently and inter-dependently”2; complex systems or mechanisms for delivery of the intervention; an intervention that is difficult to describe and replicate; complex explanatory pathways, either physiological or psychosocial; and a degree of uncertainty about the mechanism of action of the intervention or its “active ingredient.”2

We systematically examined the use of qualitative approaches alongside randomised controlled trials of complex healthcare interventions to provide an overview of current practice in this area and ways of identifying qualitative studies undertaken alongside randomised controlled trials. We also explored how trial teams could improve the quality of qualitative studies linked to randomised controlled trials and how synergies between qualitative approaches and randomised controlled trials can be maximised.

Methods

We obtained a list of all randomised controlled trials published in English during 2001-3 and included in the register of the Cochrane Effective Practice and Organisation of Care Review Group.12 From the list for each year (492 randomised controlled trials in total) we sampled every fifth study to obtain a sample of 100 trials (33 or 34 trials from each year). We chose this approach for several reasons. Firstly, we wanted a sample of recently published trials of more complex interventions as we assumed that the use of qualitative methods alongside such trials has increased in recent years. Secondly, the randomised controlled trials needed to have been published sufficiently long ago to allow associated qualitative studies also to have been published. Thirdly, we thought the Cochrane Effective Practice and Organisation of Care Review Group register, with its specific focus on interventions to change professional practice and the organisation of care, more likely to include a higher proportion of relatively complex randomised controlled trials than databases such as Medline or Embase. Details on studies included in this register and the search strategies used to locate them are available elsewhere.12

Where the sampled report was not the primary paper for the randomised controlled trial, the primary report was located. We attempted to identify all published and unpublished qualitative studies linked to these randomised controlled trials. We defined a qualitative study as any study that used qualitative methods for data collection or analysis. We initially checked the primary randomised controlled trial for citations of qualitative studies. We then located the primary randomised controlled trial in PubMed and searched for related studies and other studies published by the authors of the randomised controlled trial. We also located the randomised controlled trial in the Science and Social Science Citation Index and checked the list of studies citing the paper. Any potentially relevant titles and abstracts were examined and full papers obtained where necessary. Finally, we contacted the authors of the randomised controlled trials for information on any published or unpublished qualitative studies linked to their trials. We received responses for 76 of the 100 papers.

Two reviewers used a standard form to extract data from each “case”—that is, the randomised controlled trial and any qualitative studies. This included descriptions of the randomised controlled trials and qualitative studies; the quality of the randomised controlled trials and the qualitative studies; and information on the approaches (if any) used by the authors to combine the findings of the randomised controlled trials and qualitative studies. The quality of the trials was assessed using the quality checklist of the Cochrane Effective Practice and Organisation of Care Review Group.13 The quality of the qualitative studies was assessed using a modified checklist from the critical appraisal skills programme.14 These modifications included further details on whether the qualitative approach was justified and appropriate to the research question, whether the research context was described adequately, and items to differentiate adequate reporting of methods from the appropriateness of those methods, in relation to the research question. We summarised the findings of the study narratively.

Results

Thirty of the 100 included randomised controlled trials had qualitative work associated with them. Nineteen of these qualitative studies were published, either as stand alone papers or within another paper. In 23 (77%) of the 30 cases in which qualitative approaches were used, the researchers employed qualitative methods for both data collection and analysis. In the remaining seven cases some form of qualitative data collection (for example, group discussions or individual interviews) was used, but no formal analysis of these data using qualitative approaches was reported. Most of the qualitative studies (n=25) were carried out before or during the randomised controlled trial (figure⇓). An explicit theoretical basis for the intervention was reported in 12 of the 30 cases.

Randomised controlled trials that included qualitative research

The 30 trials that included qualitative research were carried out in a variety of settings, from general practices to communities and consumers’ homes. Twenty four of the trials were carried out in primary care and the remaining six trials evaluated interventions in secondary care or across a mix of levels.

The trials dealt with a wide range of healthcare issues, the most common being mental health, the appropriate use of medicines, and sexual health. All the trials were carried out in high income countries. The methodological quality of the trials that included qualitative research was similar to those without such studies.

Qualitative studies

Objectives of the studies

The objectives of the qualitative studies varied widely (table 1⇓). The 16 studies done either before the trial, or before and during the trial, had one or more of the following objectives: to explore the knowledge, attitudes, or practices of the target groups about the topic in question; to explore the illness experience of consumers; to develop the intervention; and to develop the instrument used to measure the effects of the intervention in the randomised controlled trial.

Table 1

Characteristics of qualitative studies included in this review and their level of integration with accompanying randomised controlled trials (studies that used only qualitative methods of data collection are not described)

The nine qualitative studies done during the trials had a wide range of objectives. These included describing the intervention as delivered and exploring issues influencing the effects of the intervention, the illness experience of consumers, participants’ experiences of the intervention (box 2), and reasons for refusal to participate in the trial (box 3).

Box 2 Example of a qualitative study exploring how trial participants experienced the intervention (adapted from Harrison et al 200615)

Background

Researchers carried out a randomised controlled trial in the United Kingdom to evaluate the effects of joint teleconsultations on hospital follow-up appointments. They concluded that patients’ overall satisfaction was higher for teleconsultations than for conventional outpatient appointments. A proportion of patients were, however, dissatisfied with their teleconsultation. It was not possible to determine from the trial findings the reasons for satisfaction or dissatisfaction or how satisfaction could be increased in other telemedicine programmes. The qualitative study aimed to answer these questions.

Qualitative methods used

Semistructured individual interviews were carried out with 24 patients within one month of their teleconsultations. The researchers used the framework approach to carry out a thematic analysis of these data.

Findings

Joint teleconsultations were, overall, highly acceptable to patients for several reasons. These included enhanced customer care, such as enhanced convenience, reduced cost, and increased punctuality. Most patients also appreciated the presence of the general practitioner in the consultation, feeling that this improved communication between the specialist and generalist and allowed the general practitioner to summarise and interpret the consultation for the patient. However, one patient stated that they had been excluded from the consultation. Other patients were dissatisfied with parts of the consultation because they would have preferred to be examined directly by the specialist. Some patients found that the technology interfered with their communication with the doctor—for instance, because of lack of synchronisation between sound and vision. The authors of the study discussed how some of these factors could be tackled in joint teleconsultations in the future.

Box 3 Example of a qualitative study exploring why people declined to participate in a randomised controlled trial (adapted from Petty et al 200116)

Background

Researchers carried out a randomised controlled trial in the UK to evaluate the effectiveness of pharmacist run clinical drug reviews in patients aged 65 years or more from general practices. Eligible participants were contacted by post, sent one reminder, and contacted by telephone if no response had been made. Twenty six per cent (n=68) of those contacted by telephone did not wish to participate in the study. The aim of the qualitative study was to identify reasons for non-participation.

Qualitative methods used

Researchers used unstructured questions to ask the 68 patients their reasons for not wishing to participate. The responses were recorded in writing. The researchers independently carried out a thematic analysis of these data, then agreed on a set of categories.

Results from the qualitative study

Ten broad categories of reasons for non-participation were identified. These included administrative categories such as difficulties in reading the invitation letter and not being available at the time suggested. Other categories were tied to behavioural factors such as confusion or lack of understanding of the trial, negative attitudes to health care, and mistrust of the objectives of the trial. The authors suggested that these factors needed to be addressed to increase the number of patients consenting to studies on drug review.

Of the four qualitative studies carried out after the trial, two explored participants’ experiences of the intervention, one explored factors influencing the effects of the intervention (box 4), and one analysed the process for development of the intervention.

Box 4 Example of a qualitative study exploring the reasons for the findings of the trial (adapted from Davies et al 200217 and Graham et al 200418)

Background

Researchers evaluated a strategy for implementing guidelines for nursing care during labour in hospital in Canada. The results of the trial were mixed. The researchers then carried out a qualitative study to explore why this guideline was introduced successfully in some settings but not in others.

Qualitative methods used

A case study approach was used, with individual interviews and group discussions done with nurses, nurse administrators, and nurse educators at the study sites. Interviews were audiorecorded and transcribed and a form of thematic analysis was applied.

Results from the qualitative study

A wide range of factors related to the study settings, the recipients of the guideline, and the characteristics of the guideline interacted to affect implementation of the guideline. Important factors included changes to the external environment of clinical practice; leadership and the availability of equipment in the study settings; concerns among the health professionals targeted; and strategies used to promote uptake of the guideline. The authors concluded that more attention was needed to identify organisational barriers to change and to address these using tailored implementation strategies.

Methodological approach, sampling, data collection, and data analysis

The methodological approaches of the included studies were heterogeneous. Whereas 19 of the qualitative studies did not refer to any specific methodological approach, 11 mentioned approaches such as grounded theory, ethnography, action research, and narrative approaches. Ten studies used several methods for data collection (most often combinations of individual interviews and group discussions), 10 utilised individual interviews only, five used focus groups only, and two used different forms of observation. The remaining three studies were unpublished and we were not able to obtain further information on data collection from the authors.

A number of studies inadequately reported several aspects of the methods: 13 did not describe their sampling approach and the remainder used a mix of purposive, convenience, and random sampling. In 14 studies we could find no information on the approach used to analyse data. Where such information was reported, thematic or content analysis or framework analysis was utilised (n=10) and, more rarely, a grounded theory approach (n=2). Four studies used other approaches.

Where the findings of the trial and qualitative studies were reported in separate papers, the link between the two was not always clear from the papers themselves. Sixteen of the qualitative studies shared authors with the report of the randomised controlled trial. Only nine papers explicitly described some level of linkage between the study teams.

In two of the studies the researchers stated that they had used a “mixed method” approach. Our review of the studies indicated some integration in the interpretation of the results from the trials and the qualitative research in eight cases although the analysis of the data was carried out separately. In most cases (n=13), however, we found no evidence of integration at the level of interpretation.

Quality of the qualitative studies

Ten qualitative studies (including the seven with no formal analysis of the qualitative data) did not provide sufficient data to allow assessment of methodological quality. Quality assessment was therefore carried out on 20 studies (table 2⇓). This showed high variability, with the most common weaknesses including lack of a clear justification for the qualitative approach used (no information in 16 studies); inadequate descriptions of context, sampling, data collection, and analysis methods; little reflection on the researcher’s role in the research process (no information in 17 studies); lack of clarity on how ethical issues had been taken into consideration (no information in 15 studies), and insufficient evidence to support the claims made in the paper.

Table 2

Methodological quality of 20 of 30 included qualitative studies that had sufficient information to carry out a quality assessment

We did not identify any relation between interventions that reported using an explicit theoretical framework and the quality of qualitative studies carried out alongside trials of those interventions.

Discussion

Qualitative studies undertaken alongside randomised controlled trials of interventions to change organisation and practice remain uncommon. Less than one third of recently completed trials of relatively complex interventions in the Cochrane Effective Practice and Organisation of Care register included some form of qualitative research. Of these, only about two thirds were published studies. This is surprising given the nature of these interventions and the growing awareness of the role that qualitative research can play in the design and evaluation of interventions.5 8 19 20 Furthermore, contacts with authors suggested that many valued the findings of qualitative studies. Why then are qualitative approaches not used more extensively alongside trials? Constraints on resources and poor access to relevant expertise were mentioned by study authors in response to our requests for information on qualitative studies. It has also been suggested that linear models for evaluating interventions may impede the use of qualitative approaches. These models, it is suggested, view such evaluation as passing through a series of phases from the development of hypotheses to efficacy trials and then effectiveness trials. This may contribute to the view that earlier phases of research, such as efficacy trials, do not need to incorporate qualitative studies to explore the effects of contextual and other moderating factors. Such methods are seen as important only in the later phases of evaluation.21

Although much has been written on qualitative process evaluation alongside trials of complex interventions, the largest group of qualitative studies identified were those carried out before trials. Firstly, this suggests that reviewers who aim to understand better the effects of interventions through examining qualitative process evaluations may locate little data. Secondly, it indicates the need for more attention to this aspect of trial design.

The rigour of qualitative studies undertaken alongside randomised controlled trials, or at least the reporting of methods used, is an important concern. We identified major shortcomings in many of these studies, particularly issues of sampling, analysis, and critical analysis of the researchers’ roles. Interestingly an explicit theoretical basis for the intervention was reported in over a third of cases—a higher proportion than reported in recent reviews on the use of theory in implementation research.22 23 Twice as many of the randomised controlled trials that included qualitative work also had a clearly specified theoretical basis (40%) compared with randomised controlled trials without any qualitative work (20%). However, the use of theory is by no means the norm in studies in this specialty (only 27% of randomised controlled trials did so explicitly) and it remains unclear whether interventions based explicitly on a particular theoretical approach are more likely to be effective than those designed using pragmatic processes.24 25 26

In our sample we found little evidence of explicit integration of data from qualitative studies and randomised controlled trials and few cases discussed mixed methods approaches. Such data could be integrated in several ways. Discussion of the trial findings could draw on both the qualitative and quantitative data in, for example, exploring reasons for success or failure of the intervention or for variation in effects across sites or individuals. Description of the intervention could also make explicit how qualitative approaches contributed, for example, to identifying barriers to change and developing the intervention. The extent of collaboration within trial teams between researchers from different disciplines, such as social scientists and epidemiologists, is another important aspect. The reported data did not, however, allow us to explore this adequately, and further work on the basis of case studies of trial teams is needed.

Limitations of the study

This study has several possible limitations. Firstly, we may not have identified all qualitative studies linked to the index randomised controlled trials. However, we did receive a high response rate from the authors of randomised controlled trials, and other reviews have indicated that this approach identifies the largest number of additional studies.23 All methods of identifying studies were resource intensive—a potential barrier to examining qualitative work done alongside trials. Secondly, trials sampled from the Cochrane Effective Practice and Organisation of Care database may not be representative of all randomised controlled trials evaluating interventions to change professional practice and the organisation of care. The sampled trials are unlikely to be representative of randomised controlled trials more widely but are likely to be similar, in terms of their use of qualitative methods, to other randomised controlled trials of complex interventions. Finally, our analysis is based largely on study reports. These may not reflect the extent of integration of qualitative and quantitative findings.

Conclusions

Although well conducted qualitative studies can support trial design and improve our understanding of the effects of complex interventions and the mechanisms through which changes occur, qualitative studies remain relatively uncommon alongside trials of complex interventions. Most of the qualitative studies were carried out before the trial, had important methodological shortcomings, and the findings were poorly integrated with those of the trials. This study highlights ways in which the quality and usefulness of qualitative studies carried out alongside randomised controlled trials can be improved (box 5). Further work is needed to develop further methodological and practical guidance for trial teams who plan to utilise qualitative approaches.8 27

Box 5 Improving the quality and usefulness of qualitative studies carried out alongside randomised controlled trials

What is already known on this topic

What this study adds

Notes

Cite this as: BMJ 2009;339:b3496

Footnotes

This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.

References

  1. ↵
    Campbell M, Fitzpatrick R, Haines A, Kinmonth A-L, Sandercock P, Spiegelhalter D, et al. Framework for design and evaluation of complex interventions to improve health. BMJ2000;321:694-6.
  2. ↵
    Medical Research Council. A framework for development and evaluation of RCTs for complex interventions to improve health. London: MRC, 2000.
  3. ↵
    Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ2008;337:a1655.
  4. ↵
    Hawe P, Shiell A, Riley T, Gold L. Methods for exploring implementation variation and local context within a cluster randomised community intervention trial. J Epidemiol Community Health2004;58:788-93.
  5. ↵
    Oakley A, Strange V, Bonell C, Allen E, Stephenson J, Team RS. Process evaluation in randomised controlled trials of complex interventions. BMJ2006;332:413-6.
  6. ↵
    Murphy E, Dingwall R, Greatbatch D, Parker S, Watson P. Qualitative research methods in health technology assessment: a review of the literature. Health Technol Assess1998;2(16).
  7. ↵
    Pearson A. Cochrane Qualitative Research Methods Group. About the Cochrane Collaboration (Methods Groups). Cochrane Library2007;(2):CE000142.
  8. ↵
    Rousseau N, McColl E, Eccles M, Hall L. Qualitative methods in implementation research. In: Thorsen T, Makela M, eds. Changing professional practice. Theory and practice of clinical guidelines implementation. Copenhagen: Danish Institute for Health Services Research and Development, 1999.
  9. ↵
    Gibson G, Timlin A, Curran S, Wattis J. The scope for qualitative methods in research and clinical trials in dementia. Age Ageing2004;33:422-6.
  10. ↵
    Donovan J, Mills N, Smith M, Brindle L, Jacoby A, Peters T, et al. Quality improvement report: improving design and conduct of randomised trials by embedding them in qualitative research: ProtecT (prostate testing for cancer and treatment) study. Commentary: presenting unbiased information to patients can be difficult. BMJ2002;325:766-70.
  11. ↵
    Wainberg ML, Alfredo Gonzalez M, McKinnon K, Elkington KS, Pinto D, Gruber Mann C, et al. Targeted ethnography as a critical step to inform cultural adaptations of HIV prevention interventions for adults with severe mental illness. Soc Sci Med2007;65:296-308.
  12. ↵
    Bero L, Eccles M, Grilli R, Grimshaw J, Gruen R, Mayhew A, et al. Cochrane Effective Practice and Organisation of Care Group. About the Cochrane Collaboration (Cochrane Review Groups (CRGs)). Cochrane Library2007;(1).
  13. ↵
    Cochrane Effective Practice and Organisation of Care Review Group. The data collection checklist. Ottawa: Cochrane Effective Practice and Organisation of Care Review Group, 2007.
  14. ↵
  15. ↵
    Harrison R, MacFarlane A, Wallace P, Murray E. Patients’ perceptions of joint teleconsultations: qualitative evaluation. Health Expect2006;9:81-90.
  16. ↵
    Petty DR, Zermansky AG, Raynor DK, Lowe CJ, Buttress AD, Vail A, et al. “No thank you”: why elderly patients declined to participate in a research study. Pharm World Sci2001;23:22-7.
  17. ↵
    Davies BL, Hodnett E. Labor support: nurses’ self-efficacy and views about factors influencing implementation. J Obstet Gynecol Neonatal Nurs2002;31:48-56.
  18. ↵
    Graham ID, Logan J, Davies B, Nimrod C. Changing the use of electronic fetal monitoring and labor support: a case study of barriers and facilitators. Birth2004;31:293-301.
  19. ↵
    Oakley A, Strange V, Stephenson J, Forrest S, Monteiro H. Evaluating processes. A case study of a randomized controlled trial of sex education. Evaluation2004;10:440-62.
  20. ↵
    Riley T, Hawe P, Shiell A. Contested ground: how should qualitative evidence inform the conduct of a community intervention trial? J Health Serv Res Policy2005;10:103-10.
  21. ↵
    Glasgow RE, Lichtenstein E, Marcus AC. Why don’t we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition. Am J Public Health2003;93:1261-7.
  22. ↵
    Eccles M, Grimshaw J, Walker A, Johnston M, Pitts N. Changing the behavior of healthcare professionals: the use of theory in promoting the uptake of research findings. J Clin Epidemiol2005;58:107-12.
  23. ↵
    Farmer A, Grimshaw J, Mayhew A, McGowan J, Graham I, Driedger M, et al. Systematic reviews of knowledge translation interventions: contributions of process evaluations and contact with authors. CCOHTA final performance report. Ottawa: University of Ottawa, 2005.
  24. ↵
    Oxman AD, Fretheim A, Flottorp S. The OFF theory of research utilization. J Clin Epidemiol2005;58:113-6; discussion 117-20.
  25. ↵
    Perkins M, Jensen P, Jaccard J, Gollwitzer P, Oettingen G, Pappadopulos E, et al. Applying theory-driven approaches to understanding and modifying clinicians’ behavior: what do we know? Psychiatr Serv2007;58:342-8.
  26. ↵
    West R. Time for a change: putting the transtheoretical (stages of change) model to rest. Addiction2005;100:1036-9.
  27. ↵
    Grol R, Wensing M, Eccles M. Improving patient care. The implementation of change in clinical practice. Oxford: Butterworth Heineman, 2004.

View Abstract