The Cochrane HPV vaccine review was incomplete and ignored important evidence of bias (original) (raw)

The Cochrane HPV vaccine review was incomplete and ignored important evidence of bias

Free

Loading

  1. http://orcid.org/0000-0002-9737-0555Lars Jørgensen1,
  2. Peter C Gøtzsche1,
  3. Tom Jefferson2
  4. 1 Nordic Cochrane Centre, Rigshospitalet (dept. 7811), Copenhagen, Hovedstaden, Denmark
  5. 2 Centre for Evidence Based Medicine, University of Oxford Centre for Evidence-Based Medicine, Oxford, UK
  6. Correspondence to Dr Lars Jørgensen, Nordic Cochrane Centre, Copenhagen 2200, Denmark; lj{at}cochrane.dk

Statistics from Altmetric.com

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Introduction

The Cochrane Review conducted trial searches up until June 2017 and included 26 randomised trials with 73 428 women.1 In January 2018, we published an index of the study programmes of the HPV vaccines that included 206 comparative studies.2 As of June 2017, about a third of the 206 studies were not published and half of the completed studies listed on ClinicalTrials.gov had no results posted.2 Although we sent our index to the Cochrane group handling the Cochrane Review, the review stated that, ‘nearly all end-of-study reports have been published in the peer-reviewed literature’. When we applied the Cochrane Review’s inclusion criteria to the 206 studies, we identified 46 completed and eligible trials. The number of randomised participants could be assessed for 42 of the 46 trials and was 121 704. With nearly half of the trials and half of the participants missing, the Cochrane authors’ conclusion, ‘that the risk of reporting bias may be small’, was inappropriate. Fifteen of the 20 additional trials were listed on ClinicalTrials.gov; the Cochrane authors would therefore have identified more trials if they had searched ClinicalTrials.gov in more depth and searched additional trial registers (we searched 45 trial registers2).

The Cochrane authors stated that they ‘did not include the nine-valent vaccine [Gardasil 9] … since the randomised trials … did not incorporate an arm with a non-HPV vaccine control’. This is not correct. The only saline placebo trial of approved HPV vaccines is a Gardasil 9 trial (V503-006; NCT01047345) that was published in 2015.3 Its participants had previously been vaccinated with four-valent Gardasil, but according to the Cochrane Review protocol,4 this was not an exclusion criterion. Since many countries are shifting to Gardasil 9,5 it is unfortunate that the Gardasil 9 trial was not included in the Cochrane Review.

No included trial in the Cochrane Review used a placebo comparator

All 26 trials included in the Cochrane Review used active comparators: adjuvants (aluminium hydroxide (Al(OH)3) or amorphous aluminium hydroxyphosphate sulphate) or hepatitis vaccines.

Adjuvants are not regulated separately from their vaccine antigens. According to the Food and Drug Administration, adjuvants are unreliable comparators.6 One HPV vaccine manufacturer (GlaxoSmithKline that produces Cervarix) states that its aluminium-based comparator induces harms: ‘higher incidences of myalgia might namely be attributable to the higher content of aluminium in the HPV vaccine [450 micrograms Al(OH)3] than the content of aluminium in the HAV (hepatitis A) vaccine [225 micrograms Al(OH)3]’.7 The comparator hepatitis vaccines also used the HPV vaccines’ aluminium-based adjuvant.

The Cochrane authors mistakenly used the term placebo to describe the active comparators. They acknowledged that ‘The comparison of the risks of adverse events was compromised by the use of different products (adjuvants and hepatitis vaccines) administered to participants in the control group’. Nevertheless, this statement can easily be overlooked, as it comes after 7500 words about other issues in the discussion and under the heading ‘Potential biases in the review process’. Active comparators was not a bias in the review process but a bias in the design of the HPV vaccine trials.

The use of active comparators probably increased the occurrence of harms in the comparator groups and thereby masked harms caused by the HPV vaccines. It is noteworthy that many women were excluded from the trials if they had received the adjuvants before or had a history of immunological or nervous system disorders; for example, in the PATRICIA Trial with 18 644 women8 and the FUTURE II trial with 12 167 women.9 These exclusion criteria lowered the external validity of the trials and suggest that the vaccine manufacturers were worried about harms caused by the adjuvants. The criteria are not listed as warnings on the package inserts of the HPV vaccines,10–12 which may have led to more vaccine-related harms in clinical practice than in the trials.

The included HPV vaccine trials used composite surrogate outcomes for cervical cancer

In line with WHO recommendations,13 the Cochrane Review was based on composite surrogate outcomes: ‘cervical intraepithelial neoplasia grade two and above [CIN2+], CIN grade three and above [CIN3+], and adenocarcinoma-in-situ (AIS)’.1 The use of such outcomes seemed reasonable for a preliminary assessment of HPV vaccine benefits, but the outcomes can be difficult to interpret. If there were clinically important differences in the severity of the cervical lesions in the two compared groups, they may not have been apparent in the composite outcomes of CIN2+ and CIN3+. The Cochrane authors did not describe any cervical cancers in the 26 trials, although cancers did occur in the trials; for example, in the ClinicalTrials.gov entry for the VIVIANE Trial, one case of ‘Adenocarcinoma of the cervix’ and one case of ‘Cervix cancer metastatic’ are listed in the HPV vaccine group (see ‘Results: Serious Adverse Events’).14 Furthermore, the relationship between CIN2 and cervical cancer is not clear-cut. Most CIN2 lesions in women below age 30 years regress spontaneously; an active surveillance approach has therefore been recommended for this group.15 The Cochrane Review’s 26 trials mainly included women below age 30 years and used frequent cervical screening (often every 6 months) that did not reflect real life practice (often every 3–5 years5).

The Cochrane Review incompletely assessed serious and systemic adverse events

The Cochrane authors reported that they made a ‘Particular effort’ to assess serious adverse events and performed a sensitivity analysis that gave them ‘confidence that published and registry or website-sourced data are similar for the same study’.1 This seems unlikely. As an example, the PATRICIA Trial publication only included two-thirds (1400/2028) of the serious adverse events listed on ClinicalTrials.gov. The Cochrane authors included 701 vs 699 serious adverse events (1400) from the PATRICIA Trial publication (see the Cochrane Reviews’ ‘Figure 10, Analysis 7.6.2’) and 835 vs 829 serious adverse events from its ClinicalTrials.gov entry (see ‘Comparison 7, Analysis 6: 7.6.2’; both analyses were called ‘7.6.2’). We found 1046 vs 982 serious adverse events (2028) when we summarised the data from ClinicalTrials.gov (see ‘Results: Serious Adverse Events’).16

The Cochrane authors concluded with ‘high certainty’ that the risk of serious adverse events was similar in the HPV vaccine groups and the comparator groups. However, the authors failed to mention that several of the included trials did not report serious adverse events for the whole trial period. For example, FUTURE I,17 FUTURE II9 and FUTURE III18—which in total included 21 441 women with up to 4 years follow-up—only reported serious adverse events occurring within 14 days postvaccination. Furthermore, the Cochrane authors did not explain what the serious adverse events consisted of or whether some of them were more common in the HPV vaccine groups.

The Cochrane authors found more deaths in the HPV vaccine groups than in the comparator groups. The death rate was significantly increased in women older than 25 years (risk ratio (RR) 2.36, 95% CI 1.10 to 5.03; no absolute numbers were provided for this subgroup analysis, but the total numbers of deaths were 51 in the HPV vaccine groups and 39 in the comparator groups). The Cochrane authors suggested that this was a chance occurrence since there was no pattern in the causes of death or in the time between vaccine administration and date of death. However, as the Cochrane Review only included randomised trials, the authors cannot rule out that the increase could be caused by the HPV vaccines. A death may be coded in a way that does not raise suspicion that the vaccine caused it; for example, a ‘traumatic head injury’ or ‘drowning’ could have been caused by a ‘syncope,’ which is a recognised harm.10–12 As of May 2018, WHO’s pharmacovigilance database—VigiBase, managed by the Uppsala Monitoring Centre (UMC)—contained 499 deaths reported as related to HPV vaccination.19

The Cochrane authors concluded that, ‘Systemic events with general mild symptoms were similarly frequent in vaccinated recipients and placebo or control vaccine recipients’. Their Analysis 7.5 showed a non-significant increase in systemic events: RR 1.02 (95% CI 0.98 to 1.07) with a total of 9137 vs 9054 events. The Cochrane authors did not include all of their trials that were eligible for systemic events in Analysis 7.5; for example, the PATRICIA Trial was not included. On ClinicalTrials.gov, PATRICIA has 7129 vs 6557 systemic events listed under ‘Results: Other Adverse Events (General disorders)’, which in itself is a significantly increased risk: RR 1.09 (95% CI 1.07 to 1.11).16

The Cochrane authors ‘planned requesting data from data owners, to fill in gaps with available unpublished data’, but ‘due to constraints in time and other resources’ they were unable to do so.1 Considering that 7 years passed from the publication of the Cochrane protocol in 20114 to the Cochrane Review in 2018,1 lack of time seems a poor excuse for not trying to obtain unpublished trial documents and data. More importantly, harms cannot be assessed reliably in published trial documents—especially in journal publications of industry funded trials where even serious harms often are missing.20 One reason may be the space restrictions that most medical journals have. As an example, the journal publication for the PATRICIA Trial is 14 pages long8 while its publicly available corresponding clinical study report is over 7000 pages long;21 although it is an interim report that has been shortened. Clinical study reports are usually confidential documents, but they can be requested from the European Medicines Agency (EMA) and ClinicalStudyDataRequest.com.

Despite the mentioned examples of reporting bias, the Cochrane authors judged all trials at low risk of reporting bias (see the Cochrane Review’s ‘figure 4: ‘Risk of bias’ summary’).

The Cochrane authors referred to many observational studies in their discussion that found no safety signals of harms associated with the HPV vaccines.1 They cited the WHO’s Global Advisory Committee on Vaccine Safety that expressed ‘concerns about unjustified claims of harms’. The Cochrane authors did not mention a study from 2017 by the WHO UMC that found serious harms following HPV vaccination overlapping with two syndromes: postural orthostatic tachycardia syndrome (POTS) and complex regional pain syndrome (CRPS).22 The WHO UMC provided part of the rationale for EMA’s investigation of POTS and CRPS in 2016.23 As of May 2018, the WHO UMC VigiBase contained 526 cases of POTS and 168 cases of CRPS reported related to HPV vaccination.19

The Cochrane authors did not investigate whether the included trial data reported cases of POTS, CRPS or other safety signals. Instead, the authors cited EMA, which concluded that ‘No causal relation could be established’ between POTS or CRPS and the HPV vaccines.1 The EMA’s conclusion was based on the HPV vaccine manufacturers’ own unverified assessments23 that only included half of the eligible trials.2 Furthermore, the HPV vaccine manufacturers search strategies for POTS and CRPS were inadequate and led to cases being overlooked.24 As an example, in 2014, the Danish Medicines Agency (DMA) asked the HPV vaccine co-manufacturer Sanofi-Pasteur-MSD to search for specific POTS-related symptoms in its database (including dizziness, palpitations, rapid heart rate, tremor, fatigue and fainting). The manufacturer only searched for ‘postural dizziness’, ‘orthostatic intolerance’ and ‘palpitations and dizziness’. The DMA discovered this because only 3 of 26 Danish reports of POTS showed up in Sanofi’s searches.24 As another example, EMA identified six possible cases of POTS and CRPS related to Gardasil 9 that Merck had not identified.25

Industry trial funding and other conflicts of interest

The Cochrane authors assessed the impact of industry funding ‘by meta-regression. No significant effects were observed’.1 They stated that, ‘All but one of the trials was funded by the vaccine manufacturers’, which is not correct. According to ClinicalTrials.gov, this particular trial (‘CVT’ or ‘Costa Rica trial’1) was sponsored by GlaxoSmithKline.26 Therefore, all included trials were funded by the HPV vaccine manufacturers and the meta-regression was meaningless.

The Cochrane Collaboration aims to be free from conflicts of interest related to the manufacturers of the reviewed products.27 Most of the 14 Cochrane authors on the first published protocol for the Cochrane Review had major conflicts of interest related to the HPV vaccine manufacturers.28 The Cochrane Review only has four authors; three of whom had such conflicts of interest a decade ago. The review’s first author currently leads EMA’s ‘post-marketing surveillance of HPV vaccination effects in non-Nordic member states of the European Union’, which is funded by Sanofi-Pasteur-MSD that was the co-manufacturer of Gardasil.

Cochrane’s public relations of the review were uncritical

The announcement of the Cochrane Review on Cochrane.org under ‘News’ included a ‘Science Media Centre roundup of third-party expert reaction to this review’.29 Six experts were cited—all from the UK; although the Cochrane Collaboration is an international organisation. Two of the experts had financial conflicts of interest with the HPV vaccine manufactures. A third expert was responsible for vaccinations in Public Health England that promotes the HPV vaccines. The experts highlighted the ‘intensive and rigorous Cochrane analysis’, ‘that the HPV vaccine is the most effective way for young girls to protect themselves against cervical cancer’, and that, ‘the vaccine causes no serious side-effects’. No expert criticised the review. In our view, this is not balanced and people with conflicts of interest in relation to the manufacturers should not be quoted in relation to a Cochrane review. Richard Smith—the former editor of the _British Medical Journal_—described medical journals as an extension of the marketing arm of the drug industry.30 We are concerned that some observers may see Cochrane Reviews in the same light when Cochrane publishes such public relations messages.

Conclusion

Part of the Cochrane Collaboration’s motto is ‘Trusted evidence’. We do not find the Cochrane HPV vaccine review to be ‘Trusted evidence’, as it was influenced by reporting bias and biassed trial designs. We believe that the Cochrane Review does not meet the standards for Cochrane Reviews or the needs of the citizens or healthcare providers that consult Cochrane Reviews to make ‘Informed decisions’, which also is part of Cochrane’s motto. We recommend that authors of Cochrane Reviews make every effort to identify all trials and their limitations and conduct reviews accordingly.

Acknowledgments

The authors thank Karsten Juhl Jørgensen for helpful comments.

References

  1. ↵
  2. ↵
  3. ↵
  4. ↵
  5. ↵
  6. ↵
  7. ↵
  8. ↵
  9. ↵
  10. ↵
  11. ↵
  12. ↵
  13. ↵
  14. ↵
  15. ↵
  16. ↵
  17. ↵
  18. ↵
  19. ↵
  20. ↵
  21. ↵
  22. ↵
  23. ↵
  24. ↵
  25. ↵
    Rapporteurs’ Day 150 Joint Response Assessment Report of Gardasil 9. Procedure no. EMEA/H/C/3852. CHMP 2014.
  26. ↵
  27. ↵
  28. ↵
  29. ↵
  30. ↵

Linked Articles

Read the full text or download the PDF:

Log in using your username and password