Moderator's view: Meta-analysis: the best knowledge but not always shining gold (original) (raw)

Journal Article

CNR-IFC Clinical Epidemiology and Pathophysiology of Renal Diseases and Hypertension, Reggio Cal Unit

,

Reggio Calabria

,

Italy

Search for other works by this author on:

• PDF
• Split View
• • Article contents
  • Figures & tables
  • Video
  • Audio
  • Supplementary Data
• Cite

Cite

Carmine Zoccali, Moderator's view: Meta-analysis: the best knowledge but not always shining gold, Nephrology Dialysis Transplantation, Volume 31, Issue 6, June 2016, Pages 886–889, https://doi.org/10.1093/ndt/gfw093
Close

Navbar Search Filter Mobile Enter search term Search

Abstract

Meta-analysis has gained top status in medicine. Correctly so, well-performed meta-analysis is perceived as an unbeatable method for distilling first class medical knowledge. However, such a high status should in no way be considered as a guarantee that all information derived from meta-analyses is pure gold. Systematic reviews and meta-analyses not performed with due methodological attention abound and nephrology is not at all a protected territory. Herein I give concrete examples of meta-analyses published in major journals whose findings are not pure gold.

INTRODUCTION

Meta-analysis is now held as a powerful instrument to maximize extraction of valid knowledge from the bulk of available studies. This method is applied in the whole realm of scientific research, from physics to biology and clinical medicine. No one doubts the value of this approach. Strictly speaking, the contenders in this Polar View debate [1, 2], rather than having opposing positions, present complementary and in part overlapping concepts. In such a scenario the role of the moderator is marginal at best. I will therefore try to provide background considerations that may be useful to frame the potential and the limitations of the technique, two issues dealt with in detail by both contenders. Starting with a brief historical account of the application of meta-analysis in medicine, I will expand on one of the main achievements favoured by meta-analysis, i.e. the creation of large collaborative consortia of methodologists-investigators to produce individual level meta-analyses on major public health issues, like hypertension and hypercholesterolaemia. I will conclude in a direct and practical way by discussing some critical issues about the application of meta-analysis in medicine and in nephrology in particular.

THE DEVELOPMENT OF META-ANALYTICAL APPROACHES TO DATA SYNTHESIS IN MEDICINE

Even though the root of meta-analysis is traced back to the XVII century to Blaise Pascal and to the British astronomer George Biddel Airy [3], Karl Pearson, the mathematician who developed the concept and the calculation of the correlation coefficient, is the biostatistician credited to be the first to have combined diverse sets of medical data to compare infection and mortality rates among soldiers volunteering for vaccination against typhoid fever in various countries of the British Empire with that of soldiers who did not volunteer [4]. By regrouping observations into larger groups, he arrived at a pooled estimate of the effect of vaccination. The founder of modern biostatistics, Ronald Fisher, applied the same approach mainly to agricultural studies and subsequently to medical issues. He intuited early on the risk of bias created by the preferential publication of positive studies [5]. In 1976 the term ‘meta-analysis’ was coined to refer to ‘the statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings’ [6]. In the 1970s Richard Peto made careful numerical syntheses of randomized trials of beta-blockade used for the prevention of re-infarction [7, 8]. In these papers he noted that when the treatment effect is very variable among studies, estimating the fixed weighted average of the effects was the wrong approach [9], thus departing from previous meta-analyses which took for granted that the effect of drugs of the same class is fixed across trials. This observation paved the way for the development of the random effects model [10]. In 1974 Elwood and the founder of modern evidence-based medicine, Arthur Cochrane, performed a landmark trial testing whether aspirin may reduce recurrences of myocardial infarction [11]. The results suggested a favourable effect but failed to achieve statistical significance. Richard Peto collected data subsequent to the Cochrane trial and synthesized the whole data by a combined analysis that was published as an anonymous editorial in The Lancet in 1980 [12]. The relevance of a systematic approach aimed at reducing bias in large reviews was clearly formulated in the late 1980s. However, the terms meta-analysis and systematic review were often used interchangeably, which generated confusion. For this reason Chalmers and Altman [13] suggested that the term ‘meta-analysis’ should be limited to the process of numerical synthesis while the term systematic review should be applied to the strategies used to limit bias in the collection, evaluation and synthesis of studies composing the review. Meta-analysis may be applied to numerically summarize studies in systematic reviews but it may not be if these studies do not possess the needed requisites for formal application of numerical synthesis.

META-ANALYSIS: INTERNATIONAL COLLABORATIONS

When they conform to the high standard set by the Cochrane Collaboration [2] or by PROSPERO [3] and to the high quality of reporting recommended by the Equator initiative [Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)] [4], systematic reviews and meta-analyses undoubtedly represent the highest level of evidence for formulating recommendation for clinical practice. The value of meta-analyses is coherently recognized by major governmental agencies like the United States Preventive Services Task Force [6] and the Australian Government National Health and Medical Research Council. Thus meta-analyses are held as a powerful instrument for designing solidly based public health interventions and for guiding medical practice. The possibility of setting common goals and collaboration among disparate investigators worldwide constitutes another unprecedented opportunity for advancing clinical science and for fostering large scale research enterprises with real impact on public health and clinical medicine. Starting from the late 1980s, large networks of investigators joined forces to scrutinize available studies on major health issues. The trialists' collaborations promoted by the Oxford's Clinical Trial Service Unit and Epidemiological Studies Unit founded by Peto have produced landmark meta-analyses on various topics from the treatment of hypercholesterolaemia to cancer. For a very recent example of the reach of these collaborations, we may refer to a meta-analysis published on 30 January this year in The Lancet [14]. The SPRINT trial [15] demonstrated that in high-risk patients with hypertension a blood pressure (BP) treatment target as low as <120/80 mmHg produces a substantial risk reduction as compared with the traditional target (<140/90 mmHg) [16]. However, because diabetics were not included in SPRINT, this trial left open the question of whether more intensive treatment may be beneficial also in diabetics, an issue left unsolved by the ACCORD trial [17]. The hypertension trialists' collaboration had already built a powerful database of hypertension trials including carefully checked individual patients' data [14]. Therefore, the crucial question of whether diabetics may benefit from intensive drug treatment could be addressed in this database. Even though the achieved BP level (133/81 mmHg) in this meta-analysis was higher than that achieved in SPRINT (121/69 mmHg), this meta-analysis provided clear evidence that the benefits of more intensive BP control extend to diabetic patients. Meta-analyses based on individual patients are increasingly performed and this praiseworthy development of the technique eventually provides at a reasonable cost the sort of solid information we need to apply in clinical practice.

THE LANDING FIELD OF META-ANALYSES IS NO GUARANTEE OF THEIR QUALITY

Meta-analysis has gained top status in medicine. Correctly so, well-performed meta-analysis is perceived as an unbeatable method for distilling first class medical knowledge. However, such a high status should in no way be considered as a guarantee that all information derived from meta-analyses is pure gold. Systematic reviews and meta-analyses not performed with due methodological attention abound and nephrology is not at all a protected territory [18]. Concrete examples can be found of meta-analyses published in major journals whose findings are not pure gold. I will focus on two meta-analyses in chronic kidney disease (CKD) patients published in one of the most prestigious journals: The Lancet. The first is a 2005 systematic review and meta-analysis by Casas et al. on inhibitors of the renin-angiotensin system [19], which negated that these drugs offer additional nephroprotection as compared with other categories of antihypertensive drugs. This meta-analysis stands as one of the most quoted in nephrology (as of 26 March 2016, 697 quotations). However, it presented important methodological and analytical issues that stimulated a fierce debate in a series of letters in the same journal. I will comment on some main objections because these pertain to the very criteria applied to adjudicate the validity of meta-analyses [20]. Non-compliance to these criteria produces some of the drawbacks enumerated both by the PRO and the CON contenders in this debate (Figure 1). In brief, angiotensin-converting enzyme (ACE) inhibitors and angiotensin II blockers were clustered into a single class, while evidence exists that, although similar, these drugs may have different effects on systems other than renin and angiotensin [22]. Second, BP was analysed as a categorical variable (tertiles), which reduces the analytical power and increases the possibility of residual confounding, a hypothesis suggested by the fact that in patients with the lowest antihypertensive effect by ACE inhibitors and angiotensin II receptor antagonists there was no excess risk for end-stage renal disease, an observation which defies the indisputable notion that hypertension is a strong risk factor for progression to kidney failure (Figure 1, last drawback). Furthermore, quite superficially, the ALLHAT study was combined with 11 trials enrolling patients with various degrees of proteinuria. ALLHAT was a very large trial contributing the 90% of the data to the meta-analysis and included patients with clinical characteristics (high-risk hypertensive patients where urine albumin was not measured but was most likely either normal or just modestly raised) much different from those of patients in the other trials (patients with established CKD and variable degrees of proteinuria). ALLHAT showed no greater nephroprotection by ACE inhibition as compared with other antihypertensive drugs. Therefore this trial—highly heterogeneous as compared with the other trials (Figure 1, second drawback)—contributed decisively to the conclusion that ACE inhibitors and angiotensin II blockers have unique nephroprotective properties. The conclusion by Casas et al. that ‘… treatment decisions for hypertension in renal disease should be based on the blood-pressure-lowering effect …’ weakened the nephrologists confidence on the value of drugs interfering with the renin-angiotensin system for slowing renal disease progression. Subsequently, another meta-analysis focusing on proteinuria [23] and a recent, detailed meta-analysis in diabetic patients with CKD [21] unmistakably showed that ACE inhibitors, beyond their BP-lowering effect, possess unique nephroprotective effects as compared with other antihypertensive agents.

FIGURE 1:

Drawbacks in meta-analyses reported in references [18, 21].

The second problematic meta-analysis is the one by Jamal et al. on phosphate binders in CKD patients [24]. This meta-analysis showed that non-calcium-based phosphate binders are associated with reduced death risk as compared with calcium-based phosphate binders. An accompanying editorial in the same issue of the journal [25] sounded the death bell for calcium-based binders. However, this meta-analysis had several drawbacks (Figure 1). First, it was based on 11 studies—only 7 of which were randomized trials—of medium-low quality. All but one of these trials, comparing sevelamer with calcium binders, included a quite low number of patients. Apparently the case in favour of non-calcium binders was made stronger by a separate analysis in the same review showing that these drugs reduce coronary calcification as compared with calcium binders. However, vascular calcifications, including coronary calcification, are an unreliable surrogate end-point [26]. Thus, this highly quoted meta-analysis (141 quotations as of 26 March 2016) hardly allows a credible conclusion on the superiority of non-calcium binders as compared with calcium binders in CKD patients.

Understandably, the two contenders of this Polar View did not even start the fight. Although with different tunes, both highlighted the relevance of canonical criteria for establishing the validity of meta-analysis. My final recommendation is to not lessen your intellectual vigilance when reading a meta-analysis. Also, meta-analyses landing in top-level medical journals may present problems making their interpretation problematic or even defying interpretation. Thus, we should remember to focus on the construct of the content (the meta-analysis) and ignore the shininess of the container (the journal).

CONFLICT OF INTEREST STATEMENT

None declared.

(See related articles by Mudge et al. Pro: Meta-analysis: the case for. Nephrol Dial Transplant 2016; 31: 875–880; Esterhuizen and Thabane. Con: Meta-analysis: some key limitations and potential solutions. Nephrol Dial Transplant 2016; 31: 882–885)

REFERENCES

1

Mudge

DW

,

Webster

ACJD

.

Meta-analysis: the case for

.

Nephrol Dial Transplant

2016

;

31

:

875

–

880

2

Esterhuizen

T

,

Thabane

L

.

Meta-analysis. CON: some key limitations and possible solution

.

Nephrol Dial Transplant

2016

;

31

:

882

–

885

3

O'Rourke

K

.

An historical perspective on meta-analysis: dealing quantitatively with varying study results

.

J R Soc Med

2007

;

100

:

579

–

582

4

Pearson

K

.

Report on certain enteric fever inoculation statistics

.

BMJ

1904

;

2

:

1243

–

1246

5

Sterling

TD

.

Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa

.

J Am Stat Assoc

1959

;

54

:

30

–

34

6

Glass

GV

.

Primary, secondary and meta-analysis of research

.

Educ Res

1976

;

10

:

3

–

8

7

Peto

R

,

Pike

MC

,

Armitage

P

et al. .

Design and analysis of randomized clinical trials requiring prolonged observation of each patient. I. Introduction and design

.

Br J Cancer

1976

;

34

:

585

–

612

8

Peto

R

,

Pike

MC

,

Armitage

P

et al. .

Design and analysis of randomized clinical trials requiring prolonged observation of each patient. II. Analysis and examples

.

Br J Cancer

1977

;

35

:

1

–

39

9

Peto

R

.

Why do we need systematic overviews of randomized trials?

Stat Med

1987

;

6

:

233

–

244

10

DerSimonian

R

,

Laird

N

.

Meta-analysis in clinical trials

.

Control Clin Trials

1986

;

7

:

177

–

188

11

Elwood

PC

,

Cochrane

AL

,

Burr

ML

et al. .

A randomized controlled trial of acetyl salicylic acid in the secondary prevention of mortality from myocardial infarction

.

BMJ

1974

;

1

:

436

–

440

12

Anon

.

Aspirin after myocardial infarction

.

Lancet

1980

;

1

:

1172

–

1173

13

Chalmers

I

,

Altman

DG

.

Systematic Reviews

.

London: BMJ Publications

,

1995

14

Xie

X

,

Atkins

E

,

Lv

J

et al. .

Effects of intensive blood pressure lowering on cardiovascular and renal outcomes: a systematic review and meta-analysis

.

Lancet

2016

;

387

:

435

–

443

15

Jones

DW

,

Weatherly

L

,

Hall

JE

.

SPRINT: what remains unanswered and where do we go from here?

Hypertension

2016

;

67

:

261

–

262

16

SPRINT Research Group

,

Wright

JT

,

Williamson

JD

et al. .

A randomized trial of intensive versus standard blood-pressure control

.

N Engl J Med

2015

;

373

:

2103

–

2116

17

Cushman

WC

,

Evans

GW

,

Byington

RP

et al. .

Effects of intensive blood-pressure control in type 2 diabetes mellitus

.

N Engl J Med

2010

;

362

:

1575

–

1585

18

Mrkobrada

M

,

Thiessen-Philbrook

H

,

Haynes

RB

et al. .

Need for quality improvement in renal systematic reviews

.

Clin J Am Soc Nephrol

2008

;

3

:

1102

–

1114

19

Casas

JP

,

Chua

W

,

Loukogeorgakis

S

et al. .

Effect of inhibitors of the renin-angiotensin system and other antihypertensive drugs on renal outcomes: systematic review and meta-analysis

.

Lancet

2005

;

366

:

2026

–

2033

20

Garg

AX

,

Hackam

D

,

Tonelli

M

.

Systematic review and meta-analysis: when one study is just not enough

.

Clin J Am Soc Nephrol

2008

;

3

:

253

–

260

21

Palmer

SC

,

Mavridis

D

,

Navarese

E

et al. .

Comparative efficacy and safety of blood pressure-lowering agents in adults with diabetes and kidney disease: a network meta-analysis

.

Lancet

2015

;

385

:

2047

–

2056

22

Chrysant

SG

,

Chrysant

GS

.

The pleiotropic effects of angiotensin receptor blockers

.

J Clin Hypertens (Greenwich)

2006

;

8

:

261

–

268

23

Kunz

R

,

Friedrich

C

,

Wolbers

M

et al. .

Meta-analysis: effect of monotherapy and combination therapy with inhibitors of the renin angiotensin system on proteinuria in renal disease

.

Ann Intern Med

2008

;

148

:

30

–

48

24

Jamal

SA

,

Vandermeer

B

,

Raggi

P

et al. .

Effect of calcium-based versus non-calcium-based phosphate binders on mortality in patients with chronic kidney disease: an updated systematic review and meta-analysis

.

Lancet

2013

;

382

:

1268

–

1277

25

Ortiz

A

,

Sanchez-Niño

MD

.

The demise of calcium-based phosphate binders

.

Lancet

2013

;

382

:

1232

–

1234

26

Zoccali

C

,

Bolignano

D

,

D'Arrigo

G

et al. .

Validity of vascular calcification as a screening tool and as a surrogate end point in clinical research

.

Hypertension

2015

;

66

:

3

–

9

© The Author 2016. Published by Oxford University Press on behalf of ERA-EDTA. All rights reserved.

I agree to the terms and conditions. You must accept the terms and conditions.

Submit a comment

Name

Affiliations

Comment title

Comment

You have entered an invalid code

Thank you for submitting a comment on this article. Your comment will be reviewed and published at the journal's discretion. Please check for further notifications by email.

Advertisement intended for healthcare professionals

Citations

Views

Altmetric

Metrics

Total Views 933

519 Pageviews

414 PDF Downloads

Since 1/1/2017

Citations

1 Web of Science

×

Email alerts

Citing articles via

• Latest

• Most Read

• Most Cited

More from Oxford Academic

Advertisement intended for healthcare professionals