Data Extraction Errors in Meta-analyses That Use Standardized Mean Differences

Data Extraction Errors in Meta-analyses That Use Standardized Mean
Differences

Peter C. Gøtzsche, MD, DrMedSci; Asbjørn Hróbjartsson, MD, PhD; Katja
Mari, MSc; Britta Tendal, MSc

JAMA. 2007;298:430-437.

Context Meta-analysis of trials that have used different continuous
or rating scales to record outcomes of a similar nature requires
sophisticated data handling and data transformation to a uniform
scale, the standardized mean difference (SMD). It is not known how
reliable such meta-analyses are.

Objective To study whether SMDs in meta-analyses are accurate.

Data Sources Systematic review of meta-analyses published in 2004
that reported a result as an SMD, with no language restrictions. Two
trials were randomly selected from each meta-analysis. We attempted to
replicate the results in each meta-analysis by independently
calculating SMD using Hedges adjusted g.

Data Extraction Our primary outcome was the proportion of meta-
analyses for which our result differed from that of the authors by 0.1
or more, either for the point estimate or for its confidence interval,
for at least 1 of the 2 selected trials. We chose 0.1 as cut point
because many commonly used treatments have an effect of 0.1 to 0.5,
compared with placebo.

Results Of the 27 meta-analyses included in this study, we could not
replicate the result for at least 1 of the 2 trials within 0.1 in 10
of the meta-analyses (37%), and in 4 cases, the discrepancy was 0.6 or
more for the point estimate. Common problems were erroneous number of
patients, means, standard deviations, and sign for the effect
estimate. In total, 17 meta-analyses (63%) had errors for at least 1
of the 2 trials examined. For the 10 meta-analyses with errors of at
least 0.1, we checked the data from all the trials and conducted our
own meta-analysis, using the authors' methods. Seven of these 10 meta-
analyses were erroneous (70%); 1 was subsequently retracted, and in 2
a significant difference disappeared or appeared.

Conclusions The high proportion of meta-analyses based on SMDs that
show errors indicates that although the statistical process is
ostensibly simple, data extraction is particularly liable to errors
that can negate or even reverse the findings of the study. This has
implications for researchers and implies that all readers, including
journal reviewers and policy makers, should approach such meta-
analyses with caution.

Author Affiliations: Nordic Cochrane Centre, Rigshospitalet,
Copenhagen, Denmark.


* * *
Hm. How common is it for meta-analyses to use SMDs? I will have to
keep this in mind in the future.

Marilyn

.