Re: Most published research findings are false
- From: claudiusdenk@xxxxxxxxxxxxx
- Date: Thu, 31 Jan 2008 09:50:48 -0800 (PST)
On Jan 30, 12:25 pm, claudiusd...@xxxxxxxxxxxxx wrote:
http://wanews.org/docs/falsefindings.htm
John P. A. Ioannidis
. . . .
Corollary 5: The greater the financial and other interests and
prejudices in a scientific field, the less likely the research
findings are to be true. Conflicts of interest and prejudice may
increase bias, u. Conflicts of interest are very common in biomedical
research [26], and typically they are inadequately and sparsely
reported [26,27]. Prejudice may not necessarily have financial roots.
Scientists in a given field may be prejudiced purely because of their
belief in a scientific theory or commitment to their own findings.
Many otherwise seemingly independent, university-based studies may be
conducted for no other reason than to give physicians and researchers
qualifications for promotion or tenure. Such nonfinancial conflicts
may also lead to distorted reported results and interpretations.
Prestigious investigators may suppress via the peer review process the
appearance and dissemination of findings that refute their findings,
thus condemning their field to perpetuate false dogma. Empirical
evidence on expert opinion shows that it is extremely unreliable [28].
.. . . . .
Claimed Research Findings May Often Be Simply Accurate Measures of the
Prevailing Bias
As shown, the majority of modern biomedical research is operating in
areas with very low pre- and post-study probability for true findings.
Let us suppose that in a research field there are no true findings at
all to be discovered. History of science teaches us that scientific
endeavor has often in the past wasted effort in fields with absolutely
no yield of true scientific information, at least based on our current
understanding. In such a "null field," one would ideally expect all
observed effect sizes to vary by chance around the null in the absence
of bias. The extent that observed findings deviate from what is
expected by chance alone would be simply a pure measure of the
prevailing bias.
For example, let us suppose that no nutrients or dietary patterns are
actually important determinants for the risk of developing a specific
tumor. Let us also suppose that the scientific literature has examined
60 nutrients and claims all of them to be related to the risk of
developing this tumor with relative risks in the range of 1.2 to 1.4
for the comparison of the upper to lower intake tertiles. Then the
claimed effect sizes are simply measuring nothing else but the net
bias that has been involved in the generation of this scientific
literature. Claimed effect sizes are in fact the most accurate
estimates of the net bias. It even follows that between "null fields,"
the fields that claim stronger effects (often with accompanying claims
of medical or public health importance) are simply those that have
sustained the worst biases.
For fields with very low PPV, the few true relationships would not
distort this overall picture much. Even if a few relationships are
true, the shape of the distribution of the observed effects would
still yield a clear measure of the biases involved in the field. This
concept totally reverses the way we view scientific results.
Traditionally, investigators have viewed large and highly significant
effects with excitement, as signs of important discoveries. Too large
and too highly significant effects may actually be more likely to be
signs of large bias in most fields of modern research. They should
lead investigators to careful critical thinking about what might have
gone wrong with their data, analyses, and results.
Of course, investigators working in any field are likely to resist
accepting that the whole field in which they have spent their careers
is a "null field." However, other lines of evidence, or advances in
technology and experimentation, may lead eventually to the dismantling
of a scientific field. Obtaining measures of the net bias in one field
may also be useful for obtaining insight into what might be the range
of bias operating in other fields where similar analytical methods,
technologies, and conflicts may be operating.
.. . . .
How Can We Improve the Situation?
Is it unavoidable that most research findings are false, or can we
improve the situation? A major problem is that it is impossible to
know with 100% certainty what the truth is in any research question.
In this regard, the pure "gold" standard is unattainable. However,
there are several approaches to improve the post-study probability.
Better powered evidence, e.g., large studies or low-bias meta-
analyses, may help, as it comes closer to the unknown "gold" standard.
However, large studies may still have biases and these should be
acknowledged and avoided. Moreover, large-scale evidence is impossible
to obtain for all of the millions and trillions of research questions
posed in current research. Large-scale evidence should be targeted for
research questions where the pre-study probability is already
considerably high, so that a significant research finding will lead to
a post-test probability that would be considered quite definitive.
Large-scale evidence is also particularly indicated when it can test
major concepts rather than narrow, specific questions. A negative
finding can then refute not only a specific proposed claim, but a
whole field or considerable portion thereof. Selecting the performance
of large-scale studies based on narrow-minded criteria, such as the
marketing promotion of a specific drug, is largely wasted research.
Moreover, one should be cautious that extremely large studies may be
more likely to find a formally statistical significant difference for
a trivial effect that is not really meaningfully different from the
null [32-34].
Second, most research questions are addressed by many teams, and it is
misleading to emphasize the statistically significant findings of any
single team. What matters is the totality of the evidence. Diminishing
bias through enhanced research standards and curtailing of prejudices
may also help. However, this may require a change in scientific
mentality that might be difficult to achieve. In some research
designs, efforts may also be more successful with upfront registration
of studies, e.g., randomized trials [35]. Registration would pose a
challenge for hypothesis-generating research. Some kind of
registration or networking of data collections or investigators within
fields may be more feasible than registration of each and every
hypothesis-generating experiment. Regardless, even if we do not see a
great deal of progress with registration of studies in other fields,
the principles of developing and adhering to a protocol could be more
widely borrowed from randomized controlled trials.
Finally, instead of chasing statistical significance, we should
improve our understanding of the range of R values--the pre-study odds--
where research efforts operate [10]. Before running an experiment,
investigators should consider what they believe the chances are that
they are testing a true rather than a non-true relationship.
Speculated high R values may sometimes then be ascertained. As
described above, whenever ethically acceptable, large studies with
minimal bias should be performed on research findings that are
considered relatively established, to see how often they are indeed
confirmed. I suspect several established "classics" will fail the test
[36].
Nevertheless, most new discoveries will continue to stem from
hypothesis-generating research with low or very low pre-study odds. We
should then acknowledge that statistical significance testing in the
report of a single study gives only a partial picture, without knowing
how much testing has been done outside the report and in the relevant
field at large. Despite a large statistical literature for multiple
testing corrections [37], usually it is impossible to decipher how
much data dredging by the reporting authors or other research teams
has preceded a reported research finding. Even if determining this
were feasible, this would not inform us about the pre-study odds.
Thus, it is unavoidable that one should make approximate assumptions
on how many relationships are expected to be true among those probed
across the relevant research fields and research designs. The wider
field may yield some guidance for estimating this probability for the
isolated research project. Experiences from biases detected in other
neighboring fields would also be useful to draw upon. Even though
these assumptions would be considerably subjective, they would still
be very useful in interpreting research claims and putting them in
context.
.
- References:
- Most published research findings are false
- From: claudiusdenk
- Most published research findings are false
- Prev by Date: Re: Mikey addresses content
- Next by Date: Re: Mikey addresses content
- Previous by thread: Most published research findings are false
- Next by thread: Peer review is frequently a way of controlling debate
- Index(es):
Relevant Pages
|
|
