The letter by dr.Cleophas permits me to carry the discussion one level deeper. The underlying assumption of his reasoning seems to be that all research that is called ‘randomized’ is by that very name suddenly worthwhile and its results should be published or at least made available. In particular, he refers to so-called ‘promotional trials’ that are either published or not, depending on the outcome. Let me reassure Cleophas that the best attitude towards such trials is to completely ignore them, whether they are published or not. They are usually readily recognized as they appear in (supplements of) journals with low impact scores, and have certain peculiarities: the few times that I had to pay attention to such publications, it was immediately obvious to me that they were badly written, had a hazy endpoint definition, unclear tables, numbers that did not add, and even hazier definitions of adverse outcomes. Most of them probably will only be randomized in name, and not in fact. What the worth of their ‘informed consent’ is, I do not dare to imagine. Invariably they claim substantial effects and no side effects. Knowing this information makes no difference, since it is no information at all. The demarcation therefore is not between ‘randomized’ or not, but between good and bad research. Good research is usually published or made known, and in good journals too, so one might restrict one's efforts to reading those.
One problem with certain current dogma's on meta-analysis is that one feels obliged to take ‘all data’ into account, as if they all really would count. Thereby, one no longer enjoys the old privilege of simply ignoring the corrupt, the stupid, or the impossible (like trials on homeopathy). Cleophas makes one exception: maybe ‘small’ trials should be ignored. There, I strongly disagree. The role of small trials has been undervalued, and often put in a wrong light by a certain emphasis on ‘sample size’ and p-values. An amusing discussion on small trials, and the real problem of the type III error: telling that something works when it is actually detrimental (which is different from saying that it works while it has no effect) can be found in the paper by Powell-Tuck et al.1
J.P. Vandenbroucke
Literatuur
Powell-Tuck J, MacRae KD, Healy MJR, Lennard-Jones JE, Parkins RA. A defence of the small clinical trial: evaluation of three gastroenterological studies. BMJ 1986;292:599-602.
Is selective reporting of well-designed clinical research unethical as well as unscientific?
Leiden, february 1996,
The letter by dr.Cleophas permits me to carry the discussion one level deeper. The underlying assumption of his reasoning seems to be that all research that is called ‘randomized’ is by that very name suddenly worthwhile and its results should be published or at least made available. In particular, he refers to so-called ‘promotional trials’ that are either published or not, depending on the outcome. Let me reassure Cleophas that the best attitude towards such trials is to completely ignore them, whether they are published or not. They are usually readily recognized as they appear in (supplements of) journals with low impact scores, and have certain peculiarities: the few times that I had to pay attention to such publications, it was immediately obvious to me that they were badly written, had a hazy endpoint definition, unclear tables, numbers that did not add, and even hazier definitions of adverse outcomes. Most of them probably will only be randomized in name, and not in fact. What the worth of their ‘informed consent’ is, I do not dare to imagine. Invariably they claim substantial effects and no side effects. Knowing this information makes no difference, since it is no information at all. The demarcation therefore is not between ‘randomized’ or not, but between good and bad research. Good research is usually published or made known, and in good journals too, so one might restrict one's efforts to reading those.
One problem with certain current dogma's on meta-analysis is that one feels obliged to take ‘all data’ into account, as if they all really would count. Thereby, one no longer enjoys the old privilege of simply ignoring the corrupt, the stupid, or the impossible (like trials on homeopathy). Cleophas makes one exception: maybe ‘small’ trials should be ignored. There, I strongly disagree. The role of small trials has been undervalued, and often put in a wrong light by a certain emphasis on ‘sample size’ and p-values. An amusing discussion on small trials, and the real problem of the type III error: telling that something works when it is actually detrimental (which is different from saying that it works while it has no effect) can be found in the paper by Powell-Tuck et al.1
Powell-Tuck J, MacRae KD, Healy MJR, Lennard-Jones JE, Parkins RA. A defence of the small clinical trial: evaluation of three gastroenterological studies. BMJ 1986;292:599-602.