Publication bias in action – it’s worse than you think
Publication bias is the phenomenon whereby a group with vested interests gets to influence which studies are published, or how they are characterised if they do get published
Let’s not beat around the bush; when a say “a group with vested interests” I really mean the pharmaceutical industry.
Most people have little idea how much influence the industry has on scientific publishing.
They don’t realise that (essentially) the papers reporting clinical trial results are not written by the authors named as such in the journals.
By way of a stark reminder of this, here’s a note that the psychiatrist David Healy found in Pfizer’s archive, as he told those of us who attended a talk he gave some years ago:
Those naive to the world of industrialised science might be surprised to see that a scientific paper reporting a clinical trial was “complete” while its author was still “TBD” (to be determined), but that’s the world we live in.
All the slides and commentary from David’s talk are available here. I highly recommend reading them, or otherwise watching the video.
David made some stunning points about SSRIs which I intend expanding upon in a future article or articles.
Re publication bias in particular, one of the points I made in my substack post (above) was that what makes it into journals – and what doesn’t – will have profound effects on the results of meta-analyses performed in the future.
I didn’t cite an actual example of this in that article, but via the material which David presented I came across this fascinating article from 2008:
In this study, the authors looked at all the trials submitted to the regulator for 12 anti-depressants, and determined whether and how each of those studies had been published to see if “positive” studies were treated any differently from “negative” ones.
These are the top-line results:
- They looked at 74 FDA-registered studies
- 38 studies were viewed by the FDA as “positive”. Of these, 37 (ie 97 percent) were published; one study viewed as positive was not published.
- 36 studies were viewed by the FDA as having negative or questionable results. Of these, 22 were not published and 11 were published in a way that conveyed a positive outcome. Only three (eight percent) were published in a way which conveyed the outcome as assessed by the regulator.
- According to the published literature, it appeared that 94 percent of the trials conducted were positive. By contrast, the FDA analysis showed that only 51 percent were positive.
- Separate meta-analyses of the FDA and journal data sets showed that the increase in effect size (caused by the publication bias) ranged from 11 to 69 percent for individual drugs and was 32 percent overall.
It’s important to recognise that the FDA’s determination of whether a study was to be viewed as positive or not was not necessarily reliable and / or free from industry influence.
It’s also important to note that what postive effects were reported did depend on trials which were conducted under the control of pharma, and thus subject to all the tricks with which we have become familiar.
In his talk, David referred to one particular such trick which – upon revisiting it yesterday – took my breath away in its brazenness, and about which I will write soon.
See more here substack.com
Header image: CartoonStock
Bold emphasis added