The problem with vaccine case-control studies

Written by Martin Neil and Norman Fenton on September 26, 2024. Posted in Current News

Before we go into vaccine case control studies the bias issue at hand is pretty easy to understand and it might surprise you to learn that you have probably already encountered it in everyday life, from dating and when forming friendships

Why do we believe ‘attractive’ people are more likely to be ‘mean’ than ‘nice’?

Let’s assume for simplicity that you classify people on their looks as ‘attractive’ or ‘ugly’ and on their personality as ‘nice’ of ‘mean’.

Then, the people you will tend to have dated or hang around with will either be attractive or nice.

If you are very lucky, they will be both attractive and nice. What that means is that among the attractive people you know best, mean people are over-represented.

So, the attractive people you know best are more likely to be mean than nice.

This is a simple example of Collider bias (also called Berkson’s paradox). It is illustrated in this short video:

Because we are relying on a dataset which over-represents some subjects and under-represents others, the ‘paradox’ can lead us to conclude that, for two completely unrelated factors (such as ‘looks‘ and ‘personality’) one has a causal influence on the other (‘attractive’ people are more likely to be ‘mean’).

In fact, more generally the paradox can also leads us to wrongly conclude that a true causal relationship between two factors is the opposite of what it is in reality. For example, ‘skipping lectures results in better grades’, ‘an effective treatment leads to worse outcomes’ or ‘an ineffective treatment leads to better outcomes’.

This is all illustrated in an extended version of the above video.

A case-control study – background

Now back to vaccine case control studies….

In April 2023 Xie et al published a research letter in JAMA comparing covid mortality with flu mortality, based on Veterans Affairs (VA) patients who had been hospitalised with either covid or flu (but not both).

Since the patient records included their vaccination status (for both covid and flu), the authors undertook an analysis which led them to the following conclusion about the covid vaccines:

The increased risk of death was greater among unvaccinated individuals compared with those vaccinated or boosted—findings that highlight the importance of vaccination in reducing risk of COVID-19 death.

Eyal Shahar provided a devastating response to the article in the Daily Sceptic. He noted that the analysis of vaccine efficacy can be considered as a classic ‘case-control’ study of hospitalised patients in which the covid patients were the ‘cases’ and the flu patients were the ‘controls’.

His own analysis of the data showed that, contrary to Xie et al’s conclusions, there was no significant evidence that the covid vaccines were either effective or safe.

I was aware of the Xie study at the time but paid little attention to it because its ‘reliability’ was based entirely on the laughably incorrect notion that patients were accurately diagnosed with covid or flu. We have provided evidence that many so-called ‘covid cases’ were just flu classified as covid.

The Kirsch et al paper

On 6 August 2024, Kirsch et al published a preprint which re-analysed the Xie et al data and, concluded that the findings “call into question any efficacy claims associated with either the COVID-19 or influenza vaccines”.

They also concluded that the Covid (and flu) vaccines do not reduce hospitalisation risk, and that:

“… comparing vaccination rates between those hospitalized for different vaccine-preventable diseases can serve as a practical method for validating the vaccine efficacy claims of different vaccine.”

In correspondence shared by Will Jones (of the Daily Sceptic) and Eyal Shahar, Will asked Eyal to comment on the Kirsch et al paper’s conclusions. While Eyal agreed with the general conclusion about lack of efficacy of the vaccines, he noted the following about the conclusion with respect to the effect of vaccine on risk of hospitalisation:

“When cases and controls are hospitalized, we cannot claim to have studied the vax effect on the risk of hospitalization. It is the effect on getting infected, conveniently studied in hospitalized patients. (By analogy, can we study some effect on smoking in a sample of only smokers?)”

Will’s response was:

“Isn’t the idea that a group hospitalised for flu can serve as a control for those hospitalised for Covid (and vice versa), to give an estimate of efficacy against hospitalisation?

If the Covid vaccination rate in the Covid-hospitalised is no different than the Covid vaccination rate in the flu-hospitalised then that implies the Covid vaccine is not protecting against Covid-hospitalisation. No?”

Collider bias in case control studies

Eyal pointed Will to this (rather technical) paper he had written about the effect of ‘collider bias’ in these kinds of case-control studies, and about the need to consider causal diagrams to understand what additional data were needed to avoid such bias.

Will asked me if I understood and agreed with Eyal’s concerns that the data based only on hospitalised patients could not be used to draw conclusions about the effect of vaccines on risk of hospitalisation.

I told Will that Eyal’s concern was valid and that I had also written to explain this in connection with some of the early studies of covid risk, such as the claim that smoking reduced the risk of severe covid which was subject to collider bias due to it being restricted to the set of people tested for covid.

With respect to the specific case discussed between Will and Eyal I showed that it is indeed theoretically possible that data based only on hospitalised patients apparently showing the vaccine is either effective or ineffective against hospitalisation with covid could be hiding the opposite ‘correct’ conclusion; so it could be showing it is effective while the ‘true’ effect is negative, and conversely it could be showing it is ineffective while the ‘true’ effect is positive.

There are other details in the Xie data that suggest no such reversal of the Kirsch et al conclusions (i.e. I believe their conclusions are correct), but I provided the following simple hypothetical example proving (using Bayes theorem) that such reversals are possible.

First of all, the causal diagram needed here is this one:

The ‘collider bias’ is introduced when we condition the results on the case where Hospitalised is ‘True’, i.e. where we only have data on hospitalised patients.

To see this, hypothetically, let us suppose that the vaccine is ineffective as follows (Note that I need to stress again that the following are purely hypothetical figures selected to demonstrate the collider bias paradox; there is nothing in the Xie data to suggest it is happening there):

• seven percent of those vaccinated are hospitalised with covid
• five percent of those unvaccinated are hospitalised with covid

Now suppose we have data from a case-control study of people who are hospitalised where we know the number who are hospitalised from covid or from something else. Suppose that, of those who hospitalised with ‘something else’:

• 15 percent are vaccinated
• 10 percent are unvaccinated

By Bayes (see Appendix below), it turns out that, since the study uses only data of those hospitalised (i.e. when we set the variable “Hospitalised” to always be True) we will find the following:

• 33.4 percent of those who are vaccinated are hospitalised with covid
• 34.5 percent of those who are unvaccinated are hospitalised with covid

In other words, based only on the data of hospitalised patients we would conclude (wrongly) that the unvaccinated are at a higher risk of being hospitalised with covid than the vaccinated.

As explained in my article, this does not mean that we cannot make the correct inferences about the effect of a treatment/risk factor on an outcome based only on a retrospective study of people hospitalised.

On the contrary, the causal model shows us exactly what additional estimates we need to make to avoid the bias; in this case we need to estimate what proportion of vaccinated/unvaccinated are hospitalised for any condition.

Appendix: Bayesian results

Using the Bayesian network calculation, we can compare the results of the ‘effect of vaccination’ both with and without the collider bias (that is introduced by using only data on those hospitalised).

First, we can see the unbiased effect of comparing the effect of vaccination on hospitalisation with covid:

The crucial thing to note here is that, based on the hypothetical data, those vaccinated are more likely (20.95 percent) to be hospitalised with ‘something’ than those unvaccinated (14.5 percent).

Next, we can see the (collider biased) effect of comparing the effect of vaccination on hospitalisation with covid when we restrict the calculation to only those who are hospitalised:

See how the original results of the effect of covid vaccination are ‘reversed’.

In this case those vaccinated are less likely (33.4 percent) to be hospitalised with covid than those unvaccinated (34.5 percent).

See more here substack.com

Bold emphasis added

Please Donate Below To Support Our Ongoing Work To Expose The Lies About Covid 19

PRINCIPIA SCIENTIFIC INTERNATIONAL, legally registered in the UK as a company incorporated for charitable purposes. Head Office: 27 Old Gloucester Street, London WC1N 3AX. 

Trackback from your site.