Statistical Considerations for Evaluating COVID Vaccine Protection

I refer to a recent Morbidity and Mortality Weekly Report (MMWR) to highlight a deficit in the reporting of statistical data: The failure to address the implications of the 95% confidence interval (95% CI).

The report pertains to estimates of COVID-19 vaccine effectiveness (VE) against emergency department or urgent clinic encounters, and hospitalizations among adults during SARS-CoV-2 (Delta) variant predominance. Regarding VE, the 95% CI is a measure of uncertainty associated with the level of VE obtained from the actual sample studied (the point estimate). Of course, other levels of confidence can be used for interval estimates, but 95% is typical in public health and medicine.

The 95% CI is used for generalizing or making an inference about the population, and can be pragmatically defined as a range of values around the point estimate that has a 95% probability of including the true level of VE for the population. The stricter and more theoretical definition and interpretation of confidence interval is based on “the hypothetical notion of considering the results that would be obtained if the study were repeated many times.” The size of the sample or subgroup is a key determinant of the precision of the 95% CI — smaller samples result in wider confidence intervals, which are less precise.

Best practice in statistical analysis and reporting regarding vaccine efficacy and effectiveness requires that both the 95% CI and the point estimate be fully interpreted and explained. However, it is not uncommon for researchers to merely mention the 95% CI in the methodology or results section of a report without addressing its implications for decision-making, while emphasizing the point estimate. This is a grave disservice to all.

Regarding the MMWR in question, while the point estimates of VE (mRNA) are very high, the lower limit of the 95% CI plausibly indicates that a lower level of VE can be expected if repeated studies were conducted on this population, and this is largely attributable to variation in research contexts, including personal characteristics. If this is not effectively communicated to the public, should the level of VE decrease in subsequent studies, the public may incorrectly interpret this as vaccine failure, when in fact, the lower level of VE may be accounted for by the 95% confidence intervals from previous studies.

Indeed, the MMWR supports that VE during the Delta wave “was significantly higher among Moderna vaccine recipients (95%) than among Pfizer-BioNTech (80%) or Janssen (60%) vaccine recipients.” Specifically, regarding COVID-19 hospitalization, the report notes the following VE point estimates:

• mRNA-1273 (Moderna): 95% (95% CI 92-97)
• BNT162b2 (Pfizer-BioNTech): 80% (95% CI 73-85)
• Ad26.COV2.S (Janssen/Johnson & Johnson): 60% (95% CI 31-77)

The lower limit of the 95% CI indicates that the true value of the level of VE can be as low as 92% for Moderna, 73% for Pfizer-BioNTech, and 31% for Janssen, which albeit relatively low, marginally achieves the lower bound of “>30%” generally required by the FDA to consider a vaccine worthy of being widely deployed. Attention to the lower limit of the 95% CI is a conservative approach to assessing VE. While the Moderna and Pfizer-BioNTech mRNA vaccines demonstrated high levels of protection against moderate and severe COVID-19 resulting in hospitalization, the Moderna vaccine outperformed the other two vaccines, and the difference was statistically and substantively significant. This trend was reinforced in a subsequent report which noted that: “Among U.S. adults without immunocompromising conditions, VE against COVID-19 hospitalization was higher for the Moderna vaccine (93%) than the Pfizer-BioNTech vaccine (88%) and the Janssen vaccine (71%).”

Consequently, the conclusion that, “These findings reaffirm the high protection of COVID-19 vaccines against moderate and severe COVID-19 resulting in , [urgent clinic], and hospital visits” is too general, as it suggests that these three vaccines are similarly “high” in terms of the level of effectiveness against hospitalization, which is not supported by the statistical data. Notably, the 95% CI for Janssen is very wide, and hence not precise, and therefore, it may not be a reliable estimate of VE in this context. Specifically, this wider 95% CI is a function of the relatively small number of participants in the analysis who received the Janssen vaccine. It’s important to note, however, that the Moderna, Pfizer-BioNTech, and Janssen vaccines do show evidence of “continued strong protection against COVID-19-associated hospitalization and death.”

The 95% CI is a probabilistic statement of uncertainty, which is necessary for meaningful determination of vaccine efficacy and effectiveness, however, it is widely misunderstood, misinterpreted, and underreported. Also, while acknowledging the constraints to generalizability, these data add to the emerging and consistent evidence of variability in the effectiveness of the COVID-19 vaccines, amidst growing concerns about waning vaccine-induced immunity and an increase in the number of breakthrough cases during the Delta wave. Consistent in this regard, last week, an FDA advisory panel “recommended emergency use authorization (EUA) of a booster dose of the Pfizer/BioNTech COVID-19 vaccine (Comirnaty) for individuals 65 and older, and those judged to be at high risk of severe COVID-19,” and yesterday, FDA authorized a third dose of the Pfizer vaccine for these groups. As we move forward with the booster dose in certain populations, the variability in the effectiveness of the COVID-19 vaccines is sure to fuel the debate about the need for heterologous boosting, also known as mix and match.

Rossi A. Hassad, PhD, MPH, is an epidemiologist and professor at Mercy College, in Dobbs Ferry, New York. He is a Fellow of the American College of Epidemiology and a Chartered Statistician of Britain’s Royal Statistical Society.