Review of the Cleveland Clinic Study: "Effectiveness of the Coronavirus Disease 2019 (COVID-19) Bivalent Vaccine"

A Peer Review Style pre-print Analysis

INTRODUCTION

It is important to continue to monitor the ongoing effectiveness of COVID-19 vaccination. The pre-print study "Effectiveness of the Coronavirus Disease 2019 (COVID-19) Bivalent Vaccine" Shrestha, et al. Dec 19th, 2022, medRxiv follows incidence of COVID-19 infection in employees of the Cleveland Clinic in Ohio after booster administration. This study provides an important first look at the real-world clinical effectiveness of the bivalent boosters, therefore it is critical that the data collected be carefully analyzed before claims are generated regarding vaccine efficacy. The intent of this pre-print review is to pose helpful questions, provide critical analysis, and identify potential adjustments prior to submission of this publication for further peer review.

This article in its current form has caught the attention of social media, with current ‘Tweets’ of the article over 28,500 at the time of this writing. Therefore, because of the importance to public perception regarding booster effectiveness this analysis is being published as a public document.

Bivalent refers to the vaccine containing the BA.4/5 antigen and antigen from the original variant of SARS-CoV-2.

AUTHORS SUMMARY OF RESULTS

“Summary Among 51,011 working-aged Cleveland Clinic employees, the bivalent COVID-19 vaccine booster was 30% effective in preventing infection, during the time when the virus strains dominant in the community were represented in the vaccine.”

During the 13 week study a total of 2,452 infections occurred (5% of the cohort). Prior infection was determined by a positive NAAT test. It is unclear how positive test information was collected during the study.

The 30% effectiveness in preventing infection was determined by Multivariable Cox proportional hazards regression modeling.

CONCLUSIONS OF GENERAL INTEREST

Not surprisingly, the authors determined that risk of COVID-19 infection increased as time increased relative to prior infection. This is confirmed by numerous other studies. The authors correctly state that this may be due to a time dependent reduction in antibody levels post infection. The authors, however, fail to mention that variant development over time may contribute to increased infection rates via increased immune evasion.

Surprisingly, the authors state that despite 30% effectiveness of the booster vaccine (but fail to indicate what this is relative to in their data) the increased risk of COVID-19 infection increased with the number of total vaccine doses.

STUDY LIMITATIONS, CONFOUNDING FACTORS, AND SUGGESTED CHANGES

1. Failure to identify comorbidities that increase infection rates.

The authors state:

“Our study of healthcare personnel included no children and few elderly subjects, and the majority would not have been immunocompromised.”

“Institutional data governance rules related to employee data limited our ability to supplement our dataset with additional clinical variables.”

These assumptions and failure to include the following clinical variables may significantly impact data analysis and interpretation:

• The average age of the groups was 42 years old and predominantly female (74.6%). Women have higher rates of autoimmune disease relative to men (10:1 to 2:1 depending on the disease) with an average onset age typically in the mid-30s. Therefore, it is possible that the female cohort contained significantly higher proportion of individuals with active autoimmune disease. If these individuals were undergoing treatment, this would likely be considered immunosuppressive to varying degrees, increasing susceptibility to infection relative to vaccination status-matched cohorts.

• Data skewing based on gender may be evidenced in their finding that the gender ‘male’ provided an advantage regarding infection: adjusted HR of 0.78 (95% CI 0.71-.86, p < 0.001). It has been prior reported that there is identical susceptibility between male and females regarding initial COVID infection (Representative reference: Gender Differences in Patients With COVID-19: Focus on Severity and Mortality). Therefore, without an explanation as to why males were less susceptible to initial infection in this study, it must be assumed that the female cohort had unaccounted for factors that significantly increased infection susceptibility.

• Breakthrough rates of COVID-19 infection are higher in immunocompromised individuals even when fully vaccinated. COVID-19 Breakthrough Infection Among Immunocompromised Persons

• Immunocompromised individuals are often aware of the their status regarding increased risk of both initial infection and severe infection. This awareness may be particularly true of employees of a major hospital system who likely have more medical education or awareness than a random sample population. Immunocompromised individuals, in general, are more likely to be up-to-date with vaccines and boosters as this is strongly encouraged by the medical community. Therefore without a gender and risk-matched vaccination cohort the data is difficult to interpret.

The authors state that they would expect less risky behavior (more COVID avoidance) from individuals who are up-to-date with vaccines and the bivalent booster. However, avoiding exposure may not be possible for a patient facing health-care worker, of which 70-80% may be female (Labor Force Statistics from the Current Population Survey). A “Clinical Job” was associated with higher infection risk in the data presented. Though the authors do analyze rates of infection among health care workers, they fail to identify potentially high-risk individuals who are typically up-to-date with vaccination.

Given the low statistical power of the results and high proportion of the study population (3:1) having a 2-10X higher rate of a potentially confounding diagnosis and the unusual 'protective’ effect of the male gender from infection in the data set, these results must not be broadly interpreted until matched cohorts with defined clinical variables are assessed.

• Cohort is skewed 3:1 towards a population known to have a 2-10X higher rate of a potentially confounding diagnosis and treatment course.

• Male gender ‘protective effect’ from initial infection is unsubstantiated in known literature and may indicate an impact of cohort skewing on other results.

Suggested publication adjustments: Cohorts should be re-analyzed accounting for data that includes the potential confounding factors described above. If this is not possible, a gender matched cohort should be analyzed and vaccination status should be balanced between genders. If the surprising effect of lower rates of infection in males is not eliminated by these changes, the authors should address other confounding factors such as possible lower rates of employment in patient facing roles etc. At a minimum an acknowledgement of incomplete data and skewed study cohorts should be included in the discussion of study limitations.

2. It is unclear which groups in the cumulative incidence of COVID-19 data presented (Figure 2) received the bivalent vaccine. The 3 or 3+ doses show significant statistical overlap at the 95% confidence interval as well as significant overlap with the 2 dose cohort throughout the majority of the study period. This indicates weak if any statistically significant difference. Therefore, the conclusion that vaccination beyond 2 doses leads to increased infection based on this data alone, not withstanding other caveats, is not supported by the presented data in Figure 2 or in Table 2 (below).

Figure 2:

Table 2:

Furthermore, this data fails to show prior infection in vaccinated and unvaccinated groups. An analysis should be performed based on groups separated by prior infection history and vaccination status, this is important for two reasons:

1. It is well known that vaccination reduced infection by prior variants. Lack of recent prior infection due to prior vaccination may increase relative susceptibility to new variants as indicated by the study data. This is critical towards the end of the study where BQ.1 and BQ1.1 were the predominant variants that have higher immune evasiveness relative to BA4/5 where the least protective effect of 3 and 3+ doses was seen. Again, separating groups based on receipt of the bivalent booster would clarify the findings.

2. If the groups seeking initial vaccination were skewed towards higher susceptibility due to exposure rates or comorbidities, the arrival of highly immune evasive variants, particularly during the later half of the study, would increase rates of infection in groups that failed to obtain the bivalent booster and/or a 3rd or more dose. Therefore, the data in the two dose cohort as presented is unlikely to reflect risk induced by vaccination alone.

ADDITIONAL POINTS:

• Study start date of September 12th represents the first day of bivalent vaccine viability. The authors correctly indicate that 'positive vaccination status’ should not be assumed until day 7. Two issues arise from this:

  1. Exposure to virus within the 7 days post vaccination, prior to ‘flipped vaccination status’ (per author’s description) would likely not yield a positive test until after ‘flipped’ vaccination status and would not indicate failure of the vaccine but rather unfortunate exposure timing. Therefore, a number of days (roughly 5-7) immediately post ‘flipped’ vaccine status should be removed from the analysis as COVID-19+ boosted individuals as the bivalent booster is known to not be effective during that exposure period.

  2. In the graphical representation of the data, in both figures it is indicated that COVID-19+ is counted from ‘Day 0’ or first day of bivalent availability. This is inconsistent with the text that describes data analysis regarding the bivalent booster ‘flipped’ status being 7 days post vaccination. It should be clarified in all analysis which timepoint is being used by the authors in their analysis to represent ‘flipped’ bivalent vaccination status.

• Graphical data analysis fails to show bivalent booster data separated from 2 or 3+ (likely booster) data, and therefore does not support the authors conclusions that more vaccinations increase the risk of infection. Labeling should ensure it is clear regarding bivalent recipients versus others who already had 3 or 4 shots as a result of high-risk of exposure or immunocompromised status.

• In the discussion two research papers are cited as a potential explanation of mechanism regarding the authors conclusions. However, the findings of both articles call into question the results of this study referencing the findings.

  1. “A large study found that those who had an Omicron variant infection after previously receiving three doses of vaccine had a higher risk of reinfection than those who had an Omicron variant infection after previously receiving two doses of vaccine [21].” *Note, the number links the study in discussion.

  • This pre-print study fails to show that 3 vaccinations increase the risk of reinfection relative to no vaccination (see table below) until additional hazard ratio adjustments based on the number of tests performed shifted the study results to be significant. This may only indicate increased rates of asymptomatic infection among the 3 dose vaccination group which performed fewer tests. Indeed, there was no severe illness reported in the study.

    

  • The authors state: “The results of this additional study confirm the relative differences in incidence of reinfection observed in the first two studies, with incidence being lowest among the two-dose cohort and highest among the unvaccinated cohort.” Therefore, the study failed to indicate that there is increased risk of infection in those that receive two doses of the vaccine relative to no vaccine doses, which is directly counter to the findings of presented in the study from the Cleveland Clinic.

2. Another study found that receipt of two or three doses of a mRNA vaccine following prior COVID-19 was associated with a higher risk of reinfection than receipt of a single dose [7]. *Note, the number links the study in discussion.

According to the findings of the cited study (in reference 7), the authors indicate that “Risk of COVID-19 increased with time since proximate SARS-CoV-2 exposure, and boosting protected those >6 months since prior infection or vaccination” which is potentially counter to the findings of the authors citing this study (they did not provide time-frame since last vaccination) and potentially an erroneous citation as it does not support the statement made.

The referenced study results continue: “In multivariable analysis, boosting was independently associated with lower risk of COVID-19 among those vaccinated but not previously infected (hazard ratio [HR], .43; 95% confidence interval [CI], .41–.46) as well as those previously infected (HR, .66; 95% CI, .58–.76).” This is both counter to the study refencing this article’s results regarding their findings and counter to the findings of the original study. It is clearly shown in Figure 3 (below) that boosting is protective from infection.

Finally the referenced study states: “Among those previously infected, receipt of 2 compared with 1 dose of vaccine was associated with higher risk of COVID-19 (HR, 1.54; 95% CI, 1.21–1.97).” Where in Figure 4 the data shows that two doses is no more protective than zero doses of the vaccine during the Omicron wave, indicating the importance of a 3rd dose or booster for reducing infection, counter to the study citing this article.

It is expected that two prior doses of the vaccine would not be as protective as three doses during the Omicron wave, especially as it may have been protective against prior infection relative to 0 doses, increasing the likelihood of exposure immunity in the 0 and 1 dose groups.

• It is unclear how positive tests were confirmed or how testing was performed during the study. It is also unclear if testing was at the behest of the study subject or performed at regular intervals. The authors should provide this information.

• A generous time of 90 days between infections was granted and two positive tests in this period were not counted as two separate events. Repeat infections, often with different variants during this period of the pandemic have been documented as early as 21 days. This may skew results as it may miss reinfections during the 13 week period.

SUMMARY

Overall, the data presented in the study is self-contradictory The bivalent booster was found to confer significant, albeit reduced protection from infection while patient cohorts with 3 or more vaccinations were found to have a higher risk of infection than those who had 2 or fewer doses and did not receive the bivalent vaccine. References used to support the authors findings directly contradict their conclusions. The suggested adjustments above may resolve this contradiction in the study data. The authors should address this significant contradiction in their findings before final publication.

Further research that resolves occupational exposure, cohort skewing, self-reporting errors, and demonstrates clear statistical significance is critical before making assessments regarding bivalent booster efficacy and infection risks.