Rxivist logo

Bias, precision and timeliness of historical (background) rate comparison methods for vaccine safety monitoring: an empirical multi-database analysis

By Xintong Li, Lana Lai, Anna Ostropolets, Faaizah Arshad, Eng Hooi Tan, Paula Casajust, Thamir M Alshammari, Talita Duarte-Salles, Evan P. Minty, Carlos Areia, Nicole Pratt, Patrick B. Ryan, George M Hripcsak, Marc A Suchard, Martijn J Schuemie, DANIEL PRIETO-ALHAMBRA

Posted 14 Jul 2021
medRxiv DOI: 10.1101/2021.07.10.21258463

Using real-world data and past vaccination data, we conducted a large-scale experiment to quantify bias, precision and timeliness of different study designs to estimate historical background (expected) compared to post-vaccination (observed) rates of safety events for several vaccines. We used negative (not causally related) and positive control outcomes. The latter were synthetically generated true safety signals with incident rate ratios ranging from 1.5 to 4. Observed vs. expected analysis using within-database historical background rates is a sensitive but unspecific method for the identification of potential vaccine safety signals. Despite good discrimination, most analyses showed a tendency to overestimate risks, with 20%-100% type 1 error, but low (0% to 20%) type 2 error in the large databases included in our study. Efforts to improve the comparability of background and post-vaccine rates, including age-sex adjustment and anchoring background rates around a visit, reduced type 1 error and improved precision but residual systematic error persisted. Additionally, empirical calibration dramatically reduced type 1 to nominal but came at the cost of increasing type 2 error. Our study found that within-database background rate comparison is a sensitive but unspecific method to identify vaccine safety signals. The method is positively biased, with low (<=20%) type 2 error, and 20% to 100% of negative control outcomes were incorrectly identified as safety signals due to type 1 error. Age-sex adjustment and anchoring background rate estimates around a healthcare visit are useful strategies to reduce false positives, with little impact on type 2 error. Sufficient sensitivity was reached for the identification of safety signals by month 1-2 for vaccines with quick uptake (e.g., seasonal influenza), but much later (up to month 9) for vaccines with slower uptake (e.g., varicella-zoster or papillomavirus). Finally, we reported that empirical calibration using negative control outcomes reduces type 1 error to nominal at the cost of increasing type 2 error.

Download data

  • Downloaded 88 times
  • Download rankings, all-time:
    • Site-wide: 154,304
    • In epidemiology: 6,357
  • Year to date:
    • Site-wide: 94,541
  • Since beginning of last month:
    • Site-wide: 45,980

Altmetric data

Downloads over time

Distribution of downloads per paper, site-wide