In the USA, the CDC (Centers for Disease Control and Prevention) compiles data on national vaccine uptake, but reporting can sometimes be delayed. Surveys that measure attitudes and behavior towards COVID-19 vaccines can fill a gap when there is a lag in real-time data, and can inform government responses to the epidemic. However, some surveys diverge substantially in their findings.
The authors of a new paper published in Nature today find that in May 2021, Delphi–Facebook’s COVID-19 symptom tracker (250,000 responses per week) overestimated vaccine uptake by 17 percentage points, and a survey from the US Census Bureau (75,000 responses per wave) overestimated vaccine uptake by 14 percentage points compared to benchmark estimates from the CDC.
These overestimation errors go orders of magnitude beyond the statistical uncertainty provided by the surveys. A survey by Axios–Ipsos also overestimated uptake, but by a smaller amount (4.2 percentage points in May 2021)—despite being the smallest survey (about 1,000 responses per week). These findings indicate that bigger is not always better when it comes to datasets, if we fail to account for data quality.
The authors note that COVID-19 vaccine uptake was not the primary focus of any of the surveys. For example, Delphi-Facebook intends to measure changes in COVID-related behaviors over time. However, the bias in estimates of vaccine uptake in the two large surveys indicates that they are not representative of the US adult population. Lack of statistical representativeness may also be causing bias in other survey outcomes they suggest.