Image Credit: Scientific Frontline / stock image
Scientific Frontline: Extended "At a Glance" Summary: Simulated Epidemics and Hospitalization Forecasts
The Core Concept: Researchers have developed a new method combining multiple mathematical models into an "ensemble forecast" to predict hospital admissions during pandemics more reliably.
Key Distinction/Mechanism: Instead of relying on a single model, this method weighs several models differently based on the disease's spread rate and their past performance under similar simulated conditions, allowing for a broader range of possibilities.
Major Frameworks/Components:
- Utilization of synthetic data from 324 simulated epidemics generated by a complex, agent-based model of COVID-19 disease transmission.
- Evaluation of 14 individual mathematical models to assess performance under various conditions, such as human mobility and disease characteristics.
- Development of six different ensemble methods, with the most accurate one assigning dynamic weights to individual models based on their situational efficacy.
- Analysis of differences between individual model forecasts to gauge the reliability of the combined ensemble forecast.
Branch of Science: Mathematical Modeling, Epidemiology, and Public Health.
Future Application: The method can be utilized in future pandemics to provide decision-makers with more accurate forecasts of hospital bed requirements and an indication of the forecast's uncertainty.
Why It Matters: Early in an outbreak, limited data creates high uncertainty. This method mitigates that uncertainty, improving the reliability of crucial healthcare resource planning and demonstrating that analyzing the variability among different models provides valuable insight into overall forecast reliability.
A mathematical model is a simplified description of reality that can be used to solve practical problems, predict future events, or explain how different systems work.
During pandemics, mathematical models can be valuable tools for simulating disease transmission and, among other things, forecasting how many hospital beds will be needed. Early in an outbreak, however, there is often considerable uncertainty about how the disease spreads, resulting in uncertain forecasts.
“At the beginning of a pandemic, the amount of data available is very limited. This creates uncertainty in both the models and their assumptions,” says Philip Gerlee, professor in the Department of Mathematical Sciences at Chalmers University of Technology and the University of Gothenburg.
Combining Models Produces More Reliable Results
To investigate how different models perform under different conditions, researchers in the Department of Mathematical Sciences at Chalmers University of Technology and the University of Gothenburg, in collaboration with École Polytechnique in France, generated synthetic data from 324 simulated epidemics.
The epidemics were generated using a complex agent-based model of disease transmission based on COVID-19.
“By varying factors such as the characteristics of the disease and human mobility, we were able to create a wide variety of epidemic scenarios,” says Philip Gerlee.
The tests showed that none of the 14 individual models evaluated performed best in every situation. Their performance varied depending on factors such as how quickly the disease was spreading. The researchers therefore also developed a new method that combines forecasts from different models into a so-called ensemble forecast.
“This allows you to cover a wider range of possibilities rather than relying on a single model,” says Philip Gerlee.
Ensemble methods have previously been shown to produce more reliable forecasts than individual models. What is novel about the new method is that the different models are given different weights depending on how quickly the disease is spreading and how well the models performed under similar conditions in the simulated epidemics.
“With our method, you could say that we listen most closely to the model that has proved to work best under the specific conditions at that point in time, although the other models still have some influence,” says Philip Gerlee.
Of the six different ensemble methods the researchers tested on synthetic data, the new method was the most accurate. It also performed well when tested on real data from the COVID-19 pandemic.
At the same time, the tests showed that a more advanced method does not necessarily produce the most accurate forecast in every situation.
“Although our new method performed best on synthetic data, it was surprising that the simplest possible method for creating an ensemble forecast—taking the median of all the individual model forecasts—performed almost as well, and even better on real data,” says Philip Gerlee.
Differences Can Indicate Uncertainty
The researchers were also able to show that differences between the forecasts produced by different models can provide information about the reliability of the combined forecast.
“By statistically analyzing the difference between ensemble forecasts and the actual outcomes, we see that when the individual model forecasts differ substantially, there is, on average, a larger discrepancy between the ensemble forecast and the outcome,” says Philip Gerlee.
The differences between the model forecasts could therefore provide an indication of when uncertainty in the combined forecast is greater—information that could be valuable to decision-makers using the forecast.
“We believe that our results could be useful in future pandemics. Synthetic data can be used to evaluate the suitability of existing models, while the relationship between variability within an ensemble and forecast accuracy can provide important information about the reliability of the combined forecast,” says Philip Gerlee.
Published in journal: Communications Medicine
Title: Evaluation of respiratory disease hospitalisation forecasts using synthetic outbreak data
Authors: Grégoire Béchade, Torbjörn Lundh, and Philip Gerlee
Source/Credit: Chalmers University of Technology | Julia Romell
Edited by: Scientific Frontline
Reference Number: cosc091526_01
