comparing human and model based forecasts of covid 19 in germany and poland CORD-Papers-2022-06-02 (Version 1)

Title: Comparing human and model-based forecasts of COVID-19 in Germany and Poland
Abstract: Forecasts based on epidemiological modelling have played an important role in shaping public policy throughout the COVID-19 pandemic. This modelling combines knowledge about infectious disease dynamics with the subjective opinion of the researcher who develops and refines the model and often also adjusts model outputs. Developing a forecast model is difficult resource- and time-consuming. It is therefore worth asking what modelling is able to add beyond the subjective opinion of the researcher alone. To investigate this we analysed different real-time forecasts of cases of and deaths from COVID-19 in Germany and Poland over a 1-4 week horizon submitted to the German and Polish Forecast Hub. We compared crowd forecasts elicited from researchers and volunteers against a) forecasts from two semi-mechanistic models based on common epidemiological assumptions and b) the ensemble of all other models submitted to the Forecast Hub. We found crowd forecasts despite being overconfident to outperform all other methods across all forecast horizons when forecasting cases (weighted interval score relative to the Hub ensemble 2 weeks ahead: 0.89). Forecasts based on computational models performed comparably better when predicting deaths (rel. WIS 1.26) suggesting that epidemiological modelling and human judgement can complement each other in important ways.
Published: 2021-12-05
DOI: 10.1101/2021.12.01.21266598
DOI_URL: http://doi.org/10.1101/2021.12.01.21266598
Author Name: Bosse N I
Author link: https://covid19-data.nist.gov/pid/rest/local/author/bosse_n_i
Author Name: Abbott S
Author link: https://covid19-data.nist.gov/pid/rest/local/author/abbott_s
Author Name: Bracher J
Author link: https://covid19-data.nist.gov/pid/rest/local/author/bracher_j
Author Name: Hain H
Author link: https://covid19-data.nist.gov/pid/rest/local/author/hain_h
Author Name: Quilty B J
Author link: https://covid19-data.nist.gov/pid/rest/local/author/quilty_b_j
Author Name: Jit M
Author link: https://covid19-data.nist.gov/pid/rest/local/author/jit_m
Author Name: Centre for the Mathematical Modelling of Infectious Diseases COVID Working Group
Author link: https://covid19-data.nist.gov/pid/rest/local/author/centre_for_the_mathematical_modelling_of_infectious_diseases_covid_working_group
Author Name: van Leeuwen E
Author link: https://covid19-data.nist.gov/pid/rest/local/author/van_leeuwen_e
Author Name: Cori A
Author link: https://covid19-data.nist.gov/pid/rest/local/author/cori_a
Author Name: Funk S
Author link: https://covid19-data.nist.gov/pid/rest/local/author/funk_s
sha: 11c56adfc0b2ea9947cf2d73bee59b8cc0bf2ea3
license: medrxiv
source_x: MedRxiv; WHO
source_x_url: https://www.who.int/
url: http://medrxiv.org/cgi/content/short/2021.12.01.21266598v1?rss=1 https://doi.org/10.1101/2021.12.01.21266598
has_full_text: TRUE
Keywords Extracted from Text Content: German human COVID-19 Germany and 1-4 volunteers EpiForecasts participants ξ O φ Hub-ensemble-all PredictIt medRxiv preprint A.9 medRxiv preprint Figure 19 Hub-ensemble-with-all A.9.1 Funk 90%coverage over-than R t . Germany Figure 3 − log(value2 Hub-ensemble-mean Viboud log(value3 over-or innermost Hub-ensemble-real hub-ensemble B, D Germany and Figure 4 19B Hub-ensemble-with-renewalmean meta-science Hub-ensemble-with-all-mean medRxiv preprint Figure 9 Hub Stan Development Team, 2020) with r obs Covid-19 scoringutils R Figure 1C Figure 2 people B over-or underpredict Bias φ ∼ 1 volunteers medRxiv preprint 17 Hub-ensemble-with-crowd hard-to-predict Figure 1 F (y) UK epiforecasts.io/covid participants German Hub-ensemble-withconvolution-mean Human Hub ensemble (= 1 Ärzteblatt ReplicationMarkets medRxiv preprint Figure 10 Figures 1H covid.german.forecasts R hub-ensemble-with-X Figure 1B Bracher − log(value3 Forsal.pl (2020) ) SI Ganyani US I t Hub-ensemble linelist α-quantiles line A.11 Line − 1 https://doi.org/10.1101/2021.12.01.21266598 doi Figures 1A O t Figures 6 and 7) EpiForecasts medRxiv preprint Figure 21 WIS human σ(log(value4 Hoogeveen CDF Wolffram 7G hub-ensemble-with-all medRxiv preprint Table 2 medRxiv preprint Figure 1B Deuschel I c t Figure 1A μ Bertsimas COVID-19 F medRxiv preprint Figure 13 Figure 6B Hub-ensemble-with-crowdmean 1A I obs SIR-type Recchia medRxiv preprint Hub-ensemble stan https://doi.org/10.1101 https://doi.org/10 medRxiv preprint A.9.2 Americas Germany ( Figure 7A H. upper https://cmmid-lshtm.shinyapps.io/crowd-forecast/ medRxiv |μ golem R Riutort-Mayol Figures 4B Görgen Bodych CSET Foretell humans 16/137/109 & 16/136/46 Centre NIB HPRU Bill 210758/Z/18/Z UK SIMCARD
Extracted Text Content in Record: First 5000 Characters:Forecasts based on epidemiological modelling have played an important role in shaping public policy throughout the COVID-19 pandemic. This modelling combines knowledge about infectious disease dynamics with the subjective opinion of the researcher who develops and refines the model and often also adjusts model outputs. Developing a forecast model is difficult, resource-and time-consuming. It is therefore worth asking what modelling is able to add beyond the subjective opinion of the researcher alone. To investigate this, we analysed different real-time forecasts of cases of and deaths from COVID-19 in Germany and Poland over a 1-4 week horizon submitted to the German and Polish Forecast Hub. We compared crowd forecasts elicited from researchers and volunteers, against a) forecasts from two semi-mechanistic models based on common epidemiological assumptions and b) the ensemble of all other models submitted to the Forecast Hub. We found crowd forecasts, despite being overconfident, to outperform all other methods across all forecast horizons when forecasting cases (weighted interval score relative to the Hub ensemble 2 weeks ahead: 0.89). Forecasts based on computational models performed comparably better when predicting deaths (rel. WIS 1.26), suggesting that epidemiological modelling and human judgement can complement each other in important ways. Since March 2020, forecasts of COVID-19 from multiple teams have been collected, aggregated and compared by Forecast Hubs such as the US Forecast Hub (Cramer et al., 2021; Cramer et al., 2020) , the German and Polish Forecast Hub (Bracher, We created and submitted the following forecasts to the German and Polish Forecast Hub: 1) a direct human forecast (henceforth called "crowd forecast"), elicited from participants through a web application (Bosse, Abbott, EpiForecasts, et al., 2020b) and 2) two semi-mechanistic model-based forecasts ("renewal model" and "convolution model") informed by basic assumptions about COVID-19 epidemiology. While the two semi-mechanistic forecasts were necessarily shaped by our implicit assumptions and decisions, they were designed such as to minimise the amount of human intervention involved. For example, we refrained from adjusting model outputs or refining the models based on past performance. Forecasts were created in real time over a period of 21 weeks from Infectious disease modelling has a long tradition and has helped inform public health decisions both through scenario modelling, as well as actual forecasts of (among others) influenza (Biggerstaff et al., 2016; e.g. McGowan et al., 2019; Shaman & Karspeck, 2012) , dengue fever (Colón-González et al., 2021; e.g. Johansson et al., 2019; Yamana et al., 2016) , ebola (Funk et al., 2019; e.g. Viboud et al., 2018) , chikungunya (e.g. Del Valle et al., 2018; Farrow et al., 2017) and now COVID-19 (Bracher, Wolffram, Deuschel, Görgen, Ketterer, Ullrich, Abbott, Barbarossa, Bertsimas, Bhatia, Bodych, Bosse, Burgard, Castro, et al., 2021; Bracher, Wolffram, Deuschel, Görgen, Ketterer, Ullrich, Abbott, Barbarossa, Bertsimas, Bhatia, Bodych, Bosse, Burgard, Fiedler, et al., 2021; Cramer et al., 2021; Cramer et al., 2020; European Covid-19 Forecast Hub, 2021; e.g. Funk et al., 2020) . Applications of epidemiological models differ in the way they make statements about the future. Forecasts aim to predict the future as it will occur, while scenario modelling and projections aim to represent what the future could look like under certain scenario assumptions or if conditions stayed the same as they were in the past. Forecasts can be judged by comparing them against observed data. Since it is much harder to fairly assess the accuracy and usefulness of projections and scenario modelling in the same way, this work focuses on forecasts, which represent only a subset of all epidemiological modelling. into a single forecast, e.g. by taking the mean or median of all forecasts. These ensemble forecasts usually tend to perform better and more consistently than individual forecasts (see e.g. Yamana et al. (2016) ; ). Individual computational models usually rely to varying degrees on mechanistic assumptions about infectious disease dynamics (such as SIR-type compartmental models that aim to represent how individuals move from being susceptible to infected and then recovered or dead). Some are more statistical in nature (such as time series models that detect statistical patterns without explicitly modelling disease dynamics). How exactly such a mathematical or computational model is constructed and which assumptions are made depends on subjective opinion and judgement of the researcher who develops and refines the model. Models are commonly adjusted and improved based on whether the model output looks plausible to the researchers involved. The process of model construction and refinement is laborious and time-consuming, and it is therefore worth asking what modelling can add beyond the subjective judgment o
PDF JSON Files: document_parses/pdf_json/11c56adfc0b2ea9947cf2d73bee59b8cc0bf2ea3.json
G_ID: comparing_human_and_model_based_forecasts_of_covid_19_in_germany_and_poland