covid scraper an open source toolset for automatically scraping and processing global CORD-Papers-2021-10-25 (Version 1)

Title: COVID-Scraper: An open-source toolset for automatically scraping and processing global multi-scale spatiotemporal COVID-19 records
Abstract: In 2019, COVID-19 quickly spread across the world, infecting billions of people and disrupting the normal lives of citizens in every country. Governments, organizations, and research institutions all over the world are dedicating vast resources to research effective strategies to fight this rapidly propagating virus. With virus testing, most countries publish the number of confirmed cases, dead cases, recovered cases, and locations routinely through various channels and forms. This important data source has enabled researchers worldwide to perform different COVID-19 scientific studies, such as modeling this virus's spreading patterns, developing prevention strategies, and studying the impact of COVID-19 on other aspects of society. However, one major challenge is that there is no standardized, updated, and high-quality data product that covers COVID-19 cases data internationally. This is because different countries may publish their data in unique channels, formats, and time intervals, which hinders researchers from fetching necessary COVID-19 datasets effectively, especially for fine-scale studies. Although existing solutions such as John's Hopkins COVID-19 Dashboard and 1point3acres COVID-19 tracker are widely used, it is difficult for users to access their original dataset and customize those data to meet specific requirements in categories, data structure, and data source selection. To address this challenge, we developed a toolset using cloud-based web scraping to extract, refine, unify, and store COVID-19 cases data at multiple scales for all available countries around the world automatically. The toolset then publishes the data for public access in an effective manner, which could offer users a real time COVID-19 dynamic dataset with a global view. Two case studies are presented about how to utilize the datasets. This toolset can also be easily extended to fulfill other purposes with its open-source nature. CCBY
Published: 2021
Journal: IEEE Access
Author Name: Lan, H
Author link: https://covid19-data.nist.gov/pid/rest/local/author/lan_h
Author Name: Sha, D
Author link: https://covid19-data.nist.gov/pid/rest/local/author/sha_d
Author Name: Malarvizhi, A S
Author link: https://covid19-data.nist.gov/pid/rest/local/author/malarvizhi_a_s
Author Name: Liu, Y
Author link: https://covid19-data.nist.gov/pid/rest/local/author/liu_y
Author Name: Li, Y
Author link: https://covid19-data.nist.gov/pid/rest/local/author/li_y
Author Name: Meister, N
Author link: https://covid19-data.nist.gov/pid/rest/local/author/meister_n
Author Name: Liu, Q
Author link: https://covid19-data.nist.gov/pid/rest/local/author/liu_q
Author Name: Wang, Z
Author link: https://covid19-data.nist.gov/pid/rest/local/author/wang_z
Author Name: Yang, J
Author link: https://covid19-data.nist.gov/pid/rest/local/author/yang_j
Author Name: Yang, C
Author link: https://covid19-data.nist.gov/pid/rest/local/author/yang_c
license: unk
license_url: [unknown license]
source_x: WHO
source_x_url: https://www.who.int/
who_covidence_id: #covidwho-1266262
has_full_text: FALSE
G_ID: covid_scraper_an_open_source_toolset_for_automatically_scraping_and_processing_global
S2 ID: 235476063