https://doi.or g/10.31449/inf.v46i4.4181 Informatica 46 (2022) 449–456 449 Ranking Effectiveness of Non-Pharmaceutical In terventions Against COVID-19: A Review David Susič ⋆ 1 , Janez T omšič 1 , and Matjaž Gams 1 1 Department of Intelligent Systems, Jožef Stefan Institute, Jamova cesta 39, 1000 Ljubljana, Slovenia E-mail: david.susic@ijs.si, janez2001@gmail.com, matjaz.gams@ijs.si Keywords: Non-pharmaceutical interventions, COVID-19, SARS-CoV -2 Received: May 13, 2022 In this r eview , we examine 34 studies based on experimental data that estimate and compar e the effective- ness of 12 non-pharmaceutical government interventions against COVID-19 based on cases, deaths, and/or transmission rates to assess their overall effectiveness. The studies r eviewed ar e based on daily country- level data and cover four to 200 countries and r egions worldwide with varying time intervals, spanning the period between December 2019 and August 2021. W e found that the overall most effective interventions ar e r estrictions on gatherings, workplace closing, public information campaigns, and school closing, while the least effective ar e close public transport, contact tracing, and testing policy . Povzetek: Pr edstavljen je pr egled 34 objav , ki analizirajo uspešnost ukr epov pr oti kovidu. 1 Intr oduction Looking back at the first months of 2020, it is clear that the pandemic COVID-19 caught the world unprepared. Ini- tially , it was unclear how contagious the virus was, how quickly it would spread, how to protect against it, and how to prevent hospital overload. T o combat the spread of the virus, governments began introducing various non- pharmaceutical interventions (NPIs). It quickly became clear that some NPIs had a stronger impact on containing the pandemic than others. As a result, researchers around the world have begun to study the ef fectiveness of NPIs in dif ferent geopolitical regions. Despite the vaccine being developed in the last half of 2020, the spread of COVID- 19 and the number of infections are still a major burden to society . As of May 2022, there have been 6.25 million COVID-19-related deaths worldwide [1]. In this paper we extend our earlier work [2]. W e review related work on the ef fectiveness of NPIs implemented in dif ferent countries and over dif ferent time periods, with the goal of assessing and ranking their overall ef fectiveness. There is some similar work in the literature [3, 4, 5], but in this work we only consider studies in which conclusive evi- dence of the ef fectiveness of at least two NPIs was found. In addition, we do not include simulation-based studies. Un- like the two reviews mentioned above, our review includes time intervals from the third and fourth waves and, to the best of our knowledge, is the most up-to-date review in this regard. The rest of this paper is structured as follows. Section 2 presents methodology for selecting the papers and ranking the ef fectiveness of the NPIs. In section 3, we present and analyse the results. Section 4 describes the limitations of our study . W e conclude the paper in section 5. 2 Methodology The first step in our research was to establish the criteria for selecting the papers to be included and to create a unified ranking system that would allow us to compare the rankings of NPIs in related work. 2.1 Eligibility criteria In this review , we considered 12 NPIs from the Oxford COVID-19 Government Response T racker (OxCGR T) [6]: school closing (C1), workplace closing (C2), cancel pub- lic events (C3), restrictions on gatherings (C4), close pub- lic transport (C5), stay at home requirements (C6), restric- tions on internal movement (C7), international travel con- trols (C8), public information campaigns (H1), testing pol- icy (H2), contact tracing (H3), and facial coverings (H6). The letters C and H correspond to containment and closur e policies and health system policies , respectively . The 12 selected NPIs were chosen because they have been imple- mented most frequently and therefore cover the majority of all measures implemented worldwide. W e searched for papers written in English and published up to May 2022. W e searched Google Scholar for published studies and MedRxiv for preprints. For a study to be in- cluded in this review , it had to meet the following criteria: – studies the ef fect on COVID-19 related deaths, cases or transmission rate, – compares NPIs that map to at least two OxCGR T NPIs, – is based on experimental data and not based on fore- casts/simulatons, and – was conducted on a geographical region level (one or more), meaning that studies that only focus on selected 450 Informatica 46 (2022) 449–456 D. Susič et al. groups of people (e.g. people from Universities only) [7, 8] were not included. All papers included in this review are listed in T able 3 along with their respective study settings. In the cases where the study used NPI information from a database other than OxCGR T , the NPIs first had to be mapped from the other database to the OxCGR T , based on the descriptions of the interventions in both of the documentations. If mul- tiple NPIs corresponded to one OxCGR T NPI, their scores were averaged. In contrast, if a single NPI corresponded to more than one OxCGR T NPI, its score was applied to all corresponding OxCGR T NPIs. 2.2 Ranking the effectiveness T o rank the ef fectiveness of the NPIs, we used a scale of one to four , with one and four representing the most and least ef- fective NPIs studied, respectively . The ef fectiveness scores from each study were first ranked and then divided into four equally sized bins, with the most ef fective NPIs in bin one and the least ef fective NPIs in bin four . The bin number corresponds directly to the value on our ef fectiveness scale. Note that in some studies, some of the bins may be empty , resulting in this value not being assigned to an NPI. In the Bendavid et al. study [9], the estimated impacts were reported separately for each country studied. In this case, the values were first averaged across countries and then ranked. In the work of Askitas et al. [26], the NPIs were clas- sified descriptively only . C1, C2, C3, and C4 were found to be the most ef fective NPIs and were given a value of one. The ef fect of C6 was judged to decrease over time and was therefore given a value of two. C8 was judged to be less ef fective and was given a value of three, while C5 and C7 were judged to be negligibly ef fective and were given a value of four . Li et al. [10] calculated the estimated ef fects one, two, and three weeks after the implementation. In this case, the scores were averaged across all three cases. In the work of Liu et al. [1 1], the ef fectiveness of NPIs was estimated in two scenarios, where NPIs are imple- mented at their maximum stringency or at any stringency . The NPIs were then described as either strong, moderate, or weak in both of the scenarios. The NPIs graded strong in at least any stringency scenario were assigned value one, NPIs graded strong in maximum stringency scenario only were assigned value two. All NPIs graded moderate were assigned value three, and all NPIs graded weak were as- signed value four . In the study by W ibbens et al. [12], the ef fectiveness of NPIs was assessed at dif ferent intensity levels. They were first rated separately at the highest intensity and at an in- termediate intensity . Then, their overall ranking was calcu- lated as the average of the two. The estimated ef fects of NPIs from all studies reviewed are summarised in T able 3. In studies in which ef fects were estimated but could not be ranked [13, 14, 15, 16], all NPIs were assigned a value 2. In studies in which fewer than four NPIs were considered [13, 14, 17, 18, 19, 20, 21, 22, 23, 24, 25], values were also assigned on the basis of descriptive ranking. 3 Results Among the 34 studies selected in this review , there are 14 works that deal with cases incidence [13, 14, 16, 21, 24, 26, 27, 28, 29, 30, 31, 32, 33, 34], 1 1 works that deal with reproduction number [10, 1 1, 18, 20, 22, 34, 35, 36, 37, 38, 39], seven works that deal with infection growth rate [9, 12, 17, 19, 25, 40, 41], and nine works that deal with mortality [15, 16, 21, 23, 28, 29, 31, 40, 42]. Note that some works deal with more than one outcome and are thus men- tioned more than once. Most of the works analyse time in- tervals before the vaccination, however two studies [31, 34] analyse time intervals when vaccines are used. Eventhough some papers consider only a few selected countries, 24 of the works include either all US states or at least 50 countries worldwide. Boxplots of the ef fectiveness values of the NPIs are shown in Figure 1. Each box extends from the lower to the upper quartile of the NPI data, with a line at the median. The whiskers extending from the box show the range of the data. The most ef fective NPIs overall are restrictions on gatherings (C4), workplace closing (C2), public informa- tion campaigns (H1), and school closing (C1) with mean ef- fectiveness value of 1.91, 1.92, 2.0, and 2.08, respectively . The NPIs with moderate ef fectiveness are stay at home re- quirements (C6), cancel public events (C3), restrictions on internal movement (C7), facial coverings (H6), and inter - national traven controls (C8) with mean ef fectiveness value of 2.25, 2.54, 2.58, 2.63, and 2.75, respectively . The least ef fective NPIs are close public transport (C5), contact trac- ing (H3), and testing policy (H2), with mean ef fectiveness value of 3.33, 3.33, and 3.75, respectively . At this point it is important to note that Herby et al. [5] determined that lockdowns, which we find to have a moderate ef fect, only reduced deaths by 0.2–2.9 %. 4 Limitations This review has the following limitations. Because the studies included in the review are based on experimental data, the NPIs are always used simultaneously , whereas the final results of the NPI ef fects are reported individu- ally . Because combinations of NPIs active at the same time were very similar in dif ferent regions and time intervals, it is sometimes dif ficult to justify treating the ef fects sepa- rately . In some papers, NPIs were not ranked, so these NPIs re- ceive the same value in our study . In addition, some ef fec- tiveness values were assigned based on descriptive ranking. Results are reported here as steady-state rankings, even though the ef fects of NPIs will change as they are imple- Informatica 46 (2022) 449–456 451 Figure 1: Boxplot of the NPIs’ ef fectiveness. V alue one corresponds to the maximum and four to the minimum ef fective- ness. The numbers in parentheses indicate the number of times the NPIs occurred in the studies examined. mented (e.g., as people stop complying with restrictions on gatherings, as vaccines are developed, etc.). In addition, the time intervals studied vary in length, and the ef fects could dif fer between short and long intervals, as the ef fects of some NPIs diminish over time [43]. The NPIs are imple- mented with dif ferent stringency according to the Oxford database. This means that our results apply only to the av- erage levels of stringency at which the NPIs can be imple- mented. Some NPIs may be much more ef fective (less ef- fective) when implemented with higher (lower) stringency . 5 Conclusion In this work, we reviewed 34 studies that assessed the ef- fectiveness of 12 non-pharmaceutical interventions against COVID-19. The studies are all based on experimental data and cover up to 200 countries and regions worldwide with dif ferent time intervals covering time span between Decem- ber , 2019 and August, 2021. W e found that the overall most ef fective interventions are restrictions on gatherings, work- place closing, public information campaigns, and school closing. The interventions with moderate impact are stay at home requirements, cancel public events, restrictions on in- ternal movement, facial coverings, and international travel controls. The interventions with the least amount of impact are close public transport, contact tracing, and testing pol- icy . 452 Informatica 46 (2022) 449–456 D. Susič et al. Acknowledgement The paper was supported by the ISE-EMH project funded by the program Interreg V -A Italy-Slovenia 2014-2020. The authors acknowledge the financial support from the Slovenian Research Agency , research core funding No. P2- 0209. Refer ences [1] WHO COVID-19 Dashboard, Geneva: W orld Health Or ganization, https://covid19.who.int/. Accesed May 2022. [2] J. T omšič, D. Susič, and M. Gams (2021). Ef fective- ness of non-pharmaceutical interventions in handling the COVID-19 pandemic: review of related studies. In Pr oceedings of the 24th international multiconfer - ence Information Society - IS 2021 , vol. D, pp. 46–51. [3] A. Mendez-Brito, C. El Bcheraoui, and F . Pozo- Martin (2021). Systematic review of empirical studies comparing the ef fectiveness of non- pharmaceutical interventions against COVID-19. Journal of Infection , vol. 83, no. 3, pp. 281–293, https://doi.or g/10.1016/j.jinf.2021.06.018. [4] I. A youni, J. Maatoug, W . Dhouib et al. (2021). Ef fective public health measures to mitigate the spread of COVID-19: a systematic review . BMC Public Health , vol. 21, no. 1, pp. 10–15, https://doi.or g/10.1 186/s12889-021-1 1 1 1 1-1. [5] J. Herby , L. Jonung, and S. H. Hanke (2022). A liter - ature review and meta-analysis of the ef fects of lock- downs on COVID-19 mortality . Studies in Applied Economics , vol. 200. [6] T . Hale, N. Angrist, and R. Goldszmidt (2021). A global panel database of pandemic policies (Ox- ford COVID-19 Government Response T racker). Na- tur e Human Behaviour , vol. 5, pp. 4, pp. 529–538, https://doi.or g/10.1016/j.jinf.2021.06.018. [7] Z. Niu and G. Scarciotti (2021). Ranking the ef fectiveness of non-pharmaceutical interven- tions to counter COVID-19 in UK univer - sities with vaccinated population. medRxiv , https://doi.or g/10.1 101/2021.1 1.07.21266028. [8] H. H. Suh, J. Meehan, L. Blaisdell, and L. Browne (2021). Non-pharmaceutical interventions and COVID-19 cases in US summer camps: re- sults from an American Camp Association survey . Journal of Epidemiology & Community Health , vol. 76, no. 4, pp. 327–334, https://doi.or g/10.1 136/jech- 2021-21671 1. [9] E. Bendavid, C. Oh, J. Bhattacharya, and J. P . A. Ioannidis (2021). Assessing mandatory stay-at- home and business closure ef fects on the spread of COVID-19. Eur opean Journal of Clinical In- vestigation , vol. 51, no. 4, article no. e13484, https://doi.or g/10.1 1 1 1/eci.13484. [10] Y . Li, H. Campbell, D. Kulkarni et al. (2021). The temporal association of introducing and lifting non- pharmaceutical interventions with the time-varying reproduction number (R) of SARS-CoV -2: a mod- elling study across 131 countries. The Lancet In- fectious Diseases , vol. 21, no. 2, pp. 193–202, https://doi.or g/10.1016/S1473-3099(20)30785-4. [1 1] Y . Liu, C. Mor genstern, J. Kelly et al. (2021). The im- pact of non-pharmaceutical interventions on SARS- CoV -2 transmission across 130 countries and ter - ritories. BMC medicine , vol. 19, no. 1, pp. 1–12, https://doi.or g/10.1 186/s12916-020-01872-8. [12] P . D. W ibbens, W . W . Koo, and A. M. McGa- han (2020). Which COVID policies are most ef fec- tive? A Bayesian analysis of COVID-19 by jurisdic- tion. Plos one , vol. 15, no. 12, article no. e0244177, https://doi.or g/10.1371/journal.pone.0244177. [13] P . Jüni, M. Rothenbühler , and P . Bobos et al. (2020). Impact of climate and public health interventions on the COVID-19 pandemic: a prospective co- hort study . CMAJ , vol. 192, no. 21, pp. 566–573, https://doi.or g/10.1503/cmaj.200920. [14] A. M. Jalali, S. G. Khoury , J. See et al. (2020). Delayed interventions, low com- pliance, and health disparities amplified the early spread of COVID-19. medRxiv , https://doi.or g/10.1 101/2020.07.31.20165654. [15] C. T . Lef fler , E. Ing, J. D. L ykins et al. (2020). Association of country-wide coronavirus mortality with demographics, testing, lockdowns, and public wearing of masks. The American journal of tr opical medicine and hygiene , vol. 103, no. 6, pp. 2400–241 1, https://doi.or g/10.4269/ajtmh.20-1015. [16] D. I. Papadopoulos, I. Donkov , K. Charitopoulos, and S. Bishara (2020). The impact of lockdown measures on COVID-19: a worldwide comparison. medRxiv , https://doi.or g/10.1 101/2020.05.22.20106476. [17] V . Chernozhukov , H. Kasahara, and P . Schrimpf (2021). Causal impact of masks, policies, behav- ior on early covid-19 pandemic in the U.S. Jour - nal of Econometrics , vol. 220, no. 1, pp. 23–62, https://doi.or g/10.1016/j.jeconom.2020.09.003. [18] P . Deb, D. Furceri, J. D. Ostry , and N. T awk (2021). Policy interventions, social distancing, and SARS- CoV -2 transmission in the United States: a retro- spective state-level analysis. The American journal of Informatica 46 (2022) 449–456 453 the medical sciences , vol. 361, no. 5, pp. 575–584, https://doi.or g/10.1016/j.amjms.2021.01.007. [19] S. Ebrahim, H. Ashworth, and C. Noah (2020). Reduction of COVID-19 incidence and nonphar - macologic interventions: Analysis using a US county–level policy data set. Journal of medical Internet r esear ch , vol. 22, no. 12, pp. 575–584, https://doi.or g/10.1016/j.amjms.2021.01.007. [20] S. Flaxman, S. Mishra, A. Gandy et al. (2020). Es- timating the ef fects of non-pharmaceutical interven- tions on COVID-19 in Europe. Natur e , vol. 584, no. 7820, pp. 257–261, https://doi.or g/10.1038/s41586- 020-2405-7. [21] P . R. Hunter , F . J. Colón-González, J. Brainard, and S. Rushton (2021). Impact of non-pharmaceutical interventions against COVID-19 in Europe in 2020: a quasi-experimental non-equivalent group and time series design study . Eur o Surveill , vol. 26, no. 28, https://doi.or g/10.2807/1560- 7917.ES.2021.26.28.2001401. [22] A. M. Olney , J. Smith, S. Sen et al. (2021). Esti- mating the ef fect of social distancing interventions on COVID-19 in the United States. American Jour - nal of Epidemiology , vol. 190, no. 8, pp. 1504–1509, https://doi.or g/10.1093/aje/kwaa293. [23] D. Piovani, M. N. Christodoulou, A. Hadjidemetriou et al. (2021). Ef fect of early application of so- cial distancing interventions on COVID-19 mor - tality over the first pandemic wave: an analy- sis of longitudinal data from 37 countries. Jour - nal of Infection , vol. 82, no. 1, pp. 133–142, https://doi.or g/10.1016/j.jinf.2020.1 1.033. [24] P . D. W ibbens, W . W . Koo, and A. M. McGahan (2020). Evaluation on dif ferent non-pharmaceutical interventions during COVID-19 pandemic: an analy- sis of 139 countries. J Infect , vol. 81, no. 3, pp. 70–71, https://doi.or g/10.1016/j.jinf.2020.06. [25] X. Zhang and M. E. W arner (2020). COVID-19 pol- icy dif ferences across US states: shutdowns, re- opening, and mask mandates. Int J Envir on Res Public Health , vol. 17, no. 24, article no. 9520, https://doi.or g/10.3390/ijerph17249520. [26] N. Askitas, K. T atsiramos, and B. V erheyden (2021). Estimating worldwide ef fects of non-pharmaceutical interventions on COVID-19 incidence and popula- tion mobility patterns using a multiple-event study . Scientific r eports , vol. 1 1, no. 1, article no. 1972, https://doi.or g/10.1038/s41598-021-81442-x. [27] N. Banholzer , E. V an W eenen, A. Lison, A. Cenedese, A. Seeliger et al. (2021). Estimating the ef fects of non- pharmaceutical interventions on the number of new infections with COVID-19 during the first epidemic wave. PLOS ONE , vol. 16, no. 6, article no. e0252827, https://doi.or g/10.1371/journal.pone.0252827. [28] R. Chaudhry et al. (2021). A country level anal- ysis measuring the impact of government actions, country preparedness and socioeconomic factors on COVID-19 mortality and related health out- comes. eClinicalMedicine , vol 25, article no. 100464, https://doi.or g/10.1016/j.eclinm.2020.100464. [29] P . Deb, D. Furceri, J. D. Ostry , and N. T awk (2020). The effect of containment measur es on the COVID-19 pandemic. Centre for Economic Policy Research. [30] N. Islam, S. Sharp, G. Chowell et al. (2020). Physical distancing interventions and incidence of coronavirus disease 2019: natural experiment in 149 countries. British Medical Journal , vol. 370, article no. m2743, https://doi.or g/10.1 136/bmj.m2743. [31] M. Sharma, S. Mindermann, C. Rogers-Smith et al. (2021). Understanding the ef fectiveness of govern- ment interventions against the resur gence of COVID- 19 in Europe. Natur e Communications , vol. 12, no. 1, article no. 5820, https://doi.or g/10.1038/s41467-021- 26013-4. [32] L. Y . Chan, B. Y uan, and M. Convertino (2021). COVID-19 non-pharmaceutical intervention portfo- lio ef fectiveness and risk communication predomi- nance. Sci Rep , vol. 1 1, no. 1, article no. 10605, https://doi.or g/10.1038/s41598-021-88309-1. [33] Y . Gokmen, C. Baskici, and Y . Ercil (2021). Ef fects of non-pharmaceutical interventions against COVID-19: A cross-country analysis. The International Journal of Health Planning and Management , vol. 36, no. 4, pp. 1 178–1 188, https://doi.or g/10.1002/hpm.3164. [34] H. Li, L. W ang, M. Zhang, Y . Lu, and W . W ang (2022). Ef fects of vaccination and non- pharmaceutical interventions and their lag times on the COVID-19 pandemic: comparison of eight countries. PLOS Neglected T r opical Dis- eases , vol. 16, no. 1, article no. e0010101, https://doi.or g/10.1371/journal.pntd.0010101. [35] Y . Bo, C. Guo, C. Lin et al. (2021). Ef fec- tiveness of non-pharmaceutical interventions on COVID-19 transmission in 190 countries from 23 January to 13 April 2020. International Journal of Infectious Diseases , vol. 102, pp. 247–253, https://doi.or g/10.1016/j.ijid.2020.10.066. [36] J. M. Brauner , S. Mindermann, M. Sharma et al. (2021). Inferring the ef fectiveness of gov- ernment interventions against COVID-19. Sci- ence , vol. 371, no. 6531, article no. eabd9338, https://doi.or g/10.1 126/science.abd9338. 454 Informatica 46 (2022) 449–456 D. Susič et al. [37] R. T . Esra, L. Jamesion, M. P . Fox et al. (2020). Evaluating the impact of non-pharmaceutical inter - ventions for SARS-CoV -2 on a global scale. medRxiv , https://doi.or g/10.1 101/2020.07.30.20164939. [38] N. Haug, L. Geyrhofer , A. Londei et al. (2020). Rank- ing the ef fectiveness of worldwide COVID-19 gov- ernment interventions. Nat Hum Behav , vol. 4, no. 12, pp. 1303–1312, https://doi.or g/10.1038/s41562-020- 01009-0. [39] W . C. Koh, L. Naing, J. W ong et al. (2020). Estimating the impact of physical distancing measures in contain- ing COVID-19: an empirical analysis. International Journal of Infectious Diseases , vol. 100, pp. 42–49, https://doi.or g/10.1016/j.ijid.2020.08.026. [40] Y . Li, M. Li, M. Rice et al. (2021). The impact of policy measures on human mobility , COVID-19 cases, and mortality in the US: a spatiotemporal per - spective. International Journal of Envir onmental Re- sear ch and Public Health , vol. 18, no. 3, article no. 996, https://doi.or g/10.3390/ijerph18030996. [41] F . Pozo-Martin, H. W eishaar , F . Cristea et al. (2021). The impact of non-pharmaceutical in- terventions on COVID-19 epidemic growth in the 37 OECD member states. Eur opean jour - nal of epidemiology , vol. 82, no. 1, pp. 1–12, https://doi.or g/10.1016/j.jinf.2020.1 1.033. [42] J. Stokes, A. J. T urner , L. Anselmi et al. (2020). The relative ef fects of non-pharmaceutical in- terventions on early Covid-19 mortality: nat- ural experiment in 130 countries. medRxiv , https://doi.or g/10.1 101/2020.10.05.20206888. [43] F . Zhou, T . J. Hu, X. Y . Zhang et al. (2022). The association of intensity and duration of non- pharmacological interventions and implementa- tion of vaccination with COVID-19 infection, death, and excess mortality: natural experiment in 22 European countries. Journal of Infection and Public Health , vol. 15, no. 5, pp. 499-507, https://doi.or g/10.1016/j.jiph.2022.03.01 1. Informatica 46 (2022) 449–456 455 T able 1: Studies included in this review . Authors NPI data sour ce Countries cover ed T ime interval Askitas et al. [26] OxCGR T 175 countries unclear Banholzer et al. [27] Collected by the authors USA, Canada, Australia and 17 EU countries Feb – May , 2020 Bendavid et al. [9] COVID-19 policy data- bank 10 countries Dec, 2019 – June, 2020 Bo et al. [35] Collected by the authors 190 countries Jan – Apr , 2020 Brauner et al. [36] Collected by the authors 41 countries Jan – May , 2020 Chan et al. [32] WHO and John Hopkins University 50 countries Dec, 2019 – June, 2020 Chaudhry et al. [28] Collected by the authors 50 countries Dec, 2019 – May , 2020 Chernozhukov et al. [17] COVID T racking Project USA Mar – June, 2020 Deb et al. [29] OxCGR T 129 countries Dec, 2019 – May , 2020 Dreher et al. [18] unclear USA Dec, 2019 – Apr , 2020 Ebrahim et al. [19] Hikma Health 1320 US counties Mar – July , 2020 Esra et al. [37] WHO-PHSM 26 countries and 34 US states Dec, 2019 – May , 2020 Flaxman et al. [20] unclear 1 1 EU countries Feb – May , 2020 Gokmen et al. [33] Our W orld in Data 4 countries Dec, 2019 – June, 2020 Haug et al. [38] CCCSL 56 countries Dec, 2019 – Aug, 2020 Hunter et al. [21] IHME 30 European countries Dec, 2019 – Apr , 2020 Islam et al. [30] OxCGR T 149 countries Dec, 2019 – May , 2020 Jalali et al. [14] Collected by the authors 30 US states Mar – May , 2020 Jüni et al. [13] Collected by the authors 144 worldwide regions Dec, 2019 – Mar , 2020 Koh et al. [39] OxCGR T 170 countries Jan – May , 2020 Lef fler et al. [15] OxCGR T 200 countries Dec, 2019 – May , 2020 Li et al. (a) [10] OxCGR T 131 countries Jan – July , 2020 Li et al. (b) [40] NSF spatiotemporal center USA Mar – July , 2020 Liu et al. [1 1] OxCGR T 130 countries and terri- tories Jan – June, 2020 Olney et al. [22] Collected by the authors USA Feb – Apr , 2020 Papadopoulos et al. [16] OxCGR T 151 countries Jan – Apr , 2020 Piovani et al. [23] OxCGR T 37 members of OECF Jan – June, 2020 Pozo-Martin et al. [41] OxCGR T and WHO- PHSM 37 members of OECD Oct – Dec, 2020 Sharma et al. [31] Collected by the authors 7 EU countries Aug, 2020 – Jan, 2021 Stokes et al. [42] OxCGR T USA and 7 countries Dec, 2019 – June, 2020 W ang et al. [34] OxCGR T 139 countries Dec, 2019 – Aug, 2021 W ibbens et al. [12] OxCGR T 40 countries and US states unclear W ong et al. [24] OxCGR T 139 countries Mar – Apr , 2020 Zhang et al. [25] NY T imes and CNN USA Feb – Aug, 2020 456 Informatica 46 (2022) 449–456 D. Susič et al. T able 2: Estimation of ef fectiveness of NPIs in reviewed studies. Study C1 C2 C3 C4 C5 C6 C7 C8 H1 H2 H3 H6 Askitas et al. [26] 1 1 1 1 4 2 4 3 Banholzer et al. [27] 2 2 1 4 3 Bendavid et al. [9] 3 4 3 2 1 1 4 3 Bo et al. [35] 1 1 1 3 4 4 2 Brauner et al. [36] 1 2 1 3 Chan et al. [32] 4 4 1 1 2 Chaudhry et al. [28] 2 2 2 3 Chernozhukov et al. [17] 2 2 3 Deb et al. [29] 1 2 2 2 1 1 2 1 Dreher et al. [18] 2 2 1 Ebrahim et al. [19] 2 3 Esra et al. [37] 3 3 1 2 Flaxman et al. [20] 4 4 3 Gokmen et al. [33] 4 1 4 2 4 2 3 3 Haug et al. [38] 1 1 4 3 3 2 2 3 Hunter et al. [21] 1 2 3 Islam et al. [30] 2 2 1 4 3 3 Jalali et al. [14] 2 2 Jüni et al. [13] 2 2 2 Koh et al. [39] 1 2 2 3 Lef fler et al. [15] 2 2 2 2 Li et al. (a) [10] 1 2 1 3 4 2 3 4 Li et al. (b) [40] 2 2 3 1 Liu et al. [1 1] 1 1 2 2 4 3 1 4 3 4 4 Olney et al. [22] 2 1 1 Papadopoulos et al. [16] 2 2 2 2 Piovani et al. [23] 3 2 Pozo-Martin et al. [41] 3 2 1 4 4 Sharma et al. [31] 4 1 2 3 3 Stokes et al. [42] 1 2 3 3 W ang et al. [34] 3 3 2 2 W ibbens et al. [12] 2 1 4 3 4 2 1 3 3 4 4 W ong et al. [24] 3 2 1 Zhang et al. [25] 2 3