research objectives on covid 19

• Research article
• Open access
• Published: 04 June 2021

Coronavirus disease (COVID-19) pandemic: an overview of systematic reviews

• Israel Júnior Borges do Nascimento 1 , 2 ,
• Dónal P. O’Mathúna 3 , 4 ,
• Thilo Caspar von Groote 5 ,
• Hebatullah Mohamed Abdulazeem 6 ,
• Ishanka Weerasekara 7 , 8 ,
• Ana Marusic 9 ,
• Livia Puljak   ORCID: orcid.org/0000-0002-8467-6061 10 ,
• Vinicius Tassoni Civile 11 ,
• Irena Zakarija-Grkovic 9 ,
• Tina Poklepovic Pericic 9 ,
• Alvaro Nagib Atallah 11 ,
• Santino Filoso 12 ,
• Nicola Luigi Bragazzi 13 &
• Milena Soriano Marcolino 1

On behalf of the International Network of Coronavirus Disease 2019 (InterNetCOVID-19)

BMC Infectious Diseases volume  21 , Article number:  525 ( 2021 ) Cite this article

16k Accesses

29 Citations

13 Altmetric

Metrics details

Navigating the rapidly growing body of scientific literature on the SARS-CoV-2 pandemic is challenging, and ongoing critical appraisal of this output is essential. We aimed to summarize and critically appraise systematic reviews of coronavirus disease (COVID-19) in humans that were available at the beginning of the pandemic.

Nine databases (Medline, EMBASE, Cochrane Library, CINAHL, Web of Sciences, PDQ-Evidence, WHO’s Global Research, LILACS, and Epistemonikos) were searched from December 1, 2019, to March 24, 2020. Systematic reviews analyzing primary studies of COVID-19 were included. Two authors independently undertook screening, selection, extraction (data on clinical symptoms, prevalence, pharmacological and non-pharmacological interventions, diagnostic test assessment, laboratory, and radiological findings), and quality assessment (AMSTAR 2). A meta-analysis was performed of the prevalence of clinical outcomes.

Eighteen systematic reviews were included; one was empty (did not identify any relevant study). Using AMSTAR 2, confidence in the results of all 18 reviews was rated as “critically low”. Identified symptoms of COVID-19 were (range values of point estimates): fever (82–95%), cough with or without sputum (58–72%), dyspnea (26–59%), myalgia or muscle fatigue (29–51%), sore throat (10–13%), headache (8–12%) and gastrointestinal complaints (5–9%). Severe symptoms were more common in men. Elevated C-reactive protein and lactate dehydrogenase, and slightly elevated aspartate and alanine aminotransferase, were commonly described. Thrombocytopenia and elevated levels of procalcitonin and cardiac troponin I were associated with severe disease. A frequent finding on chest imaging was uni- or bilateral multilobar ground-glass opacity. A single review investigated the impact of medication (chloroquine) but found no verifiable clinical data. All-cause mortality ranged from 0.3 to 13.9%.

Conclusions

In this overview of systematic reviews, we analyzed evidence from the first 18 systematic reviews that were published after the emergence of COVID-19. However, confidence in the results of all reviews was “critically low”. Thus, systematic reviews that were published early on in the pandemic were of questionable usefulness. Even during public health emergencies, studies and systematic reviews should adhere to established methodological standards.

Peer Review reports

The spread of the “Severe Acute Respiratory Coronavirus 2” (SARS-CoV-2), the causal agent of COVID-19, was characterized as a pandemic by the World Health Organization (WHO) in March 2020 and has triggered an international public health emergency [ 1 ]. The numbers of confirmed cases and deaths due to COVID-19 are rapidly escalating, counting in millions [ 2 ], causing massive economic strain, and escalating healthcare and public health expenses [ 3 , 4 ].

The research community has responded by publishing an impressive number of scientific reports related to COVID-19. The world was alerted to the new disease at the beginning of 2020 [ 1 ], and by mid-March 2020, more than 2000 articles had been published on COVID-19 in scholarly journals, with 25% of them containing original data [ 5 ]. The living map of COVID-19 evidence, curated by the Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre), contained more than 40,000 records by February 2021 [ 6 ]. More than 100,000 records on PubMed were labeled as “SARS-CoV-2 literature, sequence, and clinical content” by February 2021 [ 7 ].

Due to publication speed, the research community has voiced concerns regarding the quality and reproducibility of evidence produced during the COVID-19 pandemic, warning of the potential damaging approach of “publish first, retract later” [ 8 ]. It appears that these concerns are not unfounded, as it has been reported that COVID-19 articles were overrepresented in the pool of retracted articles in 2020 [ 9 ]. These concerns about inadequate evidence are of major importance because they can lead to poor clinical practice and inappropriate policies [ 10 ].

Systematic reviews are a cornerstone of today’s evidence-informed decision-making. By synthesizing all relevant evidence regarding a particular topic, systematic reviews reflect the current scientific knowledge. Systematic reviews are considered to be at the highest level in the hierarchy of evidence and should be used to make informed decisions. However, with high numbers of systematic reviews of different scope and methodological quality being published, overviews of multiple systematic reviews that assess their methodological quality are essential [ 11 , 12 , 13 ]. An overview of systematic reviews helps identify and organize the literature and highlights areas of priority in decision-making.

In this overview of systematic reviews, we aimed to summarize and critically appraise systematic reviews of coronavirus disease (COVID-19) in humans that were available at the beginning of the pandemic.

Methodology

Research question.

This overview’s primary objective was to summarize and critically appraise systematic reviews that assessed any type of primary clinical data from patients infected with SARS-CoV-2. Our research question was purposefully broad because we wanted to analyze as many systematic reviews as possible that were available early following the COVID-19 outbreak.

Study design

We conducted an overview of systematic reviews. The idea for this overview originated in a protocol for a systematic review submitted to PROSPERO (CRD42020170623), which indicated a plan to conduct an overview.

Overviews of systematic reviews use explicit and systematic methods for searching and identifying multiple systematic reviews addressing related research questions in the same field to extract and analyze evidence across important outcomes. Overviews of systematic reviews are in principle similar to systematic reviews of interventions, but the unit of analysis is a systematic review [ 14 , 15 , 16 ].

We used the overview methodology instead of other evidence synthesis methods to allow us to collate and appraise multiple systematic reviews on this topic, and to extract and analyze their results across relevant topics [ 17 ]. The overview and meta-analysis of systematic reviews allowed us to investigate the methodological quality of included studies, summarize results, and identify specific areas of available or limited evidence, thereby strengthening the current understanding of this novel disease and guiding future research [ 13 ].

A reporting guideline for overviews of reviews is currently under development, i.e., Preferred Reporting Items for Overviews of Reviews (PRIOR) [ 18 ]. As the PRIOR checklist is still not published, this study was reported following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2009 statement [ 19 ]. The methodology used in this review was adapted from the Cochrane Handbook for Systematic Reviews of Interventions and also followed established methodological considerations for analyzing existing systematic reviews [ 14 ].

Approval of a research ethics committee was not necessary as the study analyzed only publicly available articles.

Eligibility criteria

Systematic reviews were included if they analyzed primary data from patients infected with SARS-CoV-2 as confirmed by RT-PCR or another pre-specified diagnostic technique. Eligible reviews covered all topics related to COVID-19 including, but not limited to, those that reported clinical symptoms, diagnostic methods, therapeutic interventions, laboratory findings, or radiological results. Both full manuscripts and abbreviated versions, such as letters, were eligible.

No restrictions were imposed on the design of the primary studies included within the systematic reviews, the last search date, whether the review included meta-analyses or language. Reviews related to SARS-CoV-2 and other coronaviruses were eligible, but from those reviews, we analyzed only data related to SARS-CoV-2.

No consensus definition exists for a systematic review [ 20 ], and debates continue about the defining characteristics of a systematic review [ 21 ]. Cochrane’s guidance for overviews of reviews recommends setting pre-established criteria for making decisions around inclusion [ 14 ]. That is supported by a recent scoping review about guidance for overviews of systematic reviews [ 22 ].

Thus, for this study, we defined a systematic review as a research report which searched for primary research studies on a specific topic using an explicit search strategy, had a detailed description of the methods with explicit inclusion criteria provided, and provided a summary of the included studies either in narrative or quantitative format (such as a meta-analysis). Cochrane and non-Cochrane systematic reviews were considered eligible for inclusion, with or without meta-analysis, and regardless of the study design, language restriction and methodology of the included primary studies. To be eligible for inclusion, reviews had to be clearly analyzing data related to SARS-CoV-2 (associated or not with other viruses). We excluded narrative reviews without those characteristics as these are less likely to be replicable and are more prone to bias.

Scoping reviews and rapid reviews were eligible for inclusion in this overview if they met our pre-defined inclusion criteria noted above. We included reviews that addressed SARS-CoV-2 and other coronaviruses if they reported separate data regarding SARS-CoV-2.

Information sources

Nine databases were searched for eligible records published between December 1, 2019, and March 24, 2020: Cochrane Database of Systematic Reviews via Cochrane Library, PubMed, EMBASE, CINAHL (Cumulative Index to Nursing and Allied Health Literature), Web of Sciences, LILACS (Latin American and Caribbean Health Sciences Literature), PDQ-Evidence, WHO’s Global Research on Coronavirus Disease (COVID-19), and Epistemonikos.

The comprehensive search strategy for each database is provided in Additional file 1 and was designed and conducted in collaboration with an information specialist. All retrieved records were primarily processed in EndNote, where duplicates were removed, and records were then imported into the Covidence platform [ 23 ]. In addition to database searches, we screened reference lists of reviews included after screening records retrieved via databases.

Study selection

All searches, screening of titles and abstracts, and record selection, were performed independently by two investigators using the Covidence platform [ 23 ]. Articles deemed potentially eligible were retrieved for full-text screening carried out independently by two investigators. Discrepancies at all stages were resolved by consensus. During the screening, records published in languages other than English were translated by a native/fluent speaker.

Data collection process

We custom designed a data extraction table for this study, which was piloted by two authors independently. Data extraction was performed independently by two authors. Conflicts were resolved by consensus or by consulting a third researcher.

We extracted the following data: article identification data (authors’ name and journal of publication), search period, number of databases searched, population or settings considered, main results and outcomes observed, and number of participants. From Web of Science (Clarivate Analytics, Philadelphia, PA, USA), we extracted journal rank (quartile) and Journal Impact Factor (JIF).

We categorized the following as primary outcomes: all-cause mortality, need for and length of mechanical ventilation, length of hospitalization (in days), admission to intensive care unit (yes/no), and length of stay in the intensive care unit.

The following outcomes were categorized as exploratory: diagnostic methods used for detection of the virus, male to female ratio, clinical symptoms, pharmacological and non-pharmacological interventions, laboratory findings (full blood count, liver enzymes, C-reactive protein, d-dimer, albumin, lipid profile, serum electrolytes, blood vitamin levels, glucose levels, and any other important biomarkers), and radiological findings (using radiography, computed tomography, magnetic resonance imaging or ultrasound).

We also collected data on reporting guidelines and requirements for the publication of systematic reviews and meta-analyses from journal websites where included reviews were published.

Quality assessment in individual reviews

Two researchers independently assessed the reviews’ quality using the “A MeaSurement Tool to Assess Systematic Reviews 2 (AMSTAR 2)”. We acknowledge that the AMSTAR 2 was created as “a critical appraisal tool for systematic reviews that include randomized or non-randomized studies of healthcare interventions, or both” [ 24 ]. However, since AMSTAR 2 was designed for systematic reviews of intervention trials, and we included additional types of systematic reviews, we adjusted some AMSTAR 2 ratings and reported these in Additional file 2 .

Adherence to each item was rated as follows: yes, partial yes, no, or not applicable (such as when a meta-analysis was not conducted). The overall confidence in the results of the review is rated as “critically low”, “low”, “moderate” or “high”, according to the AMSTAR 2 guidance based on seven critical domains, which are items 2, 4, 7, 9, 11, 13, 15 as defined by AMSTAR 2 authors [ 24 ]. We reported our adherence ratings for transparency of our decision with accompanying explanations, for each item, in each included review.

One of the included systematic reviews was conducted by some members of this author team [ 25 ]. This review was initially assessed independently by two authors who were not co-authors of that review to prevent the risk of bias in assessing this study.

Synthesis of results

For data synthesis, we prepared a table summarizing each systematic review. Graphs illustrating the mortality rate and clinical symptoms were created. We then prepared a narrative summary of the methods, findings, study strengths, and limitations.

For analysis of the prevalence of clinical outcomes, we extracted data on the number of events and the total number of patients to perform proportional meta-analysis using RStudio© software, with the “meta” package (version 4.9–6), using the “metaprop” function for reviews that did not perform a meta-analysis, excluding case studies because of the absence of variance. For reviews that did not perform a meta-analysis, we presented pooled results of proportions with their respective confidence intervals (95%) by the inverse variance method with a random-effects model, using the DerSimonian-Laird estimator for τ 2 . We adjusted data using Freeman-Tukey double arcosen transformation. Confidence intervals were calculated using the Clopper-Pearson method for individual studies. We created forest plots using the RStudio© software, with the “metafor” package (version 2.1–0) and “forest” function.

Managing overlapping systematic reviews

Some of the included systematic reviews that address the same or similar research questions may include the same primary studies in overviews. Including such overlapping reviews may introduce bias when outcome data from the same primary study are included in the analyses of an overview multiple times. Thus, in summaries of evidence, multiple-counting of the same outcome data will give data from some primary studies too much influence [ 14 ]. In this overview, we did not exclude overlapping systematic reviews because, according to Cochrane’s guidance, it may be appropriate to include all relevant reviews’ results if the purpose of the overview is to present and describe the current body of evidence on a topic [ 14 ]. To avoid any bias in summary estimates associated with overlapping reviews, we generated forest plots showing data from individual systematic reviews, but the results were not pooled because some primary studies were included in multiple reviews.

Our search retrieved 1063 publications, of which 175 were duplicates. Most publications were excluded after the title and abstract analysis ( n = 860). Among the 28 studies selected for full-text screening, 10 were excluded for the reasons described in Additional file 3 , and 18 were included in the final analysis (Fig. 1 ) [ 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 ]. Reference list screening did not retrieve any additional systematic reviews.

PRISMA flow diagram

Characteristics of included reviews

Summary features of 18 systematic reviews are presented in Table 1 . They were published in 14 different journals. Only four of these journals had specific requirements for systematic reviews (with or without meta-analysis): European Journal of Internal Medicine, Journal of Clinical Medicine, Ultrasound in Obstetrics and Gynecology, and Clinical Research in Cardiology . Two journals reported that they published only invited reviews ( Journal of Medical Virology and Clinica Chimica Acta ). Three systematic reviews in our study were published as letters; one was labeled as a scoping review and another as a rapid review (Table 2 ).

All reviews were published in English, in first quartile (Q1) journals, with JIF ranging from 1.692 to 6.062. One review was empty, meaning that its search did not identify any relevant studies; i.e., no primary studies were included [ 36 ]. The remaining 17 reviews included 269 unique studies; the majority ( N = 211; 78%) were included in only a single review included in our study (range: 1 to 12). Primary studies included in the reviews were published between December 2019 and March 18, 2020, and comprised case reports, case series, cohorts, and other observational studies. We found only one review that included randomized clinical trials [ 38 ]. In the included reviews, systematic literature searches were performed from 2019 (entire year) up to March 9, 2020. Ten systematic reviews included meta-analyses. The list of primary studies found in the included systematic reviews is shown in Additional file 4 , as well as the number of reviews in which each primary study was included.

Population and study designs

Most of the reviews analyzed data from patients with COVID-19 who developed pneumonia, acute respiratory distress syndrome (ARDS), or any other correlated complication. One review aimed to evaluate the effectiveness of using surgical masks on preventing transmission of the virus [ 36 ], one review was focused on pediatric patients [ 34 ], and one review investigated COVID-19 in pregnant women [ 37 ]. Most reviews assessed clinical symptoms, laboratory findings, or radiological results.

Systematic review findings

The summary of findings from individual reviews is shown in Table 2 . Overall, all-cause mortality ranged from 0.3 to 13.9% (Fig. 2 ).

A meta-analysis of the prevalence of mortality

Clinical symptoms

Seven reviews described the main clinical manifestations of COVID-19 [ 26 , 28 , 29 , 34 , 35 , 39 , 41 ]. Three of them provided only a narrative discussion of symptoms [ 26 , 34 , 35 ]. In the reviews that performed a statistical analysis of the incidence of different clinical symptoms, symptoms in patients with COVID-19 were (range values of point estimates): fever (82–95%), cough with or without sputum (58–72%), dyspnea (26–59%), myalgia or muscle fatigue (29–51%), sore throat (10–13%), headache (8–12%), gastrointestinal disorders, such as diarrhea, nausea or vomiting (5.0–9.0%), and others (including, in one study only: dizziness 12.1%) (Figs. 3 , 4 , 5 , 6 , 7 , 8 and 9 ). Three reviews assessed cough with and without sputum together; only one review assessed sputum production itself (28.5%).

A meta-analysis of the prevalence of fever

A meta-analysis of the prevalence of cough

A meta-analysis of the prevalence of dyspnea

A meta-analysis of the prevalence of fatigue or myalgia

A meta-analysis of the prevalence of headache

A meta-analysis of the prevalence of gastrointestinal disorders

A meta-analysis of the prevalence of sore throat

Diagnostic aspects

Three reviews described methodologies, protocols, and tools used for establishing the diagnosis of COVID-19 [ 26 , 34 , 38 ]. The use of respiratory swabs (nasal or pharyngeal) or blood specimens to assess the presence of SARS-CoV-2 nucleic acid using RT-PCR assays was the most commonly used diagnostic method mentioned in the included studies. These diagnostic tests have been widely used, but their precise sensitivity and specificity remain unknown. One review included a Chinese study with clinical diagnosis with no confirmation of SARS-CoV-2 infection (patients were diagnosed with COVID-19 if they presented with at least two symptoms suggestive of COVID-19, together with laboratory and chest radiography abnormalities) [ 34 ].

Therapeutic possibilities

Pharmacological and non-pharmacological interventions (supportive therapies) used in treating patients with COVID-19 were reported in five reviews [ 25 , 27 , 34 , 35 , 38 ]. Antivirals used empirically for COVID-19 treatment were reported in seven reviews [ 25 , 27 , 34 , 35 , 37 , 38 , 41 ]; most commonly used were protease inhibitors (lopinavir, ritonavir, darunavir), nucleoside reverse transcriptase inhibitor (tenofovir), nucleotide analogs (remdesivir, galidesivir, ganciclovir), and neuraminidase inhibitors (oseltamivir). Umifenovir, a membrane fusion inhibitor, was investigated in two studies [ 25 , 35 ]. Possible supportive interventions analyzed were different types of oxygen supplementation and breathing support (invasive or non-invasive ventilation) [ 25 ]. The use of antibiotics, both empirically and to treat secondary pneumonia, was reported in six studies [ 25 , 26 , 27 , 34 , 35 , 38 ]. One review specifically assessed evidence on the efficacy and safety of the anti-malaria drug chloroquine [ 27 ]. It identified 23 ongoing trials investigating the potential of chloroquine as a therapeutic option for COVID-19, but no verifiable clinical outcomes data. The use of mesenchymal stem cells, antifungals, and glucocorticoids were described in four reviews [ 25 , 34 , 35 , 38 ].

Laboratory and radiological findings

Of the 18 reviews included in this overview, eight analyzed laboratory parameters in patients with COVID-19 [ 25 , 29 , 30 , 32 , 33 , 34 , 35 , 39 ]; elevated C-reactive protein levels, associated with lymphocytopenia, elevated lactate dehydrogenase, as well as slightly elevated aspartate and alanine aminotransferase (AST, ALT) were commonly described in those eight reviews. Lippi et al. assessed cardiac troponin I (cTnI) [ 25 ], procalcitonin [ 32 ], and platelet count [ 33 ] in COVID-19 patients. Elevated levels of procalcitonin [ 32 ] and cTnI [ 30 ] were more likely to be associated with a severe disease course (requiring intensive care unit admission and intubation). Furthermore, thrombocytopenia was frequently observed in patients with complicated COVID-19 infections [ 33 ].

Chest imaging (chest radiography and/or computed tomography) features were assessed in six reviews, all of which described a frequent pattern of local or bilateral multilobar ground-glass opacity [ 25 , 34 , 35 , 39 , 40 , 41 ]. Those six reviews showed that septal thickening, bronchiectasis, pleural and cardiac effusions, halo signs, and pneumothorax were observed in patients suffering from COVID-19.

Quality of evidence in individual systematic reviews

Table 3 shows the detailed results of the quality assessment of 18 systematic reviews, including the assessment of individual items and summary assessment. A detailed explanation for each decision in each review is available in Additional file 5 .

Using AMSTAR 2 criteria, confidence in the results of all 18 reviews was rated as “critically low” (Table 3 ). Common methodological drawbacks were: omission of prospective protocol submission or publication; use of inappropriate search strategy: lack of independent and dual literature screening and data-extraction (or methodology unclear); absence of an explanation for heterogeneity among the studies included; lack of reasons for study exclusion (or rationale unclear).

Risk of bias assessment, based on a reported methodological tool, and quality of evidence appraisal, in line with the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) method, were reported only in one review [ 25 ]. Five reviews presented a table summarizing bias, using various risk of bias tools [ 25 , 29 , 39 , 40 , 41 ]. One review analyzed “study quality” [ 37 ]. One review mentioned the risk of bias assessment in the methodology but did not provide any related analysis [ 28 ].

This overview of systematic reviews analyzed the first 18 systematic reviews published after the onset of the COVID-19 pandemic, up to March 24, 2020, with primary studies involving more than 60,000 patients. Using AMSTAR-2, we judged that our confidence in all those reviews was “critically low”. Ten reviews included meta-analyses. The reviews presented data on clinical manifestations, laboratory and radiological findings, and interventions. We found no systematic reviews on the utility of diagnostic tests.

Symptoms were reported in seven reviews; most of the patients had a fever, cough, dyspnea, myalgia or muscle fatigue, and gastrointestinal disorders such as diarrhea, nausea, or vomiting. Olfactory dysfunction (anosmia or dysosmia) has been described in patients infected with COVID-19 [ 43 ]; however, this was not reported in any of the reviews included in this overview. During the SARS outbreak in 2002, there were reports of impairment of the sense of smell associated with the disease [ 44 , 45 ].

The reported mortality rates ranged from 0.3 to 14% in the included reviews. Mortality estimates are influenced by the transmissibility rate (basic reproduction number), availability of diagnostic tools, notification policies, asymptomatic presentations of the disease, resources for disease prevention and control, and treatment facilities; variability in the mortality rate fits the pattern of emerging infectious diseases [ 46 ]. Furthermore, the reported cases did not consider asymptomatic cases, mild cases where individuals have not sought medical treatment, and the fact that many countries had limited access to diagnostic tests or have implemented testing policies later than the others. Considering the lack of reviews assessing diagnostic testing (sensitivity, specificity, and predictive values of RT-PCT or immunoglobulin tests), and the preponderance of studies that assessed only symptomatic individuals, considerable imprecision around the calculated mortality rates existed in the early stage of the COVID-19 pandemic.

Few reviews included treatment data. Those reviews described studies considered to be at a very low level of evidence: usually small, retrospective studies with very heterogeneous populations. Seven reviews analyzed laboratory parameters; those reviews could have been useful for clinicians who attend patients suspected of COVID-19 in emergency services worldwide, such as assessing which patients need to be reassessed more frequently.

All systematic reviews scored poorly on the AMSTAR 2 critical appraisal tool for systematic reviews. Most of the original studies included in the reviews were case series and case reports, impacting the quality of evidence. Such evidence has major implications for clinical practice and the use of these reviews in evidence-based practice and policy. Clinicians, patients, and policymakers can only have the highest confidence in systematic review findings if high-quality systematic review methodologies are employed. The urgent need for information during a pandemic does not justify poor quality reporting.

We acknowledge that there are numerous challenges associated with analyzing COVID-19 data during a pandemic [ 47 ]. High-quality evidence syntheses are needed for decision-making, but each type of evidence syntheses is associated with its inherent challenges.

The creation of classic systematic reviews requires considerable time and effort; with massive research output, they quickly become outdated, and preparing updated versions also requires considerable time. A recent study showed that updates of non-Cochrane systematic reviews are published a median of 5 years after the publication of the previous version [ 48 ].

Authors may register a review and then abandon it [ 49 ], but the existence of a public record that is not updated may lead other authors to believe that the review is still ongoing. A quarter of Cochrane review protocols remains unpublished as completed systematic reviews 8 years after protocol publication [ 50 ].

Rapid reviews can be used to summarize the evidence, but they involve methodological sacrifices and simplifications to produce information promptly, with inconsistent methodological approaches [ 51 ]. However, rapid reviews are justified in times of public health emergencies, and even Cochrane has resorted to publishing rapid reviews in response to the COVID-19 crisis [ 52 ]. Rapid reviews were eligible for inclusion in this overview, but only one of the 18 reviews included in this study was labeled as a rapid review.

Ideally, COVID-19 evidence would be continually summarized in a series of high-quality living systematic reviews, types of evidence synthesis defined as “ a systematic review which is continually updated, incorporating relevant new evidence as it becomes available ” [ 53 ]. However, conducting living systematic reviews requires considerable resources, calling into question the sustainability of such evidence synthesis over long periods [ 54 ].

Research reports about COVID-19 will contribute to research waste if they are poorly designed, poorly reported, or simply not necessary. In principle, systematic reviews should help reduce research waste as they usually provide recommendations for further research that is needed or may advise that sufficient evidence exists on a particular topic [ 55 ]. However, systematic reviews can also contribute to growing research waste when they are not needed, or poorly conducted and reported. Our present study clearly shows that most of the systematic reviews that were published early on in the COVID-19 pandemic could be categorized as research waste, as our confidence in their results is critically low.

Our study has some limitations. One is that for AMSTAR 2 assessment we relied on information available in publications; we did not attempt to contact study authors for clarifications or additional data. In three reviews, the methodological quality appraisal was challenging because they were published as letters, or labeled as rapid communications. As a result, various details about their review process were not included, leading to AMSTAR 2 questions being answered as “not reported”, resulting in low confidence scores. Full manuscripts might have provided additional information that could have led to higher confidence in the results. In other words, low scores could reflect incomplete reporting, not necessarily low-quality review methods. To make their review available more rapidly and more concisely, the authors may have omitted methodological details. A general issue during a crisis is that speed and completeness must be balanced. However, maintaining high standards requires proper resourcing and commitment to ensure that the users of systematic reviews can have high confidence in the results.

Furthermore, we used adjusted AMSTAR 2 scoring, as the tool was designed for critical appraisal of reviews of interventions. Some reviews may have received lower scores than actually warranted in spite of these adjustments.

Another limitation of our study may be the inclusion of multiple overlapping reviews, as some included reviews included the same primary studies. According to the Cochrane Handbook, including overlapping reviews may be appropriate when the review’s aim is “ to present and describe the current body of systematic review evidence on a topic ” [ 12 ], which was our aim. To avoid bias with summarizing evidence from overlapping reviews, we presented the forest plots without summary estimates. The forest plots serve to inform readers about the effect sizes for outcomes that were reported in each review.

Several authors from this study have contributed to one of the reviews identified [ 25 ]. To reduce the risk of any bias, two authors who did not co-author the review in question initially assessed its quality and limitations.

Finally, we note that the systematic reviews included in our overview may have had issues that our analysis did not identify because we did not analyze their primary studies to verify the accuracy of the data and information they presented. We give two examples to substantiate this possibility. Lovato et al. wrote a commentary on the review of Sun et al. [ 41 ], in which they criticized the authors’ conclusion that sore throat is rare in COVID-19 patients [ 56 ]. Lovato et al. highlighted that multiple studies included in Sun et al. did not accurately describe participants’ clinical presentations, warning that only three studies clearly reported data on sore throat [ 56 ].

In another example, Leung [ 57 ] warned about the review of Li, L.Q. et al. [ 29 ]: “ it is possible that this statistic was computed using overlapped samples, therefore some patients were double counted ”. Li et al. responded to Leung that it is uncertain whether the data overlapped, as they used data from published articles and did not have access to the original data; they also reported that they requested original data and that they plan to re-do their analyses once they receive them; they also urged readers to treat the data with caution [ 58 ]. This points to the evolving nature of evidence during a crisis.

Our study’s strength is that this overview adds to the current knowledge by providing a comprehensive summary of all the evidence synthesis about COVID-19 available early after the onset of the pandemic. This overview followed strict methodological criteria, including a comprehensive and sensitive search strategy and a standard tool for methodological appraisal of systematic reviews.

In conclusion, in this overview of systematic reviews, we analyzed evidence from the first 18 systematic reviews that were published after the emergence of COVID-19. However, confidence in the results of all the reviews was “critically low”. Thus, systematic reviews that were published early on in the pandemic could be categorized as research waste. Even during public health emergencies, studies and systematic reviews should adhere to established methodological standards to provide patients, clinicians, and decision-makers trustworthy evidence.

Availability of data and materials

All data collected and analyzed within this study are available from the corresponding author on reasonable request.

World Health Organization. Timeline - COVID-19: Available at: https://www.who.int/news/item/29-06-2020-covidtimeline . Accessed 1 June 2021.

COVID-19 Dashboard by the Center for Systems Science and Engineering (CSSE) at Johns Hopkins University (JHU). Available at: https://coronavirus.jhu.edu/map.html . Accessed 1 June 2021.

Anzai A, Kobayashi T, Linton NM, Kinoshita R, Hayashi K, Suzuki A, et al. Assessing the Impact of Reduced Travel on Exportation Dynamics of Novel Coronavirus Infection (COVID-19). J Clin Med. 2020;9(2):601.

Chinazzi M, Davis JT, Ajelli M, Gioannini C, Litvinova M, Merler S, et al. The effect of travel restrictions on the spread of the 2019 novel coronavirus (COVID-19) outbreak. Science. 2020;368(6489):395–400. https://doi.org/10.1126/science.aba9757 .

Article   CAS   PubMed   PubMed Central   Google Scholar  

Fidahic M, Nujic D, Runjic R, Civljak M, Markotic F, Lovric Makaric Z, et al. Research methodology and characteristics of journal articles with original data, preprint articles and registered clinical trial protocols about COVID-19. BMC Med Res Methodol. 2020;20(1):161. https://doi.org/10.1186/s12874-020-01047-2 .

EPPI Centre . COVID-19: a living systematic map of the evidence. Available at: http://eppi.ioe.ac.uk/cms/Projects/DepartmentofHealthandSocialCare/Publishedreviews/COVID-19Livingsystematicmapoftheevidence/tabid/3765/Default.aspx . Accessed 1 June 2021.

NCBI SARS-CoV-2 Resources. Available at: https://www.ncbi.nlm.nih.gov/sars-cov-2/ . Accessed 1 June 2021.

Gustot T. Quality and reproducibility during the COVID-19 pandemic. JHEP Rep. 2020;2(4):100141. https://doi.org/10.1016/j.jhepr.2020.100141 .

Article   PubMed   PubMed Central   Google Scholar  

Kodvanj, I., et al., Publishing of COVID-19 Preprints in Peer-reviewed Journals, Preprinting Trends, Public Discussion and Quality Issues. Preprint article. bioRxiv 2020.11.23.394577; doi: https://doi.org/10.1101/2020.11.23.394577 .

Dobler CC. Poor quality research and clinical practice during COVID-19. Breathe (Sheff). 2020;16(2):200112. https://doi.org/10.1183/20734735.0112-2020 .

Article   Google Scholar  

Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 2010;7(9):e1000326. https://doi.org/10.1371/journal.pmed.1000326 .

Lunny C, Brennan SE, McDonald S, McKenzie JE. Toward a comprehensive evidence map of overview of systematic review methods: paper 1-purpose, eligibility, search and data extraction. Syst Rev. 2017;6(1):231. https://doi.org/10.1186/s13643-017-0617-1 .

Pollock M, Fernandes RM, Becker LA, Pieper D, Hartling L. Chapter V: Overviews of Reviews. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.1 (updated September 2020). Cochrane. 2020. Available from www.training.cochrane.org/handbook .

Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al. Cochrane handbook for systematic reviews of interventions version 6.1 (updated September 2020). Cochrane. 2020; Available from www.training.cochrane.org/handbook .

Pollock M, Fernandes RM, Newton AS, Scott SD, Hartling L. The impact of different inclusion decisions on the comprehensiveness and complexity of overviews of reviews of healthcare interventions. Syst Rev. 2019;8(1):18. https://doi.org/10.1186/s13643-018-0914-3 .

Pollock M, Fernandes RM, Newton AS, Scott SD, Hartling L. A decision tool to help researchers make decisions about including systematic reviews in overviews of reviews of healthcare interventions. Syst Rev. 2019;8(1):29. https://doi.org/10.1186/s13643-018-0768-8 .

Hunt H, Pollock A, Campbell P, Estcourt L, Brunton G. An introduction to overviews of reviews: planning a relevant research question and objective for an overview. Syst Rev. 2018;7(1):39. https://doi.org/10.1186/s13643-018-0695-8 .

Pollock M, Fernandes RM, Pieper D, Tricco AC, Gates M, Gates A, et al. Preferred reporting items for overviews of reviews (PRIOR): a protocol for development of a reporting guideline for overviews of reviews of healthcare interventions. Syst Rev. 2019;8(1):335. https://doi.org/10.1186/s13643-019-1252-9 .

Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Open Med. 2009;3(3):e123–30.

Krnic Martinic M, Pieper D, Glatt A, Puljak L. Definition of a systematic review used in overviews of systematic reviews, meta-epidemiological studies and textbooks. BMC Med Res Methodol. 2019;19(1):203. https://doi.org/10.1186/s12874-019-0855-0 .

Puljak L. If there is only one author or only one database was searched, a study should not be called a systematic review. J Clin Epidemiol. 2017;91:4–5. https://doi.org/10.1016/j.jclinepi.2017.08.002 .

Article   PubMed   Google Scholar  

Gates M, Gates A, Guitard S, Pollock M, Hartling L. Guidance for overviews of reviews continues to accumulate, but important challenges remain: a scoping review. Syst Rev. 2020;9(1):254. https://doi.org/10.1186/s13643-020-01509-0 .

Covidence - systematic review software. Available at: https://www.covidence.org/ . Accessed 1 June 2021.

Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008.

Borges do Nascimento IJ, et al. Novel Coronavirus Infection (COVID-19) in Humans: A Scoping Review and Meta-Analysis. J Clin Med. 2020;9(4):941.

Article   PubMed Central   Google Scholar  

Adhikari SP, Meng S, Wu YJ, Mao YP, Ye RX, Wang QZ, et al. Epidemiology, causes, clinical manifestation and diagnosis, prevention and control of coronavirus disease (COVID-19) during the early outbreak period: a scoping review. Infect Dis Poverty. 2020;9(1):29. https://doi.org/10.1186/s40249-020-00646-x .

Cortegiani A, Ingoglia G, Ippolito M, Giarratano A, Einav S. A systematic review on the efficacy and safety of chloroquine for the treatment of COVID-19. J Crit Care. 2020;57:279–83. https://doi.org/10.1016/j.jcrc.2020.03.005 .

Li B, Yang J, Zhao F, Zhi L, Wang X, Liu L, et al. Prevalence and impact of cardiovascular metabolic diseases on COVID-19 in China. Clin Res Cardiol. 2020;109(5):531–8. https://doi.org/10.1007/s00392-020-01626-9 .

Article   CAS   PubMed   Google Scholar  

Li LQ, Huang T, Wang YQ, Wang ZP, Liang Y, Huang TB, et al. COVID-19 patients’ clinical characteristics, discharge rate, and fatality rate of meta-analysis. J Med Virol. 2020;92(6):577–83. https://doi.org/10.1002/jmv.25757 .

Lippi G, Lavie CJ, Sanchis-Gomar F. Cardiac troponin I in patients with coronavirus disease 2019 (COVID-19): evidence from a meta-analysis. Prog Cardiovasc Dis. 2020;63(3):390–1. https://doi.org/10.1016/j.pcad.2020.03.001 .

Lippi G, Henry BM. Active smoking is not associated with severity of coronavirus disease 2019 (COVID-19). Eur J Intern Med. 2020;75:107–8. https://doi.org/10.1016/j.ejim.2020.03.014 .

Lippi G, Plebani M. Procalcitonin in patients with severe coronavirus disease 2019 (COVID-19): a meta-analysis. Clin Chim Acta. 2020;505:190–1. https://doi.org/10.1016/j.cca.2020.03.004 .

Lippi G, Plebani M, Henry BM. Thrombocytopenia is associated with severe coronavirus disease 2019 (COVID-19) infections: a meta-analysis. Clin Chim Acta. 2020;506:145–8. https://doi.org/10.1016/j.cca.2020.03.022 .

Ludvigsson JF. Systematic review of COVID-19 in children shows milder cases and a better prognosis than adults. Acta Paediatr. 2020;109(6):1088–95. https://doi.org/10.1111/apa.15270 .

Lupia T, Scabini S, Mornese Pinna S, di Perri G, de Rosa FG, Corcione S. 2019 novel coronavirus (2019-nCoV) outbreak: a new challenge. J Glob Antimicrob Resist. 2020;21:22–7. https://doi.org/10.1016/j.jgar.2020.02.021 .

Marasinghe, K.M., A systematic review investigating the effectiveness of face mask use in limiting the spread of COVID-19 among medically not diagnosed individuals: shedding light on current recommendations provided to individuals not medically diagnosed with COVID-19. Research Square. Preprint article. doi : https://doi.org/10.21203/rs.3.rs-16701/v1 . 2020 .

Mullins E, Evans D, Viner RM, O’Brien P, Morris E. Coronavirus in pregnancy and delivery: rapid review. Ultrasound Obstet Gynecol. 2020;55(5):586–92. https://doi.org/10.1002/uog.22014 .

Pang J, Wang MX, Ang IYH, Tan SHX, Lewis RF, Chen JIP, et al. Potential Rapid Diagnostics, Vaccine and Therapeutics for 2019 Novel coronavirus (2019-nCoV): a systematic review. J Clin Med. 2020;9(3):623.

Rodriguez-Morales AJ, Cardona-Ospina JA, Gutiérrez-Ocampo E, Villamizar-Peña R, Holguin-Rivera Y, Escalera-Antezana JP, et al. Clinical, laboratory and imaging features of COVID-19: a systematic review and meta-analysis. Travel Med Infect Dis. 2020;34:101623. https://doi.org/10.1016/j.tmaid.2020.101623 .

Salehi S, Abedi A, Balakrishnan S, Gholamrezanezhad A. Coronavirus disease 2019 (COVID-19): a systematic review of imaging findings in 919 patients. AJR Am J Roentgenol. 2020;215(1):87–93. https://doi.org/10.2214/AJR.20.23034 .

Sun P, Qie S, Liu Z, Ren J, Li K, Xi J. Clinical characteristics of hospitalized patients with SARS-CoV-2 infection: a single arm meta-analysis. J Med Virol. 2020;92(6):612–7. https://doi.org/10.1002/jmv.25735 .

Yang J, Zheng Y, Gou X, Pu K, Chen Z, Guo Q, et al. Prevalence of comorbidities and its effects in patients infected with SARS-CoV-2: a systematic review and meta-analysis. Int J Infect Dis. 2020;94:91–5. https://doi.org/10.1016/j.ijid.2020.03.017 .

Bassetti M, Vena A, Giacobbe DR. The novel Chinese coronavirus (2019-nCoV) infections: challenges for fighting the storm. Eur J Clin Investig. 2020;50(3):e13209. https://doi.org/10.1111/eci.13209 .

Article   CAS   Google Scholar  

Hwang CS. Olfactory neuropathy in severe acute respiratory syndrome: report of a case. Acta Neurol Taiwanica. 2006;15(1):26–8.

Google Scholar  

Suzuki M, Saito K, Min WP, Vladau C, Toida K, Itoh H, et al. Identification of viruses in patients with postviral olfactory dysfunction. Laryngoscope. 2007;117(2):272–7. https://doi.org/10.1097/01.mlg.0000249922.37381.1e .

Rajgor DD, Lee MH, Archuleta S, Bagdasarian N, Quek SC. The many estimates of the COVID-19 case fatality rate. Lancet Infect Dis. 2020;20(7):776–7. https://doi.org/10.1016/S1473-3099(20)30244-9 .

Wolkewitz M, Puljak L. Methodological challenges of analysing COVID-19 data during the pandemic. BMC Med Res Methodol. 2020;20(1):81. https://doi.org/10.1186/s12874-020-00972-6 .

Rombey T, Lochner V, Puljak L, Könsgen N, Mathes T, Pieper D. Epidemiology and reporting characteristics of non-Cochrane updates of systematic reviews: a cross-sectional study. Res Synth Methods. 2020;11(3):471–83. https://doi.org/10.1002/jrsm.1409 .

Runjic E, Rombey T, Pieper D, Puljak L. Half of systematic reviews about pain registered in PROSPERO were not published and the majority had inaccurate status. J Clin Epidemiol. 2019;116:114–21. https://doi.org/10.1016/j.jclinepi.2019.08.010 .

Runjic E, Behmen D, Pieper D, Mathes T, Tricco AC, Moher D, et al. Following Cochrane review protocols to completion 10 years later: a retrospective cohort study and author survey. J Clin Epidemiol. 2019;111:41–8. https://doi.org/10.1016/j.jclinepi.2019.03.006 .

Tricco AC, Antony J, Zarin W, Strifler L, Ghassemi M, Ivory J, et al. A scoping review of rapid review methods. BMC Med. 2015;13(1):224. https://doi.org/10.1186/s12916-015-0465-6 .

COVID-19 Rapid Reviews: Cochrane’s response so far. Available at: https://training.cochrane.org/resource/covid-19-rapid-reviews-cochrane-response-so-far . Accessed 1 June 2021.

Cochrane. Living systematic reviews. Available at: https://community.cochrane.org/review-production/production-resources/living-systematic-reviews . Accessed 1 June 2021.

Millard T, Synnot A, Elliott J, Green S, McDonald S, Turner T. Feasibility and acceptability of living systematic reviews: results from a mixed-methods evaluation. Syst Rev. 2019;8(1):325. https://doi.org/10.1186/s13643-019-1248-5 .

Babic A, Poklepovic Pericic T, Pieper D, Puljak L. How to decide whether a systematic review is stable and not in need of updating: analysis of Cochrane reviews. Res Synth Methods. 2020;11(6):884–90. https://doi.org/10.1002/jrsm.1451 .

Lovato A, Rossettini G, de Filippis C. Sore throat in COVID-19: comment on “clinical characteristics of hospitalized patients with SARS-CoV-2 infection: a single arm meta-analysis”. J Med Virol. 2020;92(7):714–5. https://doi.org/10.1002/jmv.25815 .

Leung C. Comment on Li et al: COVID-19 patients’ clinical characteristics, discharge rate, and fatality rate of meta-analysis. J Med Virol. 2020;92(9):1431–2. https://doi.org/10.1002/jmv.25912 .

Li LQ, Huang T, Wang YQ, Wang ZP, Liang Y, Huang TB, et al. Response to Char’s comment: comment on Li et al: COVID-19 patients’ clinical characteristics, discharge rate, and fatality rate of meta-analysis. J Med Virol. 2020;92(9):1433. https://doi.org/10.1002/jmv.25924 .

Download references

Acknowledgments

We thank Catherine Henderson DPhil from Swanscoe Communications for pro bono medical writing and editing support. We acknowledge support from the Covidence Team, specifically Anneliese Arno. We thank the whole International Network of Coronavirus Disease 2019 (InterNetCOVID-19) for their commitment and involvement. Members of the InterNetCOVID-19 are listed in Additional file 6 . We thank Pavel Cerny and Roger Crosthwaite for guiding the team supervisor (IJBN) on human resources management.

This research received no external funding.

Author information

Authors and affiliations.

University Hospital and School of Medicine, Universidade Federal de Minas Gerais, Belo Horizonte, Minas Gerais, Brazil

Israel Júnior Borges do Nascimento & Milena Soriano Marcolino

Medical College of Wisconsin, Milwaukee, WI, USA

Israel Júnior Borges do Nascimento

Helene Fuld Health Trust National Institute for Evidence-based Practice in Nursing and Healthcare, College of Nursing, The Ohio State University, Columbus, OH, USA

Dónal P. O’Mathúna

School of Nursing, Psychotherapy and Community Health, Dublin City University, Dublin, Ireland

Department of Anesthesiology, Intensive Care and Pain Medicine, University of Münster, Münster, Germany

Thilo Caspar von Groote

Department of Sport and Health Science, Technische Universität München, Munich, Germany

Hebatullah Mohamed Abdulazeem

School of Health Sciences, Faculty of Health and Medicine, The University of Newcastle, Callaghan, Australia

Ishanka Weerasekara

Department of Physiotherapy, Faculty of Allied Health Sciences, University of Peradeniya, Peradeniya, Sri Lanka

Cochrane Croatia, University of Split, School of Medicine, Split, Croatia

Ana Marusic, Irena Zakarija-Grkovic & Tina Poklepovic Pericic

Center for Evidence-Based Medicine and Health Care, Catholic University of Croatia, Ilica 242, 10000, Zagreb, Croatia

Livia Puljak

Cochrane Brazil, Evidence-Based Health Program, Universidade Federal de São Paulo, São Paulo, Brazil

Vinicius Tassoni Civile & Alvaro Nagib Atallah

Yorkville University, Fredericton, New Brunswick, Canada

Santino Filoso

Laboratory for Industrial and Applied Mathematics (LIAM), Department of Mathematics and Statistics, York University, Toronto, Ontario, Canada

Nicola Luigi Bragazzi

You can also search for this author in PubMed   Google Scholar

Contributions

IJBN conceived the research idea and worked as a project coordinator. DPOM, TCVG, HMA, IW, AM, LP, VTC, IZG, TPP, ANA, SF, NLB and MSM were involved in data curation, formal analysis, investigation, methodology, and initial draft writing. All authors revised the manuscript critically for the content. The author(s) read and approved the final manuscript.

Corresponding author

Correspondence to Livia Puljak .

Ethics declarations

Ethics approval and consent to participate.

Not required as data was based on published studies.

Consent for publication

Not applicable.

Competing interests

The authors declare no conflict of interest.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: appendix 1..

Search strategies used in the study.

Additional file 2: Appendix 2.

Adjusted scoring of AMSTAR 2 used in this study for systematic reviews of studies that did not analyze interventions.

Additional file 3: Appendix 3.

List of excluded studies, with reasons.

Additional file 4: Appendix 4.

Table of overlapping studies, containing the list of primary studies included, their visual overlap in individual systematic reviews, and the number in how many reviews each primary study was included.

Additional file 5: Appendix 5.

A detailed explanation of AMSTAR scoring for each item in each review.

Additional file 6: Appendix 6.

List of members and affiliates of International Network of Coronavirus Disease 2019 (InterNetCOVID-19).

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Borges do Nascimento, I.J., O’Mathúna, D.P., von Groote, T.C. et al. Coronavirus disease (COVID-19) pandemic: an overview of systematic reviews. BMC Infect Dis 21 , 525 (2021). https://doi.org/10.1186/s12879-021-06214-4

Download citation

Received : 12 April 2020

Accepted : 19 May 2021

Published : 04 June 2021

DOI : https://doi.org/10.1186/s12879-021-06214-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Coronavirus
• Evidence-based medicine
• Infectious diseases

BMC Infectious Diseases

ISSN: 1471-2334

• Submission enquiries: [email protected]
• General enquiries: [email protected]

The Impact of COVID-19 on the Careers of Women in Academic Sciences, Engineering, and Medicine (2021)

Chapter: 8 major findings and research questions, 8 major findings and research questions, introduction.

The COVID-19 pandemic, which began in late 2019, created unprecedented global disruption and infused a significant level of uncertainty into the lives of individuals, both personally and professionally, around the world throughout 2020. The significant effect on vulnerable populations, such as essential workers and the elderly, is well documented, as is the devastating effect the COVID-19 pandemic had on the economy, particularly brick-and-mortar retail and hospitality and food services. Concurrently, the deaths of unarmed Black people at the hands of law enforcement officers created a heightened awareness of the persistence of structural injustices in U.S. society.

Against the backdrop of this public health crisis, economic upheaval, and amplified social consciousness, an ad hoc committee was appointed to review the potential effects of the COVID-19 pandemic on women in academic science, technology, engineering, mathematics, and medicine (STEMM) during 2020. The committee’s work built on the National Academies of Sciences, Engineering, and Medicine report Promising Practices for Addressing the Underrepresentation of Women in Science, Engineering, and Medicine: Opening Doors (the Promising Practices report), which presents evidence-based recommendations to address the well-established structural barriers that impede the advancement of women in STEMM. However, the committee recognized that none of the actions identified in the Promising Practices report were conceived within the context of a pandemic, an economic downturn, or the emergence of national protests against structural racism. The representation and vitality of academic women in STEMM had already warranted national attention prior to these events, and the COVID-19

pandemic appeared to represent an additional risk to the fragile progress that women had made in some STEMM disciplines. Furthermore, the future will almost certainly hold additional, unforeseen disruptions, which underscores the importance of the committee’s work.

In times of stress, there is a risk that the divide will deepen between those who already have advantages and those who do not. In academia, senior and tenured academics are more likely to have an established reputation, a stable salary commitment, and power within the academic system. They are more likely, before the COVID-19 pandemic began, to have established professional networks, generated data that can be used to write papers, and achieved financial and job security. While those who have these advantages may benefit from a level of stability relative to others during stressful times, those who were previously systemically disadvantaged are more likely to experience additional strain and instability.

As this report has documented, during 2020 the COVID-19 pandemic had overall negative effects on women in academic STEMM in areas such productivity, boundary setting and boundary control, networking and community building, burnout rates, and mental well-being. The excessive expectations of caregiving that often fall on the shoulders of women cut across career timeline and rank (e.g., graduate student, postdoctoral scholar, non-tenure-track and other contingent faculty, tenure-track faculty), institution type, and scientific discipline. Although there have been opportunities for innovation and some potential shifts in expectations, increased caregiving demands associated with the COVID-19 pandemic in 2020, such as remote working, school closures, and childcare and eldercare, had disproportionately negative outcomes for women.

The effects of the COVID-19 pandemic on women in STEMM during 2020 are understood better through an intentionally intersectional lens. Productivity, career, boundary setting, mental well-being, and health are all influenced by the ways in which social identities are defined and cultivated within social and power structures. Race and ethnicity, sexual orientation, gender identity, academic career stage, appointment type, institution type, age, and disability status, among many other factors, can amplify or diminish the effects of the COVID-19 pandemic for a given person. For example, non-cisgender women may be forced to return to home environments where their gender identity is not accepted, increasing their stress and isolation, and decreasing their well-being. Women of Color had a higher likelihood of facing a COVID-19–related death in their family compared with their white, non-Hispanic colleagues. The full extent of the effects of the COVID-19 pandemic for women of various social identities was not fully understood at the end of 2020.

Considering the relative paucity of women in many STEMM fields prior to the COVID-19 pandemic, women are more likely to experience academic isolation, including limited access to mentors, sponsors, and role models that share gender, racial, or ethnic identities. Combining this reality with the physical isolation stipulated by public health responses to the COVID-19 pandemic,

women in STEMM were subject to increasing isolation within their fields, networks, and communities. Explicit attention to the early indicators of how the COVID-19 pandemic affected women in academic STEMM careers during 2020, as well as attention to crisis responses throughout history, may provide opportunities to mitigate some of the long-term effects and potentially develop a more resilient and equitable academic STEMM system.

MAJOR FINDINGS

Given the ongoing nature of the COVID-19 pandemic, it was not possible to fully understand the entirety of the short- or long-term implications of this global disruption on the careers of women in academic STEMM. Having gathered preliminary data and evidence available in 2020, the committee found that significant changes to women’s work-life boundaries and divisions of labor, careers, productivity, advancement, mentoring and networking relationships, and mental health and well-being have been observed. The following findings represent those aspects that the committee agreed have been substantiated by the preliminary data, evidence, and information gathered by the end of 2020. They are presented either as Established Research and Experiences from Previous Events or Impacts of the COVID-19 Pandemic during 2020 that parallel the topics as presented in the report.

Established Research and Experiences from Previous Events

___________________

1 This finding is primarily based on research on cisgender women and men.

Impacts of the COVID-19 Pandemic during 2020

Research questions.

While this report compiled much of the research, data, and evidence available in 2020 on the effects of the COVID-19 pandemic, future research is still needed to understand all the potential effects, especially any long-term implications. The research questions represent areas the committee identified for future research, rather than specific recommendations. They are presented in six categories that parallel the chapters of the report: Cross-Cutting Themes; Academic Productivity and Institutional Responses; Work-Life Boundaries and Gendered Divisions of Labor; Collaboration, Networking, and Professional Societies; Academic Leadership and Decision-Making; and Mental Health and Well-being. The committee hopes the report will be used as a basis for continued understanding of the impact of the COVID-19 pandemic in its entirety and as a reference for mitigating impacts of future disruptions that affect women in academic STEMM. The committee also hopes that these research questions may enable academic STEMM to emerge from the pandemic era a stronger, more equitable place for women. Therefore, the committee identifies two types of research questions in each category; listed first are those questions aimed at understanding the impacts of the disruptions from the COVID-19 pandemic, followed by those questions exploring the opportunities to help support the full participation of women in the future.

Cross-Cutting Themes

• What are the short- and long-term effects of the COVID-19 pandemic on the career trajectories, job stability, and leadership roles of women, particularly of Black women and other Women of Color? How do these effects vary across institutional characteristics, 2 discipline, and career stage?

2 Institutional characteristics include different institutional types (e.g., research university, liberal arts college, community college), locales (e.g., urban, rural), missions (e.g., Historically Black Colleges and Universities, Hispanic-Serving Institutions, Asian American/Native American/Pacific Islander-Serving Institutions, Tribal Colleges and Universities), and levels of resources.

• How did the confluence of structural racism, economic hardships, and environmental disruptions affect Women of Color during the COVID-19 pandemic? Specifically, how did the murder of George Floyd, Breonna Taylor, and other Black citizens impact Black women academics’ safety, ability to be productive, and mental health?
• How has the inclusion of women in leadership and other roles in the academy influenced the ability of institutions to respond to the confluence of major social crises during the COVID-19 pandemic?
• How can institutions build on the involvement women had across STEMM disciplines during the COVID-19 pandemic to increase the participation of women in STEMM and/or elevate and support women in their current STEMM-related positions?
• How can institutions adapt, leverage, and learn from approaches developed during 2020 to attend to challenges experienced by Women of Color in STEMM in the future?

Academic Productivity and Institutional Responses

• How did the institutional responses (e.g., policies, practices) that were outlined in the Major Findings impact women faculty across institutional characteristics and disciplines?
• What are the short- and long-term effects of faculty evaluation practices and extension policies implemented during the COVID-19 pandemic on the productivity and career trajectories of members of the academic STEMM workforce by gender?
• What adaptations did women use during the transition to online and hybrid teaching modes? How did these techniques and adaptations vary as a function of career stage and institutional characteristics?
• What are examples of institutional changes implemented in response to the COVID-19 pandemic that have the potential to reduce systemic barriers to participation and advancement that have historically been faced by academic women in STEMM, specifically Women of Color and other marginalized women in STEMM? How might positive institutional responses be leveraged to create a more resilient and responsive higher education ecosystem?
• How can or should funding arrangements be altered (e.g., changes in funding for research and/or mentorship programs) to support new ways of interaction for women in STEMM during times of disruption, such as the COVID-19 pandemic?

Work-Life Boundaries and Gendered Divisions of Labor

• How do different social identities (e.g., racial; socioeconomic status; culturally, ethnically, sexually, or gender diverse; immigration status; parents of young children and other caregivers; women without partners) influence the management of work-nonwork boundaries? How did this change during the COVID-19 pandemic?
• How have COVID-19 pandemic-related disruptions affected progress toward reducing the gender gap in academic STEMM labor-force participation? How does this differ for Women of Color or women with caregiving responsibilities?
• How can institutions account for the unique challenges of women faculty with parenthood and caregiving responsibilities when developing effective and equitable policies, practices, or programs?
• How might insights gained about work-life boundaries during the COVID-19 pandemic inform how institutions develop and implement supportive resources (e.g., reductions in workload, on-site childcare, flexible working options)?

Collaboration, Networking, and Professional Societies

• What were the short- and long-term effects of the COVID-19 pandemic-prompted switch from in-person conferences to virtual conferences on conference culture and climate, especially for women in STEMM?
• How will the increase in virtual conferences specifically affect women’s advancement and career trajectories? How will it affect women’s collaborations?
• How has the shift away from attending conferences and in-person networking changed longer-term mentoring and sponsoring relationships, particularly in terms of gender dynamics?
• How can institutions maximize the benefits of digitization and the increased use of technology observed during the COVID-19 pandemic to continue supporting women, especially marginalized women, by increasing accessibility, collaborations, mentorship, and learning?
• How can organizations that support, host, or facilitate online and virtual conferences and networking events (1) ensure open and fair access to participants who face different funding and time constraints; (2) foster virtual connections among peers, mentors, and sponsors; and (3) maintain an inclusive environment to scientists of all backgrounds?
• What policies, practices, or programs can be developed to help women in STEMM maintain a sense of support, structure, and stability during and after periods of disruption?

Academic Leadership and Decision-Making

• What specific interventions did colleges and universities initiate or prioritize to ensure that women were included in decision-making processes during responses to the COVID-19 pandemic?
• How effective were colleges and universities that prioritized equity-minded leadership, shared leadership, and crisis leadership styles at mitigating emerging and potential negative effects of the COVID-19 pandemic on women in their communities?
• What specific aspects of different leadership models translated to more effective strategies to advance women in STEMM, particularly during the COVID-19 pandemic?
• How can examples of intentional inclusion of women in decision-making processes during the COVID-19 pandemic be leveraged to develop the engagement of women as leaders at all levels of academic institutions?
• What are potential “top-down” structural changes in academia that can be implemented to mitigate the adverse effects of the COVID-19 pandemic or other disruptions?
• How can academic leadership, at all levels, more effectively support the mental health needs of women in STEMM?

Mental Health and Well-being

• What is the impact of the COVID-19 pandemic and institutional responses on the mental health and well-being of members of the academic STEMM workforce as a function of gender, race, and career stage?
• How are tools and diagnostic tests to measure aspects of wellbeing, including burnout and insomnia, used in academic settings? How does this change during times of increased stress, such as the COVID-19 pandemic?
• How might insights gained about mental health during the COVID-19 pandemic be used to inform preparedness for future disruptions?
• How can programs that focus on changes in biomarkers of stress and mood dysregulation, such as levels of sleep, activity, and texting patterns, be developed and implemented to better engage women in addressing their mental health?
• What are effective interventions to address the health of women academics in STEMM that specifically account for the effects of stress on women? What are effective interventions to mitigate the excessive levels of stress for Women of Color?

This page intentionally left blank.

The spring of 2020 marked a change in how almost everyone conducted their personal and professional lives, both within science, technology, engineering, mathematics, and medicine (STEMM) and beyond. The COVID-19 pandemic disrupted global scientific conferences and individual laboratories and required people to find space in their homes from which to work. It blurred the boundaries between work and non-work, infusing ambiguity into everyday activities. While adaptations that allowed people to connect became more common, the evidence available at the end of 2020 suggests that the disruptions caused by the COVID-19 pandemic endangered the engagement, experience, and retention of women in academic STEMM, and may roll back some of the achievement gains made by women in the academy to date.

The Impact of COVID-19 on the Careers of Women in Academic Sciences, Engineering, and Medicine identifies, names, and documents how the COVID-19 pandemic disrupted the careers of women in academic STEMM during the initial 9-month period since March 2020 and considers how these disruptions - both positive and negative - might shape future progress for women. This publication builds on the 2020 report Promising Practices for Addressing the Underrepresentation of Women in Science, Engineering, and Medicine to develop a comprehensive understanding of the nuanced ways these disruptions have manifested. The Impact of COVID-19 on the Careers of Women in Academic Sciences, Engineering, and Medicine will inform the academic community as it emerges from the pandemic to mitigate any long-term negative consequences for the continued advancement of women in the academic STEMM workforce and build on the adaptations and opportunities that have emerged.

READ FREE ONLINE

Welcome to OpenBook!

You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

Do you want to take a quick tour of the OpenBook's features?

Show this book's table of contents , where you can jump to any chapter by name.

...or use these buttons to go back to the previous chapter or skip to the next one.

Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

Switch between the Original Pages , where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

To search the entire text of this book, type in your search term here and press Enter .

Share a link to this book page on your preferred social network or via email.

View our suggested citation for this chapter.

Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

Get Email Updates

Do you enjoy reading reports from the Academies online for free ? Sign up for email notifications and we'll let you know about new publications in your areas of interest when they're released.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 11 February 2021

Methodological quality of COVID-19 clinical research

• Richard G. Jung   ORCID: orcid.org/0000-0002-8570-6736 1 , 2 , 3   na1 ,
• Pietro Di Santo 1 , 2 , 4 , 5   na1 ,
• Cole Clifford 6 ,
• Graeme Prosperi-Porta 7 ,
• Stephanie Skanes 6 ,
• Annie Hung 8 ,
• Simon Parlow 4 ,
• Sarah Visintini   ORCID: orcid.org/0000-0001-6966-1753 9 ,
• F. Daniel Ramirez   ORCID: orcid.org/0000-0002-4350-1652 1 , 4 , 10 , 11 ,
• Trevor Simard 1 , 2 , 3 , 4 , 12 &
• Benjamin Hibbert   ORCID: orcid.org/0000-0003-0906-1363 2 , 3 , 4  

Nature Communications volume  12 , Article number:  943 ( 2021 ) Cite this article

13k Accesses

93 Citations

238 Altmetric

Metrics details

• Infectious diseases
• Public health

The COVID-19 pandemic began in early 2020 with major health consequences. While a need to disseminate information to the medical community and general public was paramount, concerns have been raised regarding the scientific rigor in published reports. We performed a systematic review to evaluate the methodological quality of currently available COVID-19 studies compared to historical controls. A total of 9895 titles and abstracts were screened and 686 COVID-19 articles were included in the final analysis. Comparative analysis of COVID-19 to historical articles reveals a shorter time to acceptance (13.0[IQR, 5.0–25.0] days vs. 110.0[IQR, 71.0–156.0] days in COVID-19 and control articles, respectively; p  < 0.0001). Furthermore, methodological quality scores are lower in COVID-19 articles across all study designs. COVID-19 clinical studies have a shorter time to publication and have lower methodological quality scores than control studies in the same journal. These studies should be revisited with the emergence of stronger evidence.

Similar content being viewed by others

A meta-analysis on global change drivers and the risk of infectious disease

Long COVID: major findings, mechanisms and recommendations

Refining the impact of genetic evidence on clinical success

Introduction.

The severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) pandemic spread globally in early 2020 with substantial health and economic consequences. This was associated with an exponential increase in scientific publications related to the coronavirus disease 2019 (COVID-19) in order to rapidly elucidate the natural history and identify diagnostic and therapeutic tools 1 .

While a need to rapidly disseminate information to the medical community, governmental agencies, and general public was paramount—major concerns have been raised regarding the scientific rigor in the literature 2 . Poorly conducted studies may originate from failure at any of the four consecutive research stages: (1) choice of research question relevant to patient care, (2) quality of research design 3 , (3) adequacy of publication, and (4) quality of research reports. Furthermore, evidence-based medicine relies on a hierarchy of evidence, ranging from the highest level of randomized controlled trials (RCT) to the lowest level of case series and case reports 4 .

Given the implications for clinical care, policy decision making, and concerns regarding methodological and peer-review standards for COVID-19 research 5 , we performed a formal evaluation of the methodological quality of published COVID-19 literature. Specifically, we undertook a systematic review to identify COVID-19 clinical literature and matched them to historical controls to formally evaluate the following: (1) the methodological quality of COVID-19 studies using established quality tools and checklists, (2) the methodological quality of COVID-19 studies, stratified by median time to acceptance, geographical regions, and journal impact factor and (3) a comparison of COVID-19 methodological quality to matched controls.

Herein, we show that COVID-19 articles are associated with lower methodological quality scores. Moreover, in a matched cohort analysis with control articles from the same journal, we reveal that COVID-19 articles are associated with lower quality scores and shorter time from submission to acceptance. Ultimately, COVID-19 clinical studies should be revisited with the emergence of stronger evidence.

Article selection

A total of 14787 COVID-19 papers were identified as of May 14, 2020 and 4892 duplicate articles were removed. In total, 9895 titles and abstracts were screened, and 9101 articles were excluded due to the study being pre-clinical in nature, case report, case series <5 patients, in a language other than English, reviews (including systematic reviews), study protocols or methods, and other coronavirus variants with an overall inter-rater study inclusion agreement of 96.7% ( κ  = 0.81; 95% CI, 0.79–0.83). A total number of 794 full texts were reviewed for eligibility. Over 108 articles were excluded for ineligible study design or publication type (such as letter to the editors, editorials, case reports or case series <5 patients), wrong patient population, non-English language, duplicate articles, wrong outcomes and publication in a non-peer-reviewed journal. Ultimately, 686 articles were identified with an inter-rater agreement of 86.5% ( κ  = 0.68; 95% CI, 0.67–0.70) (Fig.  1 ).

A total of 14787 articles were identified and 4892 duplicate articles were removed. Overall, 9895 articles were screened by title and abstract leaving 794 articles for full-text screening. Over 108 articles were excluded, leaving a total of 686 articles that underwent methodological quality assessment.

COVID-19 literature methodological quality

Most studies originated from Asia/Oceania with 469 (68.4%) studies followed by Europe with 139 (20.3%) studies, and the Americas with 78 (11.4%) studies. Of included studies, 380 (55.4%) were case series, 199 (29.0%) were cohort, 63 (9.2%) were diagnostic, 38 (5.5%) were case–control, and 6 (0.9%) were RCTs. Most studies (590, 86.0%) were retrospective in nature, 620 (90.4%) reported the sex of patients, and 7 (2.3%) studies excluding case series calculated their sample size a priori. The method of SARS-CoV-2 diagnosis was reported in 558 studies (81.3%) and ethics approval was obtained in 556 studies (81.0%). Finally, journal impact factor of COVID-19 manuscripts was 4.7 (IQR, 2.9–7.6) with a time to acceptance of 13.0 (IQR, 5.0–25.0) days (Table  1 ).

Overall, when COVID-19 articles were stratified by study design, a mean case series score (out of 5) (SD) of 3.3 (1.1), mean NOS cohort study score (out of 8) of 5.8 (1.5), mean NOS case–control study score (out of 8) of 5.5 (1.9), and low bias present in 4 (6.4%) diagnostic studies was observed (Table  2 and Fig.  2 ). Furthermore, in the 6 RCTs in the COVID-19 literature, there was a high risk of bias with little consideration for sequence generation, allocation concealment, blinding, incomplete outcome data, and selective outcome reporting (Table  2 ).

A Distribution of COVID-19 case series studies scored using the Murad tool ( n  = 380). B Distribution of COVID-19 cohort studies scored using the Newcastle–Ottawa Scale ( n  = 199). C Distribution of COVID-19 case–control studies scored using the Newcastle–Ottawa Scale ( n  = 38). D Distribution of COVID-19 diagnostic studies scored using the QUADAS-2 tool ( n  = 63). In panel D , blue represents low risk of bias and orange represents high risk of bias.

For secondary outcomes, rapid time from submission to acceptance (stratified by median time of acceptance of <13.0 days) was associated with lower methodological quality scores for case series and cohort study designs but not for case–control nor diagnostic studies (Fig.  3A–D ). Low journal impact factor (<10) was associated with lower methodological quality scores for case series, cohort, and case–control designs (Fig.  3E–H ). Finally, studies originating from different geographical regions had no differences in methodological quality scores with the exception of cohort studies (Fig.  3I–L ). When dichotomized by high vs. low methodological quality scores, a similar trend was observed with rapid time from submission to acceptance (34.4% vs. 46.3%, p  = 0.01, Supplementary Fig.  1B ), low impact factor journals (<10) was associated with lower methodological quality score (38.8% vs. 68.0%, p  < 0.0001, Supplementary Fig.  1C ). Finally, studies originating in either Americas or Asia/Oceania was associated with higher methodological quality scores than Europe (Supplementary Fig.  1D ).

A When stratified by time of acceptance (13.0 days), increased time of acceptance was associated with higher case series score ( n  = 186 for <13 days and n  = 193 for >=13 days; p  = 0.02). B Increased time of acceptance was associated with higher NOS cohort score ( n  = 112 for <13 days and n  = 144 for >=13 days; p  = 0.003). C No difference in time of acceptance and case–control score was observed ( n  = 18 for <13 days and n  = 27 for >=13 days; p  = 0.34). D No difference in time of acceptance and diagnostic risk of bias (QUADAS-2) was observed ( n  = 43 for <13 days and n  = 33 for >=13 days; p  = 0.23). E When stratified by impact factor (IF ≥10), high IF was associated with higher case series score ( n  = 466 for low IF and n  = 60 for high IF; p  < 0.0001). F High IF was associated with higher NOS cohort score ( n  = 262 for low IF and n  = 68 for high IF; p  = 0.01). G No difference in IF and case–control score was observed ( n  = 62 for low IF and n  = 2 for high IF; p  = 0.052). H No difference in IF and QUADAS-2 was observed ( n  = 101 for low IF and n  = 2 for high IF; p  = 0.93). I When stratified by geographical region, no difference in geographical region and case series score was observed ( n  = 276 Asia/Oceania, n  = 135 Americas, and n  = 143 Europe/Africa; p  = 0.10). J Geographical region was associated with differences in cohort score ( n  = 177 Asia/Oceania, n  = 81 Americas, and n  = 89 Europe/Africa; p  = 0.01). K No difference in geographical region and case–control score was observed ( n  = 37 Asia/Oceania, n  = 13 Americas, and n  = 14 Europe/Africa; p  = 0.81). L No difference in geographical region and QUADAS-2 was observed ( n  = 49 Asia/Oceania, n  = 28 Americas, and n  = 28 Europe/Africa; p  = 0.34). In panels A – D , orange represents lower median time of acceptance and blue represents high median time of acceptance. In panels E – H , red is low impact factor and blue is high impact factor. In panels I – L , orange represents Asia/Oceania, blue represents Americas, and brown represents Europe. Differences in distributions were analysed by two-sided Kruskal–Wallis test. Differences in diagnostic risk of bias were quantified by Chi-squares test. p  < 0.05 was considered statistically significant.

Methodological quality score differences in COVID-19 versus historical control

We matched 539 historical control articles to COVID-19 articles from the same journal with identical study designs in the previous year for a final analysis of 1078 articles (Table  1 ). Overall, 554 (51.4%) case series, 348 (32.3%) cohort, 64 (5.9%) case–control, 106 (9.8%) diagnostic and 6 (0.6%) RCTs were identified from the 1078 total articles. Differences exist between COVID-19 and historical control articles in geographical region of publication, retrospective study design, and sample size calculation (Table  1 ). Time of acceptance was 13.0 (IQR, 5.0–25.0) days in COVID-19 articles vs. 110.0 (IQR, 71.0–156.0) days in control articles (Table  1 and Fig.  4A , p  < 0.0001). Case-series methodological quality score was lower in COVID-19 articles compared to the historical control (3.3 (1.1) vs. 4.3 (0.8); n  = 554; p  < 0.0001; Table  2 and Fig.  4B ). Furthermore, NOS score was lower in COVID-19 cohort studies (5.8 (1.6) vs. 7.1 (1.0); n  = 348; p  < 0.0001; Table  2 and Fig.  4C ) and case–control studies (5.4 (1.9) vs. 6.6 (1.0); n  = 64; p  = 0.003; Table  2 and Fig.  4D ). Finally, lower risk of bias in diagnostic studies was in 12 COVID-19 articles (23%; n  = 53) compared to 24 control articles (45%; n  = 53; p  = 0.02; Table  2 and Fig.  4E ). A similar trend was observed between COVID-19 and historical control articles when dichotomized by good vs. low methodological quality scores (Supplementary Fig.  2 ).

A Time to acceptance was reduced in COVID-19 articles compared to control articles (13.0 [IQR, 5.0–25.0] days vs. 110.0 [IQR, 71.0–156.0] days, n  = 347 for COVID-19 and n  = 414 for controls; p  < 0.0001). B When compared to historical control articles, COVID-19 articles were associated with lower case series score ( n  = 277 for COVID-19 and n  = 277 for controls; p  < 0.0001). C COVID-19 articles were associated with lower NOS cohort score compared to historical control articles ( n  = 174 for COVID-19 and n  = 174 for controls; p  < 0.0001). D COVID-19 articles were associated with lower NOS case–control score compared to historical control articles ( n  = 32 for COVID-19 and n  = 32 for controls; p  = 0.003). E COVID-19 articles were associated with higher diagnostic risk of bias (QUADAS-2) compared to historical control articles ( n  = 53 for COVID-19 and n  = 53 for controls; p  = 0.02). For panel A , boxplot captures 5, 25, 50, 75 and 95% from the first to last whisker. Orange represents COVID-19 articles and blue represents control articles. Two-sided Mann–Whitney U-test was conducted to evaluate differences in time to acceptance between COVID-19 and control articles. Differences in study quality scores were evaluated by two-sided Kruskal–Wallis test. Differences in diagnostic risk of bias were quantified by Chi-squares test. p  < 0.05 was considered statistically significant.

In this systematic evaluation of methodological quality, COVID-19 clinical research was primarily observational in nature with modest methodological quality scores. Not only were the study designs low in the hierarchy of scientific evidence, we found that COVID-19 articles were associated with a lower methodological quality scores when published with a shorter time of publication and in lower impact factor journals. Furthermore, in a matched cohort analysis with historical control articles identified from the same journal of the same study design, we demonstrated that COVID-19 articles were associated with lower quality scores and shorter time from submission to acceptance.

The present study demonstrates comparative differences in methodological quality scores between COVID-19 literature and historical control articles. Overall, the accelerated publication of COVID-19 research was associated with lower study quality scores compared to previously published historical control studies. Our research highlights major differences in study quality between COVID-19 and control articles, possibly driven in part by a combination of more thorough editorial and/or peer-review process as suggested by the time to publication, and robust study design with questions which are pertinent for clinicians and patient management 3 , 6 , 7 , 8 , 9 , 10 , 11 .

In the early stages of the COVID-19 pandemic, we speculate that an urgent need for scientific data to inform clinical, social and economic decisions led to shorter time to publication and explosion in publication of COVID-19 studies in both traditional peer-reviewed journals and preprint servers 1 , 12 . The accelerated scientific process in the COVID-19 pandemic allowed a rapid understanding of natural history of COVID-19 symptomology and prognosis, identification of tools including RT-PCR to diagnose SARS-CoV-2 13 , and identification of potential therapeutic options such as tocilizumab and convalescent plasma which laid the foundation for future RCTs 14 , 15 , 16 . A delay in publication of COVID-19 articles due to a slower peer-review process may potentially delay dissemination of pertinent information against the pandemic. Despite concerns of slow peer review, major landmark trials (i.e. RECOVERY and ACTT-1 trial) 17 , 18 published their findings in preprint servers and media releases to allow for rapid dissemination. Importantly, the data obtained in these initial studies should be revisited as stronger data emerges as lower quality studies may fundamentally risk patient safety, resource allocation and future scientific research 19 .

Unfortunately, poor evidence begets poor clinical decisions 20 . Furthermore, lower quality scientific evidence potentially undermines the public’s trust in science during this time and has been evident through misleading information and high-profile retractions 12 , 21 , 22 , 23 . For example, the benefits of hydroxychloroquine, which were touted early in the pandemic based on limited data, have subsequently failed to be replicated in multiple observational studies and RCTs 5 , 24 , 25 , 26 , 27 , 28 , 29 , 30 . One poorly designed study combined with rapid publication led to considerable investment of both the scientific and medical community—akin to quinine being sold to the public as a miracle drug during the 1918 Spanish Influenza 31 , 32 . Moreover, as of June 30, 2020, ClinicalTrials.gov listed an astonishing 230 COVID-19 trials with hydroxychloroquine/plaquenil, and a recent living systematic review of observational studies and RCTs of hydroxychloroquine or chloroquine for COVID-19 demonstrated no evidence of benefit nor harm with concerns of severe methodological flaws in the included studies 33 .

Our study has important limitations. We evaluated the methodological quality of existing studies using established checklists and tools. While it is tempting to associate methodological quality scores with reproducibility or causal inferences of the intervention, it is not possible to ascertain the impact on the study design and conduct of research nor results or conclusions in the identified reports 34 . Second, although the methodological quality scales and checklists used for the manuscript are commonly used for quality assessment in systematic reviews and meta-analyses 35 , 36 , 37 , 38 , they can only assess the methodology without consideration for causal language and are prone to limitations 39 , 40 . Other tools such as the ROBINS-I and GRADE exist to evaluate methodological quality of identified manuscripts, although no consensus currently exists for critical appraisal of non-randomized studies 41 , 42 , 43 . Furthermore, other considerations of quality such as sample size calculation, sex reporting or ethics approval are not considered in these quality scores. As such, the quality scores measured using these checklists only reflect the patient selection, comparability, diagnostic reference standard and methods to ascertain the outcome of the study. Third, the 1:1 ratio to identify our historical control articles may affect the precision estimates of our findings. Interestingly, a simulation of an increase from 1:1 to 1:4 control ratio tightened the precision estimates but did not significantly alter the point estimate 44 . Furthermore, the decision for 1:1 ratio in our study exists due to limitations of available historical control articles from the identical journal in the restricted time period combined with a large effect size and sample size in the analysis. Finally, our analysis includes early publications on COVID-19 and there is likely to be an improvement in quality of related studies and study design as the field matures and higher-quality studies. Accordingly, our findings are limited to the early body of research as it pertains to the pandemic and it is likely that over time research quality will improve over time.

In summary, the early body of peer-reviewed COVID-19 literature was composed primarily of observational studies that underwent shorter peer-review evaluation and were associated with lower methodological quality scores than comparable studies. COVID-19 clinical studies should be revisited with the emergence of stronger evidence.

A systematic literature search was conducted on May 14, 2020 (registered on June 3, 2020 at PROSPERO: CRD42020187318) and reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses. Furthermore, the cohort study was reported according to the Strengthening The Reporting of Observational Studies in Epidemiology checklist. The data supporting the findings of this study is available as Supplementary Data  1 – 2 .

Data sources and searches

The search was created in MEDLINE by a medical librarian with expertise in systematic reviews (S.V.) using a combination of key terms and index headings related to COVID-19 and translated to the remaining bibliographic databases (Supplementary Tables  1 – 3 ). The searches were conducted in MEDLINE (Ovid MEDLINE(R) ALL 1946–), Embase (Ovid Embase Classic + Embase 1947–) and the Cochrane Central Register of Controlled Trials (from inception). Search results were limited to English-only publications, and a publication date limit of January 1, 2019 to present was applied. In addition, a Canadian Agency for Drugs and Technologies in Health search filter was applied in MEDLINE and Embase to remove animal studies, and commentary, newspaper article, editorial, letter and note publication types were also eliminated. Search results were exported to Covidence (Veritas Health Innovation, Melbourne, Australia) and duplicates were eliminated using the platform’s duplicate identification feature.

Study selection, data extraction and methodological quality assessment

We included all types of COVID-19 clinical studies, including case series, observational studies, diagnostic studies and RCTs. For diagnostic studies, the reference standard for COVID-19 diagnosis was defined as a nasopharyngeal swab followed by reverse transcriptase-polymerase chain reaction in order to detect SARS-CoV-2. We excluded studies that were exploratory or pre-clinical in nature (i.e. in vitro or animal studies), case reports or case series of <5 patients, studies published in a language other than English, reviews, methods or protocols, and other coronavirus variants such as the Middle East respiratory syndrome.

The review team consisted of trained research staff with expertise in systematic reviews and one trainee. Title and abstracts were evaluated by two independent reviewers using Covidence and all discrepancies were resolved by consensus. Articles that were selected for full review were independently evaluated by two reviewers for quality assessment using a standardized case report form following the completion of a training period where all reviewers were trained with the original manuscripts which derived the tools or checklists along with examples for what were deemed high scores 35 , 36 , 37 , 38 . Following this, reviewers completed thirty full-text extractions and the two reviewers had to reach consensus and the process was repeated for the remaining manuscripts independently. When two independent reviewers were not able reach consensus, a third reviewer (principal investigator) provided oversight in the process to resolve the conflicted scores.

First and corresponding author names, date of publication, title of manuscript and journal of publication were collected for all included full-text articles. Journal impact factor was obtained from the 2018 InCites Journal Citation Reports from Clarivate Analytics. Submission and acceptance dates were collected in manuscripts when available. Other information such as study type, prospective or retrospective study, sex reporting, sample size calculation, method of SARS-CoV-2 diagnosis and ethics approval was collected by the authors. Methodological quality assessment was conducted using the Newcastle–Ottawa Scale (NOS) for case–control and cohort studies 37 , QUADAS-2 tool for diagnostic studies 38 , Cochrane risk of bias for RCTs 35 and a score derived by Murad et al. for case series studies 36 .

Identification of historical control from identified COVID-19 articles

Following the completion of full-text extraction of COVID-19 articles, we obtained a historical control group by identifying reports matched in a 1:1 fashion. From the eligible COVID-19 article, historical controls were identified by searching the same journal in a systematic fashion by matching the same study design (“case series”, “cohort”, “case control” or “diagnostic”) starting in the journal edition 12 months prior to the COVID-19 article publication on the publisher website (i.e. COVID-19 article published on April 2020, going backwards to April 2019) and proceeding forward (or backward if a specific article type was not identified) in a temporal fashion until the first matched study was identified following abstract screening by two independent reviewers. If no comparison article was found by either reviewers, the corresponding COVID-19 article was excluded from the comparison analysis. Following the identification of the historical control, data extraction and quality assessment was conducted on the identified articles using the standardized case report forms by two independent reviewers and conflicts resolved by consensus. The full dataset has been made available as Supplementary Data  1 – 2 .

Data synthesis and statistical analysis

Continuous variables were reported as mean (SD) or median (IQR) as appropriate, and categorical variables were reported as proportions (%). Continuous variables were compared using Student t -test or Mann–Whitney U-test and categorical variables including quality scores were compared by χ 2 , Fisher’s exact test, or Kruskal–Wallis test.

The primary outcome of interest was to evaluate the methodological quality of COVID-19 clinical literature by study design using the Newcastle–Ottawa Scale (NOS) for case–control and cohort studies, QUADAS-2 tool for diagnostic studies 38 , Cochrane risk of bias for RCTs 35 , and a score derived by Murad et al. for case series studies 36 . Pre-specified secondary outcomes were comparison of methodological quality scores of COVID-19 articles by (i) median time to acceptance, (ii) impact factor, (iii) geographical region and (iv) historical comparator. Time of acceptance was defined as the time between submission to acceptance which captures peer review and editorial decisions. Geographical region was stratified into continents including Asia/Oceania, Europe/Africa and Americas (North and South America). Post hoc comparison analysis between COVID-19 and historical control article quality scores were evaluated using Kruskal–Wallis test. Furthermore, good quality of NOS was defined as 3+ on selection and 1+ on comparability, and 2+ on outcome/exposure domains and high-quality case series scores was defined as a score ≥3.5. Due to a small sample size of identified RCTs, they were not included in the comparison analysis.

The finalized dataset was collected on Microsoft Excel v16.44. All statistical analyses were performed using SAS v9.4 (SAS Institute, Inc., Cary, NC, USA). Statistical significance was defined as P  < 0.05. All figures were generated using GraphPad Prism v8 (GraphPad Software, La Jolla, CA, USA).

Reporting summary

Further information on research design is available in the  Nature Research Reporting Summary linked to this article.

Data availability

The authors can confirm that all relevant data are included in the paper and in Supplementary Data  1 – 2 . The original search was conducted on MEDLINE, Embase and Cochrane Central Register of Controlled Trials.

Chen, Q., Allot, A. & Lu, Z. Keep up with the latest coronavirus research. Nature 579 , 193 (2020).

Article   ADS   CAS   Google Scholar  

Mahase, E. Covid-19: 146 researchers raise concerns over chloroquine study that halted WHO trial. BMJ https://doi.org/10.1136/bmj.m2197 (2020).

Chalmers, I. & Glasziou, P. Avoidable waste in the production and reporting of research evidence. Lancet 374 , 86–89 (2009).

Article   Google Scholar  

Burns, P. B., Rohrich, R. J. & Chung, K. C. The levels of evidence and their role in evidence-based medicine. Plast. Reconstr. Surg. 128 , 305–310 (2011).

Article   CAS   Google Scholar  

Alexander, P. E. et al. COVID-19 coronavirus research has overall low methodological quality thus far: case in point for chloroquine/hydroxychloroquine. J. Clin. Epidemiol. 123 , 120–126 (2020).

Barakat, A. F., Shokr, M., Ibrahim, J., Mandrola, J. & Elgendy, I. Y. Timeline from receipt to online publication of COVID-19 original research articles. Preprint at medRxiv https://doi.org/10.1101/2020.06.22.20137653 (2020).

Chan, A.-W. et al. Increasing value and reducing waste: addressing inaccessible research. Lancet 383 , 257–266 (2014).

Ioannidis, J. P. A. et al. Increasing value and reducing waste in research design, conduct, and analysis. Lancet 383 , 166–175 (2014).

Chalmers, I. et al. How to increase value and reduce waste when research priorities are set. Lancet 383 , 156–165 (2014).

Salman, R. A.-S. et al. Increasing value and reducing waste in biomedical research regulation and management. Lancet 383 , 176–185 (2014).

Glasziou, P. et al. Reducing waste from incomplete or unusable reports of biomedical research. Lancet 383 , 267–276 (2014).

Bauchner, H. The rush to publication: an editorial and scientific mistake. JAMA 318 , 1109–1110 (2017).

He, X. et al. Temporal dynamics in viral shedding and transmissibility of COVID-19. Nat. Med. 26 , 672–675 (2020).

Guaraldi, G. et al. Tocilizumab in patients with severe COVID-19: a retrospective cohort study. Lancet Rheumatol. 2 , e474–e484 (2020).

Duan, K. et al. Effectiveness of convalescent plasma therapy in severe COVID-19 patients. Proc. Natl Acad. Sci. USA 117 , 9490–9496 (2020).

Shen, C. et al. Treatment of 5 critically Ill patients with COVID-19 with convalescent plasma. JAMA 323 , 1582–1589 (2020).

Beigel, J. H. et al. Remdesivir for the treatment of covid-19—final report. N. Engl. J. Med. 383 , 1813–1826 (2020).

Group, R. C. et al. Dexamethasone in hospitalized patients with Covid-19—preliminary report. N. Engl. J. Med. https://doi.org/10.1056/NEJMoa2021436 (2020).

Ramirez, F. D. et al. Methodological rigor in preclinical cardiovascular studies: targets to enhance reproducibility and promote research translation. Circ. Res 120 , 1916–1926 (2017).

Heneghan, C. et al. Evidence based medicine manifesto for better healthcare. BMJ 357 , j2973 (2017).

Mehra, M. R., Desai, S. S., Ruschitzka, F. & Patel, A. N. RETRACTED: hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis. Lancet https://doi.org/10.1016/S0140-6736(20)31180-6 (2020).

Servick, K. & Enserink, M. The pandemic’s first major research scandal erupts. Science 368 , 1041–1042 (2020).

Mehra, M. R., Desai, S. S., Kuy, S., Henry, T. D. & Patel, A. N. Retraction: Cardiovascular disease, drug therapy, and mortality in Covid-19. N. Engl. J. Med. 382 , 2582–2582, https://doi.org/10.1056/NEJMoa2007621. (2020).

Article   PubMed   Google Scholar  

Boulware, D. R. et al. A randomized trial of hydroxychloroquine as postexposure prophylaxis for Covid-19. N. Engl. J. Med. 383 , 517–525 (2020).

Gautret, P. et al. Clinical and microbiological effect of a combination of hydroxychloroquine and azithromycin in 80 COVID-19 patients with at least a six-day follow up: a pilot observational study. Travel Med. Infect. Dis. 34 , 101663–101663 (2020).

Geleris, J. et al. Observational study of hydroxychloroquine in hospitalized patients with Covid-19. N. Engl. J. Med. 382 , 2411–2418 (2020).

Borba, M. G. S. et al. Effect of high vs low doses of chloroquine diphosphate as adjunctive therapy for patients hospitalized with severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection: a randomized clinical trial. JAMA Netw. Open 3 , e208857–e208857 (2020).

Mercuro, N. J. et al. Risk of QT interval prolongation associated with use of hydroxychloroquine with or without concomitant azithromycin among hospitalized patients testing positive for coronavirus disease 2019 (COVID-19). JAMA Cardiol. 5 , 1036–1041 (2020).

Molina, J. M. et al. No evidence of rapid antiviral clearance or clinical benefit with the combination of hydroxychloroquine and azithromycin in patients with severe COVID-19 infection. Médecine et. Maladies Infectieuses 50 , 384 (2020).

Group, R. C. et al. Effect of hydroxychloroquine in hospitalized patients with Covid-19. N. Engl. J. Med . 383, 2030–2040 (2020).

Shors, T. & McFadden, S. H. 1918 influenza: a Winnebago County, Wisconsin perspective. Clin. Med. Res. 7 , 147–156 (2009).

Stolberg, S. A Mad Scramble to Stock Millions of Malaria Pills, Likely for Nothing (The New York Times, 2020).

Hernandez, A. V., Roman, Y. M., Pasupuleti, V., Barboza, J. J. & White, C. M. Hydroxychloroquine or chloroquine for treatment or prophylaxis of COVID-19: a living systematic review. Ann. Int. Med. 173 , 287–296 (2020).

Glasziou, P. & Chalmers, I. Research waste is still a scandal—an essay by Paul Glasziou and Iain Chalmers. BMJ 363 , k4645 (2018).

Higgins, J. P. T. et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ 343 , d5928 (2011).

Murad, M. H., Sultan, S., Haffar, S. & Bazerbachi, F. Methodological quality and synthesis of case series and case reports. BMJ Evid. Based Med. 23 , 60–63 (2018).

Wells, G. S. B. et al. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analysis. http://wwwohrica/programs/clinical_epidemiology/oxfordasp (2004).

Whiting, P. F. et al. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann. Intern. Med. 155 , 529–536 (2011).

Sanderson, S., Tatt, I. D. & Higgins, J. P. Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int. J. Epidemiol. 36 , 666–676 (2007).

Stang, A. Critical evaluation of the Newcastle-Ottawa scale for the assessment of the quality of nonrandomized studies in meta-analyses. Eur. J. Epidemiol. 25 , 603–605 (2010).

Guyatt, G. et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J. Clin. Epidemiol. 64 , 383–394 (2011).

Quigley, J. M., Thompson, J. C., Halfpenny, N. J. & Scott, D. A. Critical appraisal of nonrandomized studies-A review of recommended and commonly used tools. J. Evaluation Clin. Pract. 25 , 44–52 (2019).

Sterne, J. A. et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ 355 , i4919 (2016).

Hamajima, N. et al. Case-control studies: matched controls or all available controls? J. Clin. Epidemiol. 47 , 971–975 (1994).

Download references

Acknowledgements

This study received no specific funding or grant from any agency in the public, commercial, or not-for-profit sectors. R.G.J. was supported by the Vanier CIHR Canada Graduate Scholarship. F.D.R. was supported by a CIHR Banting Postdoctoral Fellowship and a Royal College of Physicians and Surgeons of Canada Detweiler Travelling Fellowship. The funder/sponsor(s) had no role in design and conduct of the study, collection, analysis and interpretation of the data.

Author information

These authors contributed equally: Richard G. Jung, Pietro Di Santo.

Authors and Affiliations

CAPITAL Research Group, University of Ottawa Heart Institute, Ottawa, Ontario, Canada

Richard G. Jung, Pietro Di Santo, F. Daniel Ramirez & Trevor Simard

Vascular Biology and Experimental Medicine Laboratory, University of Ottawa Heart Institute, Ottawa, Ontario, Canada

Richard G. Jung, Pietro Di Santo, Trevor Simard & Benjamin Hibbert

Department of Cellular and Molecular Medicine, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada

Richard G. Jung, Trevor Simard & Benjamin Hibbert

Division of Cardiology, University of Ottawa Heart Institute, Ottawa, Ontario, Canada

Pietro Di Santo, Simon Parlow, F. Daniel Ramirez, Trevor Simard & Benjamin Hibbert

School of Epidemiology and Public Health, University of Ottawa, Ottawa, Ontario, Canada

Pietro Di Santo

Faculty of Medicine, University of Ottawa, Ontario, Canada

Cole Clifford & Stephanie Skanes

Department of Medicine, Cumming School of Medicine, Calgary, Alberta, Canada

Graeme Prosperi-Porta

Division of Internal Medicine, The Ottawa Hospital, Ottawa, Ontario, Canada

Berkman Library, University of Ottawa Heart Institute, Ottawa, Ontario, Canada

Sarah Visintini

Hôpital Cardiologique du Haut-Lévêque, CHU Bordeaux, Bordeaux-Pessac, France

F. Daniel Ramirez

L’Institut de Rythmologie et Modélisation Cardiaque (LIRYC), University of Bordeaux, Bordeaux, France

Department of Cardiovascular Medicine, Mayo Clinic, Rochester, MN, USA

Trevor Simard

You can also search for this author in PubMed   Google Scholar

Contributions

R.G.J., P.D.S., S.V., F.D.R., T.S. and B.H. participated in the study conception and design. Data acquisition, analysis and interpretation were performed by R.G.J., P.D.S., C.C., G.P.P., S.P., S.S., A.H., F.D.R., T.S. and B.H. Statistical analysis was performed by R.G.J., P.D.S. and B.H. The manuscript was drafted by R.G.J., P.D.S., F.D.R., T.S. and B.H. All authors approved the final version of the manuscript and agree to be accountable to all aspects of the work.

Corresponding author

Correspondence to Benjamin Hibbert .

Ethics declarations

Competing interests.

B.H. reports funding as a clinical trial investigator from Abbott, Boston Scientific and Edwards Lifesciences outside of the submitted work. The remaining authors declare no competing interests.

Additional information

Peer review information Nature Communications Ian White and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Peer reviewer reports are available.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary information, peer review file, description of additional supplementary files, supplementary data 1, supplementary data 2, reporting summary, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Jung, R.G., Di Santo, P., Clifford, C. et al. Methodological quality of COVID-19 clinical research. Nat Commun 12 , 943 (2021). https://doi.org/10.1038/s41467-021-21220-5

Download citation

Received : 16 July 2020

Accepted : 13 January 2021

Published : 11 February 2021

DOI : https://doi.org/10.1038/s41467-021-21220-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

The quality of covid-19 systematic reviews during the coronavirus 2019 pandemic: an exploratory comparison.

• Kevin T. McDermott
• Jos Kleijnen

Systematic Reviews (2024)

Gatekeeping should be conserved in the open science era

• Hugh Desmond

Synthese (2024)

Exploring COVID-19 research credibility among Spanish scientists

• Eduardo Garcia-Garzon
• Ariadna Angulo-Brunet
• Guido Corradi

Current Psychology (2024)

Primary health care research in COVID-19: analysis of the protocols reviewed by the ethics committee of IDIAPJGol, Catalonia

• Anna Moleras-Serra
• Rosa Morros-Pedros
• Ainhoa Gómez-Lumbreras

BMC Primary Care (2023)

Identifying patterns of reported findings on long-term cardiac complications of COVID-19: a systematic review and meta-analysis

• Chenya Zhao

BMC Medicine (2023)

By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

• U.S. Department of Health & Human Services

• Virtual Tour
• Staff Directory
• En Español

You are here

News releases.

Media Advisory

Thursday, April 23, 2020

NIAID strategic plan details COVID-19 research priorities

Urgent public health measures are needed to control the spread of the novel coronavirus (SARS-CoV-2) and the disease it causes, coronavirus disease 2019, or COVID-19.  Scientific research to improve our understanding of the virus and how it causes disease, and to develop strategies to mitigate illness and death, is of paramount importance. A new strategic plan from the National Institute of Allergy and Infectious Diseases (NIAID), part of the National Institutes of Health, details the institute’s plan for accelerating research to diagnose, prevent and treat COVID-19.

The NIAID Strategic Plan for COVID-19 Research has four key priorities. The first involves improving fundamental knowledge of SARS-CoV-2 and COVID-19, including studies to characterize the virus and better understand how it causes infection and disease. This research includes natural history, transmission and surveillance studies to determine why some individuals experience mild symptoms of infection while others become critically ill. The role of asymptomatic individuals in viral spread and the potential seasonality of viral circulation also need to be explored, according to the report. Additionally, small and large animal models that can recapitulate COVID-19 disease seen in humans must be developed.

NIAID’s second research priority is the development of rapid, accurate diagnostics and assays to identify and isolate COVID-19 cases and track the spread of the virus.  Molecular assays can detect low levels of SARS-CoV-2 and differentiate it from other related viruses. Researchers will work to improve the speed and accuracy of these diagnostic assays to mitigate the spread of the disease during the current outbreak and any future ones. Additionally, new and improved serologic assays to detect antibodies to the virus must be developed to enhance surveillance efforts and identify individuals who may have resolved a previous COVID-19 infection.

The third research priority is characterizing and testing potential treatments for COVID-19. These efforts will include identifying and evaluating drugs already approved for other conditions that could be repurposed to treat COVID-19 and testing novel broad-spectrum antivirals, such as remdesivir ; virus-targeted antibody-based therapies; monoclonal antibodies; and host-directed strategies to target an individual’s immune response to the virus. To optimize findings during the pandemic, multiple clinical trials will be conducted in parallel among various patient populations, including hospitalized people and outpatients.

NIAID’s fourth research priority is to develop safe and effective vaccines to protect individuals from infection and prevent future SARS-CoV-2 outbreaks.  NIAID researchers and their collaborators are adapting vaccine candidates and approaches previously employed to address the related Middle East respiratory syndrome (MERS) and severe acute respiratory syndrome (SARS) coronaviruses and applied them to the current pandemic. For example, NIAID recently launched a Phase 1 clinical trial using a vaccine platform initially developed to target MERS . NIAID will use its broad clinical trial infrastructure to advance experimental vaccines through Phase 1 safety and dosing testing and simultaneously plan for advanced clinical testing of the most promising candidates. The institute will work with government partners to ensure that any safe and effective vaccine will be manufactured in sufficient quantities to allow expedient distribution to those at highest risk for infection. 

To achieve its four priorities, NIAID will build on its current resources, research programs, clinical trials networks and collaborations with other U.S. government agencies and other key U.S. and global partners. The new strategic plan aligns with priorities set by the White House Coronavirus Task Force and represents a comprehensive and coordinated effort to develop effective biomedical tools to combat COVID-19.

Publication

NIAID Strategic Plan for COVID-19 Research

NIAID Director Anthony S. Fauci, M.D., is available to comment on the strategic plan.

NIAID conducts and supports research—at NIH, throughout the United States, and worldwide—to study the causes of infectious and immune-mediated diseases, and to develop better means of preventing, diagnosing and treating these illnesses. News releases, fact sheets and other NIAID-related materials are available on the NIAID website .

About the National Institutes of Health (NIH): NIH, the nation's medical research agency, includes 27 Institutes and Centers and is a component of the U.S. Department of Health and Human Services. NIH is the primary federal agency conducting and supporting basic, clinical, and translational medical research, and is investigating the causes, treatments, and cures for both common and rare diseases. For more information about NIH and its programs, visit www.nih.gov .

NIH…Turning Discovery Into Health ®

Connect with Us

• More Social Media from NIH

The COVID-19 research landscape: Measuring topics and collaborations using scientific literature

Affiliations.

• 1 Institute of Medical Information, Chinese Academy of Medical Sciences.
• 2 Digital China Health Technologies Co. Ltd., Beijing, China.
• PMID: 33120818
• PMCID: PMC7581087
• DOI: 10.1097/MD.0000000000022849

Objectives: The Coronavirus Disease 2019 (COVID-19) caused heavy burdens and brought tremendous challenges to global public health. This study aimed to investigate collaboration relationships, research topics, and research trends on COVID-19 using scientific literature.

Method: COVID-19-related articles published from January 1 to July 1, 2020 were retrieved from PubMed database. A total of 27,370 articles were included. Excel 2010, Medical Text Indexer (MTI), VOSviewer, and D3.js were used to summarize bibliometric features.

Results: The number of the COVID-19 research publications has been continuously increasing after its break. United States was the most productive and active country for COVID-19 research, with the largest number of publications and collaboration relationships. Huazhong University of Science and Technology from China was the most productive institute on the number of publications, and University of Toronto from Canada ranked as Top 1 institute for global research collaboration. Four key research topics were identified, of which the topic of epidemiology and public health interventions has gathered highest attentions. Topic of virus infection and immunity has been more focused during the early stage of COVID-19 outbreak compared with later stage. The topic popularity of clinical symptoms and diagnosis has been steady.

Conclusions: Our topic analysis results revealed that the study of drug treatment was insufficient. To achieve critical breakthroughs of this research area, more interdisciplinary, multi-institutional, and global research collaborations are needed.

Publication types

• Systematic Review
• Betacoronavirus*
• Bibliometrics*
• Biomedical Research / trends*
• Coronavirus Infections* / diagnosis
• Coronavirus Infections* / epidemiology
• Coronavirus Infections* / therapy
• Global Health
• Intersectoral Collaboration
• Pneumonia, Viral* / diagnosis
• Pneumonia, Viral* / epidemiology
• Pneumonia, Viral* / therapy
• Publishing / trends*
• Research Design / trends

Overview and objectives

On this page:

• Research team

This research study aims to describe the health outcomes of people diagnosed with COVID-19 in Queensland, over time and in relation to patient characteristics, by combining COVID-19 notification, hospital, general practice and death registry data.

General practice patient health information, in comparison to hospital data, contains additional, more detailed and up-to-date information on patient characteristics, including health conditions and medications at the time of infection.

We will be contacting patients who have or have had COVID-19 and will be inviting them to participate by giving their individual consent.

The novel coronavirus disease, named COVID-19 on 11 February 2020, is caused by SARS-CoV-2 virus.

The outbreak was declared a Public Health Emergency of International Concern on 30 January 2020. While the number of confirmed cases worldwide and in Australia is reported daily, detailed data on the outcomes of people who test positive for SARS-CoV-2, and predictors of outcomes, are still scarce.

Outcomes are likely to vary with context, including according to extensiveness of surveillance and testing, health systems functioning and population characteristics.

Evidence to date has come primarily from overseas countries that are further along in the pandemic than Australia.

Knowledge gaps in relation to COVID-19

There is limited information to date describing patient characteristics associated with outcomes, particularly in respect to the Australian population.

There is substantial variation by age, with younger people generally experiencing milder forms of disease, with a greater proportion of those with severe disease or death being older.

Importantly, these data to date have only been based on hospitalised patients , and do not include all patients who have been diagnosed with COVID-19 in the community.

Furthermore, they do not include Australian data . Australian-specific and state-specific data are essential as we continue through the epidemic as outcomes are dependent on a number of region-specific factors including population profile, health system factors and the public health actions taken by individuals and Government.

Study Objectives

The main objectives of the ATHENA COVID-19 STUDY are:

• To quantify hospital-based outcomes and deaths, including in relation to sociodemographic characteristics and comorbidities as ascertained from hospital AND general practice data.
• To estimate the strength of association between these outcomes and sociodemographic and health characteristics.

Study Sponsor

The Health Innovation, Investment and Research Office (HIIRO) of Queensland Health is responsible for consultation, development and review of State-wide research ethics and research governance policies.

HIIRO provides a central portal of contact for Researchers, HHS Human Research Ethics Committee Chairs and Members, Coordinators, Research Governance Offices/rs and study sponsors seeking advice and direction on ethical and governance issues associated with the conduct of research in Queensland Health.

Meet the team

Professor Kim Greaves (BSc, MD, FACC, FRCP) Principal Investigator & Project Lead

Professor Greaves is the Director of Cardiac Research and a Senior Staff Specialist in Cardiology at the Sunshine Coast University Hospital, Queensland. He holds a Fellowship of the Royal College of Physicians (FRCP –UK), Fellowship of the Royal Australian College of Physicians (FRACP), and completed a Doctor of Medicine in 2007.

Kim has extensive experience and publications in medical research and holds academic appointments as Professor and Associate Professor at the Griffith University, the Australian National University, Queensland University of Technology, University of Sunshine Coast, and the University of Queensland. Professor Greaves is currently focusing on health information sharing for health service planning, delivery and research, and areas of implementation science applicable to cardiovascular disease prevention.

Associate Professor Rosemary Korda (BAppSc, MAppSc, GradDipPopHlth, PhD) Principal Investigator and team lead for ANU

Data Analyses Team – Australian National University: National Centre for Epidemiology and Population Health, Research School of Population Health

Rosemary Korda is an Associate Professor at the National Centre for Epidemiology and Population Health, working in chronic disease epidemiology and health services research. She has extensive experience in the analysis of large-scale complex data, including longitudinal survey data and linked administrative health data.

Current research interests include:

• innovation in use of linked data
• inequalities in cardiovascular disease and healthcare
• unwarranted variation in care; and
• health risks of environmental exposures, including asbestos insulation and per- and poly-fluoroalkyl substances (PFAS).

In addition to her research, she has a major role in curriculum development and teaching in postgraduate population health courses at the ANU and in supervising higher degree research students. She is currently serving as an Expert Member on the Australian Government's Prostheses List Advisory Committee (PLAC).

Dr Zoltan Bourne (FRACGP, BMed, BSc) ATHENA COVID-19 Coordination Centre, Team Lead

Dr Zoltan Bourne is the Director and owner, for the past 13 years, of Medicine on Maple - a General Practice located in Maleny on the hinterland of the Sunshine Coast.  He has been a Supervisor to General Practice Registrars for both RACGP and ACRRM and a Senior Lecturer for the Griffith University School of Medicine.

With a strong commitment to health system reform, he was the inaugural General Practice Liaison Officer for the Medicare Local and PHN where he strongly advocated for the introduction of HealthPathways and the Queensland Health General Practice Smart Referral.

More recently, Zoltan has worked with Prof Kim Greaves on the General Practice Data Linkage ‘proof of concept study’ which successfully progressed to ‘The ATHENA study’ and now ‘The ATHENA COVID-19 study. Dr Bourne contributes expert advice to the project and will lead the ATHENA COVID-19 Coordination centre team.

Coordination Call Centre contact information

Patients: 07 3184 4167

General Practice: 07 4243 4813

Email: [email protected]

Last updated: 9 November 2020

• Skip to main content
• Accessibility help

Information

We use cookies to collect anonymous data to help us improve your site browsing experience.

Click 'Accept all cookies' to agree to all cookies that collect anonymous data. To only allow the cookies that make the site work, click 'Use essential cookies only.' Visit 'Set cookie preferences' to control specific cookies.

Your cookie preferences have been saved. You can change your cookie settings at any time.

Coronavirus (COVID-19) vaccine barriers and incentives to uptake: literature review

This literature review examines UK sources on barriers and incentives to uptake of COVID-19 vaccines and refers to the WHO SAGE “3Cs model” (complacency, convenience and confidence) to report findings.

Objectives and research questions

This literature review analyses UK research on vaccine uptake. It aims to shed light on the barriers that may have caused certain groups to disengage from the Covid-19 vaccination programme. The analysis of the sources presented here suggests answers to the following research questions:

• How does knowledge and understanding of Covid-19 affect vaccine uptake?
• What are the practical and physical barriers to uptake encountered by the eligible population?
• To what extent do concerns about vaccine safety cause people to disengage from the programme?
• What is the role played in vaccine uptake by broader beliefs and attitudes?
• What lessons can be learned that may help to engage lower uptake groups better?

Email: [email protected]

There is a problem

Thanks for your feedback

Your feedback helps us to improve this website. Do not give any personal information because we cannot reply to you directly.

Methodologies for COVID-19 research and data analysis

This collection has closed and is no longer accepting new submissions. 

This  BMC Medical Research Methodology collection of articles has not been sponsored and articles undergo the journal’s standard peer-review process overseen by our Guest Editors, Prof. Dr. Livia Puljak (Catholic University of Croatia in Zagreb, Croatia) and Prof. Dr. Martin Wolkewitz (University of Freiburg, Germany).  

Delays in reporting and publishing trial results during pandemics: cross sectional analysis of 2009 H1N1, 2014 Ebola, and 2016 Zika clinical trials

Pandemic events often trigger a surge of clinical trial activity aimed at rapidly evaluating therapeutic or preventative interventions. Ensuring rapid public access to the complete and unbiased trial record is...

• View Full Text

Open science saves lives: lessons from the COVID-19 pandemic

In the last decade Open Science principles have been successfully advocated for and are being slowly adopted in different research communities. In response to the COVID-19 pandemic many publishers and research...

Clinical research activities during COVID-19: the point of view of a promoter of academic clinical trials

During the COVID-19 emergency, IRST IRCCS, an Italian cancer research institute and promoter of no profit clinical studies, adapted its activities and procedures as per European and national guidelines to main...

Instruments to measure fear of COVID-19: a diagnostic systematic review

The COVID-19 pandemic has become a source of fear across the world. Measuring the level or significance of fear in different populations may help identify populations and areas in need of public health and edu...

A risk assessment tool for resumption of research activities during the COVID-19 pandemic for field trials in low resource settings

The spread of severe acute respiratory syndrome coronavirus-2 has suspended many non-COVID-19 related research activities. Where restarting research activities is permitted, investigators need to evaluate the ...

Outbreaks of publications about emerging infectious diseases: the case of SARS-CoV-2 and Zika virus

Outbreaks of infectious diseases generate outbreaks of scientific evidence. In 2016 epidemics of Zika virus emerged, and in 2020, a novel severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) caused a p...

Analysis of clinical and methodological characteristics of early COVID-19 treatment clinical trials: so much work, so many lost opportunities

The COVID-19 pandemic continues to rage on, and clinical research has been promoted worldwide. We aimed to assess the clinical and methodological characteristics of treatment clinical trials that have been set...

The unintended consequences of COVID-19 mitigation measures matter: practical guidance for investigating them

COVID-19 has led to the adoption of unprecedented mitigation measures which could trigger many unintended consequences. These unintended consequences can be far-reaching and just as important as the intended o...

Short-term real-time prediction of total number of reported COVID-19 cases and deaths in South Africa: a data driven approach

The rising burden of the ongoing COVID-19 epidemic in South Africa has motivated the application of modeling strategies to predict the COVID-19 cases and deaths. Reliable and accurate short and long-term forec...

Incorporating and addressing testing bias within estimates of epidemic dynamics for SARS-CoV-2

The disease burden of SARS-CoV-2 as measured by tests from various localities, and at different time points present varying estimates of infection and fatality rates. Models based on these acquired data may su...

COVID-19-related medical research: a meta-research and critical appraisal

Since the start of the COVID-19 outbreak, a large number of COVID-19-related papers have been published. However, concerns about the risk of expedited science have been raised. We aimed at reviewing and catego...

Use of out-of-hospital cardiac arrest registries to assess COVID-19 home mortality

In most countries, the official statistics for the coronavirus disease 2019 (COVID-19) take account of in-hospital deaths but not those that occur at home. The study’s objective was to introduce a methodology ...

Strengthening policy coding methodologies to improve COVID-19 disease modeling and policy responses: a proposed coding framework and recommendations

In recent months, multiple efforts have sought to characterize COVID-19 social distancing policy responses. These efforts have used various coding frameworks, but many have relied on coding methodologies that ...

Predictive accuracy of a hierarchical logistic model of cumulative SARS-CoV-2 case growth until May 2020

Infectious disease predictions models, including virtually all epidemiological models describing the spread of the SARS-CoV-2 pandemic, are rarely evaluated empirically. The aim of the present study was to inv...

Alternative graphical displays for the monitoring of epidemic outbreaks, with application to COVID-19 mortality

Classic epidemic curves – counts of daily events or cumulative events over time –emphasise temporal changes in the growth or size of epidemic outbreaks. Like any graph, these curves have limitations: they are ...

The Correction to this article has been published in BMC Medical Research Methodology 2020 20 :265

COVID19-world: a shiny application to perform comprehensive country-specific data visualization for SARS-CoV-2 epidemic

Data analysis and visualization is an essential tool for exploring and communicating findings in medical research, especially in epidemiological surveillance.

Social network analysis methods for exploring SARS-CoV-2 contact tracing data

Contact tracing data of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) pandemic is used to estimate basic epidemiological parameters. Contact tracing data could also be potentially used for assessin...

Establishment of a pediatric COVID-19 biorepository: unique considerations and opportunities for studying the impact of the COVID-19 pandemic on children

COVID-19, the disease caused by the highly infectious and transmissible coronavirus SARS-CoV-2, has quickly become a morbid global pandemic. Although the impact of SARS-CoV-2 infection in children is less clin...

Statistical design of Phase II/III clinical trials for testing therapeutic interventions in COVID-19 patients

Because of unknown features of the COVID-19 and the complexity of the population affected, standard clinical trial designs on treatments may not be optimal in such patients. We propose two independent clinical...

Rapid establishment of a COVID-19 perinatal biorepository: early lessons from the first 100 women enrolled

Collection of biospecimens is a critical first step to understanding the impact of COVID-19 on pregnant women and newborns - vulnerable populations that are challenging to enroll and at risk of exclusion from ...

Disease progression of cancer patients during COVID-19 pandemic: a comprehensive analytical strategy by time-dependent modelling

As the whole world is experiencing the cascading effect of a new pandemic, almost every aspect of modern life has been disrupted. Because of health emergencies during this period, widespread fear has resulted ...

A four-step strategy for handling missing outcome data in randomised trials affected by a pandemic

The coronavirus pandemic (Covid-19) presents a variety of challenges for ongoing clinical trials, including an inevitably higher rate of missing outcome data, with new and non-standard reasons for missingness....

Joint analysis of duration of ventilation, length of intensive care, and mortality of COVID-19 patients: a multistate approach

The clinical progress of patients hospitalized due to COVID-19 is often associated with severe pneumonia which may require intensive care, invasive ventilation, or extracorporeal membrane oxygenation (ECMO). T...

COVID-19 prevalence estimation by random sampling in population - optimal sample pooling under varying assumptions about true prevalence

The number of confirmed COVID-19 cases divided by population size is used as a coarse measurement for the burden of disease in a population. However, this fraction depends heavily on the sampling intensity and...

Coronavirus disease 2019 (COVID-19): an evidence map of medical literature

Since the beginning of the COVID-19 outbreak in December 2019, a substantial body of COVID-19 medical literature has been generated. As of June 2020, gaps and longitudinal trends in the COVID-19 medical litera...

Group testing performance evaluation for SARS-CoV-2 massive scale screening and testing

The capacity of the current molecular testing convention does not allow high-throughput and community level scans of COVID-19 infections. The diameter in the current paradigm of shallow tracing is unlikely to ...

Research methodology and characteristics of journal articles with original data, preprint articles and registered clinical trial protocols about COVID-19

The research community reacted rapidly to the emergence of COVID-19. We aimed to assess characteristics of journal articles, preprint articles, and registered trial protocols about COVID-19 and its causal agen...

Using online technologies to improve diversity and inclusion in cognitive interviews with young people

We aimed to assess the feasibility of using multiple technologies to recruit and conduct cognitive interviews among young people across the United States to test items measuring sexual and reproductive empower...

Towards reduction in bias in epidemic curves due to outcome misclassification through Bayesian analysis of time-series of laboratory test results: case study of COVID-19 in Alberta, Canada and Philadelphia, USA

Despite widespread use, the accuracy of the diagnostic test for SARS-CoV-2 infection is poorly understood. The aim of our work was to better quantify misclassification errors in identification of true cases of...

The semi-automation of title and abstract screening: a retrospective exploration of ways to leverage Abstrackr’s relevance predictions in systematic and rapid reviews

We investigated the feasibility of using a machine learning tool’s relevance predictions to expedite title and abstract screening.

Social media as a recruitment platform for a nationwide online survey of COVID-19 knowledge, beliefs, and practices in the United States: methodology and feasibility analysis

The COVID-19 pandemic has evolved into one of the most impactful health crises in modern history, compelling researchers to explore innovative ways to efficiently collect public health data in a timely manner....

Current methods for development of rapid reviews about diagnostic tests: an international survey

Rapid reviews (RRs) have emerged as an efficient alternative to time-consuming systematic reviews—they can help meet the demand for accelerated evidence synthesis to inform decision-making in healthcare. The s...

Methodological challenges of analysing COVID-19 data during the pandemic

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Int J Environ Res Public Health

The Impact of COVID-19 on the Sustainable Development Goals: Achievements and Expectations

Cathaysa martín-blanco.

1 Department Business Organization I, University of Granada, 18011 Granada, Spain

Montserrat Zamorano

2 Department of Civil Engineering, University of Granada, 18011 Granada, Spain

Carmen Lizárraga

3 Department of Applied Economics, University of Granada, 18011 Granada, Spain

Valentin Molina-Moreno

Associated data.

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

The COVID-19 pandemic has had a significant impact on almost all the Sustainable Development Goals (SDGs), leaving no country unaffected. It has caused a shift in political agendas, but also in lines of research. At the same time, the world is trying to make the transition to a more sustainable economic model. The research objectives of this paper are to explore the impact of COVID-19 on the fulfilment of the SDGs with regard to the research of the scientific community, and to analyze the presence of the Circular Economy (CE) in the literature. To this end, this research applies bibliometric analysis and a systematic review of the literature, using VOSviewer for data visualization. Five clusters were detected and grouped according to the three dimensions of sustainability. The extent of the effects of the health, economic and social crisis resulting from the pandemic, in addition to the climate crisis, is still uncertain, but it seems clear that the main issues are inefficient waste management, supply chain issues, adaptation to online education and energy concerns. The CE has been part of the solution to this crisis, and it is seen as an ideal model to be promoted based on the opportunities detected.

1. Introduction

Sustainable development is not a recent or novel concept in the 21st century, although it is acquiring special relevance as a consequence of the externalities of the traditional model. In fact, the concept emerged in the early 1970s as a move to protect the environment and ensure development without the associated destruction [ 1 ], and was defined by the World Commission on Environment and Development (WCED) in 1987 as “meeting the needs of the present without compromising the ability of future generations to meet their needs” [ 2 ]. There have been many supranational strategies to promote sustainable development, from Agenda 21 in 1982 [ 3 ], the Millennium Development Goals at the beginning of the 21st century [ 4 ], and the recent Sustainable Development Goals (SDGs) established in 2015, including several evaluation milestones [ 5 ]. Agreed on by 193 countries, the SDGs are operationalized through 169 targets and 213 measurable indicators that form a global action plan [ 6 ]. They aim to address the systemic barriers to sustainable development in three dimensions—social, economic and environmental—with universal application under the premise of a growing interconnected world [ 7 ]. The SDGs are classified into five groups, named the “Five Ps”: (1) People (SDG 1, no poverty; SDG 2, zero hunger; SDG 3, good health and well-being; SDG 4, quality education; SDG 5, gender equality); (2) Planet (SDG 6, drinking water and sanitation; SDG 12, responsible consumption and production; SDG 13, climate action; SDG 14, underwater life; SDG 15, life on earth); (3) Prosperity (SDG 7, clean and affordable energy; SDG 8, decent work and economic growth; SDG 9, industry, innovation and infrastructure; SDG 10, reduction of inequality; SDG 11, sustainable cities and communities), (4) Peace (SDG16, strong institutions for peace and justice) and (5) Partnership (SDG17, partnerships to achieve the goals).

The social dimension of sustainability is addressed through the “People” goals. There is no consensus on its definition due to the divergence of approaches for the study of this aspect [ 8 , 9 ], and indeed, little academic attention has been focused on this dimension [ 10 ]. Its conceptualization faces other problems due to the inclusion of soft terms such as social capital [ 8 ], which causes further difficulties in its analysis. Landorf [ 11 ] proposed that this dimension is a binomial between social equity and community sustainability, which represent the most common terms used [ 8 ]. The most recent studies take an integrated approach due to the interconnected nature of the three dimensions [ 12 ].

The “Planet” goals focus on the environmental dimension of sustainability, understanding it as a natural science concept that obeys biophysical laws, seeking the “unimpaired maintenance of human life-support systems-environmental sink and source capacities” [ 13 ]. This concept is related to the resource-limited ecological economic framework of “limits to growth” [ 14 ].

The economic dimension fits under the “prosperity” goals. For a long time, economic policies were only applied to the distribution and allocation of resources, without paying attention to the scale of extraction from nature [ 15 ]. The 17 SDGs address challenges and take actions that can be grouped under the three sub-goals of ecological economics, which aim to move towards an efficient, just and sustainable economy [ 7 ]. This shift is related to efforts to embed sustainable finance in both private and public organizations, as well as policy initiatives to encourage responsible business conduct for sustainable development [ 16 , 17 , 18 ].

Finally, the remaining two Ps (Peace and Partnership) work as facilitators for the rest of the dimensions [ 5 ]. Peace is related to SDG 16, which is focused on improving democracies and protecting human rights, whereas Partnership is associated with SDG 17, whose primary aim is to forge alliances between public and private entities in order to achieve international cooperation and to cope with global issues such as climate change or economic crises.

Nowadays, the traditional linear economic model of “take, make and throw” has become unsustainable [ 19 , 20 ], resulting in the need to transition to more sustainable socio-technical systems [ 21 , 22 ]. The externalities of the linear production model are threatening the economic and environmental sustainability of our planet, thus causing natural ecosystems to be in jeopardy [ 23 , 24 , 25 , 26 , 27 , 28 ]. Similarly, society faces high rates of unemployment, and poor working conditions, leading to social vulnerability, conceptualized through poverty and increasing inequalities [ 29 , 30 ]. Sustainability requires the development of a balanced production system, taking into consideration economic, social, environmental and technological aspects [ 31 ], with the Circular Economy (CE) being a new paradigm that contributes to the positive reconciliation of all these elements [ 23 , 32 ].

In this context, CE is defined as an industrial economy that is restorative and regenerative by concept, intention and design [ 23 , 33 , 34 ]. It brings together diverse schools of thought [ 35 ], including biomimetics [ 36 ], performance economics [ 37 ], natural capitalism [ 38 ], regenerative design [ 39 ], cradle-to-cradle [ 40 ], blue economy [ 41 ] and industrial ecology [ 42 ]. Moreover, it is considered by the new circular economy action plan (CEAP) adopted by the EU in 2020 as one of the main building blocks of the European Green Deal, Europe’s new agenda for sustainable growth, and it is a prerequisite to achieve the climate neutrality target and to halt biodiversity loss. In fact, the transition to a CE will reduce pressure on natural resources and create sustainable growth and jobs. The field of knowledge of ecological economics or the green economy is unarguably at the roots of the CE, interwoven in its three dimensions of sustainable action [ 43 , 44 ]. Therefore, actions in the CE are closely related to the achievement of the SDGs [ 45 ], in addition to sharing a bias towards degrowth and green growth research that seeks efficient allocation [ 33 ]. Thus, five SDGs have strong synergies and a direct relation with CE practices, concretely, SDG 6 (clean water and sanitation), SDG 7 (affordable and clean energy), SDG 8 (decent work and economic growth), SDG 12 (sustainable consumption and production), and SDG 15 (life on land). Meanwhile, SDG 1 (no poverty), SDG 2 (zero hunger) and SDG 14 (life below water) are impacted by CE practices for the most part indirectly. In contrast, SDG 4 (quality education), SDG 9 (industry, innovation and infrastructure), SDG 10 (reduced inequalities), SDG 13 (climate action), SDG 16 (peace, justice and strong institutions) and SDG 17 (partnerships for the goals) show a potential relationship with CE practice that could maximize their progress [ 45 ]. Figure 1 shows the relationships between the Five Ps (first circle), the SDGs (second circle), the CE (third circle) and the ecological economics goals (the three discs orbiting the larger one).

Relation between SDGs and circular economy. Source: our own elaboration adapted on Costanza et al. [ 7 ] and Schröder et al. [ 45 ].

According to Sachs et al. [ 46 ], during the period 2015–2019, the world made progress towards the SDGs at a rate of 0.5 points per year, which is not fast enough to meet the 2030 deadline. Figure 2 shows the evolution of the SDG Index score since 2010.

SDGs Index score over time, world average (2010–2021). Source: our own elaboration on Sachs et al. [ 46 ].

By 2022, the 2030 Agenda is halfway to its target date and the COVID-19 pandemic has not favored the achievement of its goals. Even if the pandemic has not completely wiped out the progress made to date on some of the targets, it has certainly has made it more challenging, with progress inching along at a rate of 0.1 points per year [ 46 ]. However, even in a world with no COVID-19, the global targets would not be met [ 47 , 48 ].

As declared by the World Health Organization on 11 March 2020, the global pandemic has been a historically unprecedented episode in terms of financial investment in research that has prompted the generation of scientific literature in all fields on sustainability, especially on the impact of the pandemic and the forecasting of possible futures [ 47 ]. Even though the impact of the pandemic on SDGs is still uncertain [ 49 , 50 ], the measures taken by governments to contain the virus spread have shattered the basis of a globalized world that relied on international trade, and forced states to compete for scarce resources. The lockdown brought about the shutdown of the economy, putting the SDGs and the transition to the CE at the center of the debate [ 49 , 51 , 52 ].

The research objectives of this paper are to explore the impact of COVID-19 on the fulfilment of the SDGs with regard to the research of the scientific community, and to analyze the presence of the circular economy (CE) in that literature. To this end, this research applies bibliometric analysis and a review of the literature, and answers the following questions:

Q1. What are the main characteristics of this line of research?

Q2. What are the main thematic areas and the most relevant publications that address the impact of COVID-19 on the SDGs?

• Q3. Who are the most productive authors, institutions, countries and journals?
• Q4. What are the main international cooperation networks?

Q5. What are the main current trends in research on COVID-19 and SDGs?

Q6. What have been the main contributions of the CE to the SDGs during COVID-19?

In general, studies that apply a systematic review of the literature are valuable to understanding the leading edge research in a field, but an additional analysis of the literature using bibliometric methods can provide results that have not been detected in the other reviews [ 53 ]. Thus, bibliometric research, analyzing large volumes of scientific data, has an increasing scientific value in responding to the current public health emergency of international concern [ 54 ]. In this case, the pandemic has generated enormous attention in the scientific community, reflected in the large volume of articles published during the last two years, but it has been shown that COVID-19-related reviews have been limited and fragmented in particular areas [ 12 ]. This paper fills this gap, applying bibliometric analysis to explore the implication of the impact of COVID-19 on the fulfillment of the SDGs and the presence of the circular economy (CE) in the literature.

2. Materials and Methods

2.1. bibliometric analysis.

Scientometric or bibliometric analysis is a research methodology whose main objective is to identify, organize and analyze metadata to examine the evolution of an area of knowledge during a specific period of time [ 55 , 56 , 57 , 58 ]. The systematic literature review can provide a state of the art and identify gaps and potential areas for future research in the literature, but the procedure followed must be replicable [ 59 ].

2.2. Methodological Procedure

To this end, this study works under the SPAR-4-SLR protocol established by Paul et al. [ 59 ], which consists of three stages and six sub-stages. Table 1 summarizes how the protocol has been applied in this research.

Summary for SPAR-4-SLR protocol application.

Source: our own elaboration according to the SPAR-4-SLR protocol established by Paul et al. [ 59 ].

The search took place in March 2022 and the study was conducted in three phases. First, search criteria were selected to identify records in the repository (identification phase). Then, having obtained the records that met the search requirements, the data were exported for analysis using VOSviewer v. 1.6.18 software (analysis and visualization phase). Finally, connections and associations between the scientific documents were established and a discussion (results and discussion phase) took place.

2.2.1. First Phase: Identification

The Scopus scientific database was used for the data search, although the main scientific repositories such as Web Of Science, PubMed and Google Scholar were consulted, following the recommendations of Harzing and Alakangas [ 60 ] and Mongeon and Paul-Hus [ 61 ]. The reasons for using Scopus were as follows. (a) It is the repository with the largest volume of information on authors, countries and institutions [ 62 , 63 ]; (b) it has the highest volume of articles meeting the scientific quality requirements for peer review [ 62 ], [ 64 ]; and (c) the coverage it provides compared to Web of Science and other repositories is greater, while its metrics are highly correlated [ 65 , 66 ]. Consequently, it has been selected by Baas et al. [ 67 ] and Donthu et al. [ 68 ] as the most suitable repository to apply this research methodology.

The search in the Scopus repository was carried out using the fields “Article title, Abstract, and Keywords”, with the following search terms selected: TITLE-ABS-KEY (“COVID-19” OR “coronavirus disease 2019” OR “SARS-CoV-2” OR “coronavirus” OR “coronavirus” OR “coronavirus infection”) AND TITLE-ABS-KEY (“sustainable development goals” OR “sustainable development” OR “sdgs” OR “sdg” OR “Agenda 2030”). As a result, a total of 2453 documents were obtained that met the search criteria.

Next, exclusion criteria were applied. Firstly, following the recommendations of Paul et al. [ 59 ], research articles only were selected, as these are published on the basis of scientific novelty and satisfy the scientific quality criterion of peer review. Consequently, the number of papers that met the search criteria was reduced to 1483. Then, a time horizon restriction to the years 2020 and 2021 was applied, as the first notification of the existence of a cluster of cases of the virus in Wuhan was made on 31 December 2019. Therefore, all previously published articles are not in line with our research objective, again reducing the number of documents meeting the search criteria to 1148. Finally, the language criterion was applied, selecting only research articles published in English, which reduced the total number of documents to 1093, which are those that make up the sample of our bibliometric and systematic review.

Therefore, the final search string was as follows: (TITLE-ABS-KEY (“COVID-19” OR “coronavirus disease 2019” OR “SARS-CoV-2” OR “coronavirus” OR “coronavirus” OR “coronavirus infection”) AND TITLE-ABS-KEY (“sustainable development goals” OR “sustainable development” OR “sdgs” OR “sdg” OR “Agenda 2030”)) AND (LIMIT-TO [PUBYEAR, 2021) OR LIMIT-TO (PUBYEAR, 2020)) AND (LIMIT-TO (DOCTYPE, “ar”)) AND (LIMIT-TO (LANGUAGE, “English”)).

2.2.2. Second Phase: Analysis and Visualization

For the analysis and visualization of the documents, VOSViewer v.1.6.18 was used, which allows clustering and word processing [ 69 , 70 ]. The tool is useful for clustering networks of a large number of documents, keywords, authors or institutions [ 71 ].

Accordingly, based on the documents that met the search criteria, the cooperation networks between authors, institutions and countries through the co-citation method were analyzed. These international networks provide insight into the relationships between researchers and the dissemination of knowledge [ 72 ], while favoring the design of new research by generating synergies that contribute to the exchange of ideas [ 73 ].

An analysis of the clustering of keywords contained in the documents on COVID-19 and SDGs using the co-occurrence method was also conducted, as these are considered representative of their content [ 74 ]. Co-occurrence is based on the fact that records sharing the same keywords are similar [ 75 , 76 ]. This allows us to analyze the evolution of the topics covered in research papers [ 77 ], creating a picture of the line of research [ 78 ].

2.2.3. Third Phase: Results and Discussion Phase

Finally, the third phase entailed analyzing authors, institutions, countries, journals and international cooperation networks, as well as keywords, to identify research trends on the impact of COVID-19 on the SDGs. Together with the systematic literature review, this contributed to resolving the research questions posed and presenting the discussion and conclusions of this research work.

3. Results and Discussion

3.1. evolution of scientific production.

This section presents the results on the main characteristics of the scientific production of the impact of COVID-19 on the SDGs in the period 2020–2021 ( Table 2 ). The total number of publications during the two years is 1093 articles, 77% published in the last year. The growing interest in the subject is shown by the 233% increase in the number of authors, 49% in the case of countries, 235% in the case of institutions and 168% in the number of journals.

General characteristics of the scientific production.

TC/A: average number of citations per article; TC/Author: average number of citations per author.

Regarding the number of citations, the articles published in 2020 received 431 citations during that first year, which is an unusually high number, highlighting the great interest in the impact of the pandemic on the SDGs. Moreover, publications in the second year received ten times more citations than those in the first year, showing a very high productivity. In contrast, the average number of authors per publication did not change substantially because of the simultaneous increase in the number of authors and publications.

3.2. Most Influential Subject Areas and Publications

This section pertains to the second question regarding the most productive areas of knowledge. Since an article can be in more than one area [ 79 ], Figure 3 shows a large number of articles, which are classified into 27 thematic areas, with 83.75% accumulated in seven areas. The Social Sciences area is the most productive, with 565 articles representing 23.91% of the total scientific literature. This is followed by Environmental Sciences ( n = 484, 20.48%), Energy ( n = 348, 14.73%), Business, Management and Accounting ( n = 162, 6.86%), Medicine ( n = 153, 6.47%), Engineering ( n = 145, 6.14%) and, finally, Economics, Econometrics and Finance ( n = 122, 5.16%). The presence of these areas is evidence of the impact of the pandemic on the three dimensions of sustainability, as it is a phenomenon studied from an economic, social and environmental perspective, as well as the interconnection between the SDGs affected. The areas involved in this subject are similar to those involved in studying the CE, as Belmonte-Ureña et al. [ 80 ] pointed out, with the nuance that in our subject, the Social Sciences have been predominant alongside the Environmental Sciences, while the more technical areas have experienced a greater focus in the subject of the CE.

Distribution of articles by subject area.

Table 3 shows the most relevant publications in this research area according to the total number of citations.

The 10 most cited articles.

SDGs: Sustainable Development Goals.

At the beginning of the pandemic, Zambrano-Monserrate et al. [ 81 ] revealed the indirect effects of COVID-19 on the environmental dimension of sustainability in two ways. On the one hand, the positive effects include the improvement of air quality and the reduction of noise and environmental pollution due to the drastic reduction in activity due to the protection measures. At the same time, however, these measures produced negative effects such as an increase in waste generation due to the growing consumption of single-use plastics, as well as a decrease in recycling and waste management. They concluded by indicating that, given the recovery of economic activity, the temporary positive effects will not be enough to offset the negative effects of pollution. In fact, Coccia [ 82 ] highlighted the threat that contamination represents for the challenge of improving resilience to future pandemics. In his study of 55 Italian provincial capitals, he concluded that persistent air pollution was the determining factor for virus transmission, rather than the effect of direct person-to-person contact. Pirouz et al. [ 84 ] also addressed the prevention of future Coronavirus epidemics through the development of a predictive model for the occurrence of positive cases. Unlike Coccia [ 82 ], they studied only one city and concluded that the determining factors were humidity and temperature, that is, the relative humidity in the main case study, with an average of 77.9%, positively affected confirmed cases, and maximum daily temperature, with an average of 15.4 °C, negatively affected confirmed cases. Additionally, this article was the first to include predictive techniques based on massive data, being part of the advance of the fourth industrial revolution. Meanwhile, Vanapalli et al. [ 83 ] focused on the problem of plastic pollution, recommending measures to improve the management of this waste, moving towards a model that uses environmentally friendly materials and increasing investment in sustainable technologies that allow progress in the transition to a circular model to fight against future pandemics. Finally, in the environmental dimension, Ibn-Mohammed et al. [ 49 ] highlighted the need to make a transition from the linear to the circular model, considering that we are facing an opportunity to promote a low-carbon economic model in a more resilient world; for this, they provide recommendations according to key sectors.

For their part, Ilyas et al. [ 85 ] considered the social dimension of sustainability, emphasizing the environmental and health risks posed by mismanagement of biomedical waste (COVID-waste). They carried out an analysis of disinfection techniques with the intention of providing information for the prevention of future pandemics, thus also focusing on improving resilience. The study by Yeasmin et al. [ 86 ] highlights one of the main effects of the pandemic: the worsening of mental health, especially in children, as a consequence of confinement. They also provided a series of recommendations to achieve SDG 3, such as the implementation of psychological intervention strategies and the improvement of the sociodemographic conditions of households, including economic security, education, childcare and job stability, all which have been severely affected by this crisis. Along the same lines, Leal Filho et al. [ 87 ] warned of other diseases that could occur, as well as the negative impact on mental health as a result of protection measures against the virus, concluding that the pandemic is a serious threat to the achievement of the SDGs due to its severe impact in all areas, and they press for greater action to accomplish the SDGs.

Finally, Amankwah-Amoah [ 88 ] centered the economic dimension of sustainability, specifically the performance of airlines in relation to their Green Business Practices (GBP). His analysis shows that some companies evaded their environmental commitments by prioritizing market survival and cost reduction. Galvani et al. [ 89 ] also discuss airlines and tourism, it is surprisingly the only article with a positive view on the effect of the pandemic, specifically on the change of humanity towards a mindset aligned with the SDGs. This coincides with the initial speculation about the positive impacts of the pandemic, but which two years later seem far from reality.

For the analysis of the impact of COVID-19 on the SDGs, it has been considered appropriate to include a column indicating the SDGs addressed by the ten most cited articles, using the new functionality of Scopus. Elsevier data science teams have built extensive keyword queries, supplemented with machine learning, to map documents to SDGs with very high precision.

Thus, six out of ten articles discuss SDG 3—good health and well-being, which is the only “People” goal studied among the ten most cited articles. Hence, goals linked to People have received insufficient attention from the science community regarding COVID-19 impacts on SDGs, despite the fact that it is a health issue. The “Planet” goals and the environmental dimension are addressed through three goals. Concretely, three articles study SDG 13—climate action, one studies SDG 14—life below water, and five study SDG 12—responsible consumption and production. Finally, in relation to the articles examining the “Prosperity” goals and the economical dimension, there are three studies that focus on SDG 8—decent work and economic growth, four on SDG 9—industry, innovation and infrastructure and only one deals with SDG 10—reduced inequalities. These results indicate that, in the first stages of the pandemic, the economic consequences of and solutions to the pandemic were the issues that received the most attention from the scientific community (measured through the number of citations received per article). In addition, these goals bear upon the CE paradigm, especially SDGs 12, 8 and 9.

3.3. Authors’, Journals’, Institutions’ and Countries’ Productivity

This section presents the productivity results of authors, institutions, countries, journals and their global cooperation networks.

Table 4 shows the ten most productive authors on the topic of the impact of the pandemic on the SDGs and their main characteristics.

The ten most productive authors.

(A): number of research articles published; (TC): total number of citations; (TC/A): average number of citations per article; (C): country; (First A): first article published; (Last A): last article published; (H index): Hirsch index, which represents the weight of an author in the line of research.

Among the ten most productive authors, seven come from Asia and one of these, Dang, T. T., leads the list, with twice as many citations as the second and third most productive authors, despite having only been published in the last year. Meanwhile, Leal Filho, W., the only European author, is the second most productive author, with the same number of articles as Dang but with only half as many citations, even though published during the two years studied. Shaw, R. and Adelodun, B. also published both years, which is not unexpected, taking into account the novelty of the subject. However, the latter two authors are the most cited among those studying the impact of the pandemic on the SDGs and have the best ratio of total number of citations to total number of published articles. Ali, S. M. focuses his studies on the impact of the pandemic on supply chains (SC) and its implications for the SDGs; specifically, he explores the drivers for improving the sustainability of SCs [ 90 ] and the challenges of maintaining the vaccine SC [ 91 ] and the humanitarian SC [ 92 ], all with a decision-making approach and in relation to a wide range of SDGs such as 8, 10, 12 and 3. Meanwhile, Allam, Z. explores the future of post-pandemic cities in terms of their socio-economic sustainability through the paradigm of “the 15-minute city” [ 93 ], smart cities through 6G [ 94 ] and the achievement of inclusive cities [ 95 ], all related to SDG 11.

Figure 4 shows the cooperation network based on the co-authorship analysis. The criteria used for clustering were: applying the fractional counting method, ignoring documents with more than 25 authors, selecting an interaction of at least two co-authored published research papers and the association strength method for normalization [ 69 ].

Authors’ cooperation network based on co-authorship.

International cooperation is generally weak. Leal Filho, W. is the only author from the ten most productive authors to appear in the cluster. Through his five documents, he establishes the connection between the cluster and an international cooperative network. Three authors are from Brazilian institutions (Quelas, O. L. G. from Universidade Federal Fluminense, Anholon, R. from Universidade Estadual de Campinas, Fritzen, B. from Universidade de Passo Fundo and Salvia, A. L. from University of Passo Fundo), Rampasso, I. S. is from Universidad Católica del Norte (Chile), Wall, T. is from Liverpool John Moores University (United Kingdom) and Doni, F. is from University of Milano-Bicocca (Italy).

Table 5 shows the ten most productive institutions. Chinese Academy of Sciences leads the ranking with 15 articles and an H index of 7, which corresponds with the high number of Chinese authors on this topic. However, with half as many articles as the former, UNSW Sydney and Texas A&M University have the best ratios of citation per articles (15 and 14.71, respectively). Moreover, the latter has a 100% cooperation index, followed by Organisation Mondiale de la Santé, University College London, UNSW Sydney and London School of Hygiene & Tropical Medicine.

The ten most productive institutions.

(C): country; (A): total number of published articles; (TC): total number of citations; (TC/A): average number of citations per published article; (H index): H index in the line of research; (CI): cooperation index; (TC/A CI): average number of citations of articles in international cooperation; (TC/A NIC): Average number of citations of articles without international cooperation.

Figure 5 shows the network of cooperation between organizations based on the co-authorship of articles. Initially, 2969 organizations were detected, but choosing those with a minimum of two documents resulted in just five organizations being connected, meaning that the network is not solid, due to of the marked lack of cooperation.

Network of cooperation between institutions based on the co-authorship of articles.

In relation to the quality of academic institutions, the most productive countries ( Table 6 ) are China, the United States and the United Kingdom. Spain, the fourth, receives fewer citations than Italy despite being more productive. In general, the cooperation index for countries is low except for Australia. This may indicate that the topic has been studied among researchers from the same country, but from different institutions and especially from the countries most affected by the pandemic such as China, the United States, the United Kingdom and Italy.

The ten most productive countries.

(C): country; (A): total number of published articles; (TC): total number of citations; (TC/A): average number of citations per published article; (H index): H index in the line of research; (NC): total number of international collaborators; (CI): cooperation index; (TC/A CI): average number of citations of articles in international cooperation; (TC/A NIC): average number of citations of articles without international cooperation.

Figure 6 shows the international collaboration networks of countries based on the co-authorship of articles. The colors show the networks and the size of the circles indicates the productivity of the networks based on the number of documents The limit was set at an interaction of at least 10 studies that were published with international co-authorship, reducing the number of countries from 141 to 47. They were grouped into six clusters. The first cluster, colored in red, is composed of 14 countries, led by Italy, Spain and Germany. The green cluster is made up of 10 countries and is led by India and South Korea. The blue cluster includes eight countries, led by Australia, and shows an interrelation between Pacific countries. The yellow cluster is made up of seven countries and is led by China, which is the country with the most publications. The purple cluster is composed of six countries and is led by the second and third most productive countries, the United Kingdom and the United States. Only Iran and Turkey are included in the light blue cluster. Clusters, most notably the purple cluster, show that connections are more frequent if the organizations are from the same continent.

Network of cooperation between countries based on the co-authorship of articles.

Table 7 shows the ten most productive journals addressing the impact of the pandemic on the SDGs, which account for 18% of the articles. According to SCImago Journal Rank, all of them belong to the first quartile and chiefly relate to the main subject areas such as energy, environmental science or the social sciences. The most prolific journal is Sustainability , which has 62% of the articles published in the top ten most productive journals and the best H-index of the articles in that area of research. Nevertheless, the journal Science Of The Total Environment , seventh in the ranking, has the highest total citations and the second best H-index after the journal Renewable And Sustainable Energy Reviews , which ranks last in terms of number of articles, but has the best quality indexes, as it is the journal with the greatest influence (3.68 impact factor). Finally, four journals are Swiss, three of which are leading the ranking, four are Dutch, two are British and one is American.

The ten most productive journals.

(A): total number of published articles; (TC): total number of citations; (TC/A): average number of citations per published article; (H index articles): H-index of the articles of the line of research; (H index journal): H-index of the journal; (SJR): SCImago Journal Rank.

3.4. Which Have Been the Most Frequently Undertaken Problems and Results? Which SDGs Have Received the Most Attention?

Co-occurrence analysis was applied using the indexed words as the unit and the fractional counting method. A thesaurus file was also introduced to eliminate search words and countries, as well as to standardize words appearing in singular and plural. A limit of at least 10 occurrences was then set, which reduced the number of terms from 4291 to 134. Five clusters emerged ( Figure 7 ), each one representing a theme, and they were ordered according to the number of documents included (for example, Cluster 1 is the first because it contains the largest number of documents). This section contains the most frequently undertaken problems and the most frequently obtained results by cluster; they are grouped according to the three dimensions of sustainability. The social dimension includes Cluster 1, focused on health, which is logical given that it is the main issue caused by a pandemic. Cluster 5 is focused on education, the economic dimension includes Cluster 2 and the environmental dimension includes the third and fourth clusters, as these refer to energy, waste management and pollution.

Keywords co-occurrence map.

3.4.1. Social Dimension

Cluster 1: public health dimension.

Colored in red, this cluster contains 42 elements. It focuses primarily on SDG 3 because public health is the central theme and it involves issues related to infection, prevention and health policy. In addition, it is connected with the consequences of lockdown and the socio-economic implications, especially in developing countries.

The key point is to develop policies that focus on prevention and build resilience to disruptive events such as the pandemic, which are expected to increase in frequency. Ilyas et al. [ 85 ] assessed the management of bio-medical waste products, considering them as health and environmental risks, for which they advocate the use of disinfection techniques. Yeasmin et al. [ 86 ], meanwhile, highlighted the impact of the pandemic on mental health, a line of research in psychology that draws attention to the effect of isolation on young people and children and its relationship with socioeconomic conditions. In addition, rising youth unemployment is increasing psychological stress [ 96 ].

Meanwhile, other disease prevention programs have seen an overall reduction in the attention they receive in developing countries [ 87 ]. For these reasons, Adelodun et al. [ 97 ] proposed monitoring virus transmission through wastewater analysis as a sustainable preventive measure, reducing the cost for these countries. On the other hand, Visconti and Morea [ 98 ] recommend the promotion of digital technologies in healthcare through public–private partnerships to reduce costs, decongest hospitals and improve disease surveillance.

Cluster 5: Educational Dimension

Cluster 5 in light blue contains 11 elements. It focuses on education, especially at the university level, and the introduction of online teaching with the implications for the teacher and the student. Thus, it is mainly linked to SDG 4, but also to SDGs 9 and 12.

Twenty-first century education has long included information and communication technologies in its procedures, but, in the wake of the pandemic, the use of online learning platforms has intensified [ 99 ]. This situation poses serious problems for access to education in developing countries and for students in rural areas [ 100 ], for whom measures are needed to address barriers such as poor access to electricity and the relevant equipment. In contrast to Wang and Huang [ 54 ], who also studied the impact of COVID-19 on SDGs, our research found that developed countries were not primarily focused on SDG 4. There are many studies from Asia, South America and Africa. For example, Rodriguez-Segura et al. [ 101 ] from Mexico and Edelhauser and Lupu-Dima [ 99 ] from Romania studied emergency remote teaching, finding that their countries were unprepared for the switch to e-learning, but that the pandemic has been a turning point in their transition. It represents an opportunity, especially for higher education, to become more sustainable and accessible, although it requires a change in the organizational culture [ 100 , 102 ].

Studies carried out on this topic have also considered the digital skills of educators. Portillo et al. [ 103 ] found that there is a digital divide among teachers according to gender, age and educational level; what is especially concerning is that they noticed lower technological competence at the lower levels of education, which are more vulnerable to deficiencies in remote teaching. Meanwhile, Tran et al. [ 104 ], investigated how to elevate students’ learning habits to achieve quality education and its relation to socioeconomic conditions.

3.4.2. Economic Dimension

Cluster 2: economic dimension.

Colored in green, this cluster contains 32 elements. Its focus is on the economic dimension of the crisis and, in particular, on tourism. Adopting a strategic approach is a key point in this cluster, as well as the necessity to innovate and use knowledge for new solutions. It is linked to SDGs 8, 9, 11 and 12.

One line of research focuses on the economic consequences of the pandemic and the need to switch to a more sustainable and resilient economic system. For example, the shutdown of air traffic during the pandemic dealt a severe economic blow to countries heavily dependent on tourism, but current air travel is a highly polluting activity that needs to be replaced by a smaller, less economically vulnerable alternative models that take into account negative externalities [ 105 ]. In general, studies are exploring and calling for measures from the tourism sector to become more transformational and transcendent in order to achieve the SDGs [ 106 ]. From a much more positive outlook, Galvani et al. [ 89 ] considered that travel and tourism now have the opportunity to offer valuable experiences and to become a means to expand the global consciousness that emerged in the aftermath of COVID-19.

Cities must be rethought using prospective technologies in order to effectively manage this and future pandemics [ 107 ]. However, policymakers must proceed cautiously when introducing new technology; for example, the proposal by Shorfuzzaman et al. [ 108 ] to apply mass video surveillance to control the spread of the virus violates rights of privacy.

The pandemic has also affected commerce and environmental awareness, forcing businesses to implement e-commerce platforms and to orient themselves to the increasing profile of responsible consumers who are informed about sustainable production and consumption [ 109 ]. Similarly, Tchetchik et al. [ 110 ] pointed out that COVID-19 has driven a change in the behavior of consumers towards greater environmental awareness, but mainly as a consequence of threat and endeavors to cope with it.

Another discussion connected with this dimension is the value of care work. Authors such as Bahn et al. [ 111 ] call for the incorporation of lessons from feminist economics into the economic system, giving a proper place to the care work that was essential during the pandemic and, in general, to the achievement of human wellbeing.

Supply chains have received massive attention due to their far-reaching implications. In developing countries, where the informal economy is predominant, economic shutdowns and vulnerable supply chains have led to increased food insecurity [ 112 ], especially regarding supply chains for perishable foods [ 113 ]. Equally important are problems in the supply chain for vaccines, which are essential to curb infections [ 89 ].

3.4.3. Environmental Dimension

Cluster 3: energy dimension.

Cluster 3 in blue contains 29 elements. It is focused on renewable energy to combat climate change and energy dependence, and the need to invest in and streamline energy policy from a global perspective is highlighted. This cluster contains the term “Circular Economy”, but this will be analyzed in a separate section due to research question six. This cluster is linked to SDGs 7, 8, 9, 12 and 13.

Madurai Elevasaran et al. [ 114 ] tracked the impact of sustainable energy on the rest of the SDGs, demonstrating that energy transition is essential to cope with the new challenges. The energy sector has been under huge strains during the pandemic, which has driven investment in renewable energy [ 115 ]; the authors call on governments to develop short- and long-term strategies for clean energy efficiency, creating a win–win solution for economic recovery and energy supply chains [ 116 ]. It is also necessary to promote research into energy storage systems and technologies that reduce energy consumption and to facilitate entrepreneurship in the sector and the creation of energy communities [ 114 ]. Moreover, studies on the relationship between the energy, water and food supply sectors are needed in order to ensure resource security [ 117 ].

Cluster 4: Waste and Pollution Dimension

This cluster, colored in yellow, consists of 20 elements. It is focused on pollution and waste management, containing terms such as air quality, disease dispersal and hospital waste management issues. In addition, it refers to the use of new technologies and big data for decision-making to face these problems. Despite the fact that the term “mental health” appears here, it was considered more accurate to move it to the first cluster, associated with the health issues. Cluster 4 is linked to SDGs 8, 9, 11, 12 and 13.

This topic focuses on analyzing the factors and effects of COVID-19 on waste management, based on the expected increase in the occurrence of epidemics [ 84 ]. Zambrano-Monserrate et al. [ 81 ] predicted the prevalence of negative indirect effects of COVID on the SDGs in the long-term outweighing the possible benefits derived from the shutdown in economic activity. They pointed to the setback in waste management and agreed with Vanapalli et al. [ 83 ] on the major problem of plastics´ consumption and hospital waste, which has not been properly treated [ 118 ]. In particular, the new technologies and prediction models are being developed to address the sustainability of location-routing problems with COVID waste [ 119 ]. This line of research calls for investment in research and development for new personal protective equipment materials that reduce waste generation, with a focus on product lifecycle strategies [ 120 ], as well as the use of bio-based solutions to cope with microplastic pollution [ 121 ]. In addition, neglected management of this waste has led to unsafe working conditions [ 122 ].

Finally, Coccia [ 82 ] considered the factors that explain the spread of the virus and pointed to air pollution as a determining element, calling for the prevention strategies in terms of sustainability science and environmental science.

3.5. Which Have Been the Main Contributions of the CE to the SDGs during COVID-19?

Fifteen articles specifically investigate the circular economy; most of them consider that the pandemic situation has created a window of opportunity for the transition to the circular model in order to achieve SDGs. This position is most strongly defended by Ibn-Mohammed et al. [ 49 ], with the fifth most productive article of the total. The other fourteen articles on the CE receive up to seven times fewer citations, occupying a much smaller space in the research topic. These articles mostly deal with particular case studies in very different sectors. Some significant cases are summarized below.

Rahman et al. [ 123 ] studied circularity in Southeast Asian ships that were being dismantled due to the shutdown in the maritime transport of goods. Ducoli et al. [ 124 ] investigated the possible use of ashes from sewage sludge contaminated by COVID-19 as a new material for construction. Hoosain et al. [ 125 ] analyzed various case studies to show how the technologies of the fourth industrial revolution are allowing the application of circularity principles in a wide variety of sectors and how these technologies are proving to be of key importance in the fight against pandemics. In the same vein, Abdul-Hamid et al. [ 126 ] looked into the optimization of palm oil production through digital technologies. Adelodun et al. [ 97 ] considered the impact on the agri-food system, highlighting the effectiveness of measures with a CE approach taken in Europe, such as short chains, which are key to achieving the sustainability of the agri-food system. Zanoletti et al. [ 127 ] discussed how to ensure the availability of critical raw materials, a growing problem since the pandemic. Girard and Nocca [ 128 ] proposed transforming urban planning using a CE approach to improve environmental quality and resilience in the face of future pandemics. Shishkin et al. [ 129 ] studied eco-design in air disinfection devices. Sparacino et al. [ 130 ] studied the integration of CE in companies, identifying them as key actors in the transition that has accelerated in the wake of the pandemic.

In order to see what circular solutions have been proposed for the challenges posed by COVID-19 to the SDGs, other literature reviews were used. Ten reviews were identified out of the 1093 documents initially detected. Table 8 summarizes the impact that the pandemic has had and in what sense. It also lists the proposed circular solutions and the barriers to making them effective. The main themes were Industry 4.0 technologies, circular models, the effectiveness of the waste hierarchy, the use of new materials and the efficiency of waste management systems.

The main contributions of the CE to the SDGs during COVID-19.

D: dimensions of sustainability addressed; COVID-19: in red, the negative effects; in green, the positive ones; EC: in green, the opportunities; in red, the barriers.

4. Conclusions

This study had two objectives; the first was to explore the impact of COVID-19 on the fulfilment of the SDGs with regard to the research of the scientific community, and the second was to analyze the presence of the CE in the literature. To this end, bibliometric analysis was carried out to answer the following questions.

Due to the novelty of the phenomenon, the period studied was very short compared to other bibliographical studies, but the number of documents is typical of much more established topics. The 1093 articles studied show a growing and lively trend, as the high number of citations shows.

The research of the topic has been characterized by multidisciplinarity, although five areas are deeply involved in its study; the social sciences area is the most productive, followed by environmental sciences, energy, business, management and accounting, medicine, engineering and, finally, economics, econometrics and finance. Thus, the interdependence between the SDGs is clear.

Consequently, although it can be seen that the impact of COVID-19 has been addressed in all the dimensions of sustainability among the 10 most cited articles, it is clear that those linked to the environmental dimension have received greater attention, despite being a virus that has mainly affected the health of the population, which links to the social dimension. This dimension has achieved the least attention, as measured by the total number of total citations.

Q3–Q4. Who are the most productive authors, institutions, countries and journals? Which are the main international cooperation networks?

The majority of the most productive authors are from Asia, the most prolific of which is Dang, T. T., far ahead of other authors such as Leal Filho, W. or Shaw, R. who are in second and third place. However, none of Dang, T. T.’s articles are among the most cited, which is the case with Leal Filho. Cooperation between authors has been very scarce, with only the work of Leal Filho being relevant. The ten most productive journals are Q1, the number one being the generalist journal Sustainability, with eight times more articles than the second most productive journal, the International Journal of Environmental Research and Public Health. In addition, the Chinese Academy of Sciences is the most productive institution and, therefore, China is the most productive country, followed by the United States and the United Kingdom. As in the case of authors, there is a lack of international cooperation between countries and institutions, building weak and fragmented networks.

This question has been difficult to answer, as the same article can cover up to six different SDGs. Scopus has introduced a new feature in its platform for mapping SDGs, but sometimes it assumes that an SDG is being addressed simply because it is mentioned, when, in fact, it receive minimal attention; this can lead to mistakes.

Five clusters were detected that together discuss the three dimensions of sustainability: economic, social and environmental. In general, all articles deal with SDG 3—good health and wellbeing, although SDG 8—decent work and economic growth, SDG 9—industry, innovation and infrastructure and SDG 12—sustainable consumption and production play a major role in the solutions.

Analyzing the SDGs from the perspective of the 5Ps, it is widely accepted that those related to Prosperity, Planet and People have been studied in depth, in that order; meanwhile, the SDGs of Partnership and Peace have received no attention at all. Therefore, it seems that scientific interest has been guided more by the concerns of the market in terms of economic recovery and improving the efficiency of companies under the slogan of sustainability.

With regard to supply chains, the studies analyzed predicted a great opportunity for the promotion of renewable energies, but the current situation of war is making smooth energy transition very difficult and, in some countries has led to the extension of the life of nuclear power plants or to pacts being made with states that violate human rights and reject peace.

As indicated from the outset, SDGs are deeply linked to the CE, which has drawn attention to the CE, especially in the environmental dimension. The CE paradigm and its tools have been part of the solutions to the economic, health and environmental crisis caused by the pandemic. It is seen as the most desirable new model on the basis of the opportunities detected. The articles detected were case studies focused on production changes and the recovery of waste in order to ensure the availability of secondary raw materials and secure supply chains.

This research suggests some limitations that offer potential areas for future lines of research. The main limitations faced by this study are the volatility of the articles (rapid changes in the number of citations and relevance of articles due to the novelty of the topic) and the difficulty in quantifying the number of SDGs addressed and the extent to which the articles are linked to the relevant SDG. For this reason, future lines of research could investigate further the topics detected and analyze the implications for each SDG and the progress being made. This study has analyzed articles only, since the peer review process guarantees a higher scientific quality, as explained in the methodology section. However, the results may be limited due to the fact that excluding other types of documents, e.g., book chapters, may affect the representation of some disciplines such as the humanities. Moreover, the research was restricted to the Scopus database, so including another database such as Google Scholar or Web of Science may significantly change the results. Furthermore, the research included the most common and general terms on the subject, which possibly excludes studies that used more specific terms. Finally, by using this kind of bibliometric analysis, the reduced number of citations interconnecting the various publications may not properly capture the impact of a publication. Indeed, these metrics do not necessarily work well for creative works and may not reflect local cultural practices. In the future, content analysis could be complementary, in order to assess the quality of the research.

Given that the CE has become a new paradigm that advocates the constitution of a new model based on the principles of sustainability, it would be interesting to propose research studies that analyze the impact of COVID-19 on the CE, indicating whether it has contributed to, or, on the contrary, paralyzed the advances that had been taking place up to that point. Neither sustainable finance nor other useful elements to alleviate the economic crisis or to improve the implementation of the CE were found to be relevant in the literature analyzed in this study; thus, future lines of research could address this gap and determine the influence of the pandemic on the concept of the CE.

Funding Statement

This research received no external funding.

Author Contributions

Conceptualization, C.M.-B., M.Z., C.L. and V.M.-M.; methodology, C.M.-B., C.L. and M.Z.; formal analysis, investigation and data curation, C.M.-B. and C.L.; writing—original draft preparation, C.M.-B.; writing—review and editing, M.Z., C.L. and V.M.-M.; visualization, C.M.-B.; supervision and project administration, M.Z. and V.M.-M. All authors have read and agreed to the published version of the manuscript.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Data availability statement, conflicts of interest.

The authors declare no conflict of interest.

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.