This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Education, is properly cited. The complete bibliographic information, a link to the original publication on http://mededu.jmir.org/, as well as this copyright and license information must be included.
The Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) framework is a validated evaluation tool used to assess the quality of scientific publications. It helps in enhancing clinicians’ decision-making process and supports production of informed healthy policy.
The purpose of this report was two-fold. First, we reviewed the interpretation of observational studies. The second purpose was to share or provide an example using the GRADE criteria.
To illustrate the use of the GRADE framework to assess publications, we selected a study evaluating the risk of spontaneous abortion (SAB) after influenza vaccine administration.
Since 2004, the Centers for Disease Control and Prevention and the Advisory Committee on Immunization Practice have recommended influenza vaccination of pregnant women. Previous studies have not found an association between influenza vaccination and SAB. However, in a recent case-control study by Donahue et al, a correlation with SAB in women who received the H1N1 influenza vaccine was identified. For women who received H1N1–containing vaccine in the previous and current influenza season, the adjusted odds ratio (aOR) for SAB was 7.7 (95% CI, 2.2-27.3), while the aOR for women not vaccinated in the previous season but vaccinated in the current season was 1.3 (95% CI, 0.7-2.7).
Our goal is to enable the readers to critique published literature using appropriate evaluation tools such as GRADE.
The Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) framework is a validated evaluation tool used to assess the quality of scientific publications. GRADE was developed by an international group of health professionals, researchers, and guideline developers to standardize the evaluation process of publications [
The present review provides a demonstration of how to use the GRADE system to critique an observational study [
We performed a brief review of a recent publication by Donahue et al [
Cases included 485 women who had SAB and 485 pregnant women in the control group. Both groups were compared to determine whether women with SAB were more likely to have received the 2010-2011 or 2011-2012 seasonal flu vaccine in the proceeding 28 days of SAB. The control cases were selected based on similar characteristics to the cases of SAB, which included maternal age group (<30 or >30 years), approximately similar date of LMP, and enrollment in the same health care plan. Adjustments between the cases and controls were made for smoking during pregnancy, diabetes type 1 or 2, obesity with a body mass index (BMI) of >30, and health care utilization in the prior 1 year. The exposure in this study was receiving the H1N1 influenza vaccine, and the observed outcome was SAB during the first trimester of pregnancy [
A regression analysis was performed in the reviewed article; however, the analysis excluded some vital demographics, which may have affected the validity of the results. Some of the demographics are shown in
Based on abstracted data from the VSD, the authors calculated an adjusted odds ratio (aOR) of 2.0 (95% CI, 1.1-3.6) for SAB within 1-28 days in both 2010-11 and 2011-12 seasons, comparing vaccinated to unvaccinated women in these seasons. The aORs for 2010-2011 and 2011-2012 were 3.7 and 1.4, respectively, in vaccinated compared with unvaccinated women. On the other hand, the aOR for women who received H1N1–containing vaccine during both the previous (2010-11) and current (2011-12) influenza seasons was 7.7 (95% CI, 2.2-27.3). Meanwhile, the aOR for groups that received the vaccination in 2011-12 but not in 2010-11 was 1.3 (95% CI, 0.7-2.7). When women with previous SAB were excluded, the aOR remained elevated at 6.5 (95% CI, 1.7-24.3); however, the sample size was small, which is represented by the wide CI value (95%, Cl, 2.2-27.3). The study concludes that there is a correlation between SAB and influenza vaccination in the preceding 1-28 days, particularly among women who had been vaccinated in the previous season.
Major differences in demographics between cases and controls.
Characteristics | Cases, n (%) | Controls, n (%) | ||
Age, 35-44 years | 157 (32.4) | 128 (26.4) | N/Aa | |
Body mass index≥30 | 134 (27.6) | 112 (23.1) | N/A | |
Race, African American | 42 (8.7) | 20 (4.1) | .008 | |
≥1 | 138 (28.5) | 125 (25.8) | .32 | |
≥2 | 43 (8.9) | 26 (5.4) | .03 | |
Smoked during pregnancy | 52 (10.7) | 34 (7.0) | .05 |
aN/A: not applicable.
To combat the risks of misinterpreting the reported stories by the news outlets and to standardize the evaluation methods of publications, we recommend using the GRADE system to assess publications. In addition to its validated effectiveness for that purpose, GRADE provides a quantitative evaluation of the evidence [
Randomized clinical trials are assigned 4 points, while observational studies receive a score of 2 points [
This is a component of the GRADE framework that addresses the methodology and execution of the study by assessing the blinding process, group allocations, follow-ups, and sparse data (missing data). One point is deducted for each problem identified in one of these elements with a maximum deduction of 3 points [
This component assesses the consistency of outcomes. A point is deducted for inconsistent results, whereas a point is added for evidence of a dose response, or if adjustment for confounding variables would have increased the effect size [
This component evaluates the issues that may hinder the generalizability of the reported outcomes for the specified population [
Moreover, race was not adjusted for in the study model, even though a significant difference between cases and controls was observed in African American women as shown in
This component measures the impact of OR, relative risk (RR), or hazard ratio (HR) to provide an estimate of the significance of the results [
Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) score: quality and interpretation.
GRADE score | Quality | Interpretation |
≤1 | Very low | Any estimate of effect is highly uncertain |
2 | Low | Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate |
3 | Moderate | Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate |
≥4 | High | Further research is very unlikely to change our confidence in the estimate of effect |
An OR >1 reflects that the exposed population is more likely to have the observed outcome. An OR <1 means that the exposure is protective. OR is usually presented with a CI wherein the bigger the sample, the smaller the CI. In cases where the CI range crosses the value of 1.0, the OR value will be impaired due to the possibility of having a null, which implies no relationship between exposure and disease [
The present review has demonstrated the use of the GRADE framework to quantitively evaluate an observational study, which has been shown to be an effective tool in assessing the study’s strengths and weaknesses. The GRADE framework is easy to use and provides a great estimate of publications’ overall quality. The reviewed study has several limitations; they include the small number of participants, unrepresentative sample, and an observation of an outcome that is rather common during the first trimester, especially between 7th and 12th weeks, of pregnancy [
Another major limitation is the failure to appropriately match case-control groups as the case group had an older population, more SAB, higher BMI, and more smokers during pregnancy. We also need to consider the possibility that some of the pregnant women included in the study might have received influenza vaccination in a nontraditional setting such as in a pharmacy and were not identified as recipients on their medical records [
Other issues noted in the study include the possible impact of missing data from the dataset, as 13.6% (66/485) of the data points from the cases and 7.2% (35/485) of the data points from the control group were missing. This is a significant number if we take into consideration the small sample size. Furthermore, some of the outcomes were a result of a post hoc analysis, which refers to an outcome that was not planned for in the study design and was simply noted at a later stage. This is still a major limitation of the study. Physicians should not base their practice on post hoc findings as the results might be flawed due to chance.
A recent survey conducted by the CDC in late 2017 found out that around two-thirds of pregnant women in the influenza season of 2017-18 had not been vaccinated against influenza. Furthermore, only 15.6% of pregnant women who visited a medical provider since July 2017 had received a recommendation for the influenza vaccination, but not offered one; while 25.7% neither received a recommendation nor an offer for the influenza vaccination, 58.7% of pregnant women received a recommendation and an offer to administer the influenza vaccine [
The CDC’s current recommendation is to vaccinate all pregnant women. It is challenging to convince the public that the reviewed study had various limitations and cannot be generalized to all pregnant women after a media blitz. The media have played a significant part in promoting this study, as multiple news outlets adopted this study’s findings with misleading headlines such as “Miscarriages linked to flu vaccine being administered during pregnancy in new study” from The Independent news agency [
The media has a very strong impact on the way we think and act as a society [
As clinicians and researchers, we will be able to facilitate the use of GRADE to analyze the study and its statistical significance. For patients, however, it may not be that easy to find their way through the maze of variables, calculations, and adjusted rates. Therefore, a collaborative discussion with the patient is necessary to explain the overall quality of such publications and recommendations to follow.
The GRADE framework is a validated tool used to quantitively assess the overall quality of publications. Through the use of GRADE, we uncovered the low-evidence score for the reviewed article. Therefore, the best course of action will be to follow the CDC’s recommendations by providing the influenza vaccine to all pregnant women. Physicians should adopt validated evidence-based tools, such as GRADE, to quantitively assess the overall quality of studies and provide evidence-based practice.
adjusted odds ratio
body mass index
Centers for Disease Control and Prevention
Grading of Recommendations, Assessment, Development, and Evaluation
Hospital Corporation of America
hazard ratio
last menstrual cycle
measles, mumps, and rubella
relative risk
spontaneous abortion
Vaccine Safety Datalink
This research was supported by Hospital Corporation of America (HCA) and an HCA-affiliated entity. The views expressed in this publication represent those of the authors and do not necessarily represent the official views of the HCA or any of its affiliated entities.
None declared.