Researchers test new Risk of Bias assessment tool for non-randomized intervention studies

Inclusion of studies at high RoB may lead to erroneous conclusions about the benefits and harms of medical interventions. Researchers undertaking systematic review and meta-analysis (SRMA) of randomized clinical trials usually measure RoB of the component studies using an established tool, but there is no widely-accepted measure of RoB for non-randomized studies. As a test of performance, the Toronto-based researchers applied the new Cochrane RoB instrument, called ACROBAT-NRSI, to 37 papers included in two widely cited systematic reviews of the use of thiazolidinediones ("glitazones" used to treat diabetes) and cyclooxygenase-2 (COX-2) inhibitors (used to treat inflammation), and risk of major cardiovascular events (myocardial infarction and heart failure) and death.

Assessment took roughly 2.5 hours per study, and inter-rater agreement was moderate to substantial (overall Kappa 0.45 to 1.0). Of the 37 studies originally included in the meta-analyses, only eight had low RoB scored by ACROBAT-NRSI. Confining meta-analysis to studies with low RoB shifted the pooled odds ratio for myocardial infarction for rosiglitazone versus pioglitazone from 1.14 [95% CI 1.07-1.24] to 1.06 [95% CI 0.99-1.13]). Estimates of pooled relative risks (RR) of cardiovascular events with COX-2 inhibitors compared with no nonsteroidal anti-inflammatory drug changed little when analyses were confined to studies with low or moderate RoB, with the exception of an increased RR associated with ibuprofen (from 1.07 (95% CI 0.97-1.18) to 1.14 (95% CI 1.03-1.26)).

Because the study was conducted by a specific two-person team and considered only two systematic reviews, the findings may not be generalizable and further testing of the new Cochrane instrument is needed. Nonetheless, the findings indicate that at least two published meta-analyses included studies with high RoB. The authors state, "[e]xclusion of studies deemed to have unacceptably high RoB may impact the findings of pooled estimates of intervention effects, altering both the statistical and clinical significance of the results."