Potential fix for hospital star rating program

Researchers found that recognizing four fundamental quality factors in the safety of care category and assigning more equal weights to the eight measures in that category would modify the rating system's current scoring methodology and produce more accurate and informative results.

"Since the star ratings introduction in 2016, the ratings have been met with skepticism because of the methodology," says David Nerenz, Ph.D., director emeritus of Henry Ford's Center for Policy and Health Services Research and the study's lead author. "In our study, we believe we've identified a simple and straightforward solution in the safety of care domain that would make these ratings more meaningful and unbiased."

The study is published online in the American Journal of Medical Quality.

Created by the Centers for Medicare & Medicaid Services (CMS), the rating system assigns a score of one to five stars -- five being the highest -- based on a set of 57 individual quality measures across seven categories: mortality, readmission, safety of care, patient experience, effectiveness of care, timeliness of care and efficient use of medical imaging.

The first four categories each account for 22 percent of a hospital's total score, while the last three each count for four percent. A complex weighting scheme is used to assign an importance to each measure in each category, and the scheme for the safety of care category makes only one of the eight measures really count.

For their study, Dr. Nerenz and his team of researchers sought to evaluate the current methodology and how an alternative approach might affect the ratings scoring. Researchers performed principal components analyses on a subset of 674 hospitals in the December 2017 national data set that reported on eight safety measures used at the time. Then they assigned equal weight to each of the measures instead of the model employed by CMS and considered the number of cases at a hospital and how it impacted the accuracy of each measure.

The effect of applying this alternative methodology was illustrated in a single hospital, whose safety category score was markedly different (+1.65) compared to CMS' current methodology (-2.35). That was enough difference to move the score from below average to above average. Assigning equal weight to each measure had a "pronounced effect," researchers wrote.

"A stated aim of the Hospital Star Ratings program is to effectively summarize available hospital performance measures on Hospital Compare into a single, summary rating," says Brian Waterman, Ph.D., MPH, vice president of Analytic Business Solutions for the Missouri Hospital Association and a study co-author. "Our study results indicate that the current form of the methodology employed by CMS likely leaves important information about safety of care effectively out of the mix."

Researchers say the designers of the ratings system were well intentioned in creating a system that would be both "simple and patient-friendly" for consumers. However, the methodology is so highly complex that it has led to confusion and "very significant consequences for the ratings of individual hospitals." Further refining the methodology, they say, would better summarize the quality measures and "offer greater upside for consumers and providers."