Yelp reviews can enhance government reports on hospital quality, study suggests

Yelp publishes online crowd-sourced reviews about local businesses and services. It is the 33rd most visited website in the United States, with 142 million unique monthly visitors. On the site, patients are able to give hospitals a ranking from one to five stars, and provide narrative components that often reflect the features of a hospital experience most important to them.

In the new study, researchers compared approximately 17,000 Yelp reviews of 1,352 hospitals to HCAHPS reviews of the same institutions. The HCAHPS (Hospital Consumer Assessment of Healthcare Providers and Systems) Survey (pronounced "H-Caps"), is a standardized survey and data-collection tool that has been in use since 2006 to measure patient perspectives of hospital care in 11 different categories.

"Nearly 75 percent of U.S. Internet users reported looking online for health information in 2012," said the study's senior author Raina M. Merchant, MD, MSHP, an assistant professor of Emergency Medicine at Penn's Perelman School of Medicine, and director of the Penn Social Media and Health Innovation Lab. "Forty-two percent reported looking at social media for health-related consumer reviews. Meanwhile, only six percent of Americans had heard of the government website where the HCAHPS survey is reported, as of 2008. This divergence presents an opportunity for online consumer reviews to augment and even improve formal rating systems such as HCAHPS and increase their use in consumer decision making."

The researchers used natural language processing techniques to analyze the text of Yelp narrative reviews of hospitals posted as of July 15, 2014 to produce underlying categories. For example, a post that contained the terms "pain, doctor, nurse, told, medication, meds, gave," was labeled under the category "pain medications." Where possible, the categories were assigned to corresponding HCAHPS domains. For example, "pain medications" was allotted to the HCAHPS domain Pain Control.

While the Yelp reviews included information about seven of the 11 HCAHPS domains, the research team uncovered 12 additional categories covered in the Yelp reviews that are not covered by the HCAHPS survey: cost of hospital visit, insurance and billing, ancillary testing, facilities, amenities, scheduling, compassion of staff, family member care, quality of nursing, quality of staff, quality of technical aspects of care, and specific type of medical care.

For example, "compassion of staff" is not an item on the HCAHPS survey but instead is broadly reported under responses to the question "How often did doctors [or nurses] communicate well with patients?" Furthermore, four of the top five Yelp topics most strongly associated with positive Yelp review ratings were not covered by HCAHPS domains. These included: caring doctors, nurses, and staff; comforting; surgery/ procedure and peri-op; and labor and delivery.

"These topics that are covered within the Yelp reviews are important because they relate to the interpersonal relationships of patients with physicians, nurses, and staff," said the study's lead author Benjamin L. Ranard, a junior fellow at the Penn Social Media and Health Innovation Lab, and a combined MD/MS in Health Policy Research student at the Perelman School of Medicine. "Prospective patients are likely to want to know how caring and comforting caregivers are in various departments of a hospital."

Two of the top five Yelp categories most strongly associated with negative Yelp review ratings -- insurance and billing, and cost of hospital visit -- are also not covered by HCAHPS domains.

Citing previous research, the authors note that formal surveys such as HCAHPS suffer from low response rates and typically entail significant delays between hospitalization and public reporting of results. Finally, even if the evaluations can supply an overall indication of patient satisfaction, they rarely identify the source of perceived problems. The researchers add that while reviews on social media sites are not currently randomized, are largely uncurated, unvalidated, and subject to gaming, they are free, continuously updated, and often reveal in precise detail what the problem or positive occurrence was that affected the patient's or family member's experience.

"Yelp reviews are in real time and often written by patients for patients," said Merchant. "In addition, patients' perceptions of what matters most to them can change over time. HCAHPS may not be able to respond as agilely to these kinds of changes as social media."

To make reviews more helpful to readers, the Penn team suggests Yelp could highlight the most helpful positive and negative reviews, a practice that is becoming increasingly popular on consumer web sites like Amazon.com.

HCAHPS survey results are publicly reported on the Centers for Medicare and Medicaid Services Hospital Compare website, which rates all US hospitals that receive Medicare payments on a variety of quality measures. HCAHPS survey scores serve as a partial basis for what CMS pays hospitals for Medicare services. Hospitals with poor HCAHPS and other scores may see cuts of up to two percent in their Medicare payments. HCAHPS scores are based on a random sample of eligible discharges.