Concordance of Hospital Ranks and Category Ratings Using the Current Technical Specification of US Hospital Star Ratings and Reasonable Alternative Specifications
View / Open Files
Publication Date
2022-05-06Journal Title
JAMA Health Forum
ISSN
2689-0186
Publisher
American Medical Association (AMA)
Type
Article
This Version
VoR
Metadata
Show full item recordCitation
Barclay, M. E., Dixon-Woods, M., & Lyratzopoulos, G. (2022). Concordance of Hospital Ranks and Category Ratings Using the Current Technical Specification of US Hospital Star Ratings and Reasonable Alternative Specifications. JAMA Health Forum https://doi.org/10.1001/jamahealthforum.2022.1006
Abstract
Importance
Centers for Medicare & Medicaid Services (CMS) Hospital Compare Star Ratings are widely-used summaries of hospital quality that can influence patient choice and organizational reputation.
Objective
We aimed to identify the impact of varying methodological choices on the calculation of the 2021 CMS Hospital Care star ratings and the ranking of hospitals.
Design, Setting, Participants
Cross-sectional study using publicly-available 2021 Hospital Compare data for 3,339 US hospitals from the October 2020 data release. We assessed change in apparent hospital performance arising when plausible alternatives to current methods are used for calculating star ratings in relation to individual measure standardization, domain derivation and domain weighting. We did this first by looking at three example changes in detail and also by using Monte Carlo simulation. We examined changes in centile of hospital ranks and in star rating overall, and also separately in CMS peer groupings defined by the number of reported quality domains.
Main Outcomes and Measures
Proportion of hospitals receiving a different star rating under the alternative technical specifications than under the current (2021) CMS approach.
Mean absolute change in centile rank of hospitals under alternative technical specifications compared with the current (2021) CMS approach.
Results
Our study included 3,339 US hospitals. Of the specific changes to current specifications that we considered, the alternative method of standardization was associated with the most substantial changes, with 55% of hospitals having their star rating reclassified. The change in domain weights was associated with the smallest differences – but even that resulted in reclassification of the star rating of one-in-four (24%) hospitals. In the simulation study, which covered a range of possible changes, on average half of hospitals (52%, IQR 44% to 60%) were assigned a different star rating from that produced by the current specification, usually into an adjacent star category.
Conclusions and Relevance
In this cross-sectional study of data on US hospitals, CMS Hospital Compare Star Ratings were found to be highly sensitive to how performance ratings are calculated, demonstrating the need for transparent justification of the technical approaches used in calculating composite performance ratings.
Sponsorship
This project was funded by MD-W’s Wellcome Trust Investigator award WT09789 and by THIS Institute’s grant from the Health Foundation. The Health Foundation is an independent charity committed to bringing about better health and health care for people in the UK. MD-W is an NIHR Senior Investigator (NF-SI-0617-10026). GL is funded by a Cancer Research UK Advanced Clinician Scientist Fellowship award (grant number C18081/A18180). The corresponding author is Matt Barclay, who led on the work and was employed at Cambridge until recently. He does not have a current affiliation as he has moved institutions.
Funder references
Wellcome Trust (097899/Z/11/Z)
Health Foundation (unknown)
Identifiers
External DOI: https://doi.org/10.1001/jamahealthforum.2022.1006
This record's URL: https://www.repository.cam.ac.uk/handle/1810/337696
Statistics
Total file downloads (since January 2020). For more information on metrics see the
IRUS guide.