NIH

This project proposes to develop and validate a summary indicator of neonatal intensive care quality delivered to very low birth weight (VLBW) infants to allow for comparative performance measurement among neonatal intensive care units (NICUs) (benchmarking). Interest in comparative quality measurement and evaluation has grown considerably because of widespread variation in clinical practice, increased availability of evidence about medical effectiveness, and increasing concerns about the cost and quality of healthcare. In other fields of medicine, benchmarking has improved the quality of healthcare institutions, providers, and patient care. The scorecard developed in this proposal will offer performance targets for improvement by showing the gaps between NICU performance and benchmarks that have been achieved. It provides a foundation for the development of public and private policy action, and a yardstick against which to measure the success of new policies.

Quality of Care.

Premature infants cared for in NICUs experience variations in utilization and quality of healthcare and in clinical outcomes that cannot readily be explained by differences in underlying clinical risk and may result in preventable morbidity and mortality. For example, rates of nosocomial infection rates, growth velocity, and treatment of persistent pulmonary hypertension vary considerably. Risk-adjusted data from the Vermont Oxford Network show up to threefold differences in mortality rates among NICUs. While these studies have some methodological limitations, available evidence suggests that an undesirable level of unexplained variation exists in NICU quality of care.

Benchmarking.

NICUs and their care practices develop in relative isolation from each other and from other stakeholders, including healthcare payers and patients. Evidence-based medicine has improved but not eradicated this phenomenon, because knowledge generation does not equal successful dissemination and implementation. Comparative quality measurement and evaluation (benchmarking) of NICUs diminishes isolation and increases accountability. Benchmarking is widely used in the business world and relates superior outcomes to the processes and structures that result in them. It improves quality by generating both intrinsic (motivation) and extrinsic (reputation, financial rewards) performance incentives.

Policy makers and healthcare payers are increasingly applying benchmarking in an attempt to increase the value of healthcare dollars. Legislation has been drafted that will make hospital reimbursement contingent upon performance. The Center for Medicare and Medicaid Services (CMS) has implemented a demonstration project to show the effectiveness of these strategies although the absence of a control group weakens the argument. Additionally, health plans frequently use length of stay or mortality to benchmark hospitals and set reimbursement rates despite the evidence that these measures are not reliable markers of quality of care.

Appropriate comparisons require that benchmarking accurately reflect provider performance, meaning that a high-ranking NICU is indeed a superior performer and vice versa. Thus, the data must be valid and reliable. Furthermore, the data must be available, easy to collect, difficult to ¿game¿ (i.e., difficult for providers to induce a systematic error on the output), and adjusted for clinical risk. The scorecard we propose for benchmarking NICUs will be a first step towards comparative measurement of the quality of care provided to VLBW infants. The proposed composite indicator will more closely reflect quality of care than uni-dimensional (often unadjusted) measures, such as length of stay or mortality.

Rationale for the Use of a Composite Quality Indicator (Scorecard).

Scorecards are widely used in health and non-health related fields. In other fields of medicine, they have improved the quality of healthcare. The Organization of Economic Cooperation and Development (OECD) has significant institutional expertise in developing, applying and evaluating composite indicators. OECD indicators are used to inform policy regarding a wide range of subjects, including education, business, and health. Similar to other disciplines, a composite indicator of care delivered to VLBW infants can be used as a tool to measure quality comparatively, provide feedback, foster improvement, inform policy, and improve patient care. Given that improving quality of care via benchmarking is a national priority, research in this subject is becoming increasingly important.

Implications of a Composite NICU Quality Indicator.

We propose that a multi-dimensional measure of quality will induce multi-dimensional improvement activities in NICU care. Scorecards induce competition on performance because hospitals care about their reputation and market share. Institutional competition for prestige would serve as a powerful stimulus to quality improvement. NICUs could apply knowledge about organizational management from fields outside of medicine (e.g., engineering, psychology) to redesign their systems of care in order to achieve better performance in many measures of quality simultaneously. Specific examples include systems which avoid work-arounds, promote safety and lead to correct actions by default. These activities could be supported by payers¿ ability to track and reward excellence if there is evidence for effectiveness. This type of measurement will benefit all stakeholders involved in the care of neonates, including patients, providers, payers, and researchers.

Patients will gain most from benchmarking as NICUs compete for quality. Benchmarking has improved patient outcomes in other fields in medicine. Report cards have reduced mortality related to coronary artery bypass grafting and improved care on a variety of measures for adults. In preliminary results, the CMS demonstration shows that high quality care results in fewer complications and readmissions; lower hospital costs; shorter length of stay; and lower mortality rates in coronary artery bypass graft patients.

Providers will benefit from the scorecard in that comparative performance measurement will be based on a rigorous measure of quality rather than the arbitrary measures currently employed by healthcare payers. The fairness of this approach promotes professional autonomy and motivation and may provide benefits to high performers in terms of reputation, patient steering or financial rewards.

Payers will have access to valid information on NICU performance and can communicate with providers about deficits and tools for improvement. Given constrained resources for healthcare, ensuring that patients receive high quality cost-effective treatments should achieve greater efficiencies.

Researchers will benefit from measuring NICU quality because a composite measure of quality, as an outcome measure, allows for the evaluation of the effectiveness of explicit financial incentives and innovative QI interventions in rigorous controlled trials. The framework we propose can be applied to future data collection, quality research and expansion to other NICU subpopulations. Moreover, the methodology used for this NICU scorecard could be translated to other healthcare settings.

Limitations of the Approach.

Composite indicators have advocates and detractors. Advocates hold that such a summary statistic can provide a meaningful assessment that is useful in attracting media interest and therefore the attention of policy makers. Detractors hold that the weighting process to combine the variables is arbitrary. All methods of measuring quality have advantages and disadvantages. The dangers in scorecards relate to oversimplification and misleading policy messages. Additionally, if indicators lack scientific soundness (e.g., length of stay) and are not transparent, they can be misused by stakeholder groups and are unlikely to produce the desired improvements in patient health status. We will counter these dangers using dissemination formats that will convey results accurately while avoiding oversimplification and by making the process of indicator development explicit and transparent to all stakeholders.

"This project will fill the gap for a summary measure of quality of care delivered to VLBW infants and allow for benchmarking NICUs on quality. The scorecard developed in this proposal will offer performance targets for improvement by showing the gaps between NICU performance and benchmarks that have been achieved. It provides a foundation for the development of public and private policy action, and a yardstick against which to measure the success of new policies.

The construction of composite indicators requires several steps, in which data are selected, weighted and aggregated to arrive at a summary score. At each of these steps, researchers face methodological choices, which may affect the results. Our methodological choices will be explicit and transparent and result in a ¿base case¿ composite indicator. Since there is disagreement among stakeholders (patients, providers, healthcare payers) about whose preferences should be used for health policy interventions, we will explore the impact of different methodological choices on the resulting NICU rankings (sensitivity analysis).

The specific research aims of our proposal include:

Aim 1: To select measures for inclusion in the scorecard of NICU quality of care

Hypothesis 1: Experts in NICU and quality research can agree on a common set of indicators.

Aim 2: To construct a scorecard and rank NICUs according to their performance

Hypothesis 2: Using a single versus a multiple imputation method for imputing missing data will not significantly alter NICU rankings.

Aim 3: To evaluate NICU rankings in response to alternative methods of scorecard construction as well as their stability over time

Hypothesis 3.1: Sensitivity analysis, in which the composite indicator will be developed via alternative methodologies will alter overall NICU rankings but there will be no significant differences in comparing the top and bottom quintile rankings of NICUs.

Hypothesis 3.2: There will be significant variation in NICU rankings throughout consecutive years but there will be no significant differences in comparing the top and bottom quintile rankings of NICUs.

Aim 4: To validate the scorecard

Hypothesis 4.1: There will be agreement between NICU rankings using the composite indicator and expert opinion (construct validity).

Hypothesis 4.2: There will be agreement between NICU rankings in the highest and lowest quintiles using the composite indicator and the Safety Attitudes Questionnaire (SAQ) (convergent validity).

Hypothesis 4.3: There will be agreement between NICU rankings using a random split sample of the 2004 and 2005 data of all participating NICUs (cross-validation).

b) Database Review

"Human Subject Involvement and Characteristics. This proposal includes several projects. The primary body of work will analyze patient and hospital level data that are routinely collected for outcomes research and quality improvement. As such, our study places no additional burden on VLBW infants.

Two parts of the proposal will directly involve human subjects: gathering of evidence via expert opinion and validation of the scorecard via the SAQ. Experts of NICU quality will participate in several tasks involving measure selection, weighting and composite indicator validation. They will be asked to participate in a web-conference for each of these tasks. Survey respondents consist of physicians and nurses at NICUs to be selected after construction of the composite indicator. We will ask NICU physician and nursing directors for their permission and help in distributing the survey among their staff. Participation is entirely voluntary and surveys will have no personal identifiers beyond the job status (physician or nurse) of the respondent.

Given the voluntary nature of participation and anonymity of questionnaire responders and the voluntary participation on the expert panels, we judge the emotional or social risks to be minimal. "

There is no anticipated direct benefit to either the experts or the practitioners responding to the SAQ. Potentially, the survey will spark discussion within a given NICU about deficits in this area of quality of care. On the other hand, developing a composite indicator of NICU quality may bring substantial benefits to future generations of premature babies. Additionally, other parties involved in the care of VLBW infants, as well as medical research in general may benefit from this project.