Briefing : measuring performance in the national health service
April 2003
The recent rejection of the new contract for consultants has been blamed on the target and performance management culture of the National Health Service (NHS). In its evidence to the Select Committee on Public Administration, the NHS Confederation estimated that there were currently 250 targets for acute trusts and over 400 for primary care trusts. The Department of Health’s own press release for the launch of Foundation Hospitals promises that ‘the best hospitals will be freed from excessive Whitehall control’. According to the NHS confederation ‘this seems a remarkable confession about the department’s assessment of its own approach’1.
Consultants have accused the target culture of distorting clinical priorities. This claim is supported by research from the London School of Hygiene and Tropical Medicine on waiting lists for coronary artery bypass surgery. The study found that waiting list initiative monies are often specified for patients on the basis of length of time on the waiting list, rather than clinical need. Patients approaching the waiting time target may be admitted at the expense of more urgent patients with shorter waiting times2.
Pressure to meet targets can also result in unintended consequences and dysfunctional behaviours, such as deliberate manipulation of data and ‘gaming’3 4. Last year the National Audit Office published a report on ‘inappropriate adjustments to waiting lists’ which included offering patients admission during known holiday dates5.
A recent survey of Accident and Emergency (A&E) departments revealed that patients are routinely ‘warehoused’ in A&E departments because of a lack of available inpatient beds6. Patients on trolleys are considered ‘admitted’ for the purposes of meeting government targets, even though the patient does not have access to facilities for hygiene or food. The practice was also found to be detrimental to patient care because the ‘admitted’ patients were taking up physical space and nursing care that was needed by A&E patients. The government’s response has been to change its definition of a bed to ‘a device that may be used to permit a patient to lie down’7. This suggests a willingness on the part of government to collude in gaming, despite knowledge of its detrimental effects on patient care.
There are also dangers that defining explicitly what is to be measured encourages organisations and individuals to ‘jump through hoops’, directing efforts only to those aspects of care in which performance will be measured8.
The Centre for Health Economics at the University of York have identified five types of unintended consequences of performance indicators9:
Tunnel vision: Concentration on areas that are included in the performance indicator scheme, to the exclusion of other important areas.
Sub-optimisation: The pursuit of narrow local objectives by managers, at the expense of the objectives of the organisation as a whole.
Myopia: Concentration on short term issues, to the exclusion of long term criteria that may only show up in performance measures in a years’ time.
Misrepresentation: The deliberate manipulation of data including ‘creative’ accounting and fraud so that reported behaviour differs from actual behaviour.
Gaming: Altering behaviour so as to obtain strategic advantage.
Individual organisational targets also fail to consider the influence of other parts of the system and need for collaborative solutions, for example, in preventing emergency admissions and delayed discharges in the over seventy fives.
Many of these problems stem from the fact that the performance indicators currently in use do not measure what they purport to be measuring, that is, the performance of the health system. Most measures are measures of outcome, such as health status, rather than of structure (staff, beds, equipment) or process (the actual delivery of care)10. However health outcomes are often determined by population characteristics and behaviours outside the direct control of individual health professionals11. The care process may have little or no consequence on disease outcome, for example if an illness has an immutable natural history, or else outcome may follow a change in process of up to ten years, for example in the management of hypertension12.
Outcome measures also fail to measure interpersonal care, which is an important aspect of the health system for patients13, or cooridnation between health professionals, which is an important attribute of effectiveness of care14.
A more valid way of measuring performance would be to use measures of structure or process that have been demonstrated to be linked to improved outcomes. For example, it could be argued that a good measure of the performance of the local health system is the existence of spare capacity (ie bed occupancy less than 85%) and an adequately staffed 24-hour district nursing service, as these factors have been shown time and time again to be the most important in regards to waiting times in A&E and cancellation of elective surgery15 16.
Such measures are less likely to produce dysfunctional behaviour and, unlike outcome measures, provide an indication of what needs to be done to improve health care17. Measures of process also provide a measure of the patient experience which is currently missing from performance indicators18.
The current barrage of targets needs to be reduced to a smaller number of measures of structure and process that have been demonstrated to be linked to outcomes. These need to be combined with a culture that trusts health care professionals to find innovative solutions to local problems. The need for accountability can be addressed through the use of external audit bodies that can provide a more comprehensive analysis of the performance of the local health system and, importantly, have been shown to produce improvements in quality19.
References
1 Select Committee on Public Administration Minutes of Evidence. Memorandum by NHS Confederation (PST13). 17 January 2003.
2 Langham S, Soljak M, Keogh B, Gill M, Thorogood M, Normand C. The cardiac waiting game: are patients prioritised on the basis of clinical need? Health Services Management
Research 1997;10: 216-224.
3 Smith P. Outcome-related performance indicators and organisational control in the public sector. British Journal of Management 1993;4(3):135-151.
4 Goddard M, Mannion R, Smith P. The NHS performance framework:Taking account of economic behaviour. Centre for Health Economics Discussion Paper 158. York:
University of York;1998.
5 National Audit Office. Inappropriate adjustments to NHS waiting lists. HC 452. Session 2001-2002. 19 December 2001.
6 British Medical Association. Waits and Measures. Improving emergency care for today’s patients. London:BMA; 2002.
7 House of Commons Hansard. 13 January 2003:Colum 498W.
8 Campbell S, Roland M, Buetow S. Defining quality of care. Social Science and Medicine 2000;51 :1611-1635.
9 Goddard M, Mannion R, Smith P. The NHS performance framework:Taking account of economic behaviour. Centre for Health Economics Discussion Paper 158. York:
University of York;1998.
10 Donabedian A. Explorations in quality assessment and monitoring. Volume 1: The definition of quality and approaches to its assessment. Ann Arbor: Health Administration Press; 1980.
11 Giuffrida A, Gravelle H, Roland M. Measuring quality of care with routine data: avoiding confusion between performance indicators and health outcomes. BMJ 1999;319:94-98.
12 Campbell S, Roland M, Buetow S. Defining quality of care. Social Science and Medicine 2000;51: 1611-1635.
13 Wensig M, Grol R, Smits A. Quality judgements by patients on general practice care: A literature analysis. Social Science and Medicine 1994; 38: 45-53.
14 Cambell S, Roland M, Buetow S. Defining quality of care. Social Science and Medicine 2000; 51(11):1611-25.
15 Bagust A, Place M, Posnett W. Dynamics of bed use in accommodating emergency admissions: stochastic simulation model. BMJ 1999;319:155-8.
16 British Medical Association. Waits and measures. Improving emergency care for today’s patients. London: BMA;2002.
17 McColl A, Roderick P, Gabbay J, Smith H, Moore M. Performance indicators for primary care groups: an evidence based approach. BMJ 1998;317:1354-60.
18 Cambell S, Roland M, Buetow S. Defining quality of care. Social Science and Medicine 2000; 51(11):1611-25.
19 Klein R. Day P. Auditing the auditors: Audit in the National Health Service. London: Nuffield Trust; 2001.