National Lung Cancer Audit (NLCA) senior clinical lead Paul Beckett describes important new changes to the way results for the audit will be analysed and presented, and how we will identify and assist trusts whose results fall below an acceptable standard.
The National Lung Cancer Audit’s (NLCA’s) core function is to compare the performance of organisations that provide care for patients with lung cancer and to use the comparisons to stimulate quality improvement. In conducting a comparison of a process or an outcome (for example, surgical resection rates), it is crucial to know whether poor performance is the result of local clinical practice or different patient characteristics (known as casemix). A lack of adjustment for casemix was a fundamental flaw of the comparative data that were published by cancer registries in past decades, and the NLCA was conceived to try to overcome this flaw by collecting a dataset that includes key clinical features such as performance status and disease stage.
When the first NLCA annual report was published in 2006, the quality of the data meant that only unadjusted data could be reported, which was similar to the previous cancer registry publications. In the following year, casemix-adjusted data were reported, but they were anonymised.
A lack of adjustment for casemix was a fundamental flaw of the comparative data that were published by cancer registries in past decades
However, by 2008 the data were of sufficient quality that casemix-adjustment became a regular feature of the NLCA reports and outlier identification. A logistic regression analysis is undertaken for outlier identification, in order to adjust for variations in age, sex, disease stage, performance status and socioeconomic status (and, more recently, comorbidity) to produce an odds ratio with 95% confidence intervals (CIs). This method indicates whether a particular result is statistically significant or not, and therefore whether an organisation is an outlier. This method of analysis is now well established and it has served the NLCA well for the past 10 years.
The LCCOP is now a core part of the NLCA, and we have refined the analysis to allow us to report a casemix-adjusted percentage
In 2013, the Lung cancer clinical outcomes publication (LCCOP) was established to provide quality assurance on lung cancer surgical practice in England. The LCCOP was initially run in conjunction with (rather than by) the NLCA and, perhaps as a result of this, a different methodology was used according to the preference of the Healthcare Quality Improvement Partnership (HQIP), which commissions both the NLCA and the LCCOP. The LCCOP methodology uses a similar casemix adjustment, but it defines two levels of outlier status: an ‘alert’ level of two standard deviations from the mean, and an ‘alarm’ level of three standard deviations from the mean.
These levels of outlier status correspond to statistical significance levels of 95% and 99.8% respectively. The LCCOP is now a core part of the NLCA, and we have refined the analysis to allow us to report a casemix-adjusted percentage, rather than an odds ratio. This percentage is more meaningful to both clinicians and the public, and the results are graphically illustrated using funnel plots (see Figure 1 in the attached Changes to the NLCA analysis methodology diagrams PDF).
Users of the NLCA will therefore notice a change in the way the data are reported
The NLCA has re-established itself after implementing new contracting and data collection arrangements, and as a result there is a need to develop a robust policy to identify outliers and to offer help to organisations whose outcomes fall below what we expect. It is not ideal to have two different methods of analysis, so we have decided to adopt the LCCOP methodology for the NLCA analysis. We believe that the benefits include:
- maintaining internal consistency across all our analyses and reporting
- ensuring consistency with HQIP’s preferred method of outlier identification
- producing casemix-adjusted metrics that are more meaningful (ie percentages rather than odds ratios).
Users of the NLCA will therefore notice a change in the way the data are reported, as shown in Figure 2 in attached PDF . Funnel plots will accompany the data tables.