Predicting non-attendance with AI

June 13, 2022 Finley Breeze

Patients not showing up to their scheduled appointments is a significant issue, estimated to cost the New Zealand healthcare system $29 million each year1.

 

With ophthalmology being a high-volume, clinic-based specialty, non-attendances are particularly detrimental to the provision of quality ophthalmic care. Opportunities to prevent irreversible progression of ocular conditions are missed, putting patients at increased risk of vision loss2,3.

 

For clinics themselves, non-attendance reduces the capacity to schedule patients, under-utilises limited health resources and contributes to longer waiting lists. Furthermore, New Zealand's non-attendance rates reveal marked disparities in access to care. Māori and Pasifika have higher rates of non-attended appointments, reflecting a healthcare system that does not effectively engage them.

 

Current strategies to improve attendance rates predominantly involve text reminders. This is a blanket, brute-force strategy and is largely ineffective in targeting higher-risk patients. Research by Te Puna Oranga (Māori Equity Strategy & Research Team) at Waikato Hospital revealed that the majority of non-attendees were unable to be contacted on their documented mobile number4. To improve access to care, it is vital to improve patient engagement and gain a deeper understanding of which patients are at high risk of non-attendance and why appointments are being missed.

 

Why is machine learning useful?

 

With around 300,000 public ophthalmology appointments in New Zealand each year, district health boards (DHBs) hold large amounts of attendance data. For each appointment, several pieces of information are recorded. Individually, these variables are not enough to predict patient non-attendance, yet collectively they can be highly effective5. Machine-learning algorithms excel in distilling such multi-dimensional big data into simple predictions.

 

Our study compared traditional statistical modelling approaches with state-of-the-art machine-learning algorithms on a dataset totalling over three million appointments. These data were stratified by DHB to compare how models performed in different regions. The study methodology aimed to address common issues in machine-learning algorithms. Complex ‘black-box’ models are often able to achieve impressive accuracy but with no explanation of how or why predictions were made. In some cases, they are prone to bias. We used methods to help us explain the most important features in the model and the impact these had on individual predictions, and a robust validation framework where the best models were trained, fine-tuned and selected using cross-validation on designated training data.

 

The results

 

Using the area under the receiver operating characteristic curve (AUROC) metric, machine learning models achieved scores of 0.7 to 0.9 on data for 19 out of 20 DHBs in predicting non-attendances. This indicates a high level of sensitivity and specificity for predictions in almost all DHBs. Many DHBs benefited from models being restricted to their data alone, suggesting there were regional patterns of non-attendance. This level of discrimination is clinically useful to improve our clinic management and develop targeted interventions.

 

To highlight how these models may be clinically useful, we can focus on the DHBs with higher ophthalmology non-attendance rates, which would benefit the most. For example, Tairāwhiti and Whanganui had non-attendance rates of 8% and 10%, respectively. Models were able to correctly predict over 70% of non-attendances and only incorrectly labelled one quarter of attending patients.

 

For these DHBs, around 1 in 3 predictions of non-attendance was correct. Since we have 1 in 10 patients not attending, we might use the model to determine which patients to contact first with an appointment reminder. Using model predictions, we would only have to contact three patients to engage with the one that would not have attended their next scheduled appointment. This potentially represents a 70% reduction in workload!

 

Using methods to improve explainability, we were able to gain insight into the importance of variables in these attendance records. This allowed big-picture insights to be made about the entire dataset, which reflected marked inequities in access to care for Māori and Pasifika. Since machine-learning models can analyse data in real time, we hope active monitoring of inequities will aid ophthalmic clinics in improving engagement with these communities.

 

With interpretable machine learning, we can take this a step further to understand why an appointment is at high risk of non-attendance and unpack predictions for that individual patient and develop targeted interventions to address their personal barriers in accessing care. Ultimately, we hope this level of predictive performance and interpretability means these models are useful in streamlining scheduling practice, improving engagement with patients, helping clinics to ensure they are providing equitable ophthalmic care and achieving better visual outcomes.

 

Further research is planned to look specifically at Waikato Hospital Ophthalmology Department data. For this, we are collecting a much richer dataset that includes exact times of appointments and bookings, attendance history of individual patients and even local weather information! Once the models are developed, we aim to make predictions on clinic schedules in real-time to truly test the model’s capabilities.

 

References

  1. Tax Payers’ Union Briefing Paper, ‘Missed Specialist Appointments: A $29 Million Bill for Taxpayers’ (2019).
  2. Hawn, S; Yuan, S; Lee, A; and Culican, S. Visual Acuity Outcomes and Loss to Follow-up in the Treatment of Amblyopia in Children From Lower Socioeconomic Backgrounds. J. Pediatr. Ophthalmol. Strabismus 1–8 (2021).
  3. Ramakrishnan, MS; Yu, Y, and VanderBeek, BL. Association of Visit Adherence and Visual Acuity in Patients With Neovascular Age-Related Macular Degeneration: Secondary Analysis of the Comparison of Age-Related Macular Degeneration Treatment Trial. JAMA Ophthalmol. 138, 237–342 (2020).
  4. Māori Equity Strategy & Research Team, Waikato District Health Board. Outpatient Attendance Improvement Project (2018).
  5. Liu D, Shin WY, Sprecher E, Conroy K, Santiago O, Wachtel G, Santillana M. Machine learning approaches to predicting no-shows in paediatric medical appointment. npj Digital Medicine. 2022 Apr 20;5(1):1-1.

 

 

Finley Breeze is a medical honours student from the University of Auckland based in Waikato who won the John Parr Trophy at the recent RANZCO 2021/2022 virtual conference for this research. He is interested in the application and implementation of artificial intelligence solutions in ophthalmology and health systems. This thesis was supervised by ophthalmologist Dr James McKelvie and Dr Michael Mayo, a senior computer science lecturer from the University of Waikato.