Visual Acuity (VA) and Intraocular Pressure (IOP)
Visual Acuity (VA) and Intraocular Pressure (IOP) are key parameters for ophthalmological practice, providing clinicians with vital information about overall ocular health and function [1]. These parameters are commonly measured in eye health services as part of an initial workup and provide clinicians with concise, critical information about eye health status, which can be tracked over time [2]. However, the collection of these data in a systematic manner for audit or research purposes can be time-consuming and costly. Large language models (LLM), a contemporary form of artificial intelligence (AI), have been proposed as a potential solution. The feasibility of using LLM for this process, outside of specialist computer science centres, is unknown.
VA measurements are important signals for quality improvement and research, serving as core quantitative indicators of how successfully a patient is being managed [3]. Accurate and consistent recording of these parameters is crucial, as they are frequently retrospectively reviewed as Key Performance Indicators (KPIs) for quality improvement initiatives. In addition, these measurements are often used in clinical trials and other research studies to assess the efficacy of interventions and to monitor patient outcomes. Therefore, it is essential that the recording and extraction of VA and IOP data are streamlined and standardised to ensure data integrity and facilitate efficient analysis.
Standardisation of VA and IOP measurements
Despite having standardised notation to represent VA, the documentation of these measurements and the context in which they were captured are subject to great heterogeneity in the format they are presented. Traditionally, these measurements were incorporated into the narrative of the written medical note. Although the advent of Electronic Medical Records (EMRs) initially offered hope for better documentation, especially for VA and IOP measurements, these functionalities have yet to achieve widespread implementation. Furthermore, EMRs that do offer such functionality still face issues with inconsistent documentation, and poor documentation quality [4]. At the Queen Elizabeth Hospital, VAs and IOPs are documented in free text by clinicians and nursing staff. Examples are provided in the Supplementary Materials section 10.1.
The unstandardised and unstructured documentation of VA and IOP presents a challenge for large-scale data retrieval for use with quality improvement and research initiatives. The automated extraction of such data into structured and standardised formats is therefore an important field of research that has great downstream implications for future work.
An automated approach to extracting visual acuity data has significant potential, particularly in the context of cataract surgery—the most common ophthalmic operation worldwide [5]. Refractive outcomes serve as a key performance indicator for assessing the quality of cataract procedures. By enabling efficient extraction of visual acuity data, this method could facilitate the creation and maintenance of large-scale outcome databases. Such databases are invaluable for optimising future outcomes, refining intraocular lens formulas, and tracking complications.
Importantly, the ability to efficiently extract and aggregate this data aligns with several objectives outlined in the RANZCO 2030 Vision framework, particularly sections. 2.6.1–2.6.6 which emphasise the need for better data access and transparency in eye healthcare [6]. AI-powered extraction of visual acuity data could serve as a foundational step in achieving this goal. These extraction techniques could evolve to support real-time data aggregation and visualisation, enabling easy consumption of key metrics by clinicians and administrators. This capability would be particularly valuable for conducting large-scale audits and maintaining comprehensive surgical registries.
Extraction of data from electronic medical records
Previous studies have explored the use of Natural Language Processing (NLP) to extract VA from electronic medical records (EMRs) [7]. Some previous works relied on EMRs which categorise VA into specific text boxes, simplifying the extraction process [8], and others have only investigated extraction of best documented VA, without extracting further context behind how the measurement was obtained [9]. Recent works have investigated addressing these limitations using transformer-based models trained on large-scale datasets to extract VA measurements [10].
The reliance on structured EMR data for VA extraction has several limitations. Firstly, it assumes that all relevant information is consistently recorded in designated text boxes, which may not always be the case. Clinicians often include important context and qualifiers within the free-text narrative of the medical note, such as the use of refractive aids, pinholes, or lenses used to obtain a best-corrected measurement during the VA assessment.
Secondly, the use of structured EMR data for VA extraction may not be generalisable to healthcare settings where such structured data is not available. Many ophthalmology practices still rely on free-text medical notes to document patient encounters, making it crucial to develop NLP techniques that can accurately extract and interpret VA information from these unstructured sources.
The focus on extracting VA values without considering the surrounding context can lead to incomplete or inaccurate interpretations of the data. To fully understand a patient’s visual function and track their progress over time, it is essential to capture not only the VA values but also the circumstances under which these measurements were obtained.
Use of Large Language Models (LLMs)
Autoregressive Large Language Models (LLM) have shown promise in performing higher-order tasks in the medical domain, as well as on free-text clinical notes [11], but they have also demonstrated remarkable performance in zero-shot learning scenarios [12]. Zero-shot learning refers to the ability of a model to perform a task without being explicitly trained on examples for that specific task. While terms such as zero-shot learning may be used variably in different contexts, for the purposes of this study it was considered to refer to the use of LLMs without specific fine-tuning with task-specific labelled data, or the provision of examples of the task to which it would be applied in the prompt. This approach is as opposed to a few-shot approach in which a prompt may contain examples of the task to which the model is being applied (e.g., notes with the specific text to extract). In the context of clinical natural language processing, this means that an LLM can answer plain English questions about a patient’s condition without requiring a large dataset of annotated examples. This capability is particularly advantageous in the medical domain, where access to large-scale, labelled data can be scarce due to ethical and privacy concerns and the effort required for manual annotation.
LLMs exist within the broader category of machine learning termed natural language processing (NLP). NLP encompasses a wide array of applications of computer analysis to human language, be it in writing or speech [13, 14] There are types of NLP that are more computationally economical than LLMs, such as rule-based approaches like regular expressions (“regex”). These approaches can have utility, such as for the collection of certain audit data [15] but tend to perform less effectively in unstructured note types with custom formatting and variable data entry methods and documentation practices, which are characteristic of elements of ophthalmology notes.
Despite this promise, the use of LLM for this task may be impeded by several factors. Using LLM where data are processed offsite is fraught, and often prohibited, for medical data. Deploying LLM locally, on hospital premises, presents an alternative approach. With conventional NLP approaches, such as random forest models based on decision trees, this may be relatively straightforward. However, LLMs are more computationally demanding, and such local deployment can require specialised equipment (e.g., graphics processing units) and building infrastructure (e.g., power sockets). While for specialised data centres and computer science institutions, these considerations are routine, the ability of a healthcare facility to deploy LLM locally for the task of VA data extraction is uncertain.
This study aims to evaluate the feasibility of using LLM to extract data from free-text healthcare documentation, namely VA, locally, at a healthcare facility. The work will also examine the performance of zero-shot applications of LLM in the extraction of VA from free-text medical notes.
link