Diabetic Eye Screening: Evaluation and Comparison of the Grading Results Between Community and Hospital Eye Services

Introduction

Globally, about 828 million people have diabetes, with approximately 5.6 million affected in the UK, accounting for around 8% of the population.^1,2 There is estimated to be 212,000 diabetics who live in Wales.² The impact of diabetes on the National Health Service (NHS) in the UK is significant, costing around £10 billion per annum.²

Diabetes has many systemic complications, one of which is diabetic retinopathy (DR), which is one of most common causes of blindness in the UK.³ It affects vision through two main forms: active proliferative retinopathy, that leads to new blood vessel growth and potential haemorrhage, and diabetic maculopathy, which disrupts central vision.⁴ DR poses significant economic costs as it impacts working-age individuals.⁵ It stems from hyperglycaemia-induced damage to retinal blood vessels, involving pericyte loss and barrier impairment, and is complicated by chronic inflammation and retinal neurodegeneration.^4,6

DR affects 30–40% of those with diabetes, for all types of diabetes.³ Early detection and intervention can prevent irreversible damage.^7,8 A robust screening program is essential to identify and refer at-risk individuals, aligning with the Wilson and Junger criteria adopted by the WHO and the UK National Screening Committee.^9–12 Continuous screening is crucial, as the duration of diabetes is the primary risk factor for developing DR.¹³

Since 2003, DR screening in Wales has been centralised at Diabetic Eye Screening Wales (DESW), an NHS provision for eligible individuals.¹⁴ Clinics are offered throughout Wales, ensuring services for patients are close to home.¹⁵ Eligibility includes anyone over 12 years old with a diabetes diagnosis registered with a General Practitioner in Wales.¹⁵ Patients are automatically invited for screening upon diagnosis from secondary or primary care.¹⁵

To date, there has been no research evaluating grading agreement across the screening pathway for DR in Wales. The primary aim of this study was to identify the agreement between primary graders (DESW) and secondary graders (community optometrists with a specialist qualification in diabetic eye disease). A secondary aim was to compare the agreement between the secondary graders and the tertiary graders (hospital eye services).

Ethics

This study adhered to the tenets outlined in the Declaration of Helsinki. The study was discussed with Swansea Bay University Health Board Research and Development department. It was deemed that no further ethical approval was required, and patients were not required to provide informed consent as the data was collected during normal service delivery. Data accessed complied with relevant data protection and privacy regulations.

Methods

Study Design and Subjects

This retrospective study assessed data collected between January 2021 and December 2023, identifying a total of 762 eyes from 486 patients. The data was obtained from referrals sent from the primary graders. In Swansea Bay Health board, all referrals are received by tertiary graders who then determine based on the grade whether they are to be seen by secondary or tertiary graders. The data obtained adhered to the RxMx grade (Table 1) which was used by all three groups: primary graders, secondary graders, and tertiary graders.¹⁶ It should be noted that maculopathy was further subdivided into active and stable maculopathy by the secondary and tertiary graders due to the ability to perform fundoscopy and acquire OCT (Optical Coherence Tomography) which are investigations not available to the primary graders. Given this, maculopathy between primary and secondary graders was only based on absence or presence of macular disease.

Table 1 The RxMx Grading System Used in England, Wales, and Northern Ireland Which is a Feature-Based Grading System

The standard procedure for patients after grading in Wales follows a three-tier system based on severity (see supplementary material 1). Primary graders from DESW initially screen all patients. Those classified as R0M0/R1M0 are re-invited biannually, due to their low risk of progression over a two-year period. Patients showing R2M0, R0M1, R1M1, or ungradable images are seen by secondary graders—qualified community optometrists, who perform further imaging and examinations. Ungradable images typically prompt a second attempt before being referred onwards. Patients classified as R3M0/R3M1 are seen directly by tertiary services for treatment and monitoring. The screening pathway is illustrated in Supplementary Material 1.

Inclusion and Exclusion Criteria

Retinal images that were excluded from this study were those deemed ungradable by DESW due to either poor picture quality, lack of a complete fundal image or abnormalities that lie outside the remit of the graders. This was because there would be no grading in place from the primary graders and thus an inability to look for interobserver agreement. Referrals received from primary and secondary graders, alongside clinic letters from tertiary graders that did not contain a complete grading score in line with the RxMx system were removed as to ensure completeness.

Training

The primary graders were non-clinical trained technicians specialising in assessing retinal images. They completed in-house training developed by a lead ophthalmologist and are required to evaluate a specific number of retinal images under supervision prior to independently grading retinal images. According to DESW guidance, any abnormalities outside of the RxMx grading system are referred to directly to Hospital Eye Services (HES).

The secondary graders were community optometrists listed with the General Optical Council who hold the College of Optometrists Professional Certificate in medical retina.¹⁷

The tertiary graders were the hospital eye service, this was composed primarily of ophthalmologists and qualified allied health professionals under direct consultant ophthalmologist supervision.

Clinical Assessment

Table 2 outlines the range of retinal imaging and clinical examinations performed by primary, secondary, and tertiary graders. Retinal imaging in the form of a two-field photograph were taken at each point of contact. The DESW cameras capture two 45-degree retinal images for each eye: one centred on the macula and the other focused on the nasal field, using a non-mydriatic Canon DGi camera equipped with either a 30D or 40D camera back. Secondary graders use various imaging machines based on the equipment available at the community optometrist’s practice.

Table 2 Clinical Examinations Performed by Primary, Secondary, and Tertiary Graders

There is currently no centralised database in Wales allowing images taken by one group to be seen by the others. The imaging modality for DESW consisted solely of retinal photographs, while secondary and tertiary graders also utilised OCT and fundoscopic examination (Table 2). The tertiary graders had the greatest range of imaging modalities including OCT-angiography, fundus fluorescence angiography (FFA), and ultra-wide field imaging in the form of Optos.¹⁸

Patients under DESW routine follow-up would adhere to the strict criteria of being seen biannually unless they met criteria meaning they were referred to the secondary or tertiary graders. Those under of the secondary and tertiary graders would be seen according to clinical need as determined by the clinician’s judgement.

Measures

Statistical Analysis

The percentage agreement was calculated by comparing the number of times both groups assign the same Rx value to an eye, divided by the total number of cases assessed by the first group. For example, between primary and secondary graders for R0, as illustrated below.

Weighted Cohen’s Kappa was performed to measure the reliability of the agreement between the secondary graders using SPSS Version 14. The weighted Cohen’s Kappa test looks at the interobserver reliability using a scale of 0 to 1.^19–21 At its extremes, the scale implies that a value of 0 is when there is no agreement, whilst between 0.81 and 1 there is almost perfect agreement. In between this range there is 0.01 to 0.2 as none to slight, 0.21–0.4 as fair, 0.41–0.6 as moderate, and 0.61 to 0.8 as substantial. The use of weighted Cohen’s Kappa was chosen as it is more sensitive to the degree of disagreement, providing a more accurate reflection of agreement in ranked categories.^22,23

Percentage agreement and weighted Cohen’s Kappa is reported for retinopathy (R1, R2, R3) and maculopathy (M0, M1) between primary and secondary graders, and secondary and tertiary graders.

Public and Patient Involvement

The study’s design incorporated input from patients and the public through collaboration with the Royal College of Ophthalmologists in Wales, as well as consultations with national organisations and local Eye Clinic Liaison Officers. Upon publication, the study’s findings will be disseminated to members of the Wales Vision Forum and Wales Council for the Blind.

Results

In accordance with the inclusion and exclusion criteria, a total of 1,030 eyes were analysed between primary and secondary graders (n=268 excluded) and 134 eyes were analysed between secondary and tertiary graders (n=38 excluded).

Agreement Between Primary and Secondary Graders Retinopathy

Percentage agreement between the retinopathy grading of the primary and secondary graders was 87.9%, 58.5%, and 45.6% for R0, R1, and R2, respectively. The R3 were not compared as patients were seen directly by tertiary graders and so no interobserver agreement would be seen. These agreement levels are illustrated in Table 3 and results in an overall percentage agreement of 60.4% (460/762). The weighted Cohen’s Kappa was 0.423 (Standard error [SE] 0.029) indicating moderate agreement.

Table 3 Grading Scores for Retinopathy Given by the Primary (DESW) and Secondary Graders (Community Optometrists) Illustrating Percentage Agreement for R0, R1, R2, R3, and Overall

Agreement Between Primary and Secondary Graders Maculopathy

Percentage agreement between primary and secondary graders was 71.7% (294/410) and 67.0% (236/352) for M0 and M1, respectively (Table 4), resulting in an overall percentage agreement of 69.6%. The weighted Cohens Kappa was 0.42 (SE, 0.017), demonstrating moderate agreement.

Table 4 Grading Scores for Maculopathy Given by the Primary (DESW) and Secondary (Community Optometrists) Graders Illustrating Percentage Agreement for M0, M1, and Overall

Agreement Between Secondary and Tertiary Graders Retinopathy

Percentage agreement between the retinopathy grading of the secondary and tertiary graders was 55.0% (11/20), 75.0% (21/28), 48.3% (14/29), and 68.4% (13/19) for R0, R1, R2, and R3, respectively. This is illustrated in Table 5 and results in an overall percentage agreement of 61.50% (59/96). The weighted Cohen’s Kappa was 0.634 (SE, 0.068) and tertiary graders illustrating percentage agreement for R0, R1, R2, R3, and overall.

Table 5 Grading Scores for Retinopathy Given by the Secondary (Community Optometrists) Graders Illustrating Percentage Agreement for R0, R1, R2, R3, and Overall

Agreement Between Secondary and Tertiary Graders Maculopathy

Percentage agreement between secondary and tertiary graders was 89.7% (26/37), 65.9% (29/44), and 68.0% (17/25) for M0, M1a, and M1s, respectively (Table 5). Overall percentage agreement was 75.0%, as shown in Table 6. The weighted Cohen’s Kappa of 0.618 (SE, 0.062) indicating substantial agreement.

Table 6 Grading Scores for Maculopathy Given by the Primary and Secondary Graders Illustrating Percentage Agreement for M0, M1a, M1s, and Overall

The weighted Cohen’s Kappa values are shown in Figure 1, for each interobserver agreement analysed with the benchmark highlighted.

Figure 1 Weighted Cohen’s Kappa value for interobserver agreement for retinopathy (R) and maculopathy (M) by primary (1) and secondary (2) graders followed by agreement between secondary (2) and tertiary (3) graders.

Discussion

Our results show a fair and substantial level of agreement according to weighted Cohen’s Kappa between primary/secondary and secondary/tertiary graders, respectively, for retinopathy grading. Evidence suggests that an agreement below 60% is inadequate for clinical reliability, while agreement of approximately 80% is generally accepted as the gold standard.^20,21

Agreement between the primary and secondary graders was greatest for the lower grades of DR (R0 and M0), with agreement decreasing as the retinopathy grade increased. The results indicate that primary graders tend to over-grade, which is seen in previous literature.^24–26This could be explained by the limited information available from fundus images alone as well as the role-specific nature of the primary grader, with over-grading being considered safer than under-grading. Furthermore, the 2013 Four Nations Diabetic Retinopathy Screening Intervals Project Study Group²⁷ noted variations in grading practices across UK programs, affecting patient assessments and treatment for diabetic retinopathy. To address this, steps should be taken to enhance quality assurance and transparency in the screening program, with clear targets for internal and external reviews.

Despite the higher percentage of agreement in the maculopathy group, concerningly there were 116 out of the 410 patients graded by primary graders as M0 which were graded by the secondary graders as M1. Resemblances can be seen in a previous study,²⁸ that highlighted the lack of OCT as fundamental to operating with retinal images alone as a screening tool for DM. The presence of DM increases the risk of sight loss between 30% and 50% after two years and thus it is imperative to have an effective screening tool.²⁹ In comparison to DR, the higher agreement levels may be due to the high prevalence of M0 grades which could inflate the overall agreement as well as the simplified categorisation of DM as either present or absent.

Introducing optical coherence tomography (OCT) alongside fundal imaging could help address discrepancies in DR screening in Wales, potentially increasing grader agreement as currently, OCT is not part of the DESW programme. Recent research suggests that introducing OCT could streamline the screening process by reducing false positives and improving cost-effectiveness.³⁰ This is supported by our results of increased agreement between secondary and tertiary graders where both OCT and fundoscopy was utilised.

With a projected increase in the number of people living with diabetes in Wales to 260,000 by 2035/2036 will come with an increase on burden of eyecare services.^2,31 Recent studies^32–34 suggest that personalised risk-based screening could be more effective than annual reviews. This challenges the current one-size-fits-all screening model, as over-grading may lead to unnecessary referrals, while under-grading increases the risk of delayed treatment, especially with capacity constraints in hospital. Changes to the service like the introduction of the extended screening interval in 2023, which allows low-risk patients to be screened biennially aims to reduce service demand.^12,15 However, a blanket approach for the low-risk patients may lead to disparities among ethnic and higher-risk groups, as studies have shown that biennial reviews may not be adequate for all populations.^35,36 For example, younger patients may present later to services and are more likely to fail to attend, compounding the issue by presenting with more advanced disease and having poorer outcomes.⁹

Strengths and Limitations

The key strength of this study is the use of real-world data without the typical synthetic environment created by studies that remove the human factors and environmental influences that can contribute to outcomes. However, in Wales, the lack of a centralised electronic patient records meant that retinas were assessed at different times using different modalities which may impact upon agreement. It is possible that some extent of the lower agreement could be attributed to the disease progression between grading sessions for both datasets. The average time between assessment by primary and secondary graders was 26 weeks. In contrast, the time between secondary and tertiary graders was 8 weeks.

Conclusion

Further analysis is required to ascertain whether these findings highlight deficiencies in current screening practices, reflect an impact of certain patient characteristics (eg severity of DR, image quality, presence of media opacity) on grading discrepancies or, are indicative of broader systemic factors. Implementing personalised risk-based screening could potentially mitigate the increasing burden of DR on healthcare services; however, it is crucial to approach this with caution to prevent the exacerbation of existing health inequalities. Additionally, the introduction of an electronic system that allows all graders to access the same images may facilitate enhanced monitoring of program reliability and the assessment of competency among graders at different levels.