Development of training-related health care software by a team of clinical educators: their experience, from conception to piloting

Abbreviation:

ES = Educational Supervisor

Introduction

Trainee Tracker (TT) is a semiautomated, online information technology (IT) system which assists clinical educators at all levels in their oversight of trainee progression through postgraduate curricula.

A major challenge for training program directors (TPDs) is keeping up-to-date with the progress of their many trainees through specialty curricula. The radiology curriculum comprises numerous milestones, of which the most notable is its nine collegiate exams, culminating in a Certificate of Completion of Training (CCT). The provisional CCT date for each trainee is estimated in advance by their subspecialty Royal College. By traditional methods, relating each trainee’s achieved milestones to deanery-prescribed timescales, and thereby identifying “trainees in difficulty” is labor-intensive and confusing. Calculating the very important CCT date is even more so. This is especially true for less than full-time trainees (LTFTs), including the trainees employed part-time and Academic Clinical Fellows. In their case, the standard 5-year training period on which the CCT date is usually based must be recalculated manually. At the LTFTs’ annual deanery appraisals, targets for each curricular milestone are often loosely estimated, making the LTFTs in difficulty harder to identify and assist.

The developers of TT felt that an IT system capable of automating these processes would be desirable and achievable. Mindful of the major obstacles encountered in high-profile National Health Service (NHS) IT ventures of recent times, TT’s development team aimed from the outset to take a contemporary, “agile” approach to the project. This would involve close, active collaboration between a small group of committed clinicians and IT experts.

This article describes the development team’s experience in producing and piloting this new system and outlines TT’s principal unique functionality.

Development process

The project was initiated and managed by a TPD of a large UK radiology training scheme. The concept first arose during informal, chance discussion between the TPD and a contact in the IT industry. They identified the system’s few core requirements as follows:

• The system should be accessible online (Figure 1).
• To minimize user-training time and costs, the system should be simple and intuitive.
• The system should map all trainees on a training program against curriculum-defined milestones (Figure 2).
• To alleviate complex manual recalculations and potential human error, the system should allow for individual trainee factors such as LTFT, career breaks, and training extensions and recalculate milestone target dates accordingly.
• To highlight potential “trainees in difficulty” as early as possible, the system should automatically alert trainees and educators of approaching and elapsed target dates by email.
• To allow easy review of overall progression of individual trainees and entire trainee bodies, the system should indicate approaching or overdue milestones with a color-coded red/amber/green rating on outcome tables with filter functions.

Figure 1 “Main menu” screen of the Trainee Tracker system. Note: The system’s functions are displayed in the main menu as “tabs”, by which the user may design milestones and training plans as well as track trainees’ progress.

Figure 2 “Track all trainees” screen of the Trainee Tracker system. Notes: Test data are shown, including fictitious names. The “track all trainees” screen displays all trainees on the system, enabling the user to gain an overview of all trainees’ progress by scrolling through the table. The training tasks showing due dates in red indicate that the task is overdue. Dr Derek Humphres is part of the 2013 intake, and is up-to-date with his training milestones. Dr Andy Bezer is from the 2011 intake. His first task is highlighted in red, indicating that this milestone is overdue. An automated email would have been sent to him and his educational supervisor accordingly to prompt a training meeting in order to help him. Abbreviations: CCT, Certificate of Completion of Training; NWL, North West London; FRCR, Fellowship of the Royal College of Radiologists.

Inevitably, the project’s first requirement was funding. A presentation of how the system would look was first given at a regional training meeting. Potentially addressing a need familiar to several other educators, the idea was well received. With this feedback and the help of contacts in the medical education arena, modest initial investment was obtained, mainly from the deanery and local training funds. Another consultant radiologist with a specialist interest in medical education joined the development team.

The Postgraduate Education Department of the TPD’s employing NHS Trust also agreed to invest in the project. The intellectual property was the subject of some discussion at the time; but ultimately, their contribution would allow the Trust the right to apply the system locally in any of its departments, but allowed the IT specialist retention of intellectual property for the purposes of later expansion.

The IT expert employed the services of an external software development company that was able to construct a functioning prototype system within budget. Once this online system was in place, a senior radiology trainee – also with an interest in teaching and training – was recruited to the development team to assist with system testing. Through iterative, scenario-based testing, the trainee and the IT expert were able to optimize the system’s target date recalculation function and other automated processes. At that time, the core development team (by then amounting to three clinical radiologists and an IT expert) was in regular contact, corresponding most days, and meeting around twice a month. It was through this close collaboration that the system could be fine-tuned according to the needs of its end users. Glitches such as occasional slow processing/target date recalculation and lack of recourse for purging obsolete data were ironed out. The functionality was simplified as much as possible. To avoid bloating the system and digression from its original purpose, the development team were consciously disciplined in abandoning any functionality considered superfluous.

Understandably, due to their funding contribution, the employing NHS Trust wished to oversee the project. The Trust was obliged to ensure due process in both TT’s development and contracting. Ordinarily, when a new IT system is commissioned, the customer produces a “systems specifications” document, which – due to the informal approach – in TT’s case, remained wanting at the time.

In this arena, the customer is also obliged to put the contract out to tender and may only proceed if at least two contractors bid. In this case, a second potential vendor did submit a tender, which the Trust considered less favorable and more expensive than TT.

The Trust also arranged for a third-party IT consultant to examine the system. The Trust and the IT consultant required a formal “test plan” and “test document”. A suitable test document was prepared by the development team, based on the System Specifications document. With this, the third-party IT consultant produced a report, including some constructive feedback. Ultimately, it was agreed with the Trust that formal testing could be done by the clinical radiologists in the development team (mainly the radiology trainee), since they would be the end users. This was important because – unlike the third-party IT consultant – the end users, viewing the system as noncritical, could apply a degree of flexibility to the test document. It was reasonable to “pass” the test items, if the system did as intended, even if the test criterion was not satisfied to the letter.

The testing process involved stressing each individual function listed in the system’s specifications document. Most items were straightforward, such as registering a new system user, logging in, password changes, and general navigation of the system. More laborious was stressing its target date recalculation and automated email alert functions. To do this, test educator and trainee information were created. Complex work patterns and training needs would be invented for the test trainees; for example, an LTFT, who worked ten clinical sessions a week for their first year, then took maternity leave for 9 months, and thereafter worked six clinical sessions a month. TT’s predicted CCT date and target curricular milestones for this trainee were then compared with the manual calculations of the development team. A few revisions to TT’s programming were needed to achieve acceptable accuracy (ie, within a week of the manually calculated target dates). Test scenarios were also created to ensure that the system correctly issued automated email alerts to given email addresses when the test milestones became overdue.

Once the system had been “passed”, a “go-live” date was set. Over the past 18 months, TT has been piloted on a large radiology training program in the UK. The project and piloting were endorsed by the local departmental and institutional ethics committees. All 30 trainees (including seven LTFTs) from years 1 to 5 were provided with information about the system and its purpose. Every trainee gave written consent to participate in the pilot.

All 20 educational supervisors (ESs) on the training scheme also agreed to participate and undergo individual user training. User training typically took 5–10 minutes each, after which it should take a “system user” approximately 1 minute to make basic data updates. Following their user training, each ES was asked for verbal feedback on the system, in terms of its layout, usability, and overall value compared with existing online training portfolios. Overall feedback from each of the users was positive. Although some had initial reservations over possible time commitments, the ESs appreciated the potential labor-saving advantages of the system and all found it easy to learn and use. No complaints or criticisms were received about the system from the trainees or ESs during the pilot.

The system was implemented at the training scheme’s annual local trainee appraisal process. Using TT, local appraisers could more easily establish each trainee’s progression over the past year, and thereby make fair and accurate outcome recommendations to the regional appraisal panel. As an adjunct to e-portfolio, TT will be trialed at the regional radiology trainee appraisal process too.

Over the course of this pilot, eight trainees, who were previously making satisfactory curricular progression, initially fell behind with their training milestones and had received either an amber or red alert. Past trainees who struggled early in their training have tended to struggle throughout; however, in this relatively short period, four of those eight trainees are again making satisfactory progression. This relatively strong recovery among these trainees partly reflects the early impact of TT. Because TT is able to identify trainees in difficulty more quickly and informs them and their supervisors automatically, their training can be adjusted and refocused according to need at an earlier stage.

TT has recently been incorporated in three other radiology training programs, an epidemiology training scheme, and (in a less medical format) by a major training provider for health care engineering and estate management in the UK. TT is currently profit making.

The system was demonstrated at the annual Royal College of Radiologists meeting of TPDs, who acknowledged the utility, time-saving capacity, and increased accuracy the system could allow. The Royal College of Radiologists and other colleges do have existing paper and electronic training portfolios, and endorsement of TT by national commissioners of training would be needed before adoption by postgraduate colleges could be considered.

Health Education England – the national commissioner of postgraduate medical education – has been holding preliminary talks prior to tendering for bespoke electronic portfolio technology. Ultimately, TT’s automated date recalculation and email alert functionality could be incorporated into existing online portfolios, depending on requirements and preference.

Discussion

It is interesting to reflect on the development process of this system in the context of other larger health care IT projects, including e-portfolio^1–5 and the attempted £12.7 billion national NHS IT program. Traditionally, health care IT projects involve a “waterfall” approach, where the customer produces at the outset an exhaustive requirements document, covering all conceivably applicable functionality. By leaving the software designer with such instructions and limited insight into the customer’s practical needs, the product would be at greater risk of being overcomplicated with redundant functionality.⁶

In contrast, the development of TT aimed to take an “agile” approach, best achieved by smaller teams of customers and designers, in continuous, adaptable collaboration.⁶ Intended to be maximally intuitive and labor saving, the content and layout of TT reflect a minimalist approach, incorporating only essential functionality. This strategy reflects more contemporary NHS approaches, as exemplified by programs such as “Code4Health” (an initiative supporting “IT fellowships” for doctors toward closer and more insightful teamwork with software designers).⁷

A close working relationship between user and designer is advantageous in several ways. Unanticipated requirements can be realized more quickly, and superfluous ones removed. Having adaptable requirements is a strength, as predicting every requirement at the outset is not always possible. This approach offered a number of other advantages. The development team was personally invested in the success of the project. Much of the development and assistance was achieved through personal contacts in the IT industry and medical education arena and relied on goodwill. Costs were, therefore, kept to a minimum, amounting to £27k for the initial development and pilot. The project was user driven, with a clear vision of the end product and its purpose. Every effort was made in avoiding superfluous functionality, which the development team considered a major flaw in other IT systems. A degree of overlap between the customer and the supplier in this case was at times convenient; for example, it allowed greater flexibility in system testing. Customer acceptance was also inherent, as the development team had personal insight into the intended users’ environment and IT skills.

Obstacles to this lean approach were identified, but proved manageable in practice. The budget was limited. The process was labor-intensive and unpaid for the clinical radiologists involved. Modest, calculated risks were recognized. We were wary of exhausting the budget, resulting in wasted money. There was also a (small) chance of losing the tendering bid. Large institutions have established IT governance and tendering rules, which can make interacting with such an organization challenge the advantages of this small-scale approach. Ultimately, the informal approach of the development team allowed most of the overheads associated with a traditional IT project in a national organization to be dispensed with, resulting in a commensurate reduction of software implementation cost and timescale.

The major advantage of TT over other electronic and online training portfolios is its automated functions. It is able to recalculate target completion dates for a trainee’s milestones according to that trainee’s individual circumstances, for example, part-time/academic trainees or trainees in difficulty who have had their CCT date and other training milestones deferred. The system is able to automatically track trainees in real time. When a particular milestone target date is approaching, the system sends an automated email (an “amber alert”) to both the trainee and their ES in order to remind them and to prompt a meeting if appropriate. Once the milestone is completed, the trainee or ES may update TT accordingly, and the milestone will be recorded as “completed” and highlighted in green on the system. If the target completion date has elapsed without completion of the respective milestone, the system sends an automated email to the trainee and their ES accordingly to prompt a further meeting to address that individual’s ongoing training needs. Although there are a variety of existing electronic milestone tracking tools, to our knowledge, none possess this particular automated, labor-saving functionality.

Conclusion

This article describes the experience of a small team of clinical educators in the development of medical training-related IT software. Through close collaboration with IT experts and other educators, an “agile” approach was taken to the development process. A simple, effective, and popular product, born of specific necessity, was achieved at low cost. The system is already commercially viable and is ready for trialing on a larger scale. The close involvement of clinical trainers in this project proved particularly advantageous and should stand as a positive example of health care IT software design.

Further planned modifications to TT include statistical data extraction capabilities, providing additional objective external appraisal of training schemes. It would allow extraction of other useful training data, such as numbers of existing LTFTs and data applicable to appraisal of ESs. It could help predict the number of trainees required in recruitment rounds for any given specialty. Ultimately, there is potential application to other specialties and careers in any country.

Acknowledgments

Rachel Abraham and London Deanery Shared Services are acknowledged for funding and business advice. Total cost to date of £27k was shared between a local educational budget (Imperial College Healthcare NHS Trust), local deanery (London Deanery Shared Services), and regional lead provider (North West London).

Dr Derfel ap Dafydd is a Locum Consultant Radiologist at the Royal Marsden Hospital. His interests include oncological imaging, teaching, and training. He was previously the Royal College of Radiologists trainee representative for the undergraduate radiology curriculum at Imperial Medical School and founder of the undergraduate Radiology Society there.

Dr Ruth Williamson is a Consultant Radiologist at Bournemouth hospital. Her interests include breast and gynecological imaging, teaching, and training. She was former lead in postgraduate radiology education in North West London.

Philip Blunt is the owner of Savernake IT, a specialist in software design, development, and implementation for NHS hospitals.

Dr Dominic Blunt is a Consultant Radiologist at Imperial College Healthcare NHS Trust, with 10 years of experience as training program director. He is currently the Regional Educational Advisor for North West London.

Disclosure

Philip Blunt is a software developer for Savernake Ltd and developed the system. Dominic M Blunt is a Consultant Radiologist, but has no financial ties with the project. The other authors report no conflicts of interest in this work.

References

1.		Hrisos S, Illing JC, Burford BC. Portfolio learning for foundation doctors: early feedback on its use in the clinical workplace. Med Educ. 2008;42(2):214–223.
2.		Johnson G, Barrett J, Jones M, Parry D, Wade W. Feedback from educational supervisors and trainees on the implementation of curricula and the assessment system for core medical training. Clin Med (Lond). 2008;8(5):484–489.
3.		Makris J, Curtis A, Main P, Irish B. Consultants’ attitudes to the assessment of GP specialty trainees during hospital placements. Educ Prim Care. 2010;21(4):236–242.
4.		Tochel C, Haig A, Hesketh A, et al. The effectiveness of portfolios for post-graduate assessment and education: BEME Guide No 12. Med Teach. 2009;31(4):299–318.
5.		Vance G, Williamson A, Frearson R, et al. Evaluation of an established learning portfolio. Clin Teach. 2013;10(1):21–26.
6.		Hoeksma J. Turning doctors into coders. BMJ. 2013;347:f5142.
7.		Code4Health. 2016. Available from: https://code4health.org.