Content Validation of an Algorithm for the Assessment, Management and Monitoring of Drug-Induced QTc Prolongation in the Psychiatric Population

Introduction

People with serious mental illness (SMI), that is, those who experience psychotic disorders such as schizophrenia and mood disorders such as bipolar and major depression, face a heightened risk for overall mortality. Compared to the general population, the mortality rate of people with SMI has been reported to be two to three times higher.¹ Numerous factors have been implicated to the shorter life expectancy of people with SMI, with cardiovascular diseases accounting for approximately 40 to 50% of premature deaths.² While the majority of sudden cardiac deaths (SCDs) in the psychiatric population are due to ischaemic heart disease and associated risk factors, about 10% are unexplained and are thought to be due to cardiac arrhythmias.^3,4

Because of their potential for prolonging the heart rate corrected QT (QTc) interval, psychotropic medications have been implicated in the increased risk of SCDs among psychiatric patients.^5,6 Besides psychotropics, other commonly used medications in people affected by SMI have also been implicated in prolonging the QTc interval, such as antiarrhythmic medications, antibiotics, antifungals and antiemetics, some of which were removed from the market because they were associated with torsades de pointes (TdP), a life-threatening polymorphic ventricular tachyarrhythmia, which is also linked to SCD.⁷

Studies have shown that people diagnosed with a SMI are often on polypharmacy making them more vulnerable to QTc-prolonging drug–drug interactions.^8–10 Furthermore, because of an ever-increasing number of medications available, clinicians may face difficulties on how to assess, manage, monitor and refer patients at risk of QTc prolongation. Poor access to health care facilities may also increase the risk of adverse outcomes derived from the use of QT prolonging drug combinations, further contributing to the high premature mortality observed in the SMI population.⁸

Although several guidelines are available to assist clinicians on how to assess, manage and monitor patients at risk of drug-induced QTc interval (QTcI) prolongation, only a few have been developed exclusively to guide mental health practitioners.^11–16 Nevertheless, it has been reported that identifying individuals that may be at risk of experiencing QTcI prolongation is challenging, especially for mental health clinicians, and particularly at the time of prescribing.^17,18 To partially address these challenges, a stepped-based algorithm was developed (referred to as the QTcI Prolongation Algorithm).¹⁹ The overall purpose in the creation of the QTcI Prolongation Algorithm was to help mental health clinicians access reliable information about QTcI-prolonging medications, to assess and document the overall risk for someone to experience drug-induced QTcI prolongation, and to provide safety and monitoring recommendations when medications known to increase the QTcI are prescribed.

Despite the robustness of the literature review on which the algorithm was built, establishing content validity is central to ensure it is scientifically sound before it is recommended for wider utilization.²⁰ As such, the main objective of this study was to determine the content validity of the QTcI Prolongation Algorithm from a panel of subject matter experts.

Methods

Most currently available QTcI prolongation guidelines recommend consultation with cardiology at some critical stages in the decision-making process,^{11,15,16,21–23} particularly when assessing individuals at risk of QTcI prolongation or when interpreting electrocardiograms, therefore the expert panel chosen consisted of cardiologists. The study included a qualitative phase using semi-structured interviews to gather information regarding the cardiologists’ approach in assessing risk of drug-induced QTcI prolongation. Following the interview, participants were provided with a copy of the QTcI Prolongation Algorithm (Appendix 1) and a brief orientation describing the steps it entails to assess the risk for QTcI prolongation. Participants were then requested to complete an online survey that included quantitative and qualitative components. Figure 1 provides a schematic representation of the study phases and methodology.

Figure 1 Schematic representation of the study design.

Study Participants and Sampling

Cardiologists from Qatar and the United Kingdom (UK) were enrolled using purposive and snowball sampling techniques. Cardiologists working in a variety of health centers in Qatar and in the UK offering specialized cardiac care, and able to speak and read English were eligible for participation. Recruitment emails were sent out to the potential participants. A participant information sheet along with a consent form was circulated electronically to all potential participants to read and understand before agreeing to participate.

Data Collection

Face-to-face semi-structured interviews lasting between 30–40 minutes were conducted with participants in person, if available, or via Skype voice/telephone calls. A topic guide for conducting the interviews was used (Appendix 2). The guide was developed by identifying key areas of enquiry based on a thorough literature review and was further pilot-tested and amended by the research team. The telephone interviews were recorded and transcribed immediately after each interview, and the in-person interviews were hand-written by the researchers.

The survey containing quantitative and qualitative components was administered via SurveyMonkey^® (Appendix 3). A 4-point Likert scale to rate participants’ opinion of each decision statement/step of the QTcI Prolongation Algorithm was used as follows: 1 = not relevant/appropriate, 2 = unable to assess relevance/appropriateness, 3 = relevant/appropriate, but needs minor alteration, and 4 = very relevant and appropriate. In addition, open-ended questions related to participants’ perceptions of specific decision points and the overall process described in the stepped-based algorithm were included.

Data Analysis

Content analysis of the cardiologists’ interview transcripts was undertaken by each researcher independently. A coding frame was developed through an iterative process. After completing the independent analysis, the researchers worked together to discuss any differences amongst their coding and come to a consensus on derived themes and subthemes.

Survey data gathered through SurveyMonkey^® was exported into an Excel datasheet for analysis. Mean scores for the appropriateness, reliability of the references, and safety of each decision step in the algorithm were used to calculate the content validity index (CVI) for these three attributes. Item-CVI (I-CVI) was used in order to assess the validity of the individual items in the algorithm. It was calculated by dividing the number of experts (cardiologists) who rated the individual steps with Likert scores of 3 or above, by the total number of experts. The cutoff point suggested for 5 or more experts was 0.78.²⁴ If the I-CVI of any of the items was less than 0.5, it was an indication of rejection. Average CVI (Ave-CVI), which assesses the validity of the instrument, was also determined for each attribute and was calculated by adding the I-CVI score of each item and dividing it by the number of items. Ave-CVIs ≥ 0.9 have excellent content validity.²⁵ Although CVI is extensively used to estimate content validity, it does not consider the possibility of inflated values. Hence, a modified kappa (k*) statistical measure was used to reduce the possibility of this risk, due to its advantage of measuring the agreement beyond the chance.²⁶ It was calculated by subtracting the chance of agreement between raters that the items are relevant (Pc; probability of random correlation coefficient) from I-CVI; and then dividing the resulting number over the maximum possible agreement beyond chance.²⁷ For this study, a cut-off point of 0.78 was used to reflect excellent content validity.

Demographic information and participant characteristics were presented as frequencies and were used in correlation tests, where applicable. Participants’ open-ended comments were analyzed using word clouding, a method used to analyze textual data by visualization. This method is easy to use and reduces the risk of bias. In word clouding, font sizes reflect how frequent words are used. As such, the words most frequently used were presented in the largest font size. The font sizes of the remainder of the words were adjusted and calculated automatically by a free software used for creating word clouds, which can be accessed at the website: https://www.wordclouds.com/.

During the word cloud analysis, efforts were made to avoid altering the participant’s own words. However, in certain instances it was necessary to merge terms that had similar meanings.

Results

As presented in Figure 1, a total of 54 cardiologists were invited to participate, of whom 21 agreed, but only 17 were interviewed (15 males and 2 females). Of those interviewed, only 12 completed the post-orientation survey (10 males and 2 females). Their median age was 43.5 years, and the majority graduated between 1990 and 2000. Out of the 12 cardiologists who completed the survey, 11 were in Qatar and 1 was from the UK.

Cardiologist Interviews

As outlined in Table 1 and described below, four major themes emerged from the interviews in regards to cardiologist assessment of drug-induced QTc prolongation:

Table 1 Emerging Themes from the Interviews

Theme 1: Reliance on Electrocardiogram (ECG) Readings

This theme focused on the extent of cardiologists’ reliance on ECG for managing patients on QTcI prolonging medications. Several cardiologists expressed that treatment decisions are made after assessing ECG findings, considering other patient-specific factors such as medical and family history, medication history, physical examination, clinical presentation and laboratory data. Cardiologists indicated that baseline QTcIs are obtained either manually (using the equation: QTcI=QT/√RR), or automatically using the reading calculated by the ECG machine. The cutoff for considering a QTcI as prolonged varied among cardiologists and ranged from intervals ≥ 460–500 milliseconds (ms). Some cardiologists also indicated that there are age and gender differences when considering QTcI cutoff points; for example, in adult females QTcIs between 460–480ms are considered normal, while for adult males QTcIs should not exceed 470ms and for children before puberty, it should not exceed 460ms. Cardiologists also emphasized the importance of obtaining a baseline ECG before initiating a QTc-prolonging medication. Some indicated they re-measure the QTcI four to eight hours after starting the QTc-prolonging medication to ensure tolerability. They also indicated that an interval of 500ms or above is considered a severe QTc-prolongation.

Theme 2: Clinically Guided Prescribing for QTc-Prolonging Medications

Cardiologists indicated that when prescribing medications, they mostly consider the patient’s characteristics as well as the clinical and medication history to avoid any potential drug-drug or drug-disease interactions that can increase the risk of QTcI prolongation, either directly (when prescribing a known medication that can cause QTcI prolongation such as haloperidol) or indirectly (when prescribing a medication that can predispose the patient to experience a QTcI prolongation event, such as diuretic-induced electrolyte imbalances). Cardiologists also highlighted the use of a variety of drug-information resources like Lexicomp, UpToDate, British National Formulary (BNF) and Medscape, but mostly for information purposes rather than for the management of QTcI prolongation. However, most cardiologists were largely unaware of information resources specific to drug-induced QTcI prolongation such as CredibleMeds^®. Some cardiologists indicated that prescribing was also guided by the assistance of clinical pharmacists.

Theme 3: Assessment of QTcI-Prolongation

Cardiologists described a variety of assessments that are utilized for assessing the risk of QTcI prolongation when prescribing medications known to prolong the QTcI, including the patient’s family history of QTc prolongation or SCD, chronic or congenital heart disease, kidney and liver dysfunction. Some also indicated that the patient’s genotype and phenotype should be taken into consideration. Cardiologists further described how they utilize physical examinations to identify signs and symptoms of QTcI prolongation, such as dizziness, loss of consciousness, syncope, bradycardia, palpitations, and arrhythmias such as ventricular tachycardia or ventricular fibrillation. Cardiologists also highlighted the importance of monitoring electrolyte disturbances, particularly serum levels of potassium, sodium, magnesium and calcium as they can contribute to the development of drug-induced QTcI prolongation.

Theme 4: Limited Availability of Protocols

Cardiologists commented that in practice there is limited availability of guidelines and protocols for the assessment and management of QTcI prolongation. There was also an overall agreement amongst cardiologists that assessment of risk is mostly based on ECG findings and clinical presentation, rather than on the use of validated risk scoring tools. There was an overall lack of awareness of a specific tool for assessing or quantifying the risk of a specific patient for experiencing QTcI prolongation. Some indicated that they did not use, or did not have time to undertake risk scoring in practice. A few mentioned the use of specific hospital-based ECG protocols or referred to published guidelines.

Quantitative Assessment of the QTcI Prolongation Algorithm

Overall, the scores indicate the steps in the algorithm to be mostly very reliable or reliable requiring minor alterations. Highest reliability scores were given to the references used to support the decision steps in the algorithm.

Appropriateness of the QTcI Prolongation Algorithm Steps

As summarized in Table 2 and illustrated in Figure 2A, the average mean scores for the appropriateness of each step of the algorithm as evaluated by the cardiologists ranged from 3.08 to 3.67. The appropriateness decision statements that had the highest overall average score (3.67) were in relation to assessing the need for ECG monitoring based on the QTc risk score and to avoiding therapy and considering cardiac consultation if ECG readings indicate QTcI ≥ 500 ms. The I-CVI and the k* for the appropriateness of the different decision statements/steps ranged from 0.83 to 1. No items to assess the appropriateness of the QTcI Prolongation Algorithm steps were rejected (ie had an I-CVI <0.5). The Ave-CVI for the appropriateness of the algorithm steps was 0.95.

Table 2 Evaluation of the Appropriateness of Each Step in the QTcI Prolongation Algorithm

Figure 2 Reliability CVI scores of the QTcI Prolongation Algorithm. (A) presents the mean score of the appropriateness rating for each step of the algorithm in addition to the I-CVI. (B) presents the mean score for the safety of each step of the algorithm in addition to the I-CVI. (C) presents the mean score of the reliability of the references used in each step of the algorithm in addition to the I-CVI. Abbreviations: CVI, Content validity index; QTcI, QTc interval prolongation; I-CVI, item level content validity index.

Safety of the QTcI Prolongation Algorithm Steps

As summarized in Table 3, and illustrated in Figure 2B, cardiologists rated the safety of the decision steps in the algorithm with mean scores ranging from 3.08 to 3.58. The safety decision statement that had the highest overall average score (3.58) was in relation to avoiding therapy and considering cardiac consultation if ECG readings indicate QTcI ≥500 ms. Values of both the I-CVI and the k* for the different safety-related decision statements/steps ranged from 0.83 to 1. No items to assess the safety of the QTcI Prolongation Algorithm steps were rejected (i.e. had an I-CVI <0.5). The Ave-CVI for the safety of the algorithm steps was 0.92.

Table 3 Evaluation of the Safety of Each Step in the QTcI Prolongation Algorithm

Reliability of the References Used in the QTcI Prolongation Algorithm Steps

As summarized in Table 4, and illustrated in Figure 2C, cardiologists rated the reliability of the references used in the various steps of the algorithm with mean scores ranging from 3.17 to 3.75. The reference reliability decision statement that had the highest overall average score (3.75) was in relation to avoiding therapy and considering cardiac consultation if ECG readings indicate QTcI ≥ 500 ms. The I-CVI and the k* for the reliability of the references’ decision statements/steps ranged from 0.83 to 1. No items to assess the reliability of the references of the QTcI Prolongation Algorithm steps were rejected (ie had an I-CVI <0.5). The Ave-CVI for the reliability of the references used in the algorithm was 0.94.

Table 4 Evaluation of the Reliability of the References Used in Each Step in the QTcI Prolongation Algorithm

Qualitative Assessment of the QTcI Prolongation Algorithm

Figure 3 provides the Word-cloud representation of the word frequency with greatest prominence in the open-ended responses from cardiologists when asked to describe their overall assessment of the QTc Prolongation Algorithm. The words that appeared more frequently included: “should be used by pharmacists” and “cumbersome/complicated”.

Figure 3 Word-cloud representation of survey open-ended responses.

Discussion

The emerging themes that resulted from the cardiologists’ interviews provided insight on how experts deal with QTcI prolongation in clinical practice. The first theme, reliance on ECG readings, was indicative that cardiologists consider this measurement to be the starting point when making decisions related to prescribing medications known to increase the QTcI. Although this is reasonable, as ECGs have been the traditional and most advocated method for assessing QTcI prolongation, it may also be impractical to perform an ECG every time a QTcI-prolonging medication is prescribed.²¹ Additionally, QTcI readings that are measured through automatic standard 12-lead ECG, which is what is most commonly used in clinical practice, have been reported to be inaccurate. Even the heart rate correction formulas that are used, such as the one developed by Bazett, can lead to over or under-corrections. This is because they do not completely eliminate the dependence of the QTcI on heart rate and are based on assumptions that the QT/heart rate relationship remains consistent between different individuals. A QT nomogram described by Isbister and Page²⁸, addresses this issue and has been reported to be more specific than the ones using the Bazett’s formula. Furthermore, a survey aimed to characterize trends in ECG monitoring practices among psychiatry residency faculty members, indicated that outpatient providers were less likely to order an ECG when prescribing antipsychotics.²⁹ Authors attributed this finding to financial issues and lack of access to ECG resources in outpatient settings. The American Heart Association’s updated guidelines for ECG monitoring in hospital settings provides recommendations on which patient populations are most likely to benefit from ECG-based QTc monitoring while hospitalized.³⁰ These controversies regarding the use of ECGs should be examined and considered when revising the QTcI Prolongation Algorithm.

The second theme, clinically guided prescribing of QTc prolonging medications, was also indicative that cardiologists follow other approaches that can support ECG-based assessments, including the patient’s clinical/medication histories, and the use of drug information resources to assess for drug interactions. However, cardiologists were largely unaware of drug information resources specific to drug-induced QTcI prolongation such as CredibleMeds^® which is considered by many as the most reliable database on QTcI prolonging drugs.^6,31 This resource has developed a risk stratification process to categorize drugs based on their relative potential for QTc prolongation and/or life-threatening ventricular arrhythmias. Cardiologists also indicated to rely on clinical pharmacists as a valuable resource for the assessment of drug-induced QTcI prolongation, a role that has been supported in the literature.^32–34 Consultation about drug factors, such as dosing, route of administration, renal elimination, and drug interactions are important considerations in which pharmacist advice may be particularly useful at the time of prescribing.

The third theme in regards to the clinical assessment for QTc-prolongation, cardiologists identified the risk factors which are routinely assessed at the time of prescribing medications with the potential to increase the QTcI, such as history of QTcI prolongation, physical signs and symptoms, and others such as renal impairment and electrolyte imbalances. However, similar to the results of the survey among psychiatry residency faculty members described above,²⁹ cardiologists did not mention well known predisposing etiologies of QTc prolongation such as family history of long QT syndromes or sudden death, or personal history of syncope.³⁵ As the QTcI Prolongation Algorithm includes a specific question on history of long QT syndromes, it may help in addressing this commonly overlooked risk in the assessment process.

The fourth theme, limited availability of protocols, is possibly related to the limited routine use of risk scoring tools in practice. Although there are several QTcI prolongation risk scoring tools published in the literature,^14,36–38 cardiologists in our study were either not aware or argued they had limited time to utilize them in practice. Some studies have shown that risk stratification and implementation of a standardized QTcI monitoring protocol can provide relevant clinical guidance in treatment decisions and result in positive patient outcomes.^29,37,39

The findings of the second part of this study, which assessed the QTcI Prolongation Algorithm’s content validity, including the overall appropriateness, safety and references used in each decision point, supported its reliability. Values of the I-CVI, Ave-CVI and k* reflect a positive inter-rater reliability as they were all above the suggested cut-off points. There were two steps that had the lowest average rating for the overall appropriateness and reliability of the references used in each of the decision steps of the algorithm. These steps were: “Assessing the drug using CredibleMeds^®” and “Recommending therapy if the risk score ≤ 7 points”. It is possible that these steps rated the lowest due to most cardiologists’ being unaware of CredibleMeds^® and not guiding prescribing decisions based on risk stratification scoring. Interestingly, both of these issues emerged as themes from the cardiologists’ interviews. Although there are various guidelines available, our findings indicate that they may not provide a practical approach to risk assessment or that there is limited awareness or availability of risk stratification protocols among health care providers. Adopting a care map such as the QTcI Prolongation Algorithm may help fill these gaps in practice.

In regards to the overall appropriateness and safety of the algorithm, the highest average rating was for “If ECG shows QTc ≥ 500 ms, avoid therapy and consider cardiac consultation”. This finding was expected as guidelines recommend cardiologist consultation as an important safety step before prescribing medications with the potential to increase the QTcI.^{11,15,16,21–23} However, it is important to note that thecut-off point for when to avoid QTc-prolonging medication varies in the literature. Certain guidelines identify prolongation with QTc values ≥ 450 ms in males and ≥ 460 ms in females with no universally agreed upon method for measuring.³¹ Nonetheless, a QTc ≥ 500 ms, is commonly considered as severely prolonged and has been linked with a two to three-fold increase in risk for TdP.⁴⁰ As such, the cut-off point used in the QTcI Prolongation Algorithm appears to be supported by the current literature.

Lastly, the word-cloud analysis of the cardiologists’ overall opinion on the QTcI Prolongation Algorithm indicated that its adoption by health care providers could improve decision making at the time of prescribing. However, it was perceived as possibly being time consuming. Utilization of electronic decision support systems can facilitate the assessment and monitoring process, by incorporating alerts when medications with known or associated risk of prolonging the QTcI are prescribed. The algorithm can be used at this point by pharmacists who can notify physicians of their assessment and recommendation. Another possibility is to integrate the algorithm as part of an electronic prescribing process. Having the algorithm at the point of prescribing can facilitate the flow of the steps by pulling existing patient information, and reduce the time needed to reach a recommendation. Additional content validation of the algorithm will be conducted with mental health clinicians as a second phase to this study. Feedback and recommendations gathered in both content validation phases will be considered and integrated into a revised version of the algorithm prior to integration into an electronic prescribing process.

Study Limitations

Participants were initially recruited from a list of cardiologists who attended a conference, followed by snowball sampling. Although this method helped in capturing interested candidates, representativeness of the sample is not guaranteed, as the true distribution of the population and of the sample is unknown. Sampling bias is also possible when using this sampling technique. Initial subjects tend to nominate people that they know well. Furthermore, the majority of the cardiologists were based in Qatar. This can impact the generalizability of the results to clinical practice outside of Qatar. To partially address this, we purposively invited cardiologists in the UK to participate and results obtained were included. Further to the limitations described above, although the CVI is extensively used to estimate content validity by researchers, this index does not consider the possibility of inflated values because of the chance agreement. Thus, we tried to address this issue by using the I-CVI and k*. These can measure content validity and inter-rater agreement, respectively, and are helpful in improving the representativeness and supporting the validity of the results of small sample sizes.^27,41 Another important limitation in the data collection process was the emergence of the COVID-19 pandemic, which limited the participation of cardiologists in the study. This limitation however provides further support for empowering other health care professionals, such as pharmacists, in assessing the risk of QTc-prolongation at the time of prescribing, particularly in settings with limited access to cardiology consultations, such as mental health services.

Conclusion

The algorithm to assess the risk of drug-induced QTcI prolongation in the psychiatric population, which was developed based on a systematic review of the literature, has shown a relatively strong content validity. Although cardiologists rely mostly on baseline ECG readings to assess the risk of drug-induced QTc prolongation, their evaluation of the algorithm is supportive towards its use to assist in the decision making process in the absence of cardiology consultation. The algorithm also has the potential to address current practice gaps and prove to be a useful tool for risk assessment in psychiatric settings or in community mental health services where cardiology consultations are limited or hard to reach. Future implementation of the algorithm using decision support systems that can be integrated into patient assessment and health management systems in mental health should be considered.