Development and Validation of a Clinical Model for Predicting Delay in Postoperative Transfer Out of the Post-Anesthesia Care Unit: A Retrospective Cohort Study

Introduction

The post-anesthesia care unit (PACU) is a specialized anesthesia care facility for monitoring postoperative resuscitation and treatment.¹ Delay in transferring the patient out of the PACU can result in a range of adverse consequences for the patient, such as increased postoperative complications, prolonged recovery time, and inefficient utilization of healthcare resources.² Given the escalating number of patients undergoing surgical procedures,³ the PACU has become an important link in the continuum of care, responsible for the postoperative monitoring and treatment of patients, thereby under tremendous pressure due to the high patient turnover. However, the use of standardized assessment tools for PACU discharge can improve PACU turnover rates, reduce complications, and ensure postoperative patient safety.⁴ However, delay in patient transfers is a persisting challenge.⁵ Currently, some clinical models for predicting the delayed recovery or prolonged length of stay in PACU among different populations have been reported.^6,7 However, the population of these studies was limited to outpatients or patients with a certain type of surgery, and the sample size was small.

Given this background, we undertook this research on predicting transfer delays from the PACU to provide medical staff with more accurate patient management plans. It also has positive social and economic significance for the effective use of medical resources and the promotion of patient recovery.

Materials and Methods

Study Participants

We collected the data of 1153 and 297 patients who underwent surgery and were treated in the anesthesia recovery room from January 1, 2022, to December 30, 2022, and from March 1, 2023, to May 30, 2023, respectively. Thus, there were a total of 1450 surgical patients in the two data cohorts of the study.

Inclusion criteria: surgical patients who (1) received general anesthesia; (2) received postoperative anesthesia recovery in the PACU; (3) were aged over 18 years old; (4) were treated in the general surgery and thoracic surgery units; (5) provided informed consent for this study; and (6) had complete medical records and follow-up data.

Exclusion criteria: (1) patients who underwent emergency surgery; (2) patients whose tracheal intubation was not removed when they entered the PACU room; (3) those who refused consent to participate in this study; (4) inaccurate or missing medical records; (5) delay in transfer was due to insufficient staff.

Observation Indicators

We collected patient information and anesthesia nursing data from the hospital’s medical record data system and anesthesia nursing system, respectively. We accessed the medical record data system for the following patient data: patient inpatient number (ID), age (years), height (cm), body weight (kg), routine investigations such as blood biochemical indicators, and comorbid diseases.

We gathered the following details from the anesthesia care system: results of the patient’s blood gas analysis during anesthesia, duration of anesthesia, intraoperative bleeding, intraoperative BIS (Bispectral Index) value, time of admission to the PACU and time of transfer out of the PACU (which were automatically recorded by the anesthesia care system), and body temperatures at the time of admission to the PACU and at the time of transfer out of the PACU (which were manually measured and then recorded by the anesthesia recovery system).

Two senior personnel for the anesthesia nursing unit independently assessed the criteria for discharge from the PACU as per the Steward Awakening Score criteria recommended by the “Implementation Rules for the Evaluation Standards for Level III General Hospital” released in China in 2019.⁸ A score > 4 was considered to be the discharge criteria for patient transfer by the designated transfer team, and the duration spent in the PACU since then was recorded. If the duration exceeded 30 minutes, the transfer was considered delayed.

Statistical Analysis

We used R 4.0.2 software for statistics and visualization. We compared clinical characteristics between groups using the Student’s t-distribution or Wilcoxon rank-sum test for continuous variables and the chi-square test or Fisher exact test for categorical variables (as appropriate). We employed LASSO regression to reduce the dimensionality of high-dimensional data and logistic regression, random forest, and eXtreme Gradient Boosting (Xgboost) to screen variables. We evaluated the model’s performance using the area under the ROC curve (AUC) and the calibration curve method. We used Net Reclassification Improvement (NRI) and Integrated Discrimination Improvement (IDI) to evaluate the differences between different models.

The model was converted into a scoring table and divided into scores according to different stage values. We developed a scoring system and verified it in the training, validation, and external data sets. We evaluated the clinical benefit of the nomogram with the help of the Decision Curve Analysis (DCA) and Clinical Impact Curve (CIC). A difference was considered statistically significant if the P value was < 0.05.

Results

Study Cohort

We enrolled a total of 1450 patients, who were divided into data cohorts A and B. Data cohort A included 1153 patients, and we randomly assigned them to a training group and a validation group in a ratio of 3:2. Data cohort B included 297 patients as an external data set. Independent teams collected data from the two cohorts with a time interval of 3 months between their collection periods (Table 1). The data were grouped based on the presence or absence of transfer delay, respectively. We analyzed each indicator between groups in different data sets. The results showed that the overall characteristics of the data in different data sets were similar (Supplementary Material - Table S1).

Table 1 Baseline Indexes

Correlation Between Variables, Collinearity Exploration, and LASSO Regression

In this study, we included a large number of variables for analysis and found that there was a definite correlation between each variable, which may be due to the inclusion of more variables with collinearity. Further exploring the collinearity among various variables, we found that variables such as the patient’s body weight, albumin, creatinine (CRE), platelets (PLTs), height, total protein (TP), activated partial thromboplastin time (APPT), neutrophils, white blood cell (WBC), hypertension, diabetes, alcohol use, age, and HB had a variance inflation factor (VIF) of > 5, indicating that the dataset included a large number of high-dimensional collinearity indicators. Using LASSO regression, a coefficient penalty was applied to each variable through regularization techniques to reduce high-dimensional data. Variables with a coefficient of 0 were identified, and ultimately, the variables for analysis included age, gender, body weight, hypertension, diabetes, alcohol use, HB, albumin, ALT, AST, glucose, duration of anesthesia, maximum BIS, minimum BIS, blood loss, PaCO2, lactic acid, temperature at the time of entry into the PACU, and temperature at the time of leaving the PACU for a follow-up study. Exploring the collinearity among the included variables again, we found variables such as albumin, body weight, hypertension, alcohol use, and diabetes had a VIF of nearly 5, but the overall collinearity improved significantly as compared to earlier (Figure 1).

Figure 1 (A and B) Correlation and clustering analyses show a significant clustering trend of the included variables, indicating the presence of multiple variables pointing to a fixed characteristic. (C) The analysis of the VIF values between the variables included suggests a high degree of covariance between the variables. (D and E) The results of lasso regression and cross-lasso regression analyses show that the number of variables decreased after regularized data processing and stabilized after cross-sampling. (F) After lasso regression analysis, the VIF values of the included variables suggest a significant improvement in the covariance between the variables as compared to the earlier analysis.

Creation of the Model

For creating model 1, we employed univariate logistic regression analysis and area under the receiver operating characteristic (ROC) curve and identified variables with a P value of < 0.05 and an area under the ROC curve > 0.6 in univariate analysis. These included the patient’s age, HB, ALT, AST, glucose, and duration of anesthesia, maximum BIS, minimum BIS, blood loss, PaCO2, HCO, temperature at the time of entry into the PACU, and temperature at the time of leaving the PACU (details are given in Table 2).

Table 2 Univariate and Multivariate Analysis

In the training dataset, model 1 yielded an area under the ROC curve (AUC) of 0.940 (0.922–0.958), and an AUC of 0.929 (0.949–0.969) in the test dataset. The calibration curves also showed better predictive efficacy and stability, with a Dxy of 0.88, a R2 of 0.704, and a Brier of 0.092 in the training dataset, and a Dxy of 0.89, a R2 of 0.731, and a Brier of 0.085 in the test dataset.

Using the Random Forest method, the RF model had an AUC of 0.916 (0.892–0.939) in the training dataset and the AUC was 0.944 (0.922–0.965) in the test dataset. Its predictive performance was slightly lower than that of logistic regression in the training dataset, and both performed similarly in the test data. We calculated the importance of variables in the random forest model and explored whether the model could be simplified. Thus, we developed models 2 and 3 using the top five variables and the top 10 variables, respectively.

The top five variables were body weight, albumin, HB, age, and minimum BIS; the top 10 variables were body weight, albumin, HB, age, minimum BIS, glucose, duration of anesthesia, HCO, ALT, and maximum BIS. The AUC values of the two models in the training dataset were 0.901 (0.878–0.924) and 0.919 (0.898–0.940), respectively. In the test data, the AUC values were 0.923 (0.900–0.947) and 0.922 (0.922–0.962), respectively.

Additionally, when we plotted the calibration curves, models 2 and 3 showed better predictive efficacy and stability. Model 2 had a Dxy of 0.802, an R2 of 0.583, and a Brier of 0.123 in the training dataset, whereas in the test dataset, the Dxy was 0.847, the R2 was 0.645, and the Brier was 0.109. Model 3 yielded a Dxy of 0.838, an R2 of 0.638, and a Brier of 0.11 in the training data set, whereas in the test data set, the Dxy was 0.884, the R2 0.705, and the Brier was 0.093.

Using the Xgboost method to train the model, the AUC was 0.974 (0.963–0.985) in the training dataset, and when we validated the model in the test data, it yielded an AUC of 0.858 (0.821–0.896). We found that the performance of the model built using the Xgboost method performed moderately in the validation set.

We used SHAP to interpret the model for variable importance and included the top five and top 10 importance variables to build models 4 and 5, respectively. The variables for model 4 were the patient’s body weight, minimum BIS, HB, duration of anesthesia, and temperature at the time of leaving the PACU. The variables in model 5 included the patient’s body weight, minimum BIS, HB, duration of anesthesia, temperature at the time of leaving the PACU, glucose, diabetes, PaCO2, ALT, and hypertension. We developed these two models separately for the training and test datasets. The AUC values in the training dataset were 0.941 (0.923–0.959) and 0.912 (0.891–0.934), respectively. In the test dataset, the AUC values were 0.960 (0.943–0.977) and 0.941 (0.920–0.961), respectively.

Model 4 had a Dxy of 0.882, an R2 of 0.706, and a Brier of 0.09 in the training dataset, whereas in the test dataset the Dxy was 0.920, the R2 was 0.77, and the Brier was 0.072. Model 5 showed a Dxy of 0.825, an R2 of 0.618, and a Brier of 0.115 in the training data set, whereas in the test data set, the Dxy was 0.881, the R2 was 0.693, and the Brier was 0.095 (details are given in Figures 2 and 3 and Supplementary Material - Figures S1 and S2).

Figure 2 (A–E) ROC curves for models 1–5 in the training set, respectively; (F–J) calibration curves of models 1–5 in the training set, respectively; (K–O) ROC curves of models 1–5 in the validation set, respectively; (P–T) calibration curves of models 1–5 in the validation set, respectively.

Figure 3 (A and B) ROC curves of the models based on the Random Forest and the Xgboost methods in the training set, respectively; (C and D) ROC curves of the models based on the Random Forest and the Xgboost methods in the validation set, respectively; (E) order of importance of the variables in the model based on Random Forest; (F–H) explanation of the importance of SHAP to the Xgboost variables in the Xgboost method.

Model Verification and Comparison

As detailed above, we created five models using logistic regression and two models using Random Forest and Xgboost, thus consisting of a total of seven models. There were no significant differences between these models in terms of AUC values, but the variables included were not identical. We attempted to further explore ways to simplify the models by extracting the common variables of the models and analyzing trends in the weights of different variables in the models. Therefore, we built a model 6, consisting of variables such as HB, ALT, glucose, duration of anesthesia, and minimum BIS.

When trained in the training dataset, model 6 yielded an AUC of 0.909 (0.887–0.932), and validation in the test data gave an AUC of 0.939 (0.919–0.959). In the train dataset, model 6 had a Dxy of 0.819, an R2 of 0.605, and a Brier of 0.117, and in the test dataset, a Dxy of 0.878, an R2 of 0.689, and a Brier of 0.098. We further examined the differences between model 6 and the other models using NRI and IDI and did not find any significant differences (Figures 4 and 5 and Table 3).

Table 3 NRI and IDI Values Compared Between Model 6 and Models 1, 2, 3, 4, and 5 in the Training and Validation Sets

Figure 4 (A) Variable UpSet plots of different models; (B) plots of the changes in order of importance of the variables among different models; (C and D) ROC and calibration curves of model 6 in the training set, respectively; (E and F) ROC and calibration curves of model 6 in the validation set, respectively.

Figure 5 (A) Differences in NRI and IDI between model 6 and models 1–5 in the training and validation sets; (B–K) comparison of the NRI distribution plots of model 6 and models 1–5 in the training and validation sets, respectively.

Model Simplification and Visualization and DCA and CIC Curves

Given the favorable predictive performance and stability of model 6 and the fact that it incorporated fewer variables, we initially selected this model for further simplification so that it can be effectively used in clinical practice. We tried to convert it into a scoring table and analyzed the relationship between the variables and outcomes in the data.

In this analysis, the stage values were delineated based on the smoothness of the curve. HB was divided into four ranges using the cutoff values of 90, 120, and 140. ALT was divided into three ranges based on the cutoff values of 35 and 60. Glucose was divided into three ranges according to the cutoff values of 5.6 and 9.1. Duration of anesthesia was divided into three areas as per the cutoff values of 90 and 180. The minimum BIS value was divided into three areas using the cutoff values of 30 and 35.

These factors were included in the regression model for training the model in the training data set. The ROC curve was plotted, and we obtained an AUC of 0.865 (0.838–0.891). The AUC was 0.895 (0.868–0.922) in the test data set and 0.899 (0.865–0.934) in the validation data.

To facilitate practical use, we converted these into scores and assigned distinct score values to different stages as follows: HB (g/L) > 140 was assigned 4 points, (120, 140) was assigned 3 points, (90, 120) was assigned 2 points, and ≤ 90 was assigned 1 point. ALT (U/L) > 60 was assigned 3 points, (35, 60) was assigned 2 points, and ≤ 35 was assigned 1 point. Glucose (mmol/L) > 9.1 was assigned 3 points, (5.9, 9.1) was assigned 2 points, and ≤ 5.9 was assigned 1 point. Duration of anesthesia (min) > 180 was assigned 3 points, (90, 180) was assigned 2 points, and ≤ 90 was assigned 1 point. Minimum BIS (score) > 35 was assigned 1 point, (30, 35) was assigned 2 points, and ≤ 30 was assigned 3 points.

After summarizing the score values and performing modeling again, we conducted ROC analyses in the training, validation, and external data sets, yielding an AUC of 0.878 (0.861–0.895) in the overall dataset, an AUC of 0.865 (0.838–0.891) in the training set, an AUC of 0.895 (0.868–0.922) in the validation set, and an AUC of 0.899 (0.865–0.934) in the external data, showing robust predictive performance across the datasets.

We plotted nomograms to visualize the models and validated them by plotting distribution plots of the entire data set. The calibration curve indicated enhanced predictive performance of the models. Furthermore, the distribution plot analysis revealed a progressive increase in the probability of delayed transfer as the score increased. We found that the DCA curves for the models were consistently above the net benefit curves for interventions in all patients, indicating that using the model to intervene in patients with high risk can yield better clinical outcomes.

We further analyzed the clinical impact curve (CIC) derived from the converted scores to evaluate clinical benefits. The curve showed that as the predicted risk increased, the two curves gradually approached each other and eventually intersected, indicating the increasing risk, or increasing transfer delays. By selecting an appropriate cutoff value, excessive medical care and wastage of medical resources can be avoided (Figure 6).

Figure 6 (A–D) ROC curves of the score model for the full dataset, the training set, the validation set, and the external data, respectively; (E) nomogram plot of the score model; (F) histogram of the scoring distribution and the predicted probability distribution; (G and H) DCA and CIC plots of the score model, respectively.

Discussion

Delay in patient transfers out of the PACU can prolong the stay of postoperative patients, increasing the risk of complications.^9,10 This also leads to a shortage of beds in the PACU, creating barriers to care and treatment at different stages of the perioperative period, adversely impacting the efficiency and utilization of the recovery room. Although there are various PACU management models, there are commonalities among them: notably, the main focus is on perioperative nursing staff who are responsible for a large amount of medical work, including monitoring patients’ condition and early warning systems, as well as the overall management framework of anesthesiologists and other personnel on call.¹¹

Earlier studies have focused on two aspects in terms of reducing PACU transfer delays. On the one hand, recommendations include strategies such as further optimization of the management model,¹ the use of new management systems,¹² optimizing the use of antagonist drugs,¹³ and improving the use of postoperative anesthetic drugs.¹⁴ On the other hand, medical interventions, such as avoiding hypothermia¹⁵ and using music therapy,¹⁶ have been recommended for patients in the PACU to reduce the occurrence of transfer delays. However, there are only a few available studies on the screening of high-risk groups with PACU transfer delays. Consequently, there is a scarcity of substantial evidence to effectively guide clinical intervention in this domain.

In this study, we included a total of 1450 patients, divided into two data cohorts, A and B. Data cohort A included 1153 patients who were randomly assigned to a training group and a validation group in a ratio of 3:2. Data cohort B included 297 patients as external data. The two data sets were collected by independent teams, with a time interval of three months between the data collection periods. The data covered multiple aspects, encompassing patient characteristics and anesthesia care aspects.

During our preliminary analysis of the collected variables, we found that the patient’s body weight, albumin, CRE, PLTs, height, TP, APPT, neutrophils, WBC, hypertension, diabetes, alcohol use, age, and HB had a variance inflation factor (VIF) of > 5. Using LASSO regression, we regularized high-dimensional data. Based on these methods, we included age, gender, body weight, hypertension, diabetes, alcohol use, HB, albumin, ALT, AST, glucose, duration of anesthesia, maximum BIS, minimum BIS, blood loss, PaCO2, lactic acid, temperature at the time of entry into the PACU, and temperature at the time of entry into the PACU.

The collected data also included details of the category of the surgical department, such as general surgery or thoracic surgery, and whether it was a lumpectomy or not. However, in the LASSO regression analysis, these two aspects did not reflect an effect on delayed transfers. This may be related to the inherent similarities in the content and procedures of general and thoracic surgery. In the case of lumpectomy, technological advancements and the use of robot-assisted techniques have narrowed the differences between the two surgical modalities. Some of the newer procedures have replaced previous conventional surgical procedures, creating a challenge in establishing valid comparisons in the summarized data cohort.

We reassessed the covariance between the included variables, and the VIFs of albumin, body weight, hypertension, alcohol use, and diabetes converged to 5. Notably, the overall covariance was substantially improved compared with the previous analysis. This verified the effect of LASSO regression on the downscaling of high-dimensional data.

We created multiple models by screening variables using logistic regression, Random Forest, and Xgboost methods. Although the variables included in each model were not identical, the analysis of the ROC curve area and the calibration curve indicated that there was not much difference between the models. It is worth noting that the model based on the Xgboost showed overall favorable performance in the validation set; that is, its AUC was 0.858 (0.821–0.896), but despite its predictive advantage in the training set, this advantage was not consistently reflected in the validation set.

Subsequent to finding no major differences between the models, we further explored simplifying the models to enable their easier clinical use. We consolidated the key variables common to all models and developed model 6, which included five variables—HB, ALT, glucose, duration of anesthesia, and minimum BIS. When trained in the training dataset, it yielded an AUC of 0.909 (0.887–0.932), and validation in the test data showed an AUC of 0.939 (0.919–0.959). Model 6 had a Dxy of 0.819, an R2 of 0.605, and a Brier of 0.117 in the training dataset, and a Dxy of 0.878, an R2 of 0.689, and a Brier of 0.098 in the test dataset.

We further analyzed the differences between model 6 and the other models using NRI¹⁷ and IDI.^18,19 The results suggested that the differences were not significant. Such results also underscore the stability and consistent predictive efficacy of key indicators across models, ie, variables such as HB, ALT, and glucose in the models were associated with delays in transfer out of the PACU. This is similar to previous research results, but there are also some differences that are detailed below.

In this study, we found that patients with medical diseases such as hypertension and diabetes were more likely to have delayed transport. We further explored the correlation between hypertension and diabetes and low body temperature and BIS value, and found that there was no clear correlation between hypertension, diabetes and low body temperature. The correlation coefficients were −0.026 and 0.004, respectively, and the P value was not statistically significant, and there was no clear correlation with BIS. The correlation coefficients were 0.027 and 0.006, respectively, and the P value was not statistically significant. The results show that even patients without comorbidities may have hypothermia and low BIS, and may also be at risk for delayed PACU transport.

One discrepancy in findings is that hypothermia has been shown to prolong the stay time of the patient in the PACU,²⁰ whereas in our study, we found that hypothermia was not a major predictor while the duration of anesthesia and the minimum intraoperative BIS²¹ were the major predictors. In other words, the longer the duration of anesthesia, the longer the postoperative delay in awakening for recovery, and the intraoperative BIS effectively reflected the anesthesia depth.²² Both of these factors significantly enhanced predictive efficacy in our study.

It has been demonstrated in previous studies that different anesthesia protocols have an effect on delayed transfers out of the PACU, and these include factors such as the dose of drugs administered^23,24 and the type of drugs.²⁵ However, in this study, we found that our model had good predictive efficacy and stability without including anesthesia-related indicators. We attribute this divergence to two key aspects: firstly, the standardization and uniformity of the anesthesia methods and selection of drugs in our hospital; and secondly, this study included patients without tracheal intubation who stayed in the PACU, which meant that these patients had been assessed for extubation by anesthesiologists, ensuring that they fulfilled the criteria for extubation before they were transferred to the PACU.

These two aspects may have allowed us to achieve a better prediction without focusing on anesthesia status, but this conclusion may not be applicable to all hospitals. Additionally, in this study, we did not pay attention to transfer delays caused by insufficient personnel or transfer handover methods. Although these have been validated in previous studies,¹² we found that our model maintained good predictive efficacy without considering these aspects. The reasons for this may be, firstly, that we excluded emergency surgeries and night-shift surgeries from our data, ie, we collected data at a time when the transfer team was well-staffed; secondly, we assumed a relatively stable transfer pattern and process within the same hospital, and this stable transfer pattern could have resulted in a stable error that may not have been reflected in the prediction model.

Previous studies have included a large cohort of patients to summarize and analyze factors related to delayed transport of PACU. In the study of Zhang et al⁶ delayed recovery of PACU after surgery was associated with old age, neurosurgery, long duration of anesthesia, ASA grade III, antibiotic use during surgery, and postoperative analgesia. These findings provide predictors of delayed recovery from PACU, particularly neurosurgery and old age, whereas in Fang et al,⁷ based on ten selected variables, A predictive model was developed including age, BMI < 21 kg/m2, American Society of Anesthesiologists Physical condition (ASA), type of surgery, chills, dementia, pain, naloxone, duration of surgery, and blood transfusion. The predictive model incorporates data on the patient’s physical condition, surgery-related conditions, patient’s mental state, pain, and anesthetic antagonists. Both studies achieved good results. The same as our study, we discussed the time of anesthesia and the general situation of patients, but the difference was that we discussed more about the situation of patients themselves, and did not discuss anesthetic drugs, antibiotics and postoperative analgesia. The reason is that in our data cohort, that is, in the anesthesia process of our research center, both antibiotics and postoperative analgesia are taken care of by the ward after the patient returns to the ward. Based on the above situation, we once again emphasize the limitations of the retrospective single-center study, which reduces the generalization ability of our study. However, this study has made an attempt to further simplify the model.

Alongside our focus on the predictive performance of the models, we also made efforts to simplify the models to improve their interpretability in the clinic. By devising a scoring system, we were able to successfully transform complex models into a user-friendly format that is easier to understand and apply. We validated the scoring system with external data sets. We concluded that although the model was simplified and more convenient to use, there was no compromise on its predictive performance or stability. The performance of model 6 may be limited in practice. Our model showed superior performance in the test set but a slight decline in the external data. This may be due to sample heterogeneity and the specificity of the study data. Therefore, the generalizability of the model needs to be considered with caution when extrapolating it to other clinical settings.

We acknowledge that there are some limitations to this study. First, our data only covered a specific time frame and a single medical institution, thus limiting the generalizability of our findings. Second, the robustness and reliability of our model still require further external validation in terms of extending the validation process to different institutions or time frames. Furthermore, the criteria for some of the indicators were decided on the basis of clinical experience and are not uniformly recognized, which may affect the significance of the results of this study.

Conclusion

In this study, our quest to predict delays in patients moving out of post-anesthesia care units yielded some results. By developing and comparing multiple models, we identified key factors associated with metastasis delay, including HB, ALT, glucose, duration of anesthesia, and minimum BIS, and created a scoring system that proved predictable and easy to interpret. However, the use of the model must be validated in a broader clinical context to ensure its reliability and generalizability.