Comparative Impact of ChatGPT and Conventional Search Tools on Clinical Reasoning Performance: A Randomized Crossover Study in Preclinical Medical Students [Letter]

Dear editor

We read with great interest the article by Nartthanarung et al titled “Comparative Impact of ChatGPT and Conventional Search Tools on Clinical Reasoning Performance: A Randomized Crossover Study in Preclinical Medical Students”.¹ The study provides timely insights into the integration of Artificial Intelligence (AI) tools within medical education and highlights the complementary role of large language models and traditional search strategies in enhancing clinical reasoning. While the findings are promising, several methodological aspects warrant further clarification and critical appraisal.

First, although the randomized crossover design strengthens internal validity by allowing participants to serve as their own controls, the potential for carryover and period effects remains a concern. The authors implemented a 60-minute washout period; however, given the cognitive nature of clinical reasoning and learning retention, this duration may be insufficient to eliminate residual learning effects between interventions. The progressive improvement in scores across phases raises the possibility that learning or practice effects, rather than the intervention itself, contributed substantially to the observed outcomes.²

Second, the sampling framework is limited to a single institution with a relatively small cohort (n = 46), which restricts the generalizability of the findings. Although the authors justified the sample size statistically, the inclusion of a homogeneous group of second-year medical students may not adequately represent variability in baseline clinical reasoning abilities across different educational settings.³

Third, the assessment of clinical reasoning using an eight-point rubric, although structured, raises questions regarding its psychometric robustness. While content validity was reviewed by faculty members, the absence of reported inter-rater reliability, construct validity, and sensitivity to change limits confidence in the measurement of such a complex construct. Clinical reasoning is inherently multidimensional, and reliance on a single rubric-based tool may oversimplify its evaluation.⁴

Furthermore, the study utilized paired t-tests for statistical analysis, which are appropriate for within-subject comparisons; however, the absence of adjustments for multiple comparisons may increase the risk of Type I error. Additionally, while effect sizes were reported, the interpretation of cumulative improvement across phases should be approached cautiously due to the potential confounding influence of repeated testing.

Another important consideration is the lack of standardization in ChatGPT prompting strategies. Participants were allowed self-directed prompting, which introduces variability and limits reproducibility. Given that prompt engineering significantly influences AI-generated outputs, this factor represents a critical source of bias that should be controlled or at least systematically documented.

Moreover, while the study acknowledges the risk of AI-generated hallucinations, there is limited exploration of how accuracy and reliability of responses were ensured during the intervention. Without verification mechanisms, it remains unclear whether improved performance reflects enhanced reasoning or reliance on potentially inaccurate synthesized information.

Lastly, the study focuses on short-term outcomes, with no assessment of long-term knowledge retention, transfer to clinical settings, or impact on higher-order reasoning skills. Future research should incorporate longitudinal designs and include objective performance measures in real or simulated clinical environments to better understand the educational implications of AI integration.

In conclusion, while this study contributes valuable preliminary evidence supporting the integration of AI tools in medical education, addressing the aforementioned methodological concerns would strengthen the validity and applicability of the findings. Rigorous, multi-center studies with standardized protocols and robust assessment frameworks are essential to establish the true educational value of AI-assisted learning.