Comparative Impact of ChatGPT and Conventional Search Tools on Clinical Reasoning Performance: A Randomized Crossover Study in Preclinical Medical Students [Response to Letter]

Dear editor

We thank Kalra for the thoughtful comments regarding our article. We appreciate the opportunity to clarify the interpretation of our findings and to place the study’s limitations in appropriate context.

First, we agree that carryover effects are an important consideration in crossover educational studies. In this setting, a washout period cannot fully remove learning once it has occurred. For that reason, our 60-minute washout period should be interpreted as a practical measure to reduce immediate recall and tool-related priming, rather than as a guarantee of complete elimination of residual learning effects. We therefore agree that carryover cannot be fully excluded. However, this does not invalidate the study. Rather, it means the findings should be interpreted cautiously as short-term, within-learner comparisons in a structured educational setting. This interpretation is consistent with published guidance for randomized crossover trials, which emphasizes transparent acknowledgment of period and carryover effects rather than assuming that any washout period can fully resolve them.¹

Second, we agree that the single-institution setting and modest sample size limit generalizability. This was an educational study conducted within one authentic classroom cohort, and we do not claim broad external validity. At the same time, the randomized crossover design was chosen precisely because it improves internal efficiency by allowing participants to serve as their own controls. In that context, the design remains appropriate for an initial study of short-term educational performance, even though confirmation in larger and multicenter cohorts is still needed.¹

Third, we agree that the eight-point rubric would be strengthened by additional psychometric evidence. Clinical reasoning is a complex construct, and rubric-based assessment should ideally be supported by broader validity evidence and reliability reporting. At the same time, the use of a structured rubric is not, in itself, a methodological weakness. On the contrary, rubric-based assessment is a recognized approach in medical education when grounded in relevant domains and applied transparently. Our intention was to use a practical structured measure aligned with the objectives of case-based learning, not to claim that this single rubric fully captures the entire construct of clinical reasoning. Future work should expand this by including formal inter-rater reliability and broader validity evidence.²

Fourth, regarding statistical analysis, we agree that repeated comparisons in crossover studies should be interpreted carefully. For that reason, our findings should not be read as proof of cumulative superiority across phases without reservation. However, we respectfully disagree that the statistical approach was inappropriate. Paired analysis is a reasonable method for within-subject comparisons in this educational design, and effect sizes were reported to complement p-values. The main point of our results was not to make an inflated confirmatory claim, but to show that performance improved across both learning conditions in the short term. We agree that future studies may benefit from more advanced crossover-specific modeling and clearer prespecification of multiplicity handling.³

Fifth, we agree that variability in prompting is relevant. Prompt formulation can influence large language model output, and this should be recognized when interpreting reproducibility. However, we deliberately allowed students to use self-directed prompting because our aim was to reflect authentic educational use rather than an artificial, tightly scripted interaction. We therefore view prompt variability not only as a limitation, but also as part of the educational realism of the study. Future research should compare standardized and self-directed prompting strategies more directly. Recent reporting guidance for studies involving large language models supports more explicit documentation of prompts, model versions, and human oversight, and we agree that this is an important direction for the field.⁴

Sixth, we agree that hallucinations and inaccurate AI-generated content are important risks. However, we do not agree that this concern negates the value of the study. Our conclusions do not endorse uncritical use of AI outputs. On the contrary, we explicitly emphasize verification, critical appraisal, and faculty oversight. This cautious position is aligned with current international guidance, which highlights the risks of false or incomplete outputs and the need for human supervision when generative AI is used in health-related contexts.⁵

Finally, we agree that the study evaluated short-term outcomes only. We therefore do not claim long-term retention, transfer to clinical practice, or broader curricular effectiveness. Instead, we view this study as an initial contribution showing that both ChatGPT-assisted and conventional search-supported learning can improve short-term clinical reasoning performance in a structured preclinical setting. That question remains educationally relevant, even while longer-term and multicenter studies are clearly warranted.

We thank the author again for the constructive comments. We believe these points help refine the interpretation of our results, while also supporting the value of the study as an early controlled evaluation of AI-supported learning in medical education.