Automating Case-Based Learning in Obstetrics and Gynecology: Validation of a Locally Deployed Large Language Model

Introduction

Standardized residency training in obstetrics and gynecology (OB/GYN) aims to develop independent clinical decision-making skills. However, modern OB/GYN has rapidly evolved into a highly complex, multidisciplinary field. Clinical presentations no longer exist in departmental silos; rather, they constantly intersect with andrology, clinical genetics, gynecologic oncology, and endocrinology. Traditional, single-discipline teaching methods often fail to recreate these intricate, cross-specialty clinical scenarios, creating a significant pedagogical gap.^1,2 Consequently, there is an urgent need to construct a comprehensive Case-Based Learning (CBL) database that can seamlessly integrate these fragmented knowledge domains.^3,4 While such a multidisciplinary CBL framework is essential for trainees to cultivate a holistic clinical perspective, manually curating high-quality teaching cases from exponentially growing medical literature remains a massive burden for clinical educators. Furthermore, systematically identifying “clinical pitfalls” presents another major bottleneck. It traditionally requires a senior expert with substantial clinical intuition to identify these cognitive blind spots—which often lead to medical errors by junior physicians—before they can be formally incorporated into educational materials.⁵

The rapidly evolving large language models (LLMs) and advanced natural language processing (NLP) tools offer transformative solutions to these educational bottlenecks. Recently, the current applications of AI in obstetrics and gynecology education have garnered increasing attention, presenting both unprecedented opportunities and unique challenges.⁶ Recent studies have shown that LLMs perform well in processing unstructured medical texts and accurately extracting key clinical information, such as the Generative Pre-trained Transformer (GPT).^7,8 The breakthrough in clinical reasoning benchmarks further validates the potential of artificial intelligence (AI) in an innovative approach to medical education, capable of mining thousands of published reports to identify core teaching points and hidden pitfalls. Pilot studies by Cook and Gim suggest that LLMs can rapidly integrate these narratives into interactive virtual patients and scalable, low-cost CBL modules, thereby significantly reducing faculty workload.^9,10

Based on the above findings, this study aims to evaluate the feasibility, reliability and teaching validity of constructing a comprehensive structured CBL database using a local LLM (Qwen3-8B) to extract information from 3678 published OB/GYN case reports. To ensure data privacy and security—a critical concern in medical education—we specifically opted for a locally deployed, smaller-parameter LLM (Qwen3-8B) rather than relying on cloud-based APIs. Ultimately, this methodological validation study sought to demonstrate how AI can be effectively integrated into the clinical faculty workflow in alignment with competence-based Medical Education (CBME) requirements, providing a scalable case for modern resident standardized training.

Materials and Methods

Study Design and Data Acquisition

In this study, a retrospective text mining framework was used to construct a structured CBL database using LLM. A comprehensive literature search was conducted in PubMed database, focusing on clinical case reports in gynecology, obstetrics, reproductive medicine, and related multidisciplinary fields. To ensure methodological transparency, the precise Boolean search strategy utilized Medical Subject Headings (MeSH) and Publication Type (PT) as follows: (“Gynecology” [MeSH] OR “Obstetrics” [MeSH] OR “Gynecologic Neoplasms” [MeSH] OR “Reproductive Medicine” [MeSH] OR “Infertility” [MeSH]) AND “Case Reports” [PT].

Instead of limiting the date of publication, we selected the first 5000 cases ranked by the “best match” (association) search of PubMed. This approach preferentially retrieved highly typical and clinically representative cases, rather than extracting them purely chronologically. Automatic retrieval was performed using a custom Python script of the Biopython Entrez module (complete source code is provided in Supplementary file 1). The initial API query returned 4984 accessible records based on the relevance threshold. To ensure data quality, two authors independently screened the records, and any discrepancies were resolved through consensus discussion with a third senior author. During this rigorous data-cleaning process, a total of 1306 records were excluded for the following reasons: non-English texts or missing abstracts (n=214), non-case reports such as letters or editorials (n=685), and cases lacking clear clinical descriptions or educational value (n=407). Ultimately, the final dataset comprising 3678 high-quality abstracts was retained for automated analysis by Qwen3. Figure 1 summarizes the overall technical flow and validation process of this study.

Figure 1 Methodological flowchart of the AI-driven Case-Based Learning (CBL) database construction. The process integrates automated data acquisition, structured information extraction using the Qwen3-8B model, subspecialty categorization, and rigorous expert validation to ensure pedagogical accuracy and clinical reliability.

LLM-Based Information Extraction and Structuring

To transform unstructured clinical case texts into teaching modules, we deployed Qwen3 LLM locally (specifically, the instruction-adjusted 8b parameter version Qwen3-8B-Instruct to ensure understanding of complex medical domains). To maximize the extraction consistency and reduce the hallucination of the model, we set Temperature to 0.1 and Top-P to 0.95 to strictly control the generated hyperparameters. This particular combination of parameters limits the token selection of the model to highly likely outputs, which is optimal for medical information extraction, as deterministic accuracy and factual authenticity must take precedence over creative text generation.

The model is configured with specific system prompts to act as a clinical education expert. LLM performed multi-token extraction of 4 core components on 3678 abstracts: (1) primary diagnoses, (2) teaching core points, (3) clinical pitfalls, and (4) difficulty levels.

Multidisciplinary Subspecialty Mapping

To analyze the fusion of different subspecialties, we implemented a rule-based NLP dictionary on an NLP framework designed to extract structured information and semantic annotations from clinical texts.^7,11–15 Based on the establishment of a medical surveillance framework, we developed a comprehensive bilingual medical dictionary that categorizes cases into different subspecialties. Furthermore, regular expression (Regex) matching was used to quantify the teaching topics in the extracted data.

Regarding subspecialty mapping, categories were defined based on our institutional residency rotation framework. Notably, Reproductive Endocrinology and Obstetrics were grouped together due to their overlapping focus on maternal-fetal transitions and fertility outcomes in our local training context. However, we acknowledge that these are clinically distinct domains, and this grouping strategy may obscure meaningful differences in case distributions between the two fields.

Human Validation and Quality Assessment

To rigorously evaluate the objective extraction correctness and subjective instructional utility of LLM, we performed manual validation. One hundred cases were randomly selected for human validation. The sample size calculation was performed based on Bonett’s method for ICC estimation,¹⁶ ensuring sufficient statistical power (>80%) with an alpha level of 0.05 to detect an ICC of 0.80 with a 95% confidence interval width of 0.15. To ensure reproducibility, see Supplementary file 2 for the exact system prompts used at the time of extraction.

All human evaluators involved in this study were attending physicians with over 10 years of clinical experience in OB/GYN and active roles as residency program directors or clinical educators.

Validation consisted of two stages:

1. Extraction accuracy and clinical reasoning error (hallucination) assessments: AI-generated results (diagnoses, teaching points, and subspecialties) were directly compared to the ground truth. To prevent bias, a separate panel of two senior medical educators established these gold standards. This panel operated independently of the stage II reviewers and reached a consensus on all 100 cases through discussion.Crucially, our system prompt explicitly instructed the LLM to infer unstated clinical pitfalls using standard medical guidelines. Therefore, we did not penalize valid, guideline-concordant inferences as errors. Instead, to ensure a rigorous pedagogical context, we quantified and redefined AI “hallucinations” and reasoning errors into two specific categories:
2. Primary Hallucinations (Guideline-Discordant Errors): Clinically incorrect inferences or recommendations. These contradict existing medical guidelines and pose potential safety risks to patients (eg, suggesting a contraindicated procedure).
3. Contextual Hallucinations (Over-extrapolations): Inferences or teaching points with general medical plausibility but lacking logical connection to the specific case narrative. This includes inappropriate over-extrapolations beyond the provided clinical context.
4. Instructional usefulness: Two independent senior experts (Reviewer A and Reviewer B) blindly rated AI-generated content on five dimensions (diagnostic accuracy, completeness of core points, subspecialty accuracy, usefulness of clinical pitfalls, and difficulty) using a 5-point Likert scale. The difficulty ratings generated by the LLM were explicitly compared with the expert-rated references.

Statistical Analysis

Statistical analyses were performed with the use of Python (version 3.12). Descriptive statistical analyses summarized baseline characteristics. To assess the correct rate of information extraction, the correct rate and hallucination rate were reported as percentages. To assess the accuracy of subspecialty mapping according to human classification, we calculated Cohen Kappa (κ). For subjective teaching evaluations (Likert scale scores), interrater reliability was quantified using intraclass correlation coefficients (ICC, two-way mixed-effects model, absolute agreement), with 95% (CI) confidence intervals reported. Bootstrap analysis (1000 samples) was used to ensure the robustness of confidence intervals.

Ethical Considerations

Ethical approval and informed consent were not required for this study. The research did not involve human subjects, real patient encounters, or animal experiments. This study solely involved the computational text-mining and pedagogical analysis of previously published, publicly available, and de-identified case report abstracts from the PubMed database. Because this research does not meet the regulatory definition of human subjects research, it does not fall under the purview of the Institutional Review Board of Wenzhou People’s Hospital, and a formal IRB exemption application was not applicable. Furthermore, the local deployment of the LLM ensured that no clinical text was transmitted to third-party cloud APIs, strictly adhering to data privacy standards.

Results

Extraction of Structured Pedagogical Components

A total of 3678 valid clinical case abstracts were processed by the LLM. As shown in Figure 2A, the most common teaching core points extracted were “differential diagnosis” (n = 1607) and “gene/genomic testing” (n = 1083). The main clinical pitfalls identified were “misdiagnosis” (n = 225) and failure to implement “individualized treatment” (n = 211), as detailed in Figure 2B. In addition, qualitative evaluation showed that the LLM could systematically synthesize these elements into structured data fields (representative cases, including an example of suboptimal AI extraction, are shown in Table 1).

Table 1 Representative Examples of Structured Teaching Cases Generated by Qwen3

Figure 2 Frequency of pedagogical elements extracted by the LLM from the clinical case abstracts. (A) Distribution of the identified teaching core points, with “Differential Diagnosis” (n = 1607) and “Genetic/Genomic Testing” (n = 1083) being the most frequently extracted. (B) Distribution of the primary clinical pitfalls identified, highlighting “Misdiagnosis” (n = 225) and the failure to implement an “Individualized Approach” (n = 211) as the most common cognitive blind spots.

Regarding difficulty levels, more than 99% of cases were classified as “hard” (52.90%, n = 1945) or “medium” (47.10%, n = 1732). Notably, only 1 case (<0.10%) was classified as “easy” (Table 2). This heavily skewed distribution likely reflects the inherent publication bias of medical case reports, which naturally tend to describe unusual, complex, or atypical presentations rather than routine clinical scenarios.

Table 2 Baseline Characteristics of the Extracted Teaching Cases

Multidisciplinary Subspecialty Mapping

Following the extraction of core components, our rule-based NLP dictionary mapped the 3678 cases into diverse subspecialties. As shown in Table 1 and Figure 3, the largest proportion of cases belonged to the Reproductive Endocrinology and Obstetrics subspecialty (28.10%, n = 1035), followed by Andrology and Male Infertility (25.20%, n = 927) and Rare Diseases and Cross-Specialties (19.90%, n = 732).

Figure 3 Distribution of the extracted case reports across multidisciplinary subspecialty categories. The bar chart illustrates the absolute frequency of cases (n = 3678) mapped to each domain, highlighting the predominance of Reproductive Endocrinology and Obstetrics (n = 1035) and Andrology and Male Infertility (n = 927).

Extraction Correctness and Pedagogical Inter-Rater Reliability

As shown in Table 3, the LLM demonstrated robust objective extraction correctness compared to human experts. The accuracy of primary diagnostic extraction was 96.0%, and the accuracy of the subspecialty mapping algorithm against manual classification was 94.0% (Cohen’s κ = 0.88).

Table 3 Extraction Accuracy, Hallucination Rate, and Pedagogical Inter-Rater Reliability (n=100 Subset)

Importantly, the overall incidence of AI hallucinations remained at a very low 3.0%. When broken down by taxonomy, primary hallucinations (guideline-discordant clinical errors) accounted for 1.0%, while contextual hallucinations (plausible but over-extrapolated points not explicitly supported by the original text) accounted for 2.0%.

However, when assessing the utility of subjective clinical pitfalls, the factual error rate representing AI hallucinations increased to 6.0%. This error predominantly occurred because the LLM struggled to accurately infer implicit clinical context that was not explicitly stated in the original text, occasionally leading to overdefensive or clinically inappropriate recommendations. Consequently, inter-rater reliability dropped significantly for these subjective nuances, resulting in an ICC of 0.65 [95% CI: 0.50–0.76] for clinical pitfalls and an ICC of 0.54 [95% CI: 0.35–0.68] for the appropriateness of difficulty levels.

Discussion

Principal Findings

This study demonstrates the capacity of a locally deployed LLM (Qwen3-8B-Instruct) to systematically curate an extensive Case-Based Learning (CBL) database comprising 3678 clinical reports. The automated extraction significantly alleviates the cognitive and temporal burden placed on clinical faculty. Our framework achieved robust objective correctness, with a 96.0% accuracy rate for diagnostic extraction and a 94.0% accuracy for multidisciplinary subspecialty mapping (Cohen’s κ = 0.88). Importantly, the overall incidence of true hallucinations remained remarkably low at 3.0% when extracting objective teaching points. Furthermore, our subspecialty mapping highlighted an increasing necessity for deep multidisciplinary integration within modern OB/GYN, as the published case report literature suggests that disciplines like andrology, clinical genetics, and oncology now constitute a substantial portion of the relevant clinical content. Traditionally, medical education often suffers from “siloed” teaching, where human educators may naturally focus on nuances within their specific subspecialty.¹⁷ By providing AI-facilitated identification of cross-disciplinary case content, the LLM effectively categorizes these existing intersections from the literature. Rather than acting as a standalone creator of integration, the model extracts these structured fields independently, thereby helping to breaks down departmental boundaries to support holistic OB/GYN training. This multi-domain integration is critical for modern competence-based medical education, as it can foster a more comprehensive view among junior physicians, equipping them to navigate complex, multi-system diseases.

Comparison with Prior Work

Our findings align with contemporary evidence highlighting the proficiency of LLMs in processing complex clinical narratives for educational applications, building upon earlier studies demonstrating their potential in medical education.^7,8,10 Furthermore, our findings are consistent with the concept of a “tacit knowledge” gap between AI and human faculty. While the LLM excelled at explicit textbook-level knowledge extraction, we observed significant performance drops when evaluating subjective clinical nuances—a phenomenon echoed in previous studies comparing AI and human clinical reasoning ratings.^18,19 For instance, when the model attempted to extrapolate clinical pitfalls, the factual error rate representing clinically inappropriate recommendations increased to 6.0%, dropping the inter-rater reliability (ICC = 0.65) for this metric. An illustrative example occurred in an emergency obstetric case during the COVID-19 pandemic (PMID: 33218391), where the AI made an overdefensive and clinically inappropriate recommendation for proactive cesarean delivery, contradicting existing ACOG guidelines. This demonstrates that while current generative AI effectively extracts explicit information, it struggles to reliably simulate the empirically-driven intuition and situational risk assessment inherent to senior clinicians.

Practical Implications

From an implementation perspective, we deliberately opted for a local deployment of the 8B-parameter model rather than relying on larger cloud-based APIs. This approach provides a secure design advantage by offering a scalable and privacy-preserving solution, eliminating the data breach risks associated with transmitting clinical narratives to external servers—a critical consideration for medical informatics. To translate these technical findings into clinical curricula, we propose a “human-in-the-loop” constructivist workflow (Figure 4) for future implementation. When educators query the database for multidisciplinary topics, the LLM produces structured data that could form the basis for CBL module development. Human faculty must subsequently review these extracted elements to filter out “primary hallucinations” and refine tacit clinical pitfalls before synthesizing them into ready-to-use modules for residents. This proposed paradigm maximizes AI’s efficiency while preserving the critical mentoring role of clinical teachers, aligning with evolving clinician educator milestones.²⁰ Furthermore, it must be emphasized that obstetrics and gynecology is a specialty characterized by ss, emergency scenarios requiring rapid decision-making, and maternal-fetal ethical conflicts. Therefore, AI hallucinations in this field may lead to much more serious consequences compared to general internal medicine, making stringent human oversight an absolute necessity.

Figure 4 The Proposed AI-Assisted Human-in-the-Loop Teaching Workflow. This schema illustrates the four-stage evolution of the CBL database. It highlights the critical “human-in-the-loop” phase (indicated by the red curved arrow), where senior educators provide direct feedback and correct potential AI hallucinations, transforming raw AI extractions into validated pedagogical tools.

Limitations

Several limitations warrant consideration. First, regarding data acquisition, the use of the PubMed “Best Match” algorithm may introduce significant selection bias, and over 99% of the extracted cases were categorized as “medium” or “hard”. This extreme tilt reflects the inherent publication bias of peer-reviewed literature, which naturally prioritizes complex or atypical scenarios. While highly advantageous for senior residents under the Competency-Based Medical Education (CBME) framework,²¹ pushing them toward expert-level clinical reasoning,^22–24 it may overwhelm junior medical students.²⁵

Second, concerning data depth, our analysis was performed exclusively at the abstract level rather than utilizing full-text case reports. Therefore, the extracted structured content should not be overinterpreted in terms of full “educational completeness”.

Third, regarding AI hallucinations and inherent biases, although we reported overall numerical hallucination rates, we did not perform a detailed sub-analysis to identify which subspecialty areas, clinical scenarios, or prompt structures were most vulnerable to errors. Additionally, the risk of subtle factual inaccuracies persists; future deployments must remain vigilant regarding systematic or demographic biases inadvertently perpetuated by LLMs in clinical narratives.^26,27

Fourth, our expert validation methodology has constraints in external validity. Evaluating only a subset of 100 out of 3678 cases limits the broader generalizability of the findings. Moreover, the validation panel consisted entirely of OB/GYN specialists from a single center. It remains to be discussed whether similar performance could be achieved across different countries and educational systems. Future iterations would also benefit from including General Practice (GP) educators to assess the utility of these cases for primary care screening.

Fifth, regarding long-term model stability, LLMs rapidly evolve over time. The locally deployed Qwen3-8B model used in this study will likely be updated, meaning the long-term stability and exact reproducibility of these extraction outputs remain uncertain.

Finally, concerning pedagogical impact, this remains a methodological validation study; we have yet to assess the database’s direct impact on resident learning outcomes or competency scores via interventional trials.

Conclusions

This study demonstrates that locally deployed LLMs offer an efficient, scalable, and secure methodological framework facilitating the construction of large-scale CBL databases. Our validation confirms that AI can achieve high extraction accuracy for objective core knowledge while maintaining low hallucination rates. However, these findings primarily establish the pedagogical acceptability of the AI-generated content rather than its definitive educational effectiveness. Consequently, oversight by human experts remains an indispensable safeguard for screening empirically driven tacit knowledge, mitigating clinical risks, and ensuring patient safety in clinical education. Future interventional research must evaluate how integrating these AI-generated modules into standardized residency training directly impacts physician competency and diagnostic performance.