Study design<\/p>\n
This study was designed as a prospective, scenario-based, double-blind, observational study to evaluate the performance of ChatGPT-5 in obstetric and gynecological emergency scenarios.<\/p>\n
A total of 15 standardized clinical scenarios were developed for this study. All scenarios were created by a single obstetrician with over 10 years of clinical experience in obstetric and gynecologic emergency management. Each case was designed in accordance with current international guidelines (ACOG, RCOG, and WHO) and represented a range of high-risk conditions, including postpartum hemorrhage, eclampsia, HELLP syndrome, uterine rupture, amniotic fluid embolism, ruptured ectopic pregnancy, and major gynecologic hemorrhages. Developing all scenarios by a single author ensured conceptual and methodological consistency across cases, while subsequent independent expert evaluations helped reduce potential subjectivity or bias. All scenarios were conducted in Turkish, as it represents the native language of the clinical setting and the standard mode of physician\u2013patient communication in obstetric and gynecologic emergencies. Although the selected cases covered a broad clinical spectrum\u2014from common and guideline-defined emergencies to complex, multidisciplinary situations\u2014no formal difficulty grading was applied. The primary aim was to ensure comprehensive coverage of both obstetric and gynecologic emergencies rather than to stratify them by difficulty level. Case complexity was inherently reflected by the nature of each scenario and its management requirements.<\/p>\n
The study was conducted between July and August 2025, and all scenarios were run using a single large language model (ChatGPT-5). All questions were asked in the same session, using the same device (a MacBook Air, M1, 2020, with 8 GB RAM and macOS Monterey), and a stable fiber internet connection (100 Mbps). All interactions with ChatGPT-5 were conducted using default system parameters (temperature\u2009=\u20090.7, top-p\u2009=\u20091.0, and maximum token length\u2009=\u20094096). These settings were deliberately maintained to emulate a realistic clinical user experience, reflecting how healthcare professionals would typically engage with the system in practice. A separate chat window was opened for each scenario, and only the first responses were analyzed. This is because the probabilistic nature of large language models allows for different responses to the same question across different trials. Therefore, the \u201cregenerate response\u201d feature was not used\u2014a total of 15 scenarios \u00d7 five questions\u2009=\u200975 responses were obtained.<\/p>\n
Evaluation process<\/p>\n
Responses generated by ChatGPT-5 were recorded and anonymized for analysis. In parallel, the same 15 clinical scenarios were independently evaluated by four clinicians: two obstetricians, one emergency medicine specialist, and one anesthesiologist.<\/p>\n
The first obstetrician (OB1\u2013POK) had 12 years and 4 months of clinical experience in obstetrics and gynecology, the second obstetrician (OB2\u2013TA) had 10 years and 2 months, the emergency physician (EM\u2013MG) had 9 years and 6 months, and the anesthesiologist (AN\u2013KA) had 11 years and 1 month of professional experience in obstetric and emergency care. All assessors had active roles in tertiary-level hospitals and routinely managed high-acuity obstetric and gynecologic emergencies.<\/p>\n
Each response was evaluated using a structured, guideline-based scoring system encompassing five key parameters:<\/p>\n
1.<\/p>\n
Diagnostic accuracy: Whether the most likely diagnosis was determined accurately and comprehensively.<\/p>\n
2.<\/p>\n
Investigations: Appropriateness and logical prioritization of recommended diagnostic tests.<\/p>\n
3.<\/p>\n
Treatment plan: Compliance with the initial treatment approach and current clinical guidelines.<\/p>\n
4.<\/p>\n
Clinical safety and applicability: Whether the recommendations are safe and applicable in obstetric emergency department settings.<\/p>\n
5.<\/p>\n
Decision complexity: Whether the response involves clinical reasoning beyond superficial pattern recognition.<\/p>\n
Each criterion was scored as 1 point (correct\/complete) or 0 points (incorrect\/incomplete). This allowed for a maximum of 5 points per case. Total scores ranged from 0 to 5, and the results were categorized as follows:<\/p>\n
Quality and readability analysis<\/p>\n
In addition to clinical relevance, ChatGPT-5\u2019s responses were evaluated for scientific quality and readability. The same four assessors (two obstetricians, one emergency medicine specialist, and one anesthesiologist) who conducted the primary clinical evaluations also performed the quality and readability assessments to ensure methodological consistency. None of the assessors were involved in the scenario development process. Prior to evaluation, a short calibration session was conducted in which five sample responses were jointly reviewed to establish scoring consistency. Each response was anonymized and labeled with coded identifiers (e.g., Response A, B, C) to maintain blinding. Scientific quality and clinical value were assessed using the modified DISCERN (mDISCERN) and Global Quality Scale (GQS) tools, following established criteria from the obstetrics and gynecology literature and international guidelines (ACOG, 2022; WHO, 2018).<\/p>\n
Readability was analyzed using standardized indices\u2014Flesch Reading Ease Score (FRES), Flesch\u2013Kincaid Grade Level (FKGL), Simple Measure of Gobbledygook (SMOG), and Coleman\u2013Liau Index (CLI)\u2014to determine the clarity and comprehensibility of the generated text. These measures allowed for an integrated evaluation of both the informational quality and accessibility of the AI-generated outputs.<\/p>\n
mDISCERN scores<\/p>\n
mDISCERN is a modified version of the original DISCERN scale and is one of the most frequently used methods for assessing electronic health information [10<\/a>, 11<\/a>]. Each \u201cyes\u201d answer is worth 1 point; the total score ranges from 0 to 5. Higher scores indicate more reliable and higher-quality information. The scale consists of five questions:<\/p>\n 1.<\/p>\n Are the purpose and scope clear?<\/p>\n 2.<\/p>\n Is there alignment between the source and guideline?<\/p>\n 3.<\/p>\n Is the information balanced and unbiased (benefits, risks, alternatives)?<\/p>\n 4.<\/p>\n Are patient safety and limitations addressed?<\/p>\n 5.<\/p>\n Are points of uncertainty and need for expert opinion acknowledged?<\/p>\n Global Quality Scale (GQS) scores<\/p>\n