The study, published in the Journal of Medical Internet Research, examined whether large language models (LLMs) could produce educational information that combines professional accuracy with language that is accessible to the public. Researchers compared materials written by human experts with responses generated by an LLM, as well as simplified versions of the AI-generated material.
Medical professionals and family members of pediatric patients were randomly assigned to evaluate one of the three material types. Participants assessed five randomly selected educational items using a 19-question survey designed to measure factors such as usefulness, credibility, and willingness to adopt the information.
Among medical professionals, AI-generated responses received an average score of 76.17 out of the possible total, compared with 72.97 for the expert-authored materials. Among pediatric patient family members, the average scores were 68.77 for the AI-generated responses and 64.18 for the expert-written material. Although the AI-generated content received numerically higher scores in both groups, the differences were not statistically significant.
The researchers also examined whether simplifying the AI-generated material would make it more acceptable to readers. Simplified responses received average scores of 72.98 among medical professionals and 65.47 among patient family members. Neither result differed significantly from the expert-authored materials after statistical adjustment.
The study found no significant differences between the AI-generated, simplified AI-generated, and expert-authored materials across the four dimensions measured by the questionnaire. The researchers also found no meaningful interaction between material type and whether it was assessed by medical professionals or pediatric patients’ family members.
One of the study’s more notable findings involved participants’ ability to identify who had produced the information. About 81% of participants who reviewed expert-authored material correctly identified it as being written by a human. In contrast, only about 7% of participants who reviewed either type of AI-generated material correctly recognized that an LLM had produced it.
The researchers said this finding raises questions about transparency when AI is used to create health education. If readers cannot distinguish AI-generated information from expert-written content, clearly identifying the source could become an important part of responsible implementation.
The study provides preliminary evidence that LLMs could have a role in producing pediatric asthma education, but the researchers stressed that the findings do not establish that AI-generated materials are medically accurate or safe for clinical use. Professional review, along with independent testing of factual accuracy, clinical safety, comprehension, and effects on patient behavior, will be needed before widespread adoption. The researchers also noted that the study involved a predominantly highly educated sample, meaning the results may not apply to all patient populations.