Comparative Evaluation of ChatGPT and Gemini in Answering Questions on Vaccines and Immunization

Authors : Ayşegül İnci Sezen, Meryem Şahin Özdemir, Yusuf Emre Özdemir
Pages : 1011-1019
Doi:10.54005/geneltip.1735723
View : 180 | Download : 349
Publication Date : 2025-10-28
Article Type : Research Paper
Abstract :Background/Aims: This study aimed to evaluate and compare the adequacy, reliability, and reproducibility of ChatGPT-4 Plus and Gemini Advanced chatbots in answering questions about vaccines and immunization. Given the growing role of artificial intelligence in healthcare communication, evaluating the performance of these models is crucial for their effective integration into public health strategies. Methods: A total of 56 questions were selected from World Health Organization educational materials and a national guideline on adult vaccination. Questions were categorized into four domains: Myths and Misconceptions, Vaccination and Immunization, Basic Principles of Vaccination, and Claims and Facts about Vaccines. Two infectious disease specialists independently rated chatbot responses using the Global Quality Score (1–5). Discrepancies were resolved by a third evaluator. Inter-rater reliability, reproducibility, and correlation of responses were also assessed. Results: ChatGPT-4 Plus received a perfect score (5 points) in 83.9% of responses, while Gemini achieved this in 71.4% of responses. Mean scores were similar (4.70±0.72 vs. 4.61±0.75, p=0.116). ChatGPT-4 Plus outperformed Gemini in the \\\"Claims and Facts\\\" category (p=0.012), while other categories showed no significant differences. Both models demonstrated substantial inter-rater agreement (κ=0.540 for ChatGPT, κ=0.572 for Gemini) and moderate correlation (r=0.521, p<0.001). Reproducibility rates were high: 94.6% for ChatGPT and 98.2% for Gemini (p=0.332). Conclusions: ChatGPT-4 Plus and Gemini Advanced provided similarly accurate and comprehensive information on vaccination, with high reproducibility and reliability. These AI chatbots may serve as useful tools for vaccine-related public education and preventing misinformation in healthcare.
Keywords : Yapay zeka, aşı, bağışıklama, ChatGPT, Gemini, sağlık iletişimi

ORIGINAL ARTICLE URL