Document Type : Original Article

Authors

• KULYASH R. ZHILISBAYEVA ¹
• NASRIN SHOKRPOUR ^{2, 3}
• MOHAMMAD MEHDI PARVIZI ^{4, 5}

¹ Department of Languages, West Kazakhstan Marat Ospanov Medical University, Aktobe, Kazakhstan

² English Department, Faculty of Paramedical Sciences, Shiraz University of Medical Sciences, Shiraz, Iran

³ Clinical Education Research Center, Shiraz University of Medical Sciences, Shiraz, Iran

⁴ Molecular Dermatology Research Center, Shiraz University of Medical Sciences, Shiraz, Iran

⁵ Department of Medical Journalism, School of Paramedical Sciences, Shiraz University of Medical Sciences, Shiraz, Iran

Abstract

Introduction: Modern large language models (LLMs) like ChatGPT (based on the GPT4 architecture) and DeepSeek offer unprecedented capabilities for generating scientific text. However, their performance in replicating structured, high-quality scientific writing, especially compared to human-authored abstracts, remains insufficiently evaluated. To compare the abstract quality produced by human authors, ChatGPT/GPT4, and DeepSeek across six evaluation criteria: Clarity, Coherence, Conciseness, Accuracy, IMRaD Structure, and Language Quality, using blinded
expert ratings and non-parametric statistical methods, specifically the Kruskal–Wallis test followed by pairwise Wilcoxon rank-sum tests with false discovery rate correction.
Methods: We selected 23 medical and healthrelated research topics, each yielding three abstracts (human, ChatGPT, DeepSeek), for a total of 69 abstracts. Three raters scored each abstract. Kruskal–Wallis tests assessed group differences; Cliff’s Delta (δ) was calculated as a nonparametric effect size for each comparison, suitable for ordinal data.
Results: Across criteria, ChatGPT and DeepSeek significantly outperformed human authors in Clarity, Coherence, IMRaD Structure, and Language Quality. In contrast, Conciseness and Accuracy showed negligible effect sizes (|δ| <0.10), suggesting parity across all three sources.
Conclusions: ChatGPT and DeepSeek achieved significantly higher scores in clarity, coherence, structure, and language quality, while showing comparable performance in conciseness and accuracy. These findings complement recent evaluations showing competitive medical and reasoning performance of DeepSeek models compared to proprietary LLMs. While shortform abstracts, expert oversight, and domain expertise remain critical, the results suggest that LLMs—particularly GPT4 and DeepSeek—can serve as effective tools in drafting scientific abstracts.

Highlights

KULYASH R. ZHILISBAYEVA

NASRIN SHOKRPOUR

Keywords

• Artificial intelligence
• Natural language processing
• Deep learning
• Abstracting and indexing
• Medical writing