DOI: 10.11118/978-80-7701-047-4-0039

WHICH AI MODEL LEADS IN SUMMARIZING FINANCIAL ARTICLES? A COMPARATIVE ANALYSIS OF GPT, MISTRAL, AND LLAMA

Jana Dannhoferová¹, Jan Přichystal¹

¹ Department of Informatics, Faculty of Business and Economics, Mendel University in Brno, Zemědělská 1, 613 00 Brno, Czech Republic

In an era where financial data grows at an unprecedented pace, effective summarization is vital for informed decision-making. This study rigorously evaluates the summarization capabilities of three advanced AI models—GPT-4o, Mistral Instruct, and Llama 3.1 8B Instruct 128k—when applied to diverse financial articles. A key contribution of this research is the development of a comprehensive evaluation framework, which assesses the models across critical dimensions, including accuracy, clarity, relevance, adherence to formatting specifications, and practical usability. While GPT consistently achieved the highest overall scores, Llama demonstrated superior performance in certain criteria, such as clarity and compression efficiency, highlighting its potential for applications where brevity and conciseness are prioritized. Despite occasional inconsistencies, Mistral excelled in generating concise summaries with high compression ratios. Our findings emphasize that the selection of an AI model should depend on specific task priorities—whether it is accuracy, brevity, or response speed. These insights underline the importance of both rigorous evaluation methodologies and careful model selection based on task-specific requirements, paving the way for more targeted applications and further research into AI-driven summarization tools in finance.

Keywords: artificial intelligence, LLMs, AI summarization, financial articles, evaluation, GPT, Mistral, Llama, metrics

pages: 39-50, online: 2025