DOI - Mendel University Press

DOI identifiers

DOI: 10.11118/ecos2026-040

AN OFFLINE RAG SYSTEM PROTOTYPE FOR SAFE QUERYING OF UNSTRUCTURED DOCUMENTS

David Vlastník1, Jaromír Landa1
1 Department of Informatics, Faculty of Business and Economics, Mendel University in Brno, Zemědělská 1, 613 00 Brno, Czech Republic

In the context of large language models (LLMs) being utilized within organizations that manage sensitive data, two notable limitations of commercial cloud solutions become evident. Firstly, there is the risk of confidential information being disclosed when content is transmitted to external servers. Secondly, there is the issue of the models‘ propensity to hallucinate in the absence of reliable context. The present paper proposes and implements a prototype of a fully offline RAG (Retrieval-Augmented Generation) system using exclusively open-source components. The architecture is predicated on the Agentic RAG approach, which substitutes linear query processing with a multi-agent system orchestrated using the LangGraph framework. A hybrid (dense + sparse) approach over the Qdrant vector database is used for search, and the parallel relational database DuckDB is used for tabular data. The system is containerized using Docker Compose and runs a local LLM via the Ollama platform. The evaluation of the output quality was conducted through a comparative analysis with three existing tools (LM Studio, AnythingLLM, NotebookLM) on a set of ten extensive PDF reports on climate change. The evaluation process involved the utilization of five complex questions, necessitating the synthesis of information from multiple sources. The implemented prototype achieved a level of accuracy comparable to the cloud-based NotebookLM tool and significantly outperformed solutions employing local LLMs. It also maintained transparency of answers through citations of source documents and ensured that data remained on the user‘s local device.

Keywords: retrieval-augmented generation, large language models, agentic RAG, vector databases, hybrid search, open-source LLM, data protection, semantic search, on-premises deployment

pages: 363-371, online: 2026



References

  1. AILOITTE. 2026. RAG vs fine-tuning: which AI approach should businesses choose? Ailoitte. https://www.ailoitte.com/insights/rag-vs-fine-tuning.
  2. CHAI. 2024. RAG 101: Ep 1 - retrieval-augmented generation (RAG), why? Medium. [Accessed: 2026-03-07] https://medium.com/@thechaiexperiment/rag-101-ep-1-retrieval-augmented-generation-rag-basics-aaf7cafdd0ce
  3. HUANG, L., YU, W., MA, W., ZHONG, W., FENG, Z., WANG, H., CHEN, Q., PENG, W., FENG, X., QIN, B., LIU, T. 2025. A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems. 43(2), 1-55. https://doi.org/10.1145/3703155 Go to original source...
  4. JAVANMARD, A. 2025. PDF data extraction benchmark 2025: comparing Docling, Unstructured, and LlamaParse for document processing pipelines. Procyons. [Accessed: 2026-01-24] https://procycons.com/en/blogs/pdf-data-extraction-benchmark/
  5. JEONG, S. et al. 2024. Adaptive-RAG: learning to adapt retrieval-augmented large language models through question complexity. In: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume 1: Long Papers. Mexico City: Association for Computational Linguistics, pp. 7036-7050. Go to original source...
  6. PATRONUSAI. 2026. RAG evaluation metrics: best practices for evaluating RAG systems. Patronus AI. [Accessed: 2026-03-10] https://www.patronus.ai/llm-testing/rag-evaluation-metrics
  7. QU, R., TU, R., BAO, F. S. 2025. Is semantic chunking worth the computational cost? In: Findings of the Association for Computational Linguistics: NAACL 2025. Albuquerque: Association for Computational Linguistics, pp. 2155-2177. https://doi.org/10.18653/v1/2025.findings-naacl.114 Go to original source...
  8. VASWANI, A. 2023. Attention is all you need. arXiv:1706.03762v7 https://arxiv.org/html/1706.03762v7
  9. VLASTNÍK, D. 2026. Prototyp RAG (Retrieval-augmented generation) pro analýzu velkého množství nestrukturovaných textů. Master's thesis. Brno: Mendel University in Brno. Faculty of Business and Economics, Department of Informatics.
  10. XU, X. et al. 2025. Agentic RAG with human-in-the-retrieval. In: 2025 IEEE 22nd International Conference on Software Architecture Companion (ICSA-C). Odense, Denmark. IEEE. https://doi.org/10.1109/ICSA-C65153.2025.00074. Go to original source...
  11. ZHANG, Y. et al. 2025. Siren's song in the AI ocean: a survey on hallucination in large language models. Computational Linguistics. 51(4), 1373-1418. https://doi.org/10.1162/COLI.a.16 Go to original source...
Return