DOI: 10.11118/ecos2026_005

AN OPEN ETL AND RAG PIPELINE FOR UNIFIED ACCESS TO HETEROGENEOUS CLIMATE DATA

Dávid Chomo¹, Jaromír Landa¹

¹ Department of Informatics, Faculty of Business and Economics, Mendel University in Brno, Zemědělská 1, 613 00 Brno, Czech Republic

Climate data are dispersed across numerous portals in incompatible formats, presenting a substantial barrier for researchers and analysts working on climate change and sustainability. This paper presents an open system that automates the processing of climate datasets end-to-end. The system covers the entire workflow, from the downloading of raw files through to format detection (NetCDF, GRIB, GeoTIFF, CSV, HDF5, and ZIP archives), and transformation into a text description with statistical characteristics. The system also creates vector embeddings and natural language querying. The ETL pipeline is orchestrated by Dagster, the vector store is provided by Qdrant with an HNSW index, and response generation takes place in a multi-level retrieval-augmented generation (RAG) pipeline combining weighted Reciprocal Rank Fusion, Maximal Marginal Relevance diversification, and a Corrective RAG validation layer. The integration of a web interface, constructed on the foundations of Vue.js 3 and FastAPI, serves to facilitate the accessibility of this functionality to non-technical users. The system achieved an average composite score of 91% on a set of ten queries, with all tests passing the 80% threshold. Following a series of optimizations, the average query latency decreased from 120 to 180 seconds to approximately 26 seconds. The primary contribution of this work lies in the integration of components that currently function independently within existing climate RAG systems, thereby reducing the barriers to accessing standardized climate data for sustainability research.

Klíčová slova: ETL pipeline, climate data, vector database, retrieval-augmented generation, semantic search, QDrant, automation, Dagster, open data, embeddings

stránky: 6-6, online: 2026