LEAFRAG - OBJECT RECOGNITION EXPLANATION THROUGH MULTIMODAL RETRIEVAL-AUGMENTED GENERATION

Traditional object recognition systems often struggle with real-world complexities despite achieving high performance in controlled environments. This thesis presents LeafRAG, a novel approach integrating multi-modal Retrieval-Augmented Generation (RAG) with botanical knowledge to enhance object recognition, using leaf identification as the primary application domain.

The implemented system features a scalable architecture for processing multimodal data sources including text, images, and PDF documents, providing not only accurate identification results but also explanatory details that enhance educational value. Its modular design incorporates advanced embedding technologies and LLMs, with vector storage and retrieval mechanisms using FAISS.

Experimental evaluation across 13 tree species yielded promising results, with the reasoning-optimized model achieving 82.14\% Top-1 accuracy and 89.29\% Top-3 accuracy. The key findings include the substantial impact of the image comparison features (+10.71\% accuracy improvement), the value of the knowledge derived from the textbook (+8. 93\% precision) and the optimal retrieval parameter of k = 25 to balance precision and response time.

These results confirm that the integration of scientific knowledge with modern AI techniques offers substantial capabilities for complex recognition tasks. Future research directions include hierarchical retrieval and validation mechanism, GraphRAG incorporation, optimized embedding strategies, and multi-image processing capabilities to address more complex identification scenarios.