Multimodal CoLRAG-TF: Triple-Filtered Retrieval for Complex PDFs

Challenges in Complex PDF Retrieval

The article highlights the inherent difficulties in performing Retrieval-Augmented Generation (RAG) over heterogeneous PDF collections. These challenges stem from several factors: the multimodal nature of content (text, images, tables), the presence of domain-specific terminology that can hinder keyword-based searches, and the critical need for multi-hop reasoning. Multi-hop reasoning requires an AI system to connect disparate pieces of evidence scattered across a document or even multiple documents to answer a complex query, which is a significant hurdle for traditional RAG approaches. The paper specifically mentions a dataset of Japanese disaster lesson PDFs, which likely contain a mix of technical reports, diagrams, and narrative accounts, making them a prime example of "complex PDFs."

Existing RAG systems often struggle with these complexities, leading to suboptimal retrieval quality, especially when queries require synthesizing information from various parts of a document. The lexical bias of keyword matching and the limitations of purely semantic embeddings in capturing intricate relationships further exacerbate these issues. The goal of Multimodal CoLRAG-TF is to overcome these limitations by introducing a more sophisticated retrieval mechanism that can robustly handle the diverse information types and reasoning demands presented by such documents.

The CoLRAG-TF Architecture and Its Innovations

Multimodal CoLRAG-TF proposes a novel four-axis fusion architecture designed to tackle the aforementioned challenges. This architecture integrates four distinct retrieval signals: dense text embeddings (for semantic similarity), BM25 keyword matching (for lexical relevance), knowledge-graph triple filtering (for structured relational reasoning), and image-based similarity (for visual content). The system constructs a multimodal index from 2,403 blocks extracted from 43 Japanese disaster lesson PDFs, utilizing a hybrid OCR pipeline and LLM-based caption generation to process both textual and visual elements effectively.

A key innovation is the extraction of 11,414 OpenIE triples, which are then indexed with FAISS. This enables sub-second triple lookup and hierarchical propagation of relevance signals, significantly enhancing compositional reasoning. Furthermore, a HippoRAG2-inspired coarse-to-fine retriever (volume → chapter → block) is employed to efficiently narrow down the search space before the final fusion scoring. Bayesian optimization was used to determine the optimal weights for each fusion axis, revealing that the knowledge-graph triple axis must dominate (α_triple = 0.44) to effectively counteract lexical bias and sustain multi-hop retrieval quality.

Performance and Broader Implications

Evaluated on a 457-pair benchmark, Multimodal CoLRAG-TF demonstrated impressive performance, achieving a Retrieval Recall of 0.9909. More significantly, it showed a 71.6% improvement in multi-hop answer similarity over single-hop queries, underscoring its effectiveness in complex reasoning tasks. The paper also highlights the applicability of its approach to visual inputs through an image-to-lesson pipeline using a vision LLM, further broadening its potential utility.

These results strongly suggest that triple-filtered multimodal fusion is not merely an incremental improvement but an essential component for structured reasoning over noisy, heterogeneous PDF documents. The framework's generalizability is emphasized, indicating its potential applicability beyond the specific disaster domain to other fields dealing with complex, multimodal information. This could pave the way for more intelligent and reliable AI systems capable of extracting and synthesizing information from diverse and challenging document types, ultimately enhancing knowledge discovery and decision-making processes across various industries.