Building an Efficient Retriever System with Limited Resources

Abstract

Despite significant advancements in Question-Answering (QA) systems based on Large Languge Models (LLMs), the issue of generating imprecise answers leading to less informative responses persists. To develop effective QA systems for open-domain datasets, particularly content-specific datasets, dense passage retrieval and the two-stage retriever-reader model remain a rational choice. However, when being applied in real-world systems, these approaches encounter challenges posed by the limitation of computational resources and training data. To address the scarcity of training data, we propose fine-tuning the pretrained BERT-based encoder using masked language modeling before employing a dual-encoder architecture—an established and efficient technique. Additionally, we introduce a modified loss function for dual-encoder training that reduces memory usage during training without compromising system performance. The new loss function is employed in a multi-stage training strategy, yielding enhanced retriever performance at each training stage. To further augment the system's capabilities, we train a cross-encoder to construct a robust retriever for domain-specific datasets. The effectiveness of these proposed techniques is validated by experiments with significant increases in performance compared to the baseline models, underscoring their potential to advance the state-of-the-art in open-domain question-answering systems.