SemSpEm: Semantic Sparse Embedding for Document Retrieval
- 주제(키워드) Document Retrieval , Sparse Representation , Semantic Embedding , Word Sense Disambiguation
- 주제(DDC) 006.31
- 발행기관 아주대학교 일반대학원
- 지도교수 이슬
- 발행년도 2026
- 학위수여년월 2026. 8
- 학위명 석사
- 학과 및 전공 일반대학원 인공지능학과
- 실제URI http://www.dcollection.net/handler/ajou/000000036379
- 본문언어 영어
- 저작권 아주대학교 논문은 저작권에 의해 보호받습니다.
초록/요약
Pre-trained language models have revolutionized information retrieval. Current retrieval systems typically employ a two-stage pipeline: sparse retrievers for efficient first-stage filtering and dense retrievers for second-stage ranking. However, this approach presents a fundamental trade-off. Traditional sparse methods—based on lexical matching—are computationally efficient but suffer from vocabulary mismatch. In contrast, dense neural approaches achieve superior effectiveness through learned semantic embeddings but with increased computational costs. To address this limitation, we propose Semantic Sparse Embedding （SemSpEm）, which integrates the computational efficiency of sparse retrieval with the semantic representation capability of modern contextual language models. In SemSpEm, we first train a contrastive bi-BERT model to learn context-sensitive semantic representations that explicitly capture word sense disambiguation （WSD）, encouraging tokens with the same sense to lie close in the embedding space while separating those with different senses. Using this trained bi-BERT model, we then construct a semantic database that stores sense-specific embeddings, and leverage it to generate sparse semantic vectors for both queries and documents by mapping their constituent words to the corresponding learned semantic representations. These sparse semantic vectors provide a semantically enriched alternative to traditional term-based indices, while preserving the efficiency benefits of sparse retrieval. In our experiments, SemSpEm achieves competitive performance on standard word sense disambiguation benchmarks and, on the MS MARCO passage ranking dataset, outperforms BM25 on the full evaluation set, with further gains observed on subsets of passages where semantic mapping confidence is high.
more목차
1 Introduction 1
2 Method 5
2.1 Phase 1: Sense Representation Learning 6
2.2 Phase 2: Document Inverted Index Construction 8
2.2.1 Named Entity Disambiguator （NED） 8
2.2.2 Word Sense Disambiguator （WSD） 8
2.2.3 Cluster Disambiguator （CD） 9
2.2.4 Pre-computation and Offline Indexing 9
2.3 Phase 3: Sense-Aware BM25 Document Retrieval 10
3 Experiments 12
3.1 Dataset and Experimental Settings 12
3.1.1 WSD Training Dataset and Negative Sampling Strategy 12
3.1.2 WSD Evaluation Dataset 13
3.1.3 Retrieval Test Dataset 14
3.1.4 Training Configuration 14
3.2 Experimental Results 14
3.2.1 WSD Performance Results 14
3.2.2 Retrieval Performance Results 15
3.2.3 Ablation Study 17
4 Related Work 19
4.1 Word Sense Disambiguation （WSD） 19
4.2 Sparse Retrieval Models 20
4.3 Dense Retrieval Models 21
4.4 Named Entity Linking （NEL） 22
5 Conclusion 23
Bibliography 25
