The essential Retrieval-Augmented Era (RAG) pipeline makes use of an encoder mannequin to seek for related paperwork when given a question.
That is additionally referred to as semantic search as a result of the encoder transforms textual content right into a high-dimensional vector illustration (referred to as an embedding) by which semantically related texts are shut collectively.
Earlier than we had Giant Language Fashions (LLMs) to create these vector embeddings, the BM25 algorithm was a very fashionable search algorithm. BM25 focuses on vital key phrases and appears for actual matches within the accessible paperwork. This strategy is named key phrase search.
If you wish to take your RAG pipeline to the following degree, you may need to attempt hybrid search. Hybrid search combines the advantages of key phrase search and semantic search to enhance search high quality.
On this article, we’ll cowl the idea and implement all three search approaches in Python.
Desk of Contents
· RAG Retrieval
∘ Keyword Search With BM25
∘ Semantic Search With Dense Embeddings
∘ Semantic Search or Hybrid Search?
∘ Hybrid Search
∘ Putting It All Together
·…