Get to know the Reranking model technique: A key tool for enterprise information retrieval system

We are now in the age of Large Language Models (LLMs). Since OpenAI released the GPT model, we’ve seen many applications emerge, such as smart search systems, knowledge management systems, chatbots, and machine translation tools. Each of these applications is powered by LLMs at their core.

These applications rely heavily on data, requiring us to find similarities and rank results before processing. While the basic and commonly used method is to find similarity based on cosine similarity between two vectors, today AIGEN will introduce a new method: the reranking model.

Before we delve into the reranking model that AIGEN uses to improve the accuracy and performance of many AI services, we need to understand the history of ranking. We’ll explore the evolution from traditional full-text search using BM25, to vector search, and finally to reranking models. We’ll conclude by comparing the metrics of each method, highlighting their pros and cons.

How many types of ranking method?

Ranking techniques play a crucial role in information retrieval and natural language processing tasks. Some of the most widely used methods include:

1. Full text search BM25

This is a probabilistic ranking function used to estimate the relevance of documents to a given search query. BM25 (Best Matching 25) is an improvement over earlier models and is widely used in search engines due to its effectiveness and simplicity.

2. Vector Similarity

This method represents documents and queries as vectors in a high-dimensional space. The similarity between a document and a query is then computed using metrics like cosine similarity or dot product. This approach is particularly useful when dealing with semantic meaning rather than just keyword matching. Methods like TF-IDF vectorization or more advanced techniques like word embeddings (e.g., Word2Vec, GloVe) are often used to create these vector representations.

3. Reranking method

This is a two-stage approach that refines the results of an initial retrieval. First, a faster method like BM25 or vector similarity retrieves a set of candidate documents. Then, a more sophisticated model, often based on large language models like BERT, reranks these candidates. This method combines the efficiency of initial retrieval with the accuracy of advanced models, allowing for more nuanced relevance judgments. It’s particularly effective for capturing complex query-document relationships but can be computationally intensive.

4. Hybrid Approaches

These combine multiple ranking techniques to leverage the strengths of each method. For example, a system might use BM25 for initial retrieval, followed by a vector similarity check, and finally a machine learning reranking step. This can provide more robust and accurate results across a variety of query types and document collections. Hybrid approaches often outperform individual methods by balancing efficiency and effectiveness.

Each of these methods has its own strengths and weaknesses, and the choice often depends on the specific requirements of the task, the nature of the data, and computational resources available. In the following sections, we’ll delve deeper into each of these techniques, exploring their mechanisms, applications, and comparative performance.

Mechanisms of each method for ranking

1. Full text search BM25

BM25 (Best Matching 25) is a widely used ranking function in full-text search systems, designed to estimate the relevance of documents to a given search query. Developed as an improvement over earlier probabilistic models, BM25 has become a standard in information retrieval due to its effectiveness and relative simplicity. It operates on the principle of term frequency-inverse document frequency (TF-IDF), but with important refinements that address some of the limitations of simpler TF-IDF models.

The BM25 algorithm calculates a relevance score for each document based on the query terms it contains. It takes into account three main factors: term frequency (how often a query term appears in a document), inverse document frequency (how rare or common the term is across all documents), and document length. BM25 applies a saturation function to term frequency, which means that the impact of repeated terms diminishes after a certain point, preventing very long documents or keyword stuffing from unduly influencing the rankings. Additionally, it normalizes scores based on document length, helping to ensure that shorter, more focused documents aren’t unfairly penalized compared to longer ones.

2. Vector Similarity

Vector Similarity is a fundamental concept in information retrieval and natural language processing, used to measure the likeness between documents or between a query and a document. In this approach, text is transformed into numerical vectors in a high-dimensional space, where each dimension typically represents a term or a semantic concept. The similarity between these vectors is then computed using various metrics, with cosine similarity being one of the most popular.

The process of creating these vectors has evolved significantly over time. Traditional methods like TF-IDF (Term Frequency-Inverse Document Frequency) create sparse vectors based on word occurrences. More advanced techniques leverage dense embeddings, such as those produced by Word2Vec, GloVe, or BERT, which capture semantic relationships between words. These dense representations allow for more nuanced similarity comparisons, often outperforming lexical matching in tasks like document retrieval or question answering.

One of the key advantages of vector similarity methods is their ability to capture semantic similarity even when exact word matches are not present. This makes them particularly effective for handling synonyms, related concepts, and even cross-lingual information retrieval. However, the effectiveness of vector similarity heavily depends on the quality of the vector representations and the chosen similarity metric. As such, ongoing research in this field focuses on developing more sophisticated embedding techniques and exploring alternative similarity measures to improve retrieval performance.

3. Reranker Method

The reranking method is a crucial component in modern Information Retrieval (IR) systems, particularly those leveraging deep learning models. It operates as part of a two-stage retrieval pipeline. In the first stage, an initial retrieval system (such as BM25 or a dense retrieval method) provides a list of candidate documents or passages. The reranking model then refines this initial list in the second stage, aiming to improve the relevance of the top results.

Reranking typically employs more sophisticated models, often based on large pre-trained language models like BERT, T5, or their multilingual variants such as mT5 and mMiniLM. These models, referred to as cross-encoder models, take both the query and each retrieved passage as input and produce a relevance score. This allows them to capture complex semantic relationships between the query and the passage, going beyond simple lexical matching. The power of these models comes at the cost of computational intensity, which is why they are applied only to a subset of initially retrieved documents (often the top 1000) rather than the entire corpus.

The effectiveness of reranking models heavily depends on the quality and quantity of the training data. They are usually fine-tuned on labeled datasets containing queries and examples of relevant documents. The MS MARCO dataset and its multilingual version, mMARCO, are popular choices for this purpose due to their large scale and diversity. Experiments have shown that reranking models, when properly trained, can significantly improve retrieval performance across multiple languages, even in zero-shot scenarios where they’re applied to languages not seen during fine-tuning.

Comparison of Ranking Methods

While each ranking method has its strengths, it’s important to recognize that no single approach is a one-size-fits-all solution. Let’s examine the pros and cons of each method and discuss why hybrid approaches are often the most effective:

Method	Speed	Accuracy	Semantic Understanding	Scalability	Training Required
BM25	Fast	Moderate	Low	High	No
Vector Similarity	Moderate	Good	High	Moderate	Depends on embedding
Reranking	Slow	Excellent	High	Low-Moderate	Yes

Test set (article)	count	bm25	multilingual-e5-large-instruct	AIGEN reranker
airesearch	557	84.92%	85.34%	91.50%
tyqida(	496	82.18%	82.41%	80.52%
iappwiki	711	84.80%	75.34%	88.70%
	1764	84.10%	80.49%	87.28%

Limitations of Individual Methods

1. BM25
   • Struggles with semantic understanding and synonyms
   • May miss relevant documents that use different terminology
2. Vector Similarity
   • Quality heavily depends on the chosen embedding method
   • Can be computationally expensive for large-scale retrieval
   • May struggle with rare words or out-of-vocabulary terms
3. Reranking
   • Computationally intensive, not suitable for real-time applications with large datasets
   • Requires high-quality training data for optimal performance
   • May introduce latency in the retrieval process

Given these limitations, many modern information retrieval systems opt for hybrid approaches that combine multiple ranking methods. Here’s why

1. Balancing efficiency and accuracy
   • Use fast methods like BM25 for initial retrieval
   • Apply vector similarity or reranking on a smaller subset of results
2. Leveraging strengths of each method
   • BM25 for handling exact matches and rare terms
   • Vector similarity for capturing semantic relationships
   • Reranking for fine-grained relevance judgments
3. Adaptability to different query types
   • Simple queries might be well-served by BM25 alone
   • Complex queries benefit from the semantic understanding of vector similarity and reranking
4. Scalability
   • Hybrid approaches can be designed to scale efficiently by applying more resource-intensive methods only when necessary

Consulting AIGEN for tailored solutions

While hybrid approaches offer a powerful framework for ranking, the optimal combination and configuration of methods can vary significantly depending on the specific use case, data characteristics, and performance requirements. This is where consulting with AI experts, such as those at AIGEN, can provide significant value:

• Design custom hybrid architectures suited to your specific needs
• Analyze your data to recommend the best method combination
• Fine-tune models to your domain
• Implement scalable solutions for large-scale applications
• Set up continuous improvement systems based on user feedback

By leveraging hybrid approaches and expert consultation, organizations can create ranking systems that are both powerful and practical, overcoming individual method limitations and achieving optimal performance for their specific use cases. Contact us for consulting with our AI experts.