The latest update in Sentence Transformers introduces an innovative way to train cross-encoder reranker models. These models are designed to evaluate the relevance between pairs of texts—such as queries paired with documents—by processing both inputs together through a shared neural network. Unlike dual-encoder models that embed texts separately and later compute similarity scores, cross-encoder models allow each text to inform the processing of the other. This method yields more precise relevance scores, albeit at the cost of increased computation, making them best suited for a second-stage reranking process in search pipelines.
Finetuning these reranker models on domain-specific data can vastly improve their performance, often exceeding that of larger, general-purpose models. By tailoring the training to focus on your unique set of queries and documents, the model learns to distinguish subtle differences in relevance that off-the-shelf models may overlook. Even a relatively small model, when finetuned on carefully curated data, can outperform competitors that are several times larger in parameter size.
The training process for these reranker models encompasses several key components:
- Dataset: Training and evaluation data can be sourced from the Hugging Face Datasets Hub or from local files in formats such as CSV, JSON, Parquet, Arrow, or SQL. There is also support for pre-processed data and adjustments to the dataset format to match the requirements of the chosen loss function.
- Loss Functions: Loss functions like Binary Cross-Entropy are employed to gauge model performance and guide the optimization process. The model’s output is directly compared to labels (or scores) provided in the dataset to compute the loss.
- Training Arguments: Customizable parameters—such as learning rates, batch sizes, warmup ratios, and other debugging options—help optimize the training process and facilitate fine control over performance and resource usage.
- Evaluator: To effectively measure the performance of the reranker, evaluators such as the CrossEncoderCorrelationEvaluator and CrossEncoderRerankingEvaluator are used. These evaluators employ metrics like NDCG, MRR, and MAP to validate improvements during and after training.
- Trainer: Integrating all components, the trainer orchestrates the finetuning process. It combines the datasets, loss functions, training parameters, and evaluators to manage the complete training loop and supports advanced features including callbacks and multi-dataset training.
A particularly important aspect of training reranker models is the method for mining hard negatives. Hard negatives are data points that are superficially similar or suggest potential relevance but do not actually answer the query. Tools such as the mine_hard_negatives function identify these challenging examples and significantly improve the model’s ability to distinguish between subtly different passages. For instance, when training on a dataset of query-answer pairs, the mining process can efficiently gather multiple negatives for each positive pair, ensuring the model is exposed to a wide variety of difficult, yet informative, examples.
An illustrative diagram below shows a comparison between embedding-based models and reranker (cross-encoder) models:
The overall training script encompasses several stages—from initializing the model and loading the dataset, to mining hard negatives and setting up evaluators, and finally to executing the trainer. During training, a typical workflow involves:
- Loading the initial model and relevant datasets from the Hugging Face Hub or local storage.
- Preprocessing data to match the expected input format of the loss function, including the removal or reordering of extraneous columns.
- Employing the hard negatives mining strategy to enrich the training dataset with challenging examples.
- Configuring various training parameters and passing them along with the model, dataset, loss function, and evaluators to a unified training pipeline.
- After training, running evaluators to compare the reranked results against baseline retrieval performance, often achieving significant improvements in metrics such as NDCG.
Evaluation results on a development set (derived from the GooAQ dataset) have demonstrated that finetuned models can achieve significantly higher NDCG scores than general-purpose rerankers. One particularly effective approach involved reranking the top 30 documents retrieved by a static embedding model. The evaluation process was conducted in two modes: one that only reranked the top retrieved documents for a realistic assessment, and another that incorporated all positive samples to benchmark the maximum achievable performance.
An additional diagram below highlights the relationship between model size and performance (measured by NDCG) on the GooAQ dataset:
Beyond the core training loop, additional resources provide extensive training examples and documentation. Training examples cover a wide range of tasks—from semantic textual similarity, natural language inference, and paraphrase mining to full-fledged reranker model distillation. These resources, along with detailed documentation on installation, quickstarts, migration guides, and API references, are invaluable for developers seeking to implement and optimize these models. For more details, readers can visit the official installation guide, the quickstart guide, or explore advanced topics such as distributed training.
This update represents a comprehensive package for researchers and practitioners interested in pushing the boundaries of semantic search and retrieval. Through careful curation of datasets, state-of-the-art loss functions, and effective training and evaluation strategies, the new approach enables the creation of highly robust and domain-tailored reranker models that can drastically enhance search performance.
For further insights and examples, consider exploring detailed guides on semantic textual similarity, natural language inference, and other related tasks available on the Sentence Transformers website.
Image credit: Hugging Face – Blog
