**USE:** Obtain authoritative information to predict effects, thus refine and support some policy. They help build and answer referenced to its sources. **DATA:** list of documents, each with a link and text content (which could be extended to multimedia). (query, documents, index) -> Search -> documents -> Infer -> (answer, references) **MEASURE:** Compression of golden answer: $`C(A) = \frac{-1}{|A|} \sum_{t_i \in \text{BPE}(A)} \log_2(\Pr(t_i|t_{ Property BM25 Mistral Embedding SigLIP ReFrAn Cutoff between relevant documents and irrelevant ones ❌ ❌ 🟢 🟢 Optimized for the target document dataset 🟡 ❌ 🟢 🟢 Optimized for answers 🔴 🟠 🟠 🟢 Quality 🔴 🟠 🟡 🟢 Latency (s) 🟢 🟡 🟡 🟡 [MTEB]: https://huggingface.co/spaces/mteb/leaderboard ## Approach 1: BM25 Documents in the dataset $`D` are indexed into a list of terms $`T(d)`. TF-IDF scoring estimates the relevance of a document for a query using term statistics: ```latex \text{TF-IDF}(d,q) = \sum_{t\in T(q)} \frac{|\{i:T(d)_i=t\}|}{|T(d)|} \ln\frac{|D|}{|\{d:d\in D,t\in T(d)\}|} ``` The first part is the term frequency TC, the second is the inverse document frequency IDF. [BM25][] tweaks the relevance score formula: ```latex \text{BM25}(d,q) = \sum_{t\in T(q)} \frac{|\{i:T(d)_i=t\}|\cdot(\alpha+1) }{ |\{i:T(d)_i=t\}| + \alpha (1-\beta+\beta \frac{|T(d)|}{\frac{\sum_{d'\in D}|T(d')|}{|D|}})}\ln\frac{|D|}{|\{d:d\in D,t\in T(d)\}|} ``` [BM25]: https://www.elastic.co/blog/practical-bm25-part-2-the-bm25-algorithm-and-its-variables ## Approach 2: Mistral embedding It likely was built similar to [RepLLaMA][]: using the vector $`\vec{V}(d)` of the end-of-stream token when passing the document to Mistral 7B, then trained on a softmax categorical loss: ```latex \mathcal{L}(q,d^+,D) = -\log\frac{\exp(\vec{V}(q)\cdot\vec{V}(d^+))}{\sum_{d\in D}\exp(\vec{V}(q)\cdot\vec{V}(d))} ``` [RepLLaMA]: https://arxiv.org/pdf/2310.08319 ## Approach 3: SigLIP Separate query and document embedding models may improve the quality of search. [CLIP][] (Contrastive Language-Image Pre-training) aligns the dot product of paired embeddings between two modalities using cross-entropy. The same can be done to align queries and answers. [SigLIP][] (Sigmoid loss Language-Image Pre-training) improves the loss function: ```latex \mathcal{L}(Q,D^+,D) = -\sum_{q\in Q, d\in D}\log\sigma( ±1(d,D^+) (\vec{V_q}(q)\cdot\vec{V_d}(d)) ) ``` where $`±1(d,D^+) = 2⟦d\in D^+⟧-1`. [NV-Embed][] seems to wield a similar approach. [CLIP]: https://arxiv.org/pdf/2103.00020 [SigLIP]: https://arxiv.org/pdf/2303.15343 [NV-Embed]: https://arxiv.org/pdf/2405.17428 ## Approach 4: ReFrAn ReFrAn (Retrieval From Answers) trains embeddings to maximize the prediction from a golden answer. ### Dataset 1. Randomly take a set of documents. Ask an LLM: “Write a question whose answer is in the documents above.” Store the document IDs and the query. 2. Take the documents and the query. Ask an LLM: “Write the answer to that question.” Store the query and answer. ### Model LLM without the attention mask, with two versions of the last layer: one for queries, one for documents. Maybe we want each token to output the target embedding. Maybe we want to store all of those embeddings (1 per token per document). The retrieval would not have significant latency assuming we store the index in a logarithmic data structure. Maybe we can mitigate the fact that it takes a lot of storage space by quantizing the document embeddings. ### Loss We make the dot product of the query over the document track the average **information gain** on the golden answer: ```latex \mathcal{L(q,a,d)} = \mathbb{E}(\vec{V_q}(q)\cdot\vec{V_d}(d) - (r(d+q,a)-r(q,a)))^2 ``` where $`-r(q,a)` is the average informational content of the answer tokens on a specific LLM given a context $`q` (thus lower is better, as it is predicted with higher accuracy), which coincides with the reward $`r_{\text{simPO}}` from [the SimPO paper][SimPO]: ```latex r(q,a) = \frac{1}{|a|} \sum_{t_i \in a} \log_2\pi_\theta(t_i|t_{ table { table-layout: fixed; margin: auto; border-collapse: collapse; text-align: center; width: 100%; } table th, table td { border-bottom: 1px solid #cacacf; padding: 0 1em; } table tr:nth-child(2n) { background-color: #fafaff; }