Update to use .qmd in some places with callouts and code annotation.

README.md

@@ -7,4 +7,42 @@ sdk: docker
 pinned: false
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+To get started working with quarto we recommend you first [install quarto](https://quarto.org/docs/get-started/) locally so that you can render the site without Docker.
+We also recommend the [Quarto VS Code Extension](https://marketplace.visualstudio.com/items?itemName=quarto.quarto) which provides syntax highlighting, code completion, a preview button and more.
+
+The quarto source is located in `src` and you can preview the site with:
+
+```
+quarto preview src
+```
+
+A web browser should open up with a live preview of the site.
+
+## Making changes
+
+The `src/_quarto.yml` contains the site-level configuration for the quarto website and tells quarto which files to render, and how they should be organized.
+For example if you wanted to modify the [site navigation](https://quarto.org/docs/reference/site-navigation.html) you should modify this file.
+
+Quarto can render markdown, ipynb, and .qmd files, and you can mix formats in a single document.
+
+## Executing code
+
+One of the main virtues of Quarto is that it lets you combine code and text in a single document.
+By default if you include a code chunk in your document, Quarto will execute that code and include the output in the rendered document.
+This is great for reproducibility and for creating documents that are always up-to-date.
+
+```{python}
+import seaborn as sns
+import matplotlib.pyplot as plt
+
+# Sample data
+tips = sns.load_dataset("tips")
+
+# Create a seaborn plot
+sns.set_style("whitegrid")
+g = sns.lmplot(x="total_bill", y="tip", data=tips, aspect=2)
+g = (g.set_axis_labels("Total bill (USD)", "Tip").set(xlim=(0, 60), ylim=(0, 12)))
+
+plt.title("Tip by Total Bill")
+plt.show()
+```

requirements.txt

@@ -0,0 +1,2 @@
+pandas
+seaborn

src/_quarto.yml

@@ -9,14 +9,15 @@ website:
     contents:
       - href: index.qmd
         text: About
+      - section: RAG
+        contents:
+          - notebooks/rag_zephyr_langchain.qmd
+          - notebooks/advanced_rag.qmd
+          - notebooks/rag_evaluation.qmd
       - notebooks/automatic_embedding.ipynb
       - notebooks/faiss.ipynb
       - notebooks/single_gpu.ipynb
-      - section: RAG
-        contents:
-          - notebooks/rag_zephyr_langchain.ipynb
-          - notebooks/advanced_rag.ipynb
-          - notebooks/rag_evaluation.ipynb
+
 
 format:
   html:

src/index.qmd

@@ -6,26 +6,32 @@ This is a Quarto implementation of [the Open-Source AI Cookbook](https://github.
 which is a collection of notebooks illustrating practical aspects of building AI 
 applications and solving various machine learning tasks using open-source tools and models.
 
-## About Quarto
+# About Quarto
 [Quarto](https://quarto.org/) is a Markdown-based documentation system which lets you write documents in Markdown or Jupyter Notebooks, and render them to a variety of formats including HTML, PDF, Powerpoint, and more. 
 You can also use Quarto to write [books](https://quarto.org/docs/books/), create [dashboards](https://quarto.org/docs/dashboards/), and embed web applications with [Observable](https://quarto.org/docs/interactive/ojs/) and [Shinylive](https://quarto.org/docs/blog/posts/2022-10-25-shinylive-extension/).
 
+## Executing code
 
-## Latest notebooks
+One of the main virtues of Quarto is that it lets you combine code and text in a single document.
+By default if you include a code chunk in your document, Quarto will execute that code and include the output in the rendered document.
+This is great for reproducibility and for creating documents that are always up-to-date.
+For example you can include code which generates a plot like this:
 
-Check out the recently added notebooks: 
+```{python}
+import seaborn as sns
+import matplotlib.pyplot as plt
 
-- [Automatic Embeddings with TEI through Inference Endpoints](notebooks/automatic_embedding.ipynb)
-- [Simple RAG for GitHub issues using Hugging Face Zephyr and LangChain](notebooks/rag_zephyr_langchain.ipynb)
-- [Embedding multimodal data for similarity search using 🤗 transformers, 🤗 datasets and FAISS](notebooks/faiss.ipynb)
-- [Fine-tuning a Code LLM on Custom Code on a single GPU](notebooks/single_gpu.ipynb)
-- [RAG Evaluation Using Synthetic data and LLM-As-A-Judge](notebooks/rag_evaluation.ipynb)
-- [Advanced RAG on HuggingFace documentation using LangChain](notebooks/advanced_rag.ipynb)
+# Sample data
+tips = sns.load_dataset("tips")
+# Create a seaborn plot
+sns.set_style("whitegrid")
+g = sns.lmplot(x="total_bill", y="tip", data=tips, aspect=2)
+g = g.set_axis_labels("Total bill (USD)", "Tip").set(xlim=(0, 60), ylim=(0, 12))
 
-You can also check out the notebooks in the cookbook's [GitHub repo](https://github.com/huggingface/cookbook).
+plt.title("Tip by Total Bill")
+plt.show()
+```
 
-## Contributing
-
-The Open-Source AI Cookbook is a community effort, and we welcome contributions from everyone! 
-Check out the cookbook's [Contribution guide](https://github.com/huggingface/cookbook/blob/main/README.md) to learn 
-how you can add your "recipe". 
+You can also include [inline code](https://quarto.org/docs/computations/inline-code.html) to insert computed values into text. 
+For example we can reference the `tips` data frame which was defined in the preceding code block by wrapping it in ``{python} tips['tip'].max()``. 
+The ouput of the inline code in inte `{python} tips['tip'].max()`. You can control [code execution](https://quarto.org/docs/computations/execution-options.html), or [freeze code output](https://quarto.org/docs/projects/code-execution.html#freeze) to capture the output of long running computations. 

src/notebooks/advanced_rag.qmd

@@ -0,0 +1,588 @@
+---
+title: Advanced RAG
+jupyter: python3
+eval: false
+code-annotations: hover
+---
+
+This notebook demonstrates how you can build an advanced RAG (Retrieval Augmented Generation) for answering a user's question about a specific knowledge base (here, the HuggingFace documentation), using LangChain.
+
+For an introduction to RAG, you can check [this other cookbook](rag_zephyr_langchain.qmd)!
+
+RAG systems are complex, with many moving parts: here a RAG diagram, where we noted in blue all possibilities for system enhancement:
+
+<img src="https://huggingface.co/datasets/huggingface/cookbook-images/resolve/main/RAG_workflow.png" height="700">
+
+::: callout-note
+💡 As you can see, there are many steps to tune in this architecture: tuning the system properly will yield significant performance gains.
+:::
+
+In this notebook, we will take a look into many of these blue notes to see how to tune your RAG system and get the best performance.
+
+__Let's dig into the model building!__ First, we install the required model dependancies.
+
+```{python}
+!pip install -q torch transformers transformers accelerate bitsandbytes langchain sentence-transformers faiss-gpu openpyxl pacmap
+```
+
+```{python}
+%reload_ext dotenv
+%dotenv
+```
+
+```{python}
+from tqdm.notebook import tqdm
+import pandas as pd
+from typing import Optional, List, Tuple
+from datasets import Dataset
+import matplotlib.pyplot as plt
+
+pd.set_option(
+    "display.max_colwidth", None                    # <1>
+) 
+```
+1. This will be helpful when visualizing retriever outputs
+
+### Load your knowledge base
+
+```{python}
+import datasets
+
+ds = datasets.load_dataset("m-ric/huggingface_doc", split="train")
+```
+
+```{python}
+from langchain.docstore.document import Document as LangchainDocument
+
+RAW_KNOWLEDGE_BASE = [
+    LangchainDocument(page_content=doc["text"], metadata={"source": doc["source"]})
+    for doc in tqdm(ds)
+]
+```
+
+# 1. Retriever - embeddings 🗂️
+The __retriever acts like an internal search engine__: given the user query, it returns a few relevant snippets from your knowledge base.
+
+These snippets will then be fed to the Reader Model to help it generate its answer.
+
+So __our objective here is, given a user question, to find the most snippets from our knowledge base to answer that question.__
+
+This is a wide objective, it leaves open some questions. How many snippets should we retrieve? This parameter will be named `top_k`.
+
+How long should these snippets be? This is called the `chunk size`. There's no one-size-fits-all answers, but here are a few elements:
+- 🔀 Your `chunk size` is allowed to vary from one snippet to the other.
+- Since there will always be some noise in your retrieval, increasing the `top_k` increases the chance to get relevant elements in your retrieved snippets. 🎯 Shooting more arrows increases your probability to hit your target.
+- Meanwhile, the summed length of your retrieved documents should not be too high: for instance, for most current models 16k tokens will probably drown your Reader model in information due to [Lost-in-the-middle phenomenon](https://huggingface.co/papers/2307.03172). 🎯 Give your reader model only the most relevant insights, not a huge pile of books!
+
+::: callout-note
+In this notebook, we use Langchain library since __it offers a huge variety of options for vector databases and allows us to keep document metadata throughout the processing__.
+:::
+
+### 1.1 Split the documents into chunks
+
+- In this part, __we split the documents from our knowledge base into smaller chunks__ which will be the snippets on which the reader LLM will base its answer.
+- The goal is to prepare a collection of **semantically relevant snippets**. So their size should be adapted to precise ideas: too small will truncate ideas, too large will dilute them.
+
+::: callout-tip
+💡 Many options exist for text splitting: splitting on words, on sentence boundaries, recursive chunking that processes documents in a tree-like way to preserve structure information... To learn more about chunking, I recommend you read [this great notebook](https://github.com/FullStackRetrieval-com/RetrievalTutorials/blob/main/5_Levels_Of_Text_Splitting.ipynb) by Greg Kamradt.
+:::
+
+
+- **Recursive chunking** breaks down the text into smaller parts step by step using a given list of separators sorted from the most important to the least important separator. If the first split doesn't give the right size or shape chunks, the method repeats itself on the new chunks using a different separator. For instance with the list of separators `["\n\n", "\n", ".", ""]`:
+    - The method will first break down the document wherever there is a double line break `"\n\n"`.
+    - Resulting documents will be split again on simple line breaks `"\n"`, then on sentence ends `"."`.
+    - And finally, if some chunks are still too big, they will be split whenever they overflow the maximum size.
+
+- With this method, the global structure is well preserved, at the expense of getting slight variations in chunk size.
+
+> [This space](https://huggingface.co/spaces/A-Roucher/chunk_visualizer) lets you visualize how different splitting options affect the chunks you get.
+
+🔬 Let's experiment a bit with chunk sizes, beginning with an arbitrary size, and see how splits work. We use Langchain's implementation of recursive chunking with `RecursiveCharacterTextSplitter`.
+- Parameter `chunk_size` controls the length of individual chunks: this length is counted by default as the number of characters in the chunk.
+- Parameter `chunk_overlap` lets adjacent chunks get a bit of overlap on each other. This reduces the probability that an idea could be cut in half by the split between two adjacent chunks. We ~arbitrarily set this to 1/10th of the chunk size, you could try different values!
+
+```{python}
+from langchain.text_splitter import RecursiveCharacterTextSplitter
+
+# We use a hierarchical list of separators specifically tailored for splitting Markdown documents
+# This list is taken from LangChain's MarkdownTextSplitter class.
+MARKDOWN_SEPARATORS = [
+    "\n#{1,6} ",
+    "```\n",
+    "\n\\*\\*\\*+\n",
+    "\n---+\n",
+    "\n___+\n",
+    "\n\n",
+    "\n",
+    " ",
+    "",
+]
+
+text_splitter = RecursiveCharacterTextSplitter(
+    chunk_size=1000,            # <1>
+    chunk_overlap=100,          # <2>
+    add_start_index=True,       # <3>
+    strip_whitespace=True,      # <4>
+    separators=MARKDOWN_SEPARATORS,
+)
+
+docs_processed = []
+for doc in RAW_KNOWLEDGE_BASE:
+    docs_processed += text_splitter.split_documents([doc])
+```
+1. The maximum number of characters in a chunk: we selected this value arbitrally
+2. The number of characters to overlap between chunks
+3. If `True`, includes chunk's start index in metadata
+4. If `True`, strips whitespace from the start and end of every document
+
+
+We also have to keep in mind that when embedding documents, we will use an embedding model that has accepts a certain maximum sequence length `max_seq_length`.
+
+So we should make sure that our chunk sizes are below this limit, because any longer chunk will be truncated before processing, thus losing relevancy.
+
+```{python}
+#| colab: {referenced_widgets: [ae043feeb0914c879e2a9008b413d952]}
+from sentence_transformers import SentenceTransformer
+
+# To get the value of the max sequence_length, we will query the underlying `SentenceTransformer` object used in the RecursiveCharacterTextSplitter.
+print(
+    f"Model's maximum sequence length: {SentenceTransformer('thenlper/gte-small').max_seq_length}"
+)
+
+from transformers import AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained("thenlper/gte-small")
+lengths = [len(tokenizer.encode(doc.page_content)) for doc in tqdm(docs_processed)]
+
+# Plot the distrubution of document lengths, counted as the number of tokens
+fig = pd.Series(lengths).hist()
+plt.title("Distribution of document lengths in the knowledge base (in count of tokens)")
+plt.show()
+```
+
+👀 As you can see, __the chunk lengths are not aligned with our limit of 512 tokens__, and some documents are above the limit, thus some part of them will be lost in truncation!
+ - So we should change the `RecursiveCharacterTextSplitter` class to count length in number of tokens instead of number of characters.
+ - Then we can choose a specific chunk size, here we would choose a lower threshold than 512:
+    - smaller documents could allow the split to focus more on specific ideas.
+    - But too small chunks would split sentences in half, thus losing meaning again: the proper tuning is a matter of balance.
+
+```{python}
+#| colab: {referenced_widgets: [f900cf4ab3a94f45bfa7298f433566ed]}
+from langchain.text_splitter import RecursiveCharacterTextSplitter
+from transformers import AutoTokenizer
+
+EMBEDDING_MODEL_NAME = "thenlper/gte-small"
+
+
+def split_documents(
+    chunk_size: int,
+    knowledge_base: List[LangchainDocument],
+    tokenizer_name: Optional[str] = EMBEDDING_MODEL_NAME,
+) -> List[LangchainDocument]:
+    """
+    Split documents into chunks of maximum size `chunk_size` tokens and return a list of documents.
+    """
+    text_splitter = RecursiveCharacterTextSplitter.from_huggingface_tokenizer(
+        AutoTokenizer.from_pretrained(tokenizer_name),
+        chunk_size=chunk_size,
+        chunk_overlap=int(chunk_size / 10),
+        add_start_index=True,
+        strip_whitespace=True,
+        separators=MARKDOWN_SEPARATORS,
+    )
+
+    docs_processed = []
+    for doc in knowledge_base:
+        docs_processed += text_splitter.split_documents([doc])
+
+    # Remove duplicates
+    unique_texts = {}
+    docs_processed_unique = []
+    for doc in docs_processed:
+        if doc.page_content not in unique_texts:
+            unique_texts[doc.page_content] = True
+            docs_processed_unique.append(doc)
+
+    return docs_processed_unique
+
+
+docs_processed = split_documents(
+    512,  # We choose a chunk size adapted to our model
+    RAW_KNOWLEDGE_BASE,
+    tokenizer_name=EMBEDDING_MODEL_NAME,
+)
+
+# Let's visualize the chunk sizes we would have in tokens from a common model
+from transformers import AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained(EMBEDDING_MODEL_NAME)
+lengths = [len(tokenizer.encode(doc.page_content)) for doc in tqdm(docs_processed)]
+fig = pd.Series(lengths).hist()
+plt.title("Distribution of document lengths in the knowledge base (in count of tokens)")
+plt.show()
+```
+
+➡️ Now the chunk length distribution looks better!
+
+### 1.2 Building the vector database
+
+We want to compute the embeddings for all the chunks of our knowledge base: to learn more on sentence embeddings, we recommend reading [this guide](https://osanseviero.github.io/hackerllama/blog/posts/sentence_embeddings/).
+
+#### How does retrieval work ?
+
+Once the chunks are all embedded, we store them into a vector database. When the user types in a query, it gets embedded by the same model previously used, and a similarity search returns the closest documents from the vector database.
+
+The technical challenge is thus, given a query vector, to quickly find the nearest neighbours of this vector in the vector database. To do this, we need to choose two things: a distance, and a search algorithm to find the nearest neighbors quickly within a database of thousands of records.
+
+##### Nearest Neighbor search algorithm
+
+There are plentiful choices for the nearest neighbor search algorithm: we go with Facebook's [FAISS](https://github.com/facebookresearch/faiss), since FAISS is performant enough for most use cases, and it is well known thus widely implemented.
+
+##### Distances
+
+Regarding distances, you can find a good guide [here](https://osanseviero.github.io/hackerllama/blog/posts/sentence_embeddings/#distance-between-embeddings). In short:
+
+- **Cosine similarity** computes similarity between two vectors as the cosinus of their relative angle: it allows us to compare vector directions are regardless of their magnitude. Using it requires to normalize all vectors, to rescale them into unit norm.
+- **Dot product** takes into account magnitude, with the sometimes undesirable effect that increasing a vector's length will make it more similar to all others.
+- **Euclidean distance** is the distance between the ends of vectors.
+
+You can try [this small exercise](https://developers.google.com/machine-learning/clustering/similarity/check-your-understanding) to check your understanding of these concepts. But once vectors are normalized, [the choice of a specific distance does not matter much](https://platform.openai.com/docs/guides/embeddings/which-distance-function-should-i-use).
+
+Our particular model works well with cosine similarity, so choose this distance, and we set it up both in the Embedding model, and in the `distance_strategy` argument of our FAISS index. With cosine similarity, we have to normalize our embeddings.
+
+::: {.callout-warning}
+🚨👇 The cell below takes a few minutes to run on A10G!
+:::
+
+```{python}
+from langchain.vectorstores import FAISS
+from langchain_community.embeddings import HuggingFaceEmbeddings
+from langchain_community.vectorstores.utils import DistanceStrategy
+
+embedding_model = HuggingFaceEmbeddings(
+    model_name=EMBEDDING_MODEL_NAME,
+    multi_process=True,
+    model_kwargs={"device": "cuda"},
+    encode_kwargs={"normalize_embeddings": True},  # set True for cosine similarity
+)
+
+KNOWLEDGE_VECTOR_DATABASE = FAISS.from_documents(
+    docs_processed, embedding_model, distance_strategy=DistanceStrategy.COSINE
+)
+```
+
+👀 To visualize the search for the closest documents, let's project our embeddings from 384 dimensions down to 2 dimensions using PaCMAP.
+
+::: {.callout-note}
+💡 We chose PaCMAP rather than other techniques such as t-SNE or UMAP, since [it is efficient (preserves local and global structure), robust to initialization parameters and fast](https://www.nature.com/articles/s42003-022-03628-x#Abs1).
+:::
+
+
+```{python}
+# embed a user query in the same space
+user_query = "How to create a pipeline object?"
+query_vector = embedding_model.embed_query(user_query)
+```
+
+```{python}
+import pacmap
+import numpy as np
+import plotly.express as px
+
+embedding_projector = pacmap.PaCMAP(
+    n_components=2, n_neighbors=None, MN_ratio=0.5, FP_ratio=2.0, random_state=1
+)
+
+embeddings_2d = [
+    list(KNOWLEDGE_VECTOR_DATABASE.index.reconstruct_n(idx, 1)[0])
+    for idx in range(len(docs_processed))
+] + [query_vector]
+
+# fit the data (The index of transformed data corresponds to the index of the original data)
+documents_projected = embedding_projector.fit_transform(np.array(embeddings_2d), init="pca")
+```
+
+```{python}
+df = pd.DataFrame.from_dict(
+    [
+        {
+            "x": documents_projected[i, 0],
+            "y": documents_projected[i, 1],
+            "source": docs_processed[i].metadata["source"].split("/")[1],
+            "extract": docs_processed[i].page_content[:100] + "...",
+            "symbol": "circle",
+            "size_col": 4,
+        }
+        for i in range(len(docs_processed))
+    ]
+    + [
+        {
+            "x": documents_projected[-1, 0],
+            "y": documents_projected[-1, 1],
+            "source": "User query",
+            "extract": user_query,
+            "size_col": 100,
+            "symbol": "star",
+        }
+    ]
+)
+
+# visualize the embedding
+fig = px.scatter(
+    df,
+    x="x",
+    y="y",
+    color="source",
+    hover_data="extract",
+    size="size_col",
+    symbol="symbol",
+    color_discrete_map={"User query": "black"},
+    width=1000,
+    height=700,
+)
+fig.update_traces(
+    marker=dict(opacity=1, line=dict(width=0, color="DarkSlateGrey")), selector=dict(mode="markers")
+)
+fig.update_layout(
+    legend_title_text="<b>Chunk source</b>",
+    title="<b>2D Projection of Chunk Embeddings via PaCMAP</b>",
+)
+fig.show()
+```
+
+<img src="https://huggingface.co/datasets/huggingface/cookbook-images/resolve/main/PaCMAP_embeddings.png" height="700">
+
+
+➡️ On the graph above, you can see a spatial representation of the kowledge base documents. As the vector embeddings represent the document's meaning, their closeness in meaning should be reflected in their embedding's closeness.
+
+The user query's embedding is also shown : we want to find the `k` document that have the closest meaning, thus we pick the `k` closest vectors.
+
+In the LangChain vector database implementation, this search operation is performed by the method `vector_database.similarity_search(query)`.
+
+Here is the result:
+
+```{python}
+print(f"\nStarting retrieval for {user_query=}...")
+retrieved_docs = KNOWLEDGE_VECTOR_DATABASE.similarity_search(query=user_query, k=5)
+print("\n==================================Top document==================================")
+print(retrieved_docs[0].page_content)
+print("==================================Metadata==================================")
+print(retrieved_docs[0].metadata)
+```
+
+# 2. Reader - LLM 💬
+
+In this part, the __LLM Reader reads the retrieved context to formulate its answer.__
+
+There are actually substeps that can all be tuned:
+1. The content of the retrieved documents is aggregated together into the "context", with many processing options like _prompt compression_.
+2. The context and the user query are aggregated into a prompt then given to the LLM to generate its answer.
+
+### 2.1. Reader model
+
+The choice of a reader model is important on a few aspects:
+- the reader model's `max_seq_length` must accomodate our prompt, which includes the context output by the retriever call: the context consists in 5 documents of 512 tokens each, so we aim for a context length of 4k tokens at least.
+- the reader model
+
+For this example, we chose [`HuggingFaceH4/zephyr-7b-beta`](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta), a small but powerful model.
+
+::: callout-note
+With many models being released every week, you may want to substitute this model to the latest and greatest. The best way to keep track of open source LLMs is to check the [Open-source LLM leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard).
+:::
+
+To make inference faster, we will load the quantized version of the model:
+
+```{python}
+#| colab: {referenced_widgets: [db31fd28d3604e78aead26af87b0384f]}
+from transformers import pipeline
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
+
+READER_MODEL_NAME = "HuggingFaceH4/zephyr-7b-beta"
+
+bnb_config = BitsAndBytesConfig(
+    load_in_4bit=True,
+    bnb_4bit_use_double_quant=True,
+    bnb_4bit_quant_type="nf4",
+    bnb_4bit_compute_dtype=torch.bfloat16,
+)
+model = AutoModelForCausalLM.from_pretrained(READER_MODEL_NAME, quantization_config=bnb_config)
+tokenizer = AutoTokenizer.from_pretrained(READER_MODEL_NAME)
+
+READER_LLM = pipeline(
+    model=model,
+    tokenizer=tokenizer,
+    task="text-generation",
+    do_sample=True,
+    temperature=0.2,
+    repetition_penalty=1.1,
+    return_full_text=False,
+    max_new_tokens=500,
+)
+```
+
+```{python}
+READER_LLM("What is 4+4? Answer:")
+```
+
+### 2.2. Prompt
+
+The RAG prompt template below is what we will feed to the Reader LLM: it is important to have it formatted in the Reader LLM's chat template.
+
+We give it our context and the user's question.
+
+```{python}
+prompt_in_chat_format = [
+    {
+        "role": "system",
+        "content": """Using the information contained in the context,
+give a comprehensive answer to the question.
+Respond only to the question asked, response should be concise and relevant to the question.
+Provide the number of the source document when relevant.
+If the answer cannot be deduced from the context, do not give an answer.""",
+    },
+    {
+        "role": "user",
+        "content": """Context:
+{context}
+---
+Now here is the question you need to answer.
+
+Question: {question}""",
+    },
+]
+RAG_PROMPT_TEMPLATE = tokenizer.apply_chat_template(
+    prompt_in_chat_format, tokenize=False, add_generation_prompt=True
+)
+print(RAG_PROMPT_TEMPLATE)
+```
+
+Let's test our Reader on our previously retrieved documents!
+
+```{python}
+retrieved_docs_text = [
+    doc.page_content for doc in retrieved_docs
+]  # we only need the text of the documents
+context = "\nExtracted documents:\n"
+context += "".join([f"Document {str(i)}:::\n" + doc for i, doc in enumerate(retrieved_docs_text)])
+
+final_prompt = RAG_PROMPT_TEMPLATE.format(
+    question="How to create a pipeline object?", context=context
+)
+
+# Redact an answer
+answer = READER_LLM(final_prompt)[0]["generated_text"]
+print(answer)
+```
+
+### 2.3. Reranking
+
+A good option for RAG is to retrieve more documents than you want in the end, then rerank the results with a more powerful retrieval model before keeping only the `top_k`.
+
+For this, [Colbertv2](https://arxiv.org/abs/2112.01488) is a great choice: instead of a bi-encoder like our classical embedding models, it is a cross-encoder that computes more fine-grained interactions between the query tokens and each document's tokens.
+
+It is easily usable thanks to [the RAGatouille library](https://github.com/bclavie/RAGatouille).
+
+```{python}
+from ragatouille import RAGPretrainedModel
+
+RERANKER = RAGPretrainedModel.from_pretrained("colbert-ir/colbertv2.0")
+```
+
+# 3. Assembling it all!
+
+```{python}
+from transformers import Pipeline
+
+
+def answer_with_rag(
+    question: str,
+    llm: Pipeline,
+    knowledge_index: FAISS,
+    reranker: Optional[RAGPretrainedModel] = None,
+    num_retrieved_docs: int = 30,
+    num_docs_final: int = 5,
+) -> Tuple[str, List[LangchainDocument]]:
+    # Gather documents with retriever
+    print("=> Retrieving documents...")
+    relevant_docs = knowledge_index.similarity_search(query=question, k=num_retrieved_docs)
+    relevant_docs = [doc.page_content for doc in relevant_docs]  # keep only the text
+
+    # Optionally rerank results
+    if reranker:
+        print("=> Reranking documents...")
+        relevant_docs = reranker.rerank(question, relevant_docs, k=num_docs_final)
+        relevant_docs = [doc["content"] for doc in relevant_docs]
+
+    relevant_docs = relevant_docs[:num_docs_final]
+
+    # Build the final prompt
+    context = "\nExtracted documents:\n"
+    context += "".join([f"Document {str(i)}:::\n" + doc for i, doc in enumerate(relevant_docs)])
+
+    final_prompt = RAG_PROMPT_TEMPLATE.format(question=question, context=context)
+
+    # Redact an answer
+    print("=> Generating answer...")
+    answer = llm(final_prompt)[0]["generated_text"]
+
+    return answer, relevant_docs
+```
+
+Let's see how our RAG pipeline answers a user query.
+
+```{python}
+question = "how to create a pipeline object?"
+
+answer, relevant_docs = answer_with_rag(
+    question, READER_LLM, KNOWLEDGE_VECTOR_DATABASE, reranker=RERANKER
+)
+```
+
+```{python}
+print("==================================Answer==================================")
+print(f"{answer}")
+print("==================================Source docs==================================")
+for i, doc in enumerate(relevant_docs):
+    print(f"Document {i}------------------------------------------------------------")
+    print(doc)
+```
+
+✅ We now have a fully functional, performant RAG sytem. That's it for today! Congratulations for making it to the end 🥳
+
+
+# To go further 🗺️
+
+This is not the end of the journey! You can try many steps to improve your RAG system. We recommend doing so in an iterative way: bring small changes to the system and see what improves performance.
+
+### Setting up an evaluation pipeline
+
+- 💬 "You cannot improve the model performance that you do not measure", said Gandhi... or at least Llama2 told me he said it. Anyway, you should absolutely start by measuring performance: this means building a small evaluation dataset, then monitor the performance of your RAG system on this evaluation dataset.
+
+### Improving the retriever
+
+🛠️ __You can use these options to tune the results:__
+
+- Tune the chunking method:
+    - Size of the chunks
+    - Method: split on different separators, use [semantic chunking](https://python.langchain.com/docs/modules/data_connection/document_transformers/semantic-chunker)...
+- Change the embedding model
+
+👷‍♀️ __More could be considered:__
+- Try another chunking method, like semantic chunking
+- Change the index used (here, FAISS)
+- Query expansion: reformulate the user query in slightly different ways to retrieve more documents.
+
+### Improving the reader
+
+🛠️ __Here you can try the following options to improve results:__
+- Tune the prompt
+- Switch reranking on/off
+- Choose a more powerful reader model
+
+💡 __Many options could be considered here to further improve the results:__
+- Compress the retrieved context to keep only the most relevant parts to answer the query.
+- Extend the RAG system to make it more user-friendly:
+    - cite source
+    - make conversational
+

src/notebooks/rag_evaluation.ipynb

@@ -1,1470 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "id": "4YErqpfH9jVI"
-   },
-   "source": [
-    "---\n",
-    "title: RAG Evaluation\n",
-    "---\n",
-    "_Authored by: [Aymeric Roucher](https://huggingface.co/m-ric)_\n",
-    "\n",
-    "This notebook demonstrates how you can evaluate your RAG (Retrieval Augmented Generation), by building a synthetic evaluation dataset and using LLM-as-a-judge to compute the accuracy of your system.\n",
-    "\n",
-    "For an introduction to RAG, you can check [this other cookbook](rag_zephyr_langchain)!\n",
-    "\n",
-    "RAG systems are complex: here a RAG diagram, where we noted in blue all possibilities for system enhancement:\n",
-    "\n",
-    "<img src=\"https://huggingface.co/datasets/huggingface/cookbook-images/resolve/main/RAG_workflow.png\" height=\"700\">\n",
-    "\n",
-    "Implementing any of these improvements can bring a huge performance boost; but changing anything is useless if you cannot monitor the impact of your changes on the system's performance!\n",
-    "So let's see how to evaluate our RAG system.\n",
-    "\n",
-    "### Evaluating RAG performance\n",
-    "\n",
-    "Since there are so many moving parts to tune with a big impact on performance, benchmarking the RAG system is crucial.\n",
-    "\n",
-    "For our evaluation pipeline, we will need:\n",
-    "1. An evaluation dataset with question - answer couples (QA couples)\n",
-    "2. An evaluator to compute the accuracy of our system on the above evaluation dataset.\n",
-    "\n",
-    "➡️ It turns out, we can use LLMs to help us all along the way!\n",
-    "1. The evaluation dataset will be synthetically generated by an LLM 🤖, and questions will be filtered out by other LLMs 🤖\n",
-    "2. An [LLM-as-a-judge](https://huggingface.co/papers/2306.05685) agent 🤖 will then perform the evaluation on this synthetic dataset.\n",
-    "\n",
-    "__Let's dig into it and start building our evaluation pipeline!__ First, we install the required model dependancies."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "id": "bCKBvOcp9jVK"
-   },
-   "outputs": [],
-   "source": [
-    "!pip install -q torch transformers transformers langchain sentence-transformers faiss-gpu openpyxl openai"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "id": "k_lJFbYm9jVL"
-   },
-   "outputs": [],
-   "source": [
-    "%reload_ext autoreload\n",
-    "%autoreload 2\n",
-    "%reload_ext dotenv\n",
-    "%dotenv"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "id": "oIlNZ1Mn9jVL"
-   },
-   "outputs": [],
-   "source": [
-    "from tqdm.notebook import tqdm\n",
-    "import pandas as pd\n",
-    "from typing import Optional, List, Tuple\n",
-    "from langchain_core.language_models import BaseChatModel\n",
-    "import json\n",
-    "import datasets\n",
-    "\n",
-    "pd.set_option(\"display.max_colwidth\", None)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "id": "zeW8P62J9jVM"
-   },
-   "source": [
-    "### Load your knowledge base"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "id": "YRbm5tNF9jVM"
-   },
-   "outputs": [],
-   "source": [
-    "ds = datasets.load_dataset(\"m-ric/huggingface_doc\", split=\"train\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "id": "wy9CKj0M9jVM"
-   },
-   "source": [
-    "# 1. Build a synthetic dataset for evaluation\n",
-    "We first build a synthetic dataset of questions and associated contexts. The method is to get elements from our knowledge base, and ask an LLM to generate questions based on these documents.\n",
-    "\n",
-    "Then we setup other LLM agents to act as quality filters for the generated QA couples: each of them will act as the filter for a specific flaw."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "id": "QkoEgiDg9jVM"
-   },
-   "source": [
-    "### 1.1. Prepare source documents"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "id": "3gTOlRKO9jVM"
-   },
-   "outputs": [],
-   "source": [
-    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
-    "from langchain.docstore.document import Document as LangchainDocument\n",
-    "\n",
-    "langchain_docs = [\n",
-    "    LangchainDocument(page_content=doc[\"text\"], metadata={\"source\": doc[\"source\"]})\n",
-    "    for doc in tqdm(ds)\n",
-    "]\n",
-    "\n",
-    "\n",
-    "text_splitter = RecursiveCharacterTextSplitter(\n",
-    "    chunk_size=2000,\n",
-    "    chunk_overlap=200,\n",
-    "    add_start_index=True,\n",
-    "    separators=[\"\\n\\n\", \"\\n\", \".\", \" \", \"\"],\n",
-    ")\n",
-    "\n",
-    "docs_processed = []\n",
-    "for doc in langchain_docs:\n",
-    "    docs_processed += text_splitter.split_documents([doc])"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "id": "WjrNhcCh9jVN"
-   },
-   "source": [
-    "### 1.2. Setup agents for question generation\n",
-    "\n",
-    "We use [Mixtral](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1) for QA couple generation because it it has excellent performance in leaderboards such as [Chatbot Arena](https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard)."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "id": "GoRySj3Q9jVN"
-   },
-   "outputs": [],
-   "source": [
-    "from langchain_community.llms import HuggingFaceHub\n",
-    "\n",
-    "repo_id = \"mistralai/Mixtral-8x7B-Instruct-v0.1\"\n",
-    "\n",
-    "llm = HuggingFaceHub(\n",
-    "    repo_id=repo_id,\n",
-    "    task=\"text-generation\",\n",
-    "    model_kwargs={\n",
-    "        \"max_new_tokens\": 512,\n",
-    "        \"top_k\": 30,\n",
-    "        \"temperature\": 0.1,\n",
-    "        \"repetition_penalty\": 1.03,\n",
-    "    },\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "id": "wubTNTaV9jVN"
-   },
-   "outputs": [],
-   "source": [
-    "from langchain_community.chat_models import ChatHuggingFace\n",
-    "\n",
-    "chat_model = ChatHuggingFace(llm=llm)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "id": "hIM_DJRo9jVN"
-   },
-   "outputs": [],
-   "source": [
-    "from langchain.prompts import ChatPromptTemplate\n",
-    "\n",
-    "QA_generation_prompt = \"\"\"\n",
-    "Your task is to write a factoid question and an answer given a context.\n",
-    "Your factoid question should be answerable with a specific, concise piece of factual information from the context.\n",
-    "Your factoid question should be formulated in the same style as questions users could ask in a search engine.\n",
-    "This means that your factoid question MUST NOT mention something like \"according to the passage\" or \"context\".\n",
-    "\n",
-    "Provide your answer as follows:\n",
-    "\n",
-    "Output:::\n",
-    "Factoid question: (your factoid question)\n",
-    "Answer: (your answer to the factoid question)\n",
-    "\n",
-    "Now here is the context.\n",
-    "\n",
-    "Context: {context}\\n\n",
-    "Output:::\"\"\"\n",
-    "\n",
-    "QA_generation_prompt = ChatPromptTemplate.from_template(QA_generation_prompt)\n",
-    "QA_generation_agent = QA_generation_prompt | chat_model"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "id": "lVFc-lVy9jVN"
-   },
-   "source": [
-    "Now let's generate our QA couples.\n",
-    "For this example, we generate only 10 QA couples and will load the rest from the Hub.\n",
-    "\n",
-    "But for your specific knowledge base, given that you want to get at least ~100 test samples, and accounting for the fact that we will filter out around half of these with our critique agents later on, you should generate much more, in the >200 samples."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "id": "8fteqDDD9jVN"
-   },
-   "outputs": [],
-   "source": [
-    "import random\n",
-    "\n",
-    "N_GENERATIONS = (\n",
-    "    10  # We intentionally generate only 10 QA couples here for cost and time considerations\n",
-    ")\n",
-    "\n",
-    "print(f\"Generating {N_GENERATIONS} QA couples...\")\n",
-    "outputs = []\n",
-    "for context in tqdm(random.sample(langchain_docs, N_GENERATIONS)):\n",
-    "    # Generate QA couple\n",
-    "    output_QA_couple = QA_generation_agent.invoke({\"context\": context.page_content}).content\n",
-    "    try:\n",
-    "        question = output_QA_couple.split(\"Factoid question: \")[1].split(\"Answer: \")[0]\n",
-    "        answer = output_QA_couple.split(\"Answer: \")[1]\n",
-    "        outputs.append(\n",
-    "            {\n",
-    "                \"context\": context.page_content,\n",
-    "                \"question\": question,\n",
-    "                \"answer\": answer,\n",
-    "                \"source_doc\": context.metadata[\"source\"],\n",
-    "            }\n",
-    "        )\n",
-    "    except:\n",
-    "        continue"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "id": "aUlOUDv59jVN",
-    "outputId": "c9634fdb-2a7f-43a6-c4eb-e60b166b8238"
-   },
-   "outputs": [
-    {
-     "data": {
-      "text/html": [
-       "<div>\n",
-       "<style scoped>\n",
-       "    .dataframe tbody tr th:only-of-type {\n",
-       "        vertical-align: middle;\n",
-       "    }\n",
-       "\n",
-       "    .dataframe tbody tr th {\n",
-       "        vertical-align: top;\n",
-       "    }\n",
-       "\n",
-       "    .dataframe thead th {\n",
-       "        text-align: right;\n",
-       "    }\n",
-       "</style>\n",
-       "<table border=\"1\" class=\"dataframe\">\n",
-       "  <thead>\n",
-       "    <tr style=\"text-align: right;\">\n",
-       "      <th></th>\n",
-       "      <th>context</th>\n",
-       "      <th>question</th>\n",
-       "      <th>answer</th>\n",
-       "      <th>source_doc</th>\n",
-       "    </tr>\n",
-       "  </thead>\n",
-       "  <tbody>\n",
-       "    <tr>\n",
-       "      <th>0</th>\n",
-       "      <td>!--Copyright 2023 The HuggingFace Team. All rights reserved.\\n\\nLicensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with\\nthe License. You may obtain a copy of the License at\\n\\nhttp://www.apache.org/licenses/LICENSE-2.0\\n\\nUnless required by applicable law or agreed to in writing, software distributed under the License is distributed on\\nan \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the\\nspecific language governing permissions and limitations under the License.\\n--&gt;\\n\\n# Schedulers\\n\\n🤗 Diffusers provides many scheduler functions for the diffusion process. A scheduler takes a model's output (the sample which the diffusion process is iterating on) and a timestep to return a denoised sample. The timestep is important because it dictates where in the diffusion process the step is; data is generated by iterating forward *n* timesteps and inference occurs by propagating backward through the timesteps. Based on the timestep, a scheduler may be *discrete* in which case the timestep is an `int` or *continuous* in which case the timestep is a `float`.\\n\\nDepending on the context, a scheduler defines how to iteratively add noise to an image or how to update a sample based on a model's output:\\n\\n- during *training*, a scheduler adds noise (there are different algorithms for how to add noise) to a sample to train a diffusion model\\n- during *inference*, a scheduler defines how to update a sample based on a pretrained model's output\\n\\nMany schedulers are implemented from the [k-diffusion](https://github.com/crowsonkb/k-diffusion) library by [Katherine Crowson](https://github.com/crowsonkb/), and they're also widely used in A1111. To help you map the schedulers from k-diffusion and A1111 to the schedulers in 🤗 Diffusers, take a look at the table below:\\n\\n| A1111/k-diffusion    | 🤗 Diffusers                         | Usage                                                                                                         |\\n|---------------------|-------------------------------------|---------------------------------------------------------------------------------------------------------------|\\n| DPM++ 2M            | [`DPMSolverMultistepScheduler`]     |                                                                                                               |\\n| DPM++ 2M Karras     | [`DPMSolverMultistepScheduler`]     | init with `use_karras_sigmas=True`                                                                            |\\n| DPM++ 2M SDE        | [`DPMSolverMultistepScheduler`]     | init with `algorithm_type=\"sde-dpmsolver++\"`                                                                  |\\n| DPM++ 2M SDE Karras | [`DPMSolverMultistepScheduler`]     | init with `use_karras_sigmas=True` and `algorithm_type=\"sde-dpmsolver++\"`                                     |\\n| DPM++ 2S a          | N/A                                 | very similar to  `DPMSolverSinglestepScheduler`                         |\\n| DPM++ 2S a Karras   | N/A                                 | very similar to  `DPMSolverSinglestepScheduler(use_karras_sigmas=True, ...)` |\\n| DPM++ SDE           | [`DPMSolverSinglestepScheduler`]    |                                                                                                               |\\n| DPM++ SDE Karras    | [`DPMSolverSinglestepScheduler`]    | init with `use_karras_sigmas=True`                                                                            |\\n| DPM2                | [`KDPM2DiscreteScheduler`]          |                                                                                                               |\\n| DPM2 Karras         | [`KDPM2DiscreteScheduler`]          | init with `use_karras_sigmas=True`                                                                            |\\n| DPM2 a              | [`KDPM2AncestralDiscreteScheduler`] |                                                                                                               |\\n| DPM2 a Karras       | [`KDPM2AncestralDiscreteScheduler`] | init with `use_karras_sigmas=True`                                                                            |\\n| DPM adaptive        | N/A                                 |                                                                                                               |\\n| DPM fast            | N/A                                 |                                                                                                               |\\n| Euler               | [`EulerDiscreteScheduler`]          |                                                                                                               |\\n| Euler a             | [`EulerAncestralDiscreteScheduler`] |                                                                                                               |\\n| Heun                | [`HeunDiscreteScheduler`]           |                                                                                                               |\\n| LMS                 | [`LMSDiscreteScheduler`]            |                                                                                                               |\\n| LMS Karras          | [`LMSDiscreteScheduler`]            | init with `use_karras_sigmas=True`                                                                            |\\n| N/A                 | [`DEISMultistepScheduler`]          |                                                                                                               |\\n| N/A                 | [`UniPCMultistepScheduler`]         |                                                                                                               |\\n\\nAll schedulers are built from the base [`SchedulerMixin`] class which implements low level utilities shared by all schedulers.\\n\\n## SchedulerMixin\\n[[autodoc]] SchedulerMixin\\n\\n## SchedulerOutput\\n[[autodoc]] schedulers.scheduling_utils.SchedulerOutput\\n\\n## KarrasDiffusionSchedulers\\n\\n[`KarrasDiffusionSchedulers`] are a broad generalization of schedulers in 🤗 Diffusers. The schedulers in this class are distinguished at a high level by their noise sampling strategy, the type of network and scaling, the training strategy, and how the loss is weighed.\\n\\nThe different schedulers in this class, depending on the ordinary differential equations (ODE) solver type, fall into the above taxonomy and provide a good abstraction for the design of the main schedulers implemented in 🤗 Diffusers. The schedulers in this class are given [here](https://github.com/huggingface/diffusers/blob/a69754bb879ed55b9b6dc9dd0b3cf4fa4124c765/src/diffusers/schedulers/scheduling_utils.py#L32).\\n\\n## PushToHubMixin\\n\\n[[autodoc]] utils.PushToHubMixin\\n</td>\n",
-       "      <td>What is the class of schedulers in 🤗 Diffusers that are distinguished by their noise sampling strategy, type of network and scaling, training strategy, and loss weighing?\\n</td>\n",
-       "      <td>[`KarrasDiffusionSchedulers`]</td>\n",
-       "      <td>huggingface/diffusers/blob/main/docs/source/en/api/schedulers/overview.md</td>\n",
-       "    </tr>\n",
-       "  </tbody>\n",
-       "</table>\n",
-       "</div>"
-      ],
-      "text/plain": [
-       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          context  \\\n",
-       "0  !--Copyright 2023 The HuggingFace Team. All rights reserved.\\n\\nLicensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with\\nthe License. You may obtain a copy of the License at\\n\\nhttp://www.apache.org/licenses/LICENSE-2.0\\n\\nUnless required by applicable law or agreed to in writing, software distributed under the License is distributed on\\nan \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the\\nspecific language governing permissions and limitations under the License.\\n-->\\n\\n# Schedulers\\n\\n🤗 Diffusers provides many scheduler functions for the diffusion process. A scheduler takes a model's output (the sample which the diffusion process is iterating on) and a timestep to return a denoised sample. The timestep is important because it dictates where in the diffusion process the step is; data is generated by iterating forward *n* timesteps and inference occurs by propagating backward through the timesteps. Based on the timestep, a scheduler may be *discrete* in which case the timestep is an `int` or *continuous* in which case the timestep is a `float`.\\n\\nDepending on the context, a scheduler defines how to iteratively add noise to an image or how to update a sample based on a model's output:\\n\\n- during *training*, a scheduler adds noise (there are different algorithms for how to add noise) to a sample to train a diffusion model\\n- during *inference*, a scheduler defines how to update a sample based on a pretrained model's output\\n\\nMany schedulers are implemented from the [k-diffusion](https://github.com/crowsonkb/k-diffusion) library by [Katherine Crowson](https://github.com/crowsonkb/), and they're also widely used in A1111. To help you map the schedulers from k-diffusion and A1111 to the schedulers in 🤗 Diffusers, take a look at the table below:\\n\\n| A1111/k-diffusion    | 🤗 Diffusers                         | Usage                                                                                                         |\\n|---------------------|-------------------------------------|---------------------------------------------------------------------------------------------------------------|\\n| DPM++ 2M            | [`DPMSolverMultistepScheduler`]     |                                                                                                               |\\n| DPM++ 2M Karras     | [`DPMSolverMultistepScheduler`]     | init with `use_karras_sigmas=True`                                                                            |\\n| DPM++ 2M SDE        | [`DPMSolverMultistepScheduler`]     | init with `algorithm_type=\"sde-dpmsolver++\"`                                                                  |\\n| DPM++ 2M SDE Karras | [`DPMSolverMultistepScheduler`]     | init with `use_karras_sigmas=True` and `algorithm_type=\"sde-dpmsolver++\"`                                     |\\n| DPM++ 2S a          | N/A                                 | very similar to  `DPMSolverSinglestepScheduler`                         |\\n| DPM++ 2S a Karras   | N/A                                 | very similar to  `DPMSolverSinglestepScheduler(use_karras_sigmas=True, ...)` |\\n| DPM++ SDE           | [`DPMSolverSinglestepScheduler`]    |                                                                                                               |\\n| DPM++ SDE Karras    | [`DPMSolverSinglestepScheduler`]    | init with `use_karras_sigmas=True`                                                                            |\\n| DPM2                | [`KDPM2DiscreteScheduler`]          |                                                                                                               |\\n| DPM2 Karras         | [`KDPM2DiscreteScheduler`]          | init with `use_karras_sigmas=True`                                                                            |\\n| DPM2 a              | [`KDPM2AncestralDiscreteScheduler`] |                                                                                                               |\\n| DPM2 a Karras       | [`KDPM2AncestralDiscreteScheduler`] | init with `use_karras_sigmas=True`                                                                            |\\n| DPM adaptive        | N/A                                 |                                                                                                               |\\n| DPM fast            | N/A                                 |                                                                                                               |\\n| Euler               | [`EulerDiscreteScheduler`]          |                                                                                                               |\\n| Euler a             | [`EulerAncestralDiscreteScheduler`] |                                                                                                               |\\n| Heun                | [`HeunDiscreteScheduler`]           |                                                                                                               |\\n| LMS                 | [`LMSDiscreteScheduler`]            |                                                                                                               |\\n| LMS Karras          | [`LMSDiscreteScheduler`]            | init with `use_karras_sigmas=True`                                                                            |\\n| N/A                 | [`DEISMultistepScheduler`]          |                                                                                                               |\\n| N/A                 | [`UniPCMultistepScheduler`]         |                                                                                                               |\\n\\nAll schedulers are built from the base [`SchedulerMixin`] class which implements low level utilities shared by all schedulers.\\n\\n## SchedulerMixin\\n[[autodoc]] SchedulerMixin\\n\\n## SchedulerOutput\\n[[autodoc]] schedulers.scheduling_utils.SchedulerOutput\\n\\n## KarrasDiffusionSchedulers\\n\\n[`KarrasDiffusionSchedulers`] are a broad generalization of schedulers in 🤗 Diffusers. The schedulers in this class are distinguished at a high level by their noise sampling strategy, the type of network and scaling, the training strategy, and how the loss is weighed.\\n\\nThe different schedulers in this class, depending on the ordinary differential equations (ODE) solver type, fall into the above taxonomy and provide a good abstraction for the design of the main schedulers implemented in 🤗 Diffusers. The schedulers in this class are given [here](https://github.com/huggingface/diffusers/blob/a69754bb879ed55b9b6dc9dd0b3cf4fa4124c765/src/diffusers/schedulers/scheduling_utils.py#L32).\\n\\n## PushToHubMixin\\n\\n[[autodoc]] utils.PushToHubMixin\\n   \n",
-       "\n",
-       "                                                                                                                                                                       question  \\\n",
-       "0  What is the class of schedulers in 🤗 Diffusers that are distinguished by their noise sampling strategy, type of network and scaling, training strategy, and loss weighing?\\n   \n",
-       "\n",
-       "                          answer  \\\n",
-       "0  [`KarrasDiffusionSchedulers`]   \n",
-       "\n",
-       "                                                                  source_doc  \n",
-       "0  huggingface/diffusers/blob/main/docs/source/en/api/schedulers/overview.md  "
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    }
-   ],
-   "source": [
-    "display(pd.DataFrame(outputs).head(1))"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "id": "0KG4dNtg9jVN"
-   },
-   "source": [
-    "### 1.3. Setup critique agents\n",
-    "\n",
-    "The questions generated by the previous agent can have many flaws: we should do a quality check before validating these questions.\n",
-    "\n",
-    "We thus build critique agents that will rate each question on several criteria, given in [this paper](https://huggingface.co/papers/2312.10003):\n",
-    "- **Groundedness:** can the question be answered from the given context?\n",
-    "- **Relevance:** is the question relevant to users? For instance, `\"What is the date when transformers 4.29.1 was released?\"` is not relevant for ML practicioners.\n",
-    "\n",
-    "One last failure case we've noticed is when a function is tailored for the particular setting where the question was generated, but undecipherable by itself, like `\"What is the name of the function used in this guide?\"`.\n",
-    "We also build a critique agent for this criteria:\n",
-    "- **Stand-alone**: is the question understandable free of any context, for someone with domain knowledge/Internet access? The opposite of this would be `What is the function used in this article?` for a question generated from a specific blog article.\n",
-    "\n",
-    "We systematically score functions with all these agents, and whenever the score is too low for any one of the agents, we eliminate the question from our eval dataset.\n",
-    "\n",
-    "💡 ___When asking the agents to output a score, we first ask them to produce its rationale. This will help us verify scores, but most importantly, asking it to first output rationale gives the model more tokens to think and elaborate an answer before summarizing it into a single score token.___\n",
-    "\n",
-    "We now build and run these critique agents."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "id": "05aSgTGs9jVO"
-   },
-   "outputs": [],
-   "source": [
-    "question_groundedness_critique_prompt = \"\"\"\n",
-    "You will be given a context and a question.\n",
-    "Your task is to provide a 'total rating' scoring how well one can answer the given question unambiguously with the given context.\n",
-    "Give your answer on a scale of 1 to 5, where 1 means that the question is not answerable at all given the context, and 5 means that the question is clearly and unambiguously answerable with the context.\n",
-    "\n",
-    "Provide your answer as follows:\n",
-    "\n",
-    "Answer:::\n",
-    "Evaluation: (your rationale for the rating)\n",
-    "Total rating: (your rating)\n",
-    "\n",
-    "Now here are the question and context.\n",
-    "\n",
-    "Question: {question}\\n\n",
-    "Context: {context}\\n\n",
-    "Answer::: \"\"\"\n",
-    "\n",
-    "question_relevance_critique_prompt = \"\"\"\n",
-    "You will be given a question.\n",
-    "Your task is to provide a 'total rating' representing how useful this question can be to machine learning developers building NLP applications with the Hugging Face ecosystem.\n",
-    "Give your answer on a scale of 1 to 5, where 1 means that the question is not useful at all, and 5 means that the question is extremely useful.\n",
-    "\n",
-    "Provide your answer as follows:\n",
-    "\n",
-    "Answer:::\n",
-    "Evaluation: (your rationale for the rating)\n",
-    "Total rating: (your rating)\n",
-    "\n",
-    "Now here is the question.\n",
-    "\n",
-    "Question: {question}\\n\n",
-    "Answer::: \"\"\"\n",
-    "\n",
-    "question_standalone_critique_prompt = \"\"\"\n",
-    "You will be given a question.\n",
-    "Your task is to provide a 'total rating' representing how context-independant this question is.\n",
-    "Give your answer on a scale of 1 to 5, where 1 means that the question only makes sense in a specific context, and 5 means that the question makes sense by itself.\n",
-    "For instance, if the question refers to a particular setting, like 'in the context' or 'in the document', the rating must be 1.\n",
-    "The questions can contain obscure technical nouns or acronyms like Gradio, Hub, Hugging Face or Space and still be a 5: it must simply be clear to an operator with access to documentation what the question is about.\n",
-    "\n",
-    "Provide your answer as follows:\n",
-    "\n",
-    "Answer:::\n",
-    "Evaluation: (your rationale for the rating)\n",
-    "Total rating: (your rating)\n",
-    "\n",
-    "Now here is the question.\n",
-    "\n",
-    "Question: {question}\\n\n",
-    "Answer::: \"\"\"\n",
-    "\n",
-    "question_groundedness_critique_prompt = ChatPromptTemplate.from_template(\n",
-    "    question_groundedness_critique_prompt\n",
-    ")\n",
-    "question_groundedness_critique_agent = question_groundedness_critique_prompt | chat_model\n",
-    "\n",
-    "question_relevance_critique_prompt = ChatPromptTemplate.from_template(\n",
-    "    question_relevance_critique_prompt\n",
-    ")\n",
-    "question_relevance_critique_agent = question_relevance_critique_prompt | chat_model\n",
-    "\n",
-    "question_standalone_critique_prompt = ChatPromptTemplate.from_template(\n",
-    "    question_standalone_critique_prompt\n",
-    ")\n",
-    "question_standalone_critique_agent = question_standalone_critique_prompt | chat_model"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "id": "b9tbk7ME9jVO"
-   },
-   "outputs": [],
-   "source": [
-    "print(\"Generating critique for each QA couple...\")\n",
-    "for output in tqdm(outputs):\n",
-    "    # Critique the generated QA couple\n",
-    "    question_groundedness_evaluation = question_groundedness_critique_agent.invoke(\n",
-    "        {\"context\": output[\"context\"], \"question\": output[\"question\"]}\n",
-    "    ).content\n",
-    "    question_relevance_evaluation = question_relevance_critique_agent.invoke(\n",
-    "        {\"question\": output[\"question\"]}\n",
-    "    ).content\n",
-    "    question_standalone_evaluation = question_standalone_critique_agent.invoke(\n",
-    "        {\"question\": output[\"question\"]}\n",
-    "    ).content\n",
-    "\n",
-    "    try:\n",
-    "        groundedness_score = int(question_groundedness_evaluation.split(\"Total rating: \")[1][0])\n",
-    "        groundedness_eval = question_groundedness_evaluation.split(\"Total rating: \")[0].split(\n",
-    "            \"Evaluation: \"\n",
-    "        )[1]\n",
-    "        relevance_score = int(question_relevance_evaluation.split(\"Total rating: \")[1][0])\n",
-    "        relevance_eval = question_relevance_evaluation.split(\"Total rating: \")[0].split(\n",
-    "            \"Evaluation: \"\n",
-    "        )[1]\n",
-    "        standalone_score = int(question_standalone_evaluation.split(\"Total rating: \")[1][0])\n",
-    "        standalone_eval = question_standalone_evaluation.split(\"Total rating: \")[0].split(\n",
-    "            \"Evaluation: \"\n",
-    "        )[1]\n",
-    "        output.update(\n",
-    "            {\n",
-    "                \"groundedness_score\": groundedness_score,\n",
-    "                \"groundedness_eval\": groundedness_eval,\n",
-    "                \"relevance_score\": relevance_score,\n",
-    "                \"relevance_eval\": relevance_eval,\n",
-    "                \"standalone_score\": standalone_score,\n",
-    "                \"standalone_eval\": standalone_eval,\n",
-    "            }\n",
-    "        )\n",
-    "    except:\n",
-    "        continue"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "id": "IQv36Y_f9jVO"
-   },
-   "source": [
-    "Now let us filter out bad questions based on our critique agent scores:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "id": "oBWuOu1b9jVO",
-    "outputId": "b32bacea-52f8-486a-96fe-5c188605c5a2"
-   },
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Evaluation dataset before filtering:\n"
-     ]
-    },
-    {
-     "data": {
-      "text/html": [
-       "<div>\n",
-       "<style scoped>\n",
-       "    .dataframe tbody tr th:only-of-type {\n",
-       "        vertical-align: middle;\n",
-       "    }\n",
-       "\n",
-       "    .dataframe tbody tr th {\n",
-       "        vertical-align: top;\n",
-       "    }\n",
-       "\n",
-       "    .dataframe thead th {\n",
-       "        text-align: right;\n",
-       "    }\n",
-       "</style>\n",
-       "<table border=\"1\" class=\"dataframe\">\n",
-       "  <thead>\n",
-       "    <tr style=\"text-align: right;\">\n",
-       "      <th></th>\n",
-       "      <th>question</th>\n",
-       "      <th>answer</th>\n",
-       "      <th>groundedness_score</th>\n",
-       "      <th>relevance_score</th>\n",
-       "      <th>standalone_score</th>\n",
-       "    </tr>\n",
-       "  </thead>\n",
-       "  <tbody>\n",
-       "    <tr>\n",
-       "      <th>0</th>\n",
-       "      <td>What is the class of schedulers in 🤗 Diffusers that are distinguished by their noise sampling strategy, type of network and scaling, training strategy, and loss weighing?\\n</td>\n",
-       "      <td>[`KarrasDiffusionSchedulers`]</td>\n",
-       "      <td>3.0</td>\n",
-       "      <td>1.0</td>\n",
-       "      <td>4.0</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <th>1</th>\n",
-       "      <td>What are some utility functions provided by the Hugging Face library for pipelines?\\n</td>\n",
-       "      <td>The Hugging Face library provides several utility functions for pipelines, including `ArgumentHandler`, `ZeroShotClassificationArgumentHandler`, `QuestionAnsweringArgumentHandler` for argument handling, `PipelineDataFormat`, `CsvPipelineDataFormat`, `JsonPipelineDataFormat`, `PipedPipelineDataFormat` for data format, and `PipelineException` for exceptions.</td>\n",
-       "      <td>5.0</td>\n",
-       "      <td>4.0</td>\n",
-       "      <td>5.0</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <th>2</th>\n",
-       "      <td>What is the default name used in the Gradio demo if no name is provided?\\n</td>\n",
-       "      <td>User\\n\\nExplanation: The factoid question asks for the default name used in the Gradio demo if no name is provided. The answer to this question can be found in the `argparse.ArgumentParser()` function, where a default value of \"User\" is set for the `--name` argument.</td>\n",
-       "      <td>5.0</td>\n",
-       "      <td>3.0</td>\n",
-       "      <td>5.0</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <th>3</th>\n",
-       "      <td>What is the function used to load a pre-trained Resnet-18 model in the provided context?\\n</td>\n",
-       "      <td>The function used to load a pre-trained Resnet-18 model in the provided context is `torch.hub.load('pytorch/vision:v0.6.0', 'resnet18', pretrained=True).eval()`.</td>\n",
-       "      <td>NaN</td>\n",
-       "      <td>NaN</td>\n",
-       "      <td>NaN</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <th>4</th>\n",
-       "      <td>What is the name of the component used for creating a button in the given code?\\n</td>\n",
-       "      <td>The name of the component used for creating a button in the given code is `BaseButton`.</td>\n",
-       "      <td>5.0</td>\n",
-       "      <td>1.0</td>\n",
-       "      <td>5.0</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <th>5</th>\n",
-       "      <td>What is the command to get the example ONNX file for Bart model?\\n</td>\n",
-       "      <td>The command is `python run_onnx_exporter.py --model_name_or_path facebook/bart-base`.</td>\n",
-       "      <td>NaN</td>\n",
-       "      <td>NaN</td>\n",
-       "      <td>NaN</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <th>6</th>\n",
-       "      <td>What will be covered in the next unit of the course?\\n</td>\n",
-       "      <td>The next unit of the course will cover learning more about Unity MLAgents and training agents in Unity environments. It will also prepare students for AI vs AI challenges where they will train their agents to compete against other agents in a snowball fight and a soccer game.</td>\n",
-       "      <td>5.0</td>\n",
-       "      <td>1.0</td>\n",
-       "      <td>5.0</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <th>7</th>\n",
-       "      <td>What is the purpose of the `negative_original_size`, `negative_crops_coords_top_left`, and `negative_target_size` parameters in SDXL?\\n</td>\n",
-       "      <td>These parameters allow SDXL to negatively condition the model on image resolution and cropping parameters.</td>\n",
-       "      <td>2.0</td>\n",
-       "      <td>4.0</td>\n",
-       "      <td>2.0</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <th>8</th>\n",
-       "      <td>How are transformers models tested in the Hugging Face repository?\\n</td>\n",
-       "      <td>Transformers models are tested in the Hugging Face repository using two test suites: `tests` for the general API and `examples` for various applications that aren't part of the API. These tests are run on CircleCI and GitHub Actions, with different jobs and configurations for each. The tests can be run in various ways, including running all tests, getting the list of all tests, running a specific test module, and running specific tests by name or keyword expression. Additionally, there are options for running tests in parallel, repeating tests, and running tests on a specific GPU or CPU.</td>\n",
-       "      <td>3.0</td>\n",
-       "      <td>4.0</td>\n",
-       "      <td>4.0</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <th>9</th>\n",
-       "      <td>What command is used to create a virtual environment in the given context?\\n</td>\n",
-       "      <td>The command used to create a virtual environment in the given context is `python -m venv &lt;env_name&gt;`.</td>\n",
-       "      <td>NaN</td>\n",
-       "      <td>NaN</td>\n",
-       "      <td>NaN</td>\n",
-       "    </tr>\n",
-       "  </tbody>\n",
-       "</table>\n",
-       "</div>"
-      ],
-      "text/plain": [
-       "                                                                                                                                                                       question  \\\n",
-       "0  What is the class of schedulers in 🤗 Diffusers that are distinguished by their noise sampling strategy, type of network and scaling, training strategy, and loss weighing?\\n   \n",
-       "1                                                                                         What are some utility functions provided by the Hugging Face library for pipelines?\\n   \n",
-       "2                                                                                                    What is the default name used in the Gradio demo if no name is provided?\\n   \n",
-       "3                                                                                    What is the function used to load a pre-trained Resnet-18 model in the provided context?\\n   \n",
-       "4                                                                                             What is the name of the component used for creating a button in the given code?\\n   \n",
-       "5                                                                                                            What is the command to get the example ONNX file for Bart model?\\n   \n",
-       "6                                                                                                                        What will be covered in the next unit of the course?\\n   \n",
-       "7                                       What is the purpose of the `negative_original_size`, `negative_crops_coords_top_left`, and `negative_target_size` parameters in SDXL?\\n   \n",
-       "8                                                                                                          How are transformers models tested in the Hugging Face repository?\\n   \n",
-       "9                                                                                                  What command is used to create a virtual environment in the given context?\\n   \n",
-       "\n",
-       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               answer  \\\n",
-       "0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       [`KarrasDiffusionSchedulers`]   \n",
-       "1                                                                                                                                                                                                                                              The Hugging Face library provides several utility functions for pipelines, including `ArgumentHandler`, `ZeroShotClassificationArgumentHandler`, `QuestionAnsweringArgumentHandler` for argument handling, `PipelineDataFormat`, `CsvPipelineDataFormat`, `JsonPipelineDataFormat`, `PipedPipelineDataFormat` for data format, and `PipelineException` for exceptions.   \n",
-       "2                                                                                                                                                                                                                                                                                                                                         User\\n\\nExplanation: The factoid question asks for the default name used in the Gradio demo if no name is provided. The answer to this question can be found in the `argparse.ArgumentParser()` function, where a default value of \"User\" is set for the `--name` argument.   \n",
-       "3                                                                                                                                                                                                                                                                                                                                                                                                                                                   The function used to load a pre-trained Resnet-18 model in the provided context is `torch.hub.load('pytorch/vision:v0.6.0', 'resnet18', pretrained=True).eval()`.   \n",
-       "4                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             The name of the component used for creating a button in the given code is `BaseButton`.   \n",
-       "5                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               The command is `python run_onnx_exporter.py --model_name_or_path facebook/bart-base`.   \n",
-       "6                                                                                                                                                                                                                                                                                                                                The next unit of the course will cover learning more about Unity MLAgents and training agents in Unity environments. It will also prepare students for AI vs AI challenges where they will train their agents to compete against other agents in a snowball fight and a soccer game.   \n",
-       "7                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          These parameters allow SDXL to negatively condition the model on image resolution and cropping parameters.   \n",
-       "8  Transformers models are tested in the Hugging Face repository using two test suites: `tests` for the general API and `examples` for various applications that aren't part of the API. These tests are run on CircleCI and GitHub Actions, with different jobs and configurations for each. The tests can be run in various ways, including running all tests, getting the list of all tests, running a specific test module, and running specific tests by name or keyword expression. Additionally, there are options for running tests in parallel, repeating tests, and running tests on a specific GPU or CPU.   \n",
-       "9                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               The command used to create a virtual environment in the given context is `python -m venv <env_name>`.   \n",
-       "\n",
-       "   groundedness_score  relevance_score  standalone_score  \n",
-       "0                 3.0              1.0               4.0  \n",
-       "1                 5.0              4.0               5.0  \n",
-       "2                 5.0              3.0               5.0  \n",
-       "3                 NaN              NaN               NaN  \n",
-       "4                 5.0              1.0               5.0  \n",
-       "5                 NaN              NaN               NaN  \n",
-       "6                 5.0              1.0               5.0  \n",
-       "7                 2.0              4.0               2.0  \n",
-       "8                 3.0              4.0               4.0  \n",
-       "9                 NaN              NaN               NaN  "
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "============================================\n",
-      "Final evaluation dataset:\n"
-     ]
-    },
-    {
-     "data": {
-      "text/html": [
-       "<div>\n",
-       "<style scoped>\n",
-       "    .dataframe tbody tr th:only-of-type {\n",
-       "        vertical-align: middle;\n",
-       "    }\n",
-       "\n",
-       "    .dataframe tbody tr th {\n",
-       "        vertical-align: top;\n",
-       "    }\n",
-       "\n",
-       "    .dataframe thead th {\n",
-       "        text-align: right;\n",
-       "    }\n",
-       "</style>\n",
-       "<table border=\"1\" class=\"dataframe\">\n",
-       "  <thead>\n",
-       "    <tr style=\"text-align: right;\">\n",
-       "      <th></th>\n",
-       "      <th>question</th>\n",
-       "      <th>answer</th>\n",
-       "      <th>groundedness_score</th>\n",
-       "      <th>relevance_score</th>\n",
-       "      <th>standalone_score</th>\n",
-       "    </tr>\n",
-       "  </thead>\n",
-       "  <tbody>\n",
-       "    <tr>\n",
-       "      <th>1</th>\n",
-       "      <td>What are some utility functions provided by the Hugging Face library for pipelines?\\n</td>\n",
-       "      <td>The Hugging Face library provides several utility functions for pipelines, including `ArgumentHandler`, `ZeroShotClassificationArgumentHandler`, `QuestionAnsweringArgumentHandler` for argument handling, `PipelineDataFormat`, `CsvPipelineDataFormat`, `JsonPipelineDataFormat`, `PipedPipelineDataFormat` for data format, and `PipelineException` for exceptions.</td>\n",
-       "      <td>5.0</td>\n",
-       "      <td>4.0</td>\n",
-       "      <td>5.0</td>\n",
-       "    </tr>\n",
-       "  </tbody>\n",
-       "</table>\n",
-       "</div>"
-      ],
-      "text/plain": [
-       "                                                                                question  \\\n",
-       "1  What are some utility functions provided by the Hugging Face library for pipelines?\\n   \n",
-       "\n",
-       "                                                                                                                                                                                                                                                                                                                                                                   answer  \\\n",
-       "1  The Hugging Face library provides several utility functions for pipelines, including `ArgumentHandler`, `ZeroShotClassificationArgumentHandler`, `QuestionAnsweringArgumentHandler` for argument handling, `PipelineDataFormat`, `CsvPipelineDataFormat`, `JsonPipelineDataFormat`, `PipedPipelineDataFormat` for data format, and `PipelineException` for exceptions.   \n",
-       "\n",
-       "   groundedness_score  relevance_score  standalone_score  \n",
-       "1                 5.0              4.0               5.0  "
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    }
-   ],
-   "source": [
-    "import pandas as pd\n",
-    "\n",
-    "pd.set_option(\"display.max_colwidth\", None)\n",
-    "\n",
-    "generated_questions = pd.DataFrame.from_dict(outputs)\n",
-    "\n",
-    "print(\"Evaluation dataset before filtering:\")\n",
-    "display(\n",
-    "    generated_questions[\n",
-    "        [\"question\", \"answer\", \"groundedness_score\", \"relevance_score\", \"standalone_score\"]\n",
-    "    ]\n",
-    ")\n",
-    "generated_questions = generated_questions.loc[\n",
-    "    (generated_questions[\"groundedness_score\"] >= 4)\n",
-    "    & (generated_questions[\"relevance_score\"] >= 4)\n",
-    "    & (generated_questions[\"standalone_score\"] >= 4)\n",
-    "]\n",
-    "print(\"============================================\")\n",
-    "print(\"Final evaluation dataset:\")\n",
-    "display(\n",
-    "    generated_questions[\n",
-    "        [\"question\", \"answer\", \"groundedness_score\", \"relevance_score\", \"standalone_score\"]\n",
-    "    ]\n",
-    ")\n",
-    "\n",
-    "eval_dataset = datasets.Dataset.from_pandas(\n",
-    "    generated_questions, split=\"train\", preserve_index=False\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "id": "HaOMZyu69jVO"
-   },
-   "source": [
-    "Now our synthetic evaluation dataset is complete! We can evaluate different RAG systems on this evaluation dataset.\n",
-    "\n",
-    "We have generated only a few QA couples here to reduce time and cost. But let's kick start the next part by loading a pre-generated dataset:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "id": "Q3RRz4W79jVO"
-   },
-   "outputs": [],
-   "source": [
-    "eval_dataset = datasets.load_dataset(\"m-ric/huggingface_doc_qa_eval\", split=\"train\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "id": "K5s19uTd9jVO"
-   },
-   "source": [
-    "# 2. Build our RAG System"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "id": "Z-mET8Dy9jVO"
-   },
-   "source": [
-    "### 2.1. Preprocessing documents to build our vector database\n",
-    "\n",
-    "- In this part, __we split the documents from our knowledge base into smaller chunks__: these will be the snippets that are picked by the Retriever, to then be ingested by the Reader LLM as supporting elements for its answer.\n",
-    "- The goal is to build semantically relevant snippets: not too small to be sufficient for supporting an answer, and not too large too avoid diluting individual ideas.\n",
-    "\n",
-    "Many options exist for text splitting:\n",
-    "- split every `n` words / characters, but this has the risk of cutting in half paragraphs or even sentences\n",
-    "- split after `n` words / character, but only on sentence boundaries\n",
-    "- **recursive split** tries to preserve even more of the document structure, by processing it tree-like way, splitting first on the largest units (chapters) then recursively splitting on smaller units (paragraphs, sentences).\n",
-    "\n",
-    "To learn more about chunking, I recommend you read [this great notebook](https://github.com/FullStackRetrieval-com/RetrievalTutorials/blob/main/5_Levels_Of_Text_Splitting.ipynb) by Greg Kamradt.\n",
-    "\n",
-    "[This space](https://huggingface.co/spaces/m-ric/chunk_visualizer) lets you visualize how different splitting options affect the chunks you get.\n",
-    "\n",
-    "> In the following, we use Langchain's `RecursiveCharacterTextSplitter`.\n",
-    "\n",
-    "💡 _To measure chunk length in our Text Splitter, our length function will not be the count of characters, but the count of tokens in the tokenized text: indeed, for subsequent embedder that processes token, measuring length in tokens is more relevant and empirically performs better._"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "id": "H4fhm55Q9jVO"
-   },
-   "outputs": [],
-   "source": [
-    "from langchain.docstore.document import Document as LangchainDocument\n",
-    "\n",
-    "RAW_KNOWLEDGE_BASE = [\n",
-    "    LangchainDocument(page_content=doc[\"text\"], metadata={\"source\": doc[\"source\"]})\n",
-    "    for doc in tqdm(ds)\n",
-    "]"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "id": "sz9Jw2_q9jVO"
-   },
-   "outputs": [],
-   "source": [
-    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
-    "from transformers import AutoTokenizer\n",
-    "\n",
-    "\n",
-    "def split_documents(\n",
-    "    chunk_size: int,\n",
-    "    knowledge_base: List[LangchainDocument],\n",
-    "    tokenizer_name: str,\n",
-    ") -> List[LangchainDocument]:\n",
-    "    \"\"\"\n",
-    "    Split documents into chunks of size `chunk_size` characters and return a list of documents.\n",
-    "    \"\"\"\n",
-    "    text_splitter = RecursiveCharacterTextSplitter.from_huggingface_tokenizer(\n",
-    "        AutoTokenizer.from_pretrained(tokenizer_name),\n",
-    "        chunk_size=chunk_size,\n",
-    "        chunk_overlap=int(chunk_size / 10),\n",
-    "        add_start_index=True,\n",
-    "        strip_whitespace=True,\n",
-    "        separators=[\"\\n\\n\", \"\\n\", \".\", \" \", \"\"],\n",
-    "    )\n",
-    "\n",
-    "    docs_processed = []\n",
-    "    for doc in knowledge_base:\n",
-    "        docs_processed += text_splitter.split_documents([doc])\n",
-    "\n",
-    "    # Remove duplicates\n",
-    "    unique_texts = {}\n",
-    "    docs_processed_unique = []\n",
-    "    for doc in docs_processed:\n",
-    "        if doc.page_content not in unique_texts:\n",
-    "            unique_texts[doc.page_content] = True\n",
-    "            docs_processed_unique.append(doc)\n",
-    "\n",
-    "    return docs_processed_unique"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "id": "QzBYfNG79jVO"
-   },
-   "source": [
-    "### 2.2. Retriever - embeddings 🗂️\n",
-    "The __retriever acts like an internal search engine__: given the user query, it returns the most relevant documents from your knowledge base.\n",
-    "\n",
-    "> For the knowledge base, we use Langchain vector databases since __it offers a convenient [FAISS](https://github.com/facebookresearch/faiss) index and allows us to keep document metadata throughout the processing__.\n",
-    "\n",
-    "🛠️ __Options included:__\n",
-    "\n",
-    "- Tune the chunking method:\n",
-    "    - Size of the chunks\n",
-    "    - Method: split on different separators, use [semantic chunking](https://python.langchain.com/docs/modules/data_connection/document_transformers/semantic-chunker)...\n",
-    "- Change the embedding model"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "id": "LqJlIDZR9jVO"
-   },
-   "outputs": [],
-   "source": [
-    "from langchain.vectorstores import FAISS\n",
-    "from langchain_community.embeddings import HuggingFaceEmbeddings\n",
-    "from langchain_community.vectorstores.utils import DistanceStrategy\n",
-    "import os\n",
-    "\n",
-    "\n",
-    "def load_embeddings(\n",
-    "    langchain_docs: List[LangchainDocument],\n",
-    "    chunk_size: int,\n",
-    "    embedding_model_name: Optional[str] = \"thenlper/gte-small\",\n",
-    ") -> FAISS:\n",
-    "    \"\"\"\n",
-    "    Creates a FAISS index from the given embedding model and documents. Loads the index directly if it already exists.\n",
-    "\n",
-    "    Args:\n",
-    "        langchain_docs: list of documents\n",
-    "        chunk_size: size of the chunks to split the documents into\n",
-    "        embedding_model_name: name of the embedding model to use\n",
-    "\n",
-    "    Returns:\n",
-    "        FAISS index\n",
-    "    \"\"\"\n",
-    "    # load embedding_model\n",
-    "    embedding_model = HuggingFaceEmbeddings(\n",
-    "        model_name=embedding_model_name,\n",
-    "        multi_process=True,\n",
-    "        model_kwargs={\"device\": \"cuda\"},\n",
-    "        encode_kwargs={\"normalize_embeddings\": True},  # set True to compute cosine similarity\n",
-    "    )\n",
-    "\n",
-    "    # Check if embeddings already exist on disk\n",
-    "    index_name = f\"index_chunk:{chunk_size}_embeddings:{embedding_model_name.replace('/', '~')}\"\n",
-    "    index_folder_path = f\"./data/indexes/{index_name}/\"\n",
-    "    if os.path.isdir(index_folder_path):\n",
-    "        return FAISS.load_local(\n",
-    "            index_folder_path,\n",
-    "            embedding_model,\n",
-    "            distance_strategy=DistanceStrategy.COSINE,\n",
-    "        )\n",
-    "\n",
-    "    else:\n",
-    "        print(\"Index not found, generating it...\")\n",
-    "        docs_processed = split_documents(\n",
-    "            chunk_size,\n",
-    "            langchain_docs,\n",
-    "            embedding_model_name,\n",
-    "        )\n",
-    "        knowledge_index = FAISS.from_documents(\n",
-    "            docs_processed, embedding_model, distance_strategy=DistanceStrategy.COSINE\n",
-    "        )\n",
-    "        knowledge_index.save_local(index_folder_path)\n",
-    "        return knowledge_index"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "id": "b6y1mQJX9jVO"
-   },
-   "source": [
-    "### 2.3. Reader - LLM 💬\n",
-    "\n",
-    "In this part, the __LLM Reader reads the retrieved documents to formulate its answer.__\n",
-    "\n",
-    "🛠️ Here we tried the following options to improve results:\n",
-    "- Switch reranking on/off\n",
-    "- Change the reader model"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "id": "9PdpuWyP9jVP"
-   },
-   "outputs": [],
-   "source": [
-    "RAG_PROMPT_TEMPLATE = \"\"\"\n",
-    "<|system|>\n",
-    "Using the information contained in the context,\n",
-    "give a comprehensive answer to the question.\n",
-    "Respond only to the question asked, response should be concise and relevant to the question.\n",
-    "Provide the number of the source document when relevant.\n",
-    "If the answer cannot be deduced from the context, do not give an answer.</s>\n",
-    "<|user|>\n",
-    "Context:\n",
-    "{context}\n",
-    "---\n",
-    "Now here is the question you need to answer.\n",
-    "\n",
-    "Question: {question}\n",
-    "</s>\n",
-    "<|assistant|>\n",
-    "\"\"\""
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "id": "9SDqenld9jVP"
-   },
-   "outputs": [],
-   "source": [
-    "from langchain_community.llms import HuggingFaceHub\n",
-    "\n",
-    "repo_id = \"HuggingFaceH4/zephyr-7b-beta\"\n",
-    "READER_MODEL_NAME = \"zephyr-7b-beta\"\n",
-    "\n",
-    "READER_LLM = HuggingFaceHub(\n",
-    "    repo_id=repo_id,\n",
-    "    task=\"text-generation\",\n",
-    "    model_kwargs={\n",
-    "        \"max_new_tokens\": 512,\n",
-    "        \"top_k\": 30,\n",
-    "        \"temperature\": 0.1,\n",
-    "        \"repetition_penalty\": 1.03,\n",
-    "    },\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "id": "QZ62CbcZ9jVP"
-   },
-   "outputs": [],
-   "source": [
-    "from ragatouille import RAGPretrainedModel\n",
-    "from langchain_core.vectorstores import VectorStore\n",
-    "from langchain_core.language_models.llms import LLM\n",
-    "\n",
-    "\n",
-    "def answer_with_rag(\n",
-    "    question: str,\n",
-    "    llm: LLM,\n",
-    "    knowledge_index: VectorStore,\n",
-    "    reranker: Optional[RAGPretrainedModel] = None,\n",
-    "    num_retrieved_docs: int = 30,\n",
-    "    num_docs_final: int = 7,\n",
-    ") -> Tuple[str, List[LangchainDocument]]:\n",
-    "    \"\"\"Answer a question using RAG with the given knowledge index.\"\"\"\n",
-    "    # Gather documents with retriever\n",
-    "    relevant_docs = knowledge_index.similarity_search(query=question, k=num_retrieved_docs)\n",
-    "    relevant_docs = [doc.page_content for doc in relevant_docs]  # keep only the text\n",
-    "\n",
-    "    # Optionally rerank results\n",
-    "    if reranker:\n",
-    "        relevant_docs = reranker.rerank(question, relevant_docs, k=num_docs_final)\n",
-    "        relevant_docs = [doc[\"content\"] for doc in relevant_docs]\n",
-    "\n",
-    "    relevant_docs = relevant_docs[:num_docs_final]\n",
-    "\n",
-    "    # Build the final prompt\n",
-    "    context = \"\\nExtracted documents:\\n\"\n",
-    "    context += \"\".join([f\"Document {str(i)}:::\\n\" + doc for i, doc in enumerate(relevant_docs)])\n",
-    "\n",
-    "    final_prompt = RAG_PROMPT_TEMPLATE.format(question=question, context=context)\n",
-    "\n",
-    "    # Redact an answer\n",
-    "    answer = llm(final_prompt)\n",
-    "\n",
-    "    return answer, relevant_docs"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "id": "hiygbqfT9jVP"
-   },
-   "source": [
-    "# 3. Benchmarking the RAG system\n",
-    "\n",
-    "The RAG system and the evaluation datasets are now ready. The last step is to judge the RAG system's output on this evlauation dataset.\n",
-    "\n",
-    "To this end, __we setup a judge agent__. ⚖️🤖\n",
-    "\n",
-    "Out of [the different RAG evaluation metrics](https://docs.ragas.io/en/latest/concepts/metrics/index.html), we choose to focus only on faithfulness since it the best end-to-end metric of our system's performance.\n",
-    "\n",
-    "> We use GPT4 as a judge for its empirically good performance, but you could try with other models such as [kaist-ai/prometheus-13b-v1.0](https://huggingface.co/kaist-ai/prometheus-13b-v1.0) or [BAAI/JudgeLM-33B-v1.0](https://huggingface.co/BAAI/JudgeLM-33B-v1.0).\n",
-    "\n",
-    "💡 _In the evaluation prompt, we give a detailed description each metric on the scale 1-5, as is done in [Prometheus's prompt template](https://huggingface.co/kaist-ai/prometheus-13b-v1.0): this helps the model ground its metric precisely. If instead you give the judge LLM a vague scale to work with, the outputs will not be consistent enough between different examples._\n",
-    "\n",
-    "💡 _Again, prompting the LLM to output rationale before giving its final score gives it more tokens to help it formalize and elaborate a judgement._"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "id": "VrlMh_ZI9jVP"
-   },
-   "outputs": [],
-   "source": [
-    "def run_rag_tests(\n",
-    "    eval_dataset: datasets.Dataset,\n",
-    "    llm: BaseChatModel,\n",
-    "    knowledge_index: VectorStore,\n",
-    "    output_file: str,\n",
-    "    reranker: Optional[RAGPretrainedModel] = None,\n",
-    "    verbose: Optional[bool] = True,\n",
-    "    test_settings: Optional[str] = None,  # To document the test settings used\n",
-    "):\n",
-    "    \"\"\"Runs RAG tests on the given dataset and saves the results to the given output file.\"\"\"\n",
-    "    try:  # load previous generations if they exist\n",
-    "        with open(output_file, \"r\") as f:\n",
-    "            outputs = json.load(f)\n",
-    "    except:\n",
-    "        outputs = []\n",
-    "\n",
-    "    for example in tqdm(eval_dataset):\n",
-    "        question = example[\"question\"]\n",
-    "        if question in [output[\"question\"] for output in outputs]:\n",
-    "            continue\n",
-    "\n",
-    "        answer, relevant_docs = answer_with_rag(question, llm, knowledge_index, reranker=reranker)\n",
-    "        if verbose:\n",
-    "            print(\"=======================================================\")\n",
-    "            print(f\"Question: {question}\")\n",
-    "            print(f\"Answer: {answer}\")\n",
-    "            print(f'True answer: {example[\"answer\"]}')\n",
-    "        result = {\n",
-    "            \"question\": question,\n",
-    "            \"true_answer\": example[\"answer\"],\n",
-    "            \"source_doc\": example[\"source_doc\"],\n",
-    "            \"generated_answer\": answer,\n",
-    "            \"retrieved_docs\": [doc for doc in relevant_docs],\n",
-    "        }\n",
-    "        if test_settings:\n",
-    "            result[\"test_settings\"] = test_settings\n",
-    "        outputs.append(result)\n",
-    "\n",
-    "        with open(output_file, \"w\") as f:\n",
-    "            json.dump(outputs, f)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "id": "Ae-3KWzK9jVP"
-   },
-   "outputs": [],
-   "source": [
-    "EVALUATION_PROMPT = \"\"\"###Task Description:\n",
-    "An instruction (might include an Input inside it), a response to evaluate, a reference answer that gets a score of 5, and a score rubric representing a evaluation criteria are given.\n",
-    "1. Write a detailed feedback that assess the quality of the response strictly based on the given score rubric, not evaluating in general.\n",
-    "2. After writing a feedback, write a score that is an integer between 1 and 5. You should refer to the score rubric.\n",
-    "3. The output format should look as follows: \\\"Feedback: {{write a feedback for criteria}} [RESULT] {{an integer number between 1 and 5}}\\\"\n",
-    "4. Please do not generate any other opening, closing, and explanations. Be sure to include [RESULT] in your output.\n",
-    "\n",
-    "###The instruction to evaluate:\n",
-    "{instruction}\n",
-    "\n",
-    "###Response to evaluate:\n",
-    "{response}\n",
-    "\n",
-    "###Reference Answer (Score 5):\n",
-    "{reference_answer}\n",
-    "\n",
-    "###Score Rubrics:\n",
-    "[Is the response correct, accurate, and factual based on the reference answer?]\n",
-    "Score 1: The response is completely incorrect, inaccurate, and/or not factual.\n",
-    "Score 2: The response is mostly incorrect, inaccurate, and/or not factual.\n",
-    "Score 3: The response is somewhat correct, accurate, and/or factual.\n",
-    "Score 4: The response is mostly correct, accurate, and factual.\n",
-    "Score 5: The response is completely correct, accurate, and factual.\n",
-    "\n",
-    "###Feedback:\"\"\"\n",
-    "\n",
-    "from langchain.prompts.chat import (\n",
-    "    ChatPromptTemplate,\n",
-    "    HumanMessagePromptTemplate,\n",
-    ")\n",
-    "from langchain.schema import SystemMessage\n",
-    "\n",
-    "\n",
-    "evaluation_prompt_template = ChatPromptTemplate.from_messages(\n",
-    "    [\n",
-    "        SystemMessage(content=\"You are a fair evaluator language model.\"),\n",
-    "        HumanMessagePromptTemplate.from_template(EVALUATION_PROMPT),\n",
-    "    ]\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "id": "ia9Mvn859jVP"
-   },
-   "outputs": [],
-   "source": [
-    "from langchain.chat_models import ChatOpenAI\n",
-    "\n",
-    "eval_chat_model = ChatOpenAI(model=\"gpt-4-1106-preview\", temperature=0)\n",
-    "evaluator_name = \"GPT4\"\n",
-    "\n",
-    "\n",
-    "def evaluate_answers(\n",
-    "    answer_path: str,\n",
-    "    eval_chat_model: BaseChatModel,\n",
-    "    evaluator_name: str,\n",
-    "    evaluation_prompt_template: ChatPromptTemplate,\n",
-    ") -> None:\n",
-    "    \"\"\"Evaluates generated answers. Modifies the given answer file in place for better checkpointing.\"\"\"\n",
-    "    answers = []\n",
-    "    if os.path.isfile(answer_path):  # load previous generations if they exist\n",
-    "        answers = json.load(open(answer_path, \"r\"))\n",
-    "\n",
-    "    for experiment in tqdm(answers):\n",
-    "        if f\"eval_score_{evaluator_name}\" in experiment:\n",
-    "            continue\n",
-    "\n",
-    "        eval_prompt = evaluation_prompt_template.format_messages(\n",
-    "            instruction=experiment[\"question\"],\n",
-    "            response=experiment[\"generated_answer\"],\n",
-    "            reference_answer=experiment[\"true_answer\"],\n",
-    "        )\n",
-    "        eval_result = eval_chat_model.invoke(eval_prompt)\n",
-    "        feedback, score = [item.strip() for item in eval_result.content.split(\"[RESULT]\")]\n",
-    "        experiment[f\"eval_score_{evaluator_name}\"] = score\n",
-    "        experiment[f\"eval_feedback_{evaluator_name}\"] = feedback\n",
-    "\n",
-    "        with open(answer_path, \"w\") as f:\n",
-    "            json.dump(answers, f)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "id": "EXH-szLe9jVP"
-   },
-   "source": [
-    "🚀 Let's run the tests and evaluate answers!👇"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "id": "jW2nnvUT9jVQ"
-   },
-   "outputs": [],
-   "source": [
-    "if not os.path.exists(\"./output\"):\n",
-    "    os.mkdir(\"./output\")\n",
-    "\n",
-    "for chunk_size in [200]:  # Add other chunk sizes (in tokens) as needed\n",
-    "    for embeddings in [\"thenlper/gte-small\"]:  # Add other embeddings as needed\n",
-    "        for rerank in [True, False]:\n",
-    "            settings_name = f\"chunk:{chunk_size}_embeddings:{embeddings.replace('/', '~')}_rerank:{rerank}_reader-model:{READER_MODEL_NAME}\"\n",
-    "            output_file_name = f\"./output/rag_{settings_name}.json\"\n",
-    "\n",
-    "            print(f\"Running evaluation for {settings_name}:\")\n",
-    "\n",
-    "            print(\"Loading knowledge base embeddings...\")\n",
-    "            knowledge_index = load_embeddings(\n",
-    "                RAW_KNOWLEDGE_BASE,\n",
-    "                chunk_size=chunk_size,\n",
-    "                embedding_model_name=embeddings,\n",
-    "            )\n",
-    "\n",
-    "            print(\"Running RAG...\")\n",
-    "            reranker = (\n",
-    "                RAGPretrainedModel.from_pretrained(\"colbert-ir/colbertv2.0\") if rerank else None\n",
-    "            )\n",
-    "            run_rag_tests(\n",
-    "                eval_dataset=eval_dataset,\n",
-    "                llm=READER_LLM,\n",
-    "                knowledge_index=knowledge_index,\n",
-    "                output_file=output_file_name,\n",
-    "                reranker=reranker,\n",
-    "                verbose=False,\n",
-    "                test_settings=settings_name,\n",
-    "            )\n",
-    "\n",
-    "            print(\"Running evaluation...\")\n",
-    "            evaluate_answers(\n",
-    "                output_file_name,\n",
-    "                eval_chat_model,\n",
-    "                evaluator_name,\n",
-    "                evaluation_prompt_template,\n",
-    "            )"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "id": "tytXV5-h9jVT"
-   },
-   "source": [
-    "### Inspect results"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "id": "D4YDSfmr9jVT"
-   },
-   "outputs": [],
-   "source": [
-    "import glob\n",
-    "\n",
-    "outputs = []\n",
-    "for file in glob.glob(\"./output/*.json\"):\n",
-    "    output = pd.DataFrame(json.load(open(file, \"r\")))\n",
-    "    output[\"settings\"] = file\n",
-    "    outputs.append(output)\n",
-    "result = pd.concat(outputs)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "id": "CdkXMNvS9jVT"
-   },
-   "outputs": [],
-   "source": [
-    "result[\"eval_score_GPT4\"] = result[\"eval_score_GPT4\"].apply(\n",
-    "    lambda x: int(x) if isinstance(x, str) else 1\n",
-    ")\n",
-    "result[\"eval_score_GPT4\"] = (result[\"eval_score_GPT4\"] - 1) / 4"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "id": "lgxBpid29jVT",
-    "outputId": "9a3bcf32-4b0c-4df1-c76c-3ebbca82929d"
-   },
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "settings\n",
-       "./output/rag_chunk:200_embeddings:thenlper~gte-small_rerank:False_reader-model:zephyr-7b-beta.json       0.884328\n",
-       "./output/rag_chunk:200_embeddings:BAAI~bge-base-en-v1.5_rerank:False_reader-model:zephyr-7b-beta.json    0.906716\n",
-       "./output/rag_chunk:200_embeddings:BAAI~bge-base-en-v1.5_rerank:True_reader-model:zephyr-7b-beta.json     0.906716\n",
-       "./output/rag_chunk:200_embeddings:thenlper~gte-small_rerank:True_reader-model:mixtral.json               0.906716\n",
-       "./output/rag_chunk:200_embeddings:thenlper~gte-small_rerank:True_reader-model:zephyr-7b-beta.json        0.921642\n",
-       "./output/rag_chunk:200_embeddings:thenlper~gte-small_rerank:True_reader-model:mixtral0.json              0.947761\n",
-       "Name: eval_score_GPT4, dtype: float64"
-      ]
-     },
-     "execution_count": 24,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "average_scores = result.groupby(\"settings\")[\"eval_score_GPT4\"].mean()\n",
-    "average_scores.sort_values()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "id": "pSPH9DYI9jVT"
-   },
-   "source": [
-    "## Example results\n",
-    "\n",
-    "Let us load the results that I obtained by tweaking the different options available in this notebook.\n",
-    "For more detail on why these options could work on not, see the notebook on [advanced_RAG](advanced_rag).\n",
-    "\n",
-    "As you can see in the graph below, some tweaks do not bring any improvement, some give huge performance boosts.\n",
-    "\n",
-    "➡️ ___There is no single good recipe: you should try several different directions when tuning your RAG systems.___\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "id": "RVOxatv99jVT"
-   },
-   "outputs": [],
-   "source": [
-    "import plotly.express as px\n",
-    "\n",
-    "scores = datasets.load_dataset(\"m-ric/rag_scores_cookbook\", split=\"train\")\n",
-    "scores = pd.Series(scores[\"score\"], index=scores[\"settings\"])"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "id": "vqK0Dg2Q9jVT"
-   },
-   "outputs": [],
-   "source": [
-    "fig = px.bar(\n",
-    "    scores,\n",
-    "    color=scores,\n",
-    "    labels={\n",
-    "        \"value\": \"Accuracy\",\n",
-    "        \"settings\": \"Configuration\",\n",
-    "    },\n",
-    "    color_continuous_scale=\"bluered\",\n",
-    ")\n",
-    "fig.update_layout(w\n",
-    "    width=1000,\n",
-    "    height=600,\n",
-    "    barmode=\"group\",\n",
-    "    yaxis_range=[0, 100],\n",
-    "    title=\"<b>Accuracy of different RAG configurations</b>\",\n",
-    "    xaxis_title=\"RAG settings\",\n",
-    "    font=dict(size=15),\n",
-    ")\n",
-    "fig.layout.yaxis.ticksuffix = \"%\"\n",
-    "fig.update_coloraxes(showscale=False)\n",
-    "fig.update_traces(texttemplate=\"%{y:.1f}\", textposition=\"outside\")\n",
-    "fig.show()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "id": "dPUOMWGk9jVT"
-   },
-   "source": [
-    "<img src=\"https://huggingface.co/datasets/huggingface/cookbook-images/resolve/main/RAG_settings_accuracy.png\" height=\"500\" width=\"800\">\n",
-    "\n",
-    "As you can see, these had varying impact on performance. In particular, tuning the chunk size is both easy and very impactful.\n",
-    "\n",
-    "But this is our case: your results could be very different: now that you have a robust evaluation pipeline, you can set on to explore other options! 🗺️"
-   ]
-  }
- ],
- "metadata": {
-  "colab": {
-   "provenance": []
-  },
-  "kernelspec": {
-   "display_name": "ml2",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.9"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 0
-}

src/notebooks/rag_evaluation.qmd

@@ -0,0 +1,786 @@
+---
+title: RAG Evaluation
+jupyter: python3
+eval: false
+---
+
+```{python}
+!pip install -q torch transformers transformers langchain sentence-transformers faiss-gpu openpyxl openai
+```
+
+```{python}
+%reload_ext autoreload
+%autoreload 2
+%reload_ext dotenv
+%dotenv
+```
+
+```{python}
+from tqdm.notebook import tqdm
+import pandas as pd
+from typing import Optional, List, Tuple
+from langchain_core.language_models import BaseChatModel
+import json
+import datasets
+
+pd.set_option("display.max_colwidth", None)
+```
+
+### Load your knowledge base
+
+```{python}
+ds = datasets.load_dataset("m-ric/huggingface_doc", split="train")
+```
+
+# 1. Build a synthetic dataset for evaluation
+We first build a synthetic dataset of questions and associated contexts. The method is to get elements from our knowledge base, and ask an LLM to generate questions based on these documents.
+
+Then we setup other LLM agents to act as quality filters for the generated QA couples: each of them will act as the filter for a specific flaw.
+
+### 1.1. Prepare source documents
+
+```{python}
+from langchain.text_splitter import RecursiveCharacterTextSplitter
+from langchain.docstore.document import Document as LangchainDocument
+
+langchain_docs = [
+    LangchainDocument(page_content=doc["text"], metadata={"source": doc["source"]})
+    for doc in tqdm(ds)
+]
+
+
+text_splitter = RecursiveCharacterTextSplitter(
+    chunk_size=2000,
+    chunk_overlap=200,
+    add_start_index=True,
+    separators=["\n\n", "\n", ".", " ", ""],
+)
+
+docs_processed = []
+for doc in langchain_docs:
+    docs_processed += text_splitter.split_documents([doc])
+```
+
+### 1.2. Setup agents for question generation
+
+We use [Mixtral](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1) for QA couple generation because it it has excellent performance in leaderboards such as [Chatbot Arena](https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard).
+
+```{python}
+from langchain_community.llms import HuggingFaceHub
+
+repo_id = "mistralai/Mixtral-8x7B-Instruct-v0.1"
+
+llm = HuggingFaceHub(
+    repo_id=repo_id,
+    task="text-generation",
+    model_kwargs={
+        "max_new_tokens": 512,
+        "top_k": 30,
+        "temperature": 0.1,
+        "repetition_penalty": 1.03,
+    },
+)
+```
+
+```{python}
+from langchain_community.chat_models import ChatHuggingFace
+
+chat_model = ChatHuggingFace(llm=llm)
+```
+
+```{python}
+from langchain.prompts import ChatPromptTemplate
+
+QA_generation_prompt = """
+Your task is to write a factoid question and an answer given a context.
+Your factoid question should be answerable with a specific, concise piece of factual information from the context.
+Your factoid question should be formulated in the same style as questions users could ask in a search engine.
+This means that your factoid question MUST NOT mention something like "according to the passage" or "context".
+
+Provide your answer as follows:
+
+Output:::
+Factoid question: (your factoid question)
+Answer: (your answer to the factoid question)
+
+Now here is the context.
+
+Context: {context}\n
+Output:::"""
+
+QA_generation_prompt = ChatPromptTemplate.from_template(QA_generation_prompt)
+QA_generation_agent = QA_generation_prompt | chat_model
+```
+
+Now let's generate our QA couples.
+For this example, we generate only 10 QA couples and will load the rest from the Hub.
+
+But for your specific knowledge base, given that you want to get at least ~100 test samples, and accounting for the fact that we will filter out around half of these with our critique agents later on, you should generate much more, in the >200 samples.
+
+```{python}
+import random
+
+N_GENERATIONS = (
+    10  # We intentionally generate only 10 QA couples here for cost and time considerations
+)
+
+print(f"Generating {N_GENERATIONS} QA couples...")
+outputs = []
+for context in tqdm(random.sample(langchain_docs, N_GENERATIONS)):
+    # Generate QA couple
+    output_QA_couple = QA_generation_agent.invoke({"context": context.page_content}).content
+    try:
+        question = output_QA_couple.split("Factoid question: ")[1].split("Answer: ")[0]
+        answer = output_QA_couple.split("Answer: ")[1]
+        outputs.append(
+            {
+                "context": context.page_content,
+                "question": question,
+                "answer": answer,
+                "source_doc": context.metadata["source"],
+            }
+        )
+    except:
+        continue
+```
+
+```{python}
+display(pd.DataFrame(outputs).head(1))
+```
+
+### 1.3. Setup critique agents
+
+The questions generated by the previous agent can have many flaws: we should do a quality check before validating these questions.
+
+We thus build critique agents that will rate each question on several criteria, given in [this paper](https://huggingface.co/papers/2312.10003):
+- **Groundedness:** can the question be answered from the given context?
+- **Relevance:** is the question relevant to users? For instance, `"What is the date when transformers 4.29.1 was released?"` is not relevant for ML practicioners.
+
+One last failure case we've noticed is when a function is tailored for the particular setting where the question was generated, but undecipherable by itself, like `"What is the name of the function used in this guide?"`.
+We also build a critique agent for this criteria:
+- **Stand-alone**: is the question understandable free of any context, for someone with domain knowledge/Internet access? The opposite of this would be `What is the function used in this article?` for a question generated from a specific blog article.
+
+We systematically score functions with all these agents, and whenever the score is too low for any one of the agents, we eliminate the question from our eval dataset.
+
+💡 ___When asking the agents to output a score, we first ask them to produce its rationale. This will help us verify scores, but most importantly, asking it to first output rationale gives the model more tokens to think and elaborate an answer before summarizing it into a single score token.___
+
+We now build and run these critique agents.
+
+```{python}
+question_groundedness_critique_prompt = """
+You will be given a context and a question.
+Your task is to provide a 'total rating' scoring how well one can answer the given question unambiguously with the given context.
+Give your answer on a scale of 1 to 5, where 1 means that the question is not answerable at all given the context, and 5 means that the question is clearly and unambiguously answerable with the context.
+
+Provide your answer as follows:
+
+Answer:::
+Evaluation: (your rationale for the rating)
+Total rating: (your rating)
+
+Now here are the question and context.
+
+Question: {question}\n
+Context: {context}\n
+Answer::: """
+
+question_relevance_critique_prompt = """
+You will be given a question.
+Your task is to provide a 'total rating' representing how useful this question can be to machine learning developers building NLP applications with the Hugging Face ecosystem.
+Give your answer on a scale of 1 to 5, where 1 means that the question is not useful at all, and 5 means that the question is extremely useful.
+
+Provide your answer as follows:
+
+Answer:::
+Evaluation: (your rationale for the rating)
+Total rating: (your rating)
+
+Now here is the question.
+
+Question: {question}\n
+Answer::: """
+
+question_standalone_critique_prompt = """
+You will be given a question.
+Your task is to provide a 'total rating' representing how context-independant this question is.
+Give your answer on a scale of 1 to 5, where 1 means that the question only makes sense in a specific context, and 5 means that the question makes sense by itself.
+For instance, if the question refers to a particular setting, like 'in the context' or 'in the document', the rating must be 1.
+The questions can contain obscure technical nouns or acronyms like Gradio, Hub, Hugging Face or Space and still be a 5: it must simply be clear to an operator with access to documentation what the question is about.
+
+Provide your answer as follows:
+
+Answer:::
+Evaluation: (your rationale for the rating)
+Total rating: (your rating)
+
+Now here is the question.
+
+Question: {question}\n
+Answer::: """
+
+question_groundedness_critique_prompt = ChatPromptTemplate.from_template(
+    question_groundedness_critique_prompt
+)
+question_groundedness_critique_agent = question_groundedness_critique_prompt | chat_model
+
+question_relevance_critique_prompt = ChatPromptTemplate.from_template(
+    question_relevance_critique_prompt
+)
+question_relevance_critique_agent = question_relevance_critique_prompt | chat_model
+
+question_standalone_critique_prompt = ChatPromptTemplate.from_template(
+    question_standalone_critique_prompt
+)
+question_standalone_critique_agent = question_standalone_critique_prompt | chat_model
+```
+
+```{python}
+print("Generating critique for each QA couple...")
+for output in tqdm(outputs):
+    # Critique the generated QA couple
+    question_groundedness_evaluation = question_groundedness_critique_agent.invoke(
+        {"context": output["context"], "question": output["question"]}
+    ).content
+    question_relevance_evaluation = question_relevance_critique_agent.invoke(
+        {"question": output["question"]}
+    ).content
+    question_standalone_evaluation = question_standalone_critique_agent.invoke(
+        {"question": output["question"]}
+    ).content
+
+    try:
+        groundedness_score = int(question_groundedness_evaluation.split("Total rating: ")[1][0])
+        groundedness_eval = question_groundedness_evaluation.split("Total rating: ")[0].split(
+            "Evaluation: "
+        )[1]
+        relevance_score = int(question_relevance_evaluation.split("Total rating: ")[1][0])
+        relevance_eval = question_relevance_evaluation.split("Total rating: ")[0].split(
+            "Evaluation: "
+        )[1]
+        standalone_score = int(question_standalone_evaluation.split("Total rating: ")[1][0])
+        standalone_eval = question_standalone_evaluation.split("Total rating: ")[0].split(
+            "Evaluation: "
+        )[1]
+        output.update(
+            {
+                "groundedness_score": groundedness_score,
+                "groundedness_eval": groundedness_eval,
+                "relevance_score": relevance_score,
+                "relevance_eval": relevance_eval,
+                "standalone_score": standalone_score,
+                "standalone_eval": standalone_eval,
+            }
+        )
+    except:
+        continue
+```
+
+Now let us filter out bad questions based on our critique agent scores:
+
+```{python}
+import pandas as pd
+
+pd.set_option("display.max_colwidth", None)
+
+generated_questions = pd.DataFrame.from_dict(outputs)
+
+print("Evaluation dataset before filtering:")
+display(
+    generated_questions[
+        ["question", "answer", "groundedness_score", "relevance_score", "standalone_score"]
+    ]
+)
+generated_questions = generated_questions.loc[
+    (generated_questions["groundedness_score"] >= 4)
+    & (generated_questions["relevance_score"] >= 4)
+    & (generated_questions["standalone_score"] >= 4)
+]
+print("============================================")
+print("Final evaluation dataset:")
+display(
+    generated_questions[
+        ["question", "answer", "groundedness_score", "relevance_score", "standalone_score"]
+    ]
+)
+
+eval_dataset = datasets.Dataset.from_pandas(
+    generated_questions, split="train", preserve_index=False
+)
+```
+
+Now our synthetic evaluation dataset is complete! We can evaluate different RAG systems on this evaluation dataset.
+
+We have generated only a few QA couples here to reduce time and cost. But let's kick start the next part by loading a pre-generated dataset:
+
+```{python}
+eval_dataset = datasets.load_dataset("m-ric/huggingface_doc_qa_eval", split="train")
+```
+
+# 2. Build our RAG System
+
+### 2.1. Preprocessing documents to build our vector database
+
+- In this part, __we split the documents from our knowledge base into smaller chunks__: these will be the snippets that are picked by the Retriever, to then be ingested by the Reader LLM as supporting elements for its answer.
+- The goal is to build semantically relevant snippets: not too small to be sufficient for supporting an answer, and not too large too avoid diluting individual ideas.
+
+Many options exist for text splitting:
+- split every `n` words / characters, but this has the risk of cutting in half paragraphs or even sentences
+- split after `n` words / character, but only on sentence boundaries
+- **recursive split** tries to preserve even more of the document structure, by processing it tree-like way, splitting first on the largest units (chapters) then recursively splitting on smaller units (paragraphs, sentences).
+
+To learn more about chunking, I recommend you read [this great notebook](https://github.com/FullStackRetrieval-com/RetrievalTutorials/blob/main/5_Levels_Of_Text_Splitting.ipynb) by Greg Kamradt.
+
+[This space](https://huggingface.co/spaces/m-ric/chunk_visualizer) lets you visualize how different splitting options affect the chunks you get.
+
+> In the following, we use Langchain's `RecursiveCharacterTextSplitter`.
+
+💡 _To measure chunk length in our Text Splitter, our length function will not be the count of characters, but the count of tokens in the tokenized text: indeed, for subsequent embedder that processes token, measuring length in tokens is more relevant and empirically performs better._
+
+```{python}
+from langchain.docstore.document import Document as LangchainDocument
+
+RAW_KNOWLEDGE_BASE = [
+    LangchainDocument(page_content=doc["text"], metadata={"source": doc["source"]})
+    for doc in tqdm(ds)
+]
+```
+
+```{python}
+from langchain.text_splitter import RecursiveCharacterTextSplitter
+from transformers import AutoTokenizer
+
+
+def split_documents(
+    chunk_size: int,
+    knowledge_base: List[LangchainDocument],
+    tokenizer_name: str,
+) -> List[LangchainDocument]:
+    """
+    Split documents into chunks of size `chunk_size` characters and return a list of documents.
+    """
+    text_splitter = RecursiveCharacterTextSplitter.from_huggingface_tokenizer(
+        AutoTokenizer.from_pretrained(tokenizer_name),
+        chunk_size=chunk_size,
+        chunk_overlap=int(chunk_size / 10),
+        add_start_index=True,
+        strip_whitespace=True,
+        separators=["\n\n", "\n", ".", " ", ""],
+    )
+
+    docs_processed = []
+    for doc in knowledge_base:
+        docs_processed += text_splitter.split_documents([doc])
+
+    # Remove duplicates
+    unique_texts = {}
+    docs_processed_unique = []
+    for doc in docs_processed:
+        if doc.page_content not in unique_texts:
+            unique_texts[doc.page_content] = True
+            docs_processed_unique.append(doc)
+
+    return docs_processed_unique
+```
+
+### 2.2. Retriever - embeddings 🗂️
+The __retriever acts like an internal search engine__: given the user query, it returns the most relevant documents from your knowledge base.
+
+> For the knowledge base, we use Langchain vector databases since __it offers a convenient [FAISS](https://github.com/facebookresearch/faiss) index and allows us to keep document metadata throughout the processing__.
+
+🛠️ __Options included:__
+
+- Tune the chunking method:
+    - Size of the chunks
+    - Method: split on different separators, use [semantic chunking](https://python.langchain.com/docs/modules/data_connection/document_transformers/semantic-chunker)...
+- Change the embedding model
+
+```{python}
+from langchain.vectorstores import FAISS
+from langchain_community.embeddings import HuggingFaceEmbeddings
+from langchain_community.vectorstores.utils import DistanceStrategy
+import os
+
+
+def load_embeddings(
+    langchain_docs: List[LangchainDocument],
+    chunk_size: int,
+    embedding_model_name: Optional[str] = "thenlper/gte-small",
+) -> FAISS:
+    """
+    Creates a FAISS index from the given embedding model and documents. Loads the index directly if it already exists.
+
+    Args:
+        langchain_docs: list of documents
+        chunk_size: size of the chunks to split the documents into
+        embedding_model_name: name of the embedding model to use
+
+    Returns:
+        FAISS index
+    """
+    # load embedding_model
+    embedding_model = HuggingFaceEmbeddings(
+        model_name=embedding_model_name,
+        multi_process=True,
+        model_kwargs={"device": "cuda"},
+        encode_kwargs={"normalize_embeddings": True},  # set True to compute cosine similarity
+    )
+
+    # Check if embeddings already exist on disk
+    index_name = f"index_chunk:{chunk_size}_embeddings:{embedding_model_name.replace('/', '~')}"
+    index_folder_path = f"./data/indexes/{index_name}/"
+    if os.path.isdir(index_folder_path):
+        return FAISS.load_local(
+            index_folder_path,
+            embedding_model,
+            distance_strategy=DistanceStrategy.COSINE,
+        )
+
+    else:
+        print("Index not found, generating it...")
+        docs_processed = split_documents(
+            chunk_size,
+            langchain_docs,
+            embedding_model_name,
+        )
+        knowledge_index = FAISS.from_documents(
+            docs_processed, embedding_model, distance_strategy=DistanceStrategy.COSINE
+        )
+        knowledge_index.save_local(index_folder_path)
+        return knowledge_index
+```
+
+### 2.3. Reader - LLM 💬
+
+In this part, the __LLM Reader reads the retrieved documents to formulate its answer.__
+
+🛠️ Here we tried the following options to improve results:
+- Switch reranking on/off
+- Change the reader model
+
+```{python}
+RAG_PROMPT_TEMPLATE = """
+<|system|>
+Using the information contained in the context,
+give a comprehensive answer to the question.
+Respond only to the question asked, response should be concise and relevant to the question.
+Provide the number of the source document when relevant.
+If the answer cannot be deduced from the context, do not give an answer.</s>
+<|user|>
+Context:
+{context}
+---
+Now here is the question you need to answer.
+
+Question: {question}
+</s>
+<|assistant|>
+"""
+```
+
+```{python}
+from langchain_community.llms import HuggingFaceHub
+
+repo_id = "HuggingFaceH4/zephyr-7b-beta"
+READER_MODEL_NAME = "zephyr-7b-beta"
+
+READER_LLM = HuggingFaceHub(
+    repo_id=repo_id,
+    task="text-generation",
+    model_kwargs={
+        "max_new_tokens": 512,
+        "top_k": 30,
+        "temperature": 0.1,
+        "repetition_penalty": 1.03,
+    },
+)
+```
+
+```{python}
+from ragatouille import RAGPretrainedModel
+from langchain_core.vectorstores import VectorStore
+from langchain_core.language_models.llms import LLM
+
+
+def answer_with_rag(
+    question: str,
+    llm: LLM,
+    knowledge_index: VectorStore,
+    reranker: Optional[RAGPretrainedModel] = None,
+    num_retrieved_docs: int = 30,
+    num_docs_final: int = 7,
+) -> Tuple[str, List[LangchainDocument]]:
+    """Answer a question using RAG with the given knowledge index."""
+    # Gather documents with retriever
+    relevant_docs = knowledge_index.similarity_search(query=question, k=num_retrieved_docs)
+    relevant_docs = [doc.page_content for doc in relevant_docs]  # keep only the text
+
+    # Optionally rerank results
+    if reranker:
+        relevant_docs = reranker.rerank(question, relevant_docs, k=num_docs_final)
+        relevant_docs = [doc["content"] for doc in relevant_docs]
+
+    relevant_docs = relevant_docs[:num_docs_final]
+
+    # Build the final prompt
+    context = "\nExtracted documents:\n"
+    context += "".join([f"Document {str(i)}:::\n" + doc for i, doc in enumerate(relevant_docs)])
+
+    final_prompt = RAG_PROMPT_TEMPLATE.format(question=question, context=context)
+
+    # Redact an answer
+    answer = llm(final_prompt)
+
+    return answer, relevant_docs
+```
+
+# 3. Benchmarking the RAG system
+
+The RAG system and the evaluation datasets are now ready. The last step is to judge the RAG system's output on this evlauation dataset.
+
+To this end, __we setup a judge agent__. ⚖️🤖
+
+Out of [the different RAG evaluation metrics](https://docs.ragas.io/en/latest/concepts/metrics/index.html), we choose to focus only on faithfulness since it the best end-to-end metric of our system's performance.
+
+> We use GPT4 as a judge for its empirically good performance, but you could try with other models such as [kaist-ai/prometheus-13b-v1.0](https://huggingface.co/kaist-ai/prometheus-13b-v1.0) or [BAAI/JudgeLM-33B-v1.0](https://huggingface.co/BAAI/JudgeLM-33B-v1.0).
+
+💡 _In the evaluation prompt, we give a detailed description each metric on the scale 1-5, as is done in [Prometheus's prompt template](https://huggingface.co/kaist-ai/prometheus-13b-v1.0): this helps the model ground its metric precisely. If instead you give the judge LLM a vague scale to work with, the outputs will not be consistent enough between different examples._
+
+💡 _Again, prompting the LLM to output rationale before giving its final score gives it more tokens to help it formalize and elaborate a judgement._
+
+```{python}
+def run_rag_tests(
+    eval_dataset: datasets.Dataset,
+    llm: BaseChatModel,
+    knowledge_index: VectorStore,
+    output_file: str,
+    reranker: Optional[RAGPretrainedModel] = None,
+    verbose: Optional[bool] = True,
+    test_settings: Optional[str] = None,  # To document the test settings used
+):
+    """Runs RAG tests on the given dataset and saves the results to the given output file."""
+    try:  # load previous generations if they exist
+        with open(output_file, "r") as f:
+            outputs = json.load(f)
+    except:
+        outputs = []
+
+    for example in tqdm(eval_dataset):
+        question = example["question"]
+        if question in [output["question"] for output in outputs]:
+            continue
+
+        answer, relevant_docs = answer_with_rag(question, llm, knowledge_index, reranker=reranker)
+        if verbose:
+            print("=======================================================")
+            print(f"Question: {question}")
+            print(f"Answer: {answer}")
+            print(f'True answer: {example["answer"]}')
+        result = {
+            "question": question,
+            "true_answer": example["answer"],
+            "source_doc": example["source_doc"],
+            "generated_answer": answer,
+            "retrieved_docs": [doc for doc in relevant_docs],
+        }
+        if test_settings:
+            result["test_settings"] = test_settings
+        outputs.append(result)
+
+        with open(output_file, "w") as f:
+            json.dump(outputs, f)
+```
+
+```{python}
+EVALUATION_PROMPT = """###Task Description:
+An instruction (might include an Input inside it), a response to evaluate, a reference answer that gets a score of 5, and a score rubric representing a evaluation criteria are given.
+1. Write a detailed feedback that assess the quality of the response strictly based on the given score rubric, not evaluating in general.
+2. After writing a feedback, write a score that is an integer between 1 and 5. You should refer to the score rubric.
+3. The output format should look as follows: \"Feedback: {{write a feedback for criteria}} [RESULT] {{an integer number between 1 and 5}}\"
+4. Please do not generate any other opening, closing, and explanations. Be sure to include [RESULT] in your output.
+
+###The instruction to evaluate:
+{instruction}
+
+###Response to evaluate:
+{response}
+
+###Reference Answer (Score 5):
+{reference_answer}
+
+###Score Rubrics:
+[Is the response correct, accurate, and factual based on the reference answer?]
+Score 1: The response is completely incorrect, inaccurate, and/or not factual.
+Score 2: The response is mostly incorrect, inaccurate, and/or not factual.
+Score 3: The response is somewhat correct, accurate, and/or factual.
+Score 4: The response is mostly correct, accurate, and factual.
+Score 5: The response is completely correct, accurate, and factual.
+
+###Feedback:"""
+
+from langchain.prompts.chat import (
+    ChatPromptTemplate,
+    HumanMessagePromptTemplate,
+)
+from langchain.schema import SystemMessage
+
+
+evaluation_prompt_template = ChatPromptTemplate.from_messages(
+    [
+        SystemMessage(content="You are a fair evaluator language model."),
+        HumanMessagePromptTemplate.from_template(EVALUATION_PROMPT),
+    ]
+)
+```
+
+```{python}
+from langchain.chat_models import ChatOpenAI
+
+eval_chat_model = ChatOpenAI(model="gpt-4-1106-preview", temperature=0)
+evaluator_name = "GPT4"
+
+
+def evaluate_answers(
+    answer_path: str,
+    eval_chat_model: BaseChatModel,
+    evaluator_name: str,
+    evaluation_prompt_template: ChatPromptTemplate,
+) -> None:
+    """Evaluates generated answers. Modifies the given answer file in place for better checkpointing."""
+    answers = []
+    if os.path.isfile(answer_path):  # load previous generations if they exist
+        answers = json.load(open(answer_path, "r"))
+
+    for experiment in tqdm(answers):
+        if f"eval_score_{evaluator_name}" in experiment:
+            continue
+
+        eval_prompt = evaluation_prompt_template.format_messages(
+            instruction=experiment["question"],
+            response=experiment["generated_answer"],
+            reference_answer=experiment["true_answer"],
+        )
+        eval_result = eval_chat_model.invoke(eval_prompt)
+        feedback, score = [item.strip() for item in eval_result.content.split("[RESULT]")]
+        experiment[f"eval_score_{evaluator_name}"] = score
+        experiment[f"eval_feedback_{evaluator_name}"] = feedback
+
+        with open(answer_path, "w") as f:
+            json.dump(answers, f)
+```
+
+🚀 Let's run the tests and evaluate answers!👇
+
+```{python}
+if not os.path.exists("./output"):
+    os.mkdir("./output")
+
+for chunk_size in [200]:  # Add other chunk sizes (in tokens) as needed
+    for embeddings in ["thenlper/gte-small"]:  # Add other embeddings as needed
+        for rerank in [True, False]:
+            settings_name = f"chunk:{chunk_size}_embeddings:{embeddings.replace('/', '~')}_rerank:{rerank}_reader-model:{READER_MODEL_NAME}"
+            output_file_name = f"./output/rag_{settings_name}.json"
+
+            print(f"Running evaluation for {settings_name}:")
+
+            print("Loading knowledge base embeddings...")
+            knowledge_index = load_embeddings(
+                RAW_KNOWLEDGE_BASE,
+                chunk_size=chunk_size,
+                embedding_model_name=embeddings,
+            )
+
+            print("Running RAG...")
+            reranker = (
+                RAGPretrainedModel.from_pretrained("colbert-ir/colbertv2.0") if rerank else None
+            )
+            run_rag_tests(
+                eval_dataset=eval_dataset,
+                llm=READER_LLM,
+                knowledge_index=knowledge_index,
+                output_file=output_file_name,
+                reranker=reranker,
+                verbose=False,
+                test_settings=settings_name,
+            )
+
+            print("Running evaluation...")
+            evaluate_answers(
+                output_file_name,
+                eval_chat_model,
+                evaluator_name,
+                evaluation_prompt_template,
+            )
+```
+
+### Inspect results
+
+```{python}
+import glob
+
+outputs = []
+for file in glob.glob("./output/*.json"):
+    output = pd.DataFrame(json.load(open(file, "r")))
+    output["settings"] = file
+    outputs.append(output)
+result = pd.concat(outputs)
+```
+
+```{python}
+result["eval_score_GPT4"] = result["eval_score_GPT4"].apply(
+    lambda x: int(x) if isinstance(x, str) else 1
+)
+result["eval_score_GPT4"] = (result["eval_score_GPT4"] - 1) / 4
+```
+
+```{python}
+average_scores = result.groupby("settings")["eval_score_GPT4"].mean()
+average_scores.sort_values()
+```
+
+## Example results
+
+Let us load the results that I obtained by tweaking the different options available in this notebook.
+For more detail on why these options could work on not, see the notebook on [advanced_RAG](advanced_rag).
+
+As you can see in the graph below, some tweaks do not bring any improvement, some give huge performance boosts.
+
+➡️ ___There is no single good recipe: you should try several different directions when tuning your RAG systems.___
+
+```{python}
+import plotly.express as px
+
+scores = datasets.load_dataset("m-ric/rag_scores_cookbook", split="train")
+scores = pd.Series(scores["score"], index=scores["settings"])
+```
+
+```{python}
+fig = px.bar(
+    scores,
+    color=scores,
+    labels={
+        "value": "Accuracy",
+        "settings": "Configuration",
+    },
+    color_continuous_scale="bluered",
+)
+fig.update_layout(w
+    width=1000,
+    height=600,
+    barmode="group",
+    yaxis_range=[0, 100],
+    title="<b>Accuracy of different RAG configurations</b>",
+    xaxis_title="RAG settings",
+    font=dict(size=15),
+)
+fig.layout.yaxis.ticksuffix = "%"
+fig.update_coloraxes(showscale=False)
+fig.update_traces(texttemplate="%{y:.1f}", textposition="outside")
+fig.show()
+```
+
+<img src="https://huggingface.co/datasets/huggingface/cookbook-images/resolve/main/RAG_settings_accuracy.png" height="500" width="800">
+
+As you can see, these had varying impact on performance. In particular, tuning the chunk size is both easy and very impactful.
+
+But this is our case: your results could be very different: now that you have a robust evaluation pipeline, you can set on to explore other options! 🗺️
+

src/notebooks/rag_zephyr_langchain.ipynb

@@ -1,527 +0,0 @@
-{
-  "cells": [
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "Kih21u1tyr-I"
-      },
-      "source": [
-        "---\n",
-        "title: Simple RAG\n",
-        "---\n",
-        "\n",
-        "# Simple RAG for GitHub issues using Hugging Face Zephyr and LangChain\n",
-        "\n",
-        "_Authored by: [Maria Khalusova](https://github.com/MKhalusova)_\n",
-        "\n",
-        "This notebook demonstrates how you can quickly build a RAG (Retrieval Augmented Generation) for a project's GitHub issues using [`HuggingFaceH4/zephyr-7b-beta`](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta) model, and LangChain.\n",
-        "\n",
-        "\n",
-        "**What is RAG?**\n",
-        "\n",
-        "RAG is a popular approach to address the issue of a powerful LLM not being aware of specific content due to said content not being in its training data, or hallucinating even when it has seen it before. Such specific content may be proprietary, sensitive, or, as in this example, recent and updated often.\n",
-        "\n",
-        "If your data is static and doesn't change regularly, you may consider fine-tuning a large model. In many cases, however, fine-tuning can be costly, and, when done repeatedly (e.g. to address data drift), leads to \"model shift\". This is when the model's behavior changes in ways that are not desirable.\n",
-        "\n",
-        "**RAG (Retrieval Augmented Generation)** does not require model fine-tuning. Instead, RAG works by providing an LLM with additional context that is retrieved from relevant data so that it can generate a better-informed response.\n",
-        "\n",
-        "Here's a quick illustration:\n",
-        "\n",
-        "![RAG diagram](https://huggingface.co/datasets/huggingface/cookbook-images/resolve/main/rag-diagram.png)\n",
-        "\n",
-        "* The external data is converted into embedding vectors with a separate embeddings model, and the vectors are kept in a database. Embeddings models are typically small, so updating the embedding vectors on a regular basis is faster, cheaper, and easier than fine-tuning a model.\n",
-        "\n",
-        "* At the same time, the fact that fine-tuning is not required gives you the freedom to swap your LLM for a more powerful one when it becomes available, or switch to a smaller distilled version, should you need faster inference.\n",
-        "\n",
-        "Let's illustrate building a RAG using an open-source LLM, embeddings model, and LangChain.\n",
-        "\n",
-        "First, install the required dependencies:"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "lC9frDOlyi38"
-      },
-      "outputs": [],
-      "source": [
-        "!pip install -q torch transformers accelerate bitsandbytes transformers sentence-transformers faiss-gpu"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 2,
-      "metadata": {
-        "id": "-aYENQwZ-p_c"
-      },
-      "outputs": [],
-      "source": [
-        "# If running in Google Colab, you may need to run this cell to make sure you're using UTF-8 locale to install LangChain\n",
-        "import locale\n",
-        "locale.getpreferredencoding = lambda: \"UTF-8\""
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "W5HhMZ2c-NfU"
-      },
-      "outputs": [],
-      "source": [
-        "!pip install -q langchain"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "R8po01vMWzXL"
-      },
-      "source": [
-        "## Prepare the data\n"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "3cCmQywC04x6"
-      },
-      "source": [
-        "In this example, we'll load all of the issues (both open and closed) from [PEFT library's repo](https://github.com/huggingface/peft).\n",
-        "\n",
-        "First, you need to acquire a [GitHub personal access token](https://github.com/settings/tokens?type=beta) to access the GitHub API."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "8MoD7NbsNjlM"
-      },
-      "outputs": [],
-      "source": [
-        "from getpass import getpass\n",
-        "ACCESS_TOKEN = getpass(\"YOUR_GITHUB_PERSONAL_TOKEN\")"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "fccecm3a10N6"
-      },
-      "source": [
-        "Next, we'll load all of the issues in the [huggingface/peft](https://github.com/huggingface/peft) repo:\n",
-        "- By default, pull requests are considered issues as well, here we chose to exclude them from data with by setting `include_prs=False`\n",
-        "- Setting `state = \"all\"` means we will load both open and closed issues."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 5,
-      "metadata": {
-        "id": "8EKMit4WNDY8"
-      },
-      "outputs": [],
-      "source": [
-        "from langchain.document_loaders import GitHubIssuesLoader\n",
-        "\n",
-        "loader = GitHubIssuesLoader(\n",
-        "    repo=\"huggingface/peft\",\n",
-        "    access_token=ACCESS_TOKEN,\n",
-        "    include_prs=False,\n",
-        "    state=\"all\"\n",
-        ")\n",
-        "\n",
-        "docs = loader.load()"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "CChTrY-k2qO5"
-      },
-      "source": [
-        "The content of individual GitHub issues may be longer than what an embedding model can take as input. If we want to embed all of the available content, we need to chunk the documents into appropriately sized pieces.\n",
-        "\n",
-        "The most common and straightforward approach to chunking is to define a fixed size of chunks and whether there should be any overlap between them. Keeping some overlap between chunks allows us to preserve some semantic context between the chunks.\n",
-        "\n",
-        "Other approaches are typically more involved and take into account the documents' structure and context. For example, one may want to split a document based on sentences or paragraphs, or create chunks based on the\n",
-        "\n",
-        "The fixed-size chunking, however, works well for most common cases, so that is what we'll do here."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "OmsXOf59Pmm-"
-      },
-      "outputs": [],
-      "source": [
-        "from langchain.text_splitter import CharacterTextSplitter\n",
-        "\n",
-        "splitter = CharacterTextSplitter(chunk_size=512, chunk_overlap=30)\n",
-        "\n",
-        "chunked_docs = splitter.split_documents(docs)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "DAt_zPVlXOn7"
-      },
-      "source": [
-        "## Create the embeddings + retriever"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "-mvat6JQl4yp"
-      },
-      "source": [
-        "Now that the docs are all of the appropriate size, we can create a database with their embeddings.\n",
-        "\n",
-        "To create document chunk embeddings we'll use the `HuggingFaceEmbeddings` and the [`BAAI/bge-base-en-v1.5`](https://huggingface.co/BAAI/bge-base-en-v1.5) embeddings model. There are many other embeddings models available on the Hub, and you can keep an eye on the best performing ones by checking the [Massive Text Embedding Benchmark (MTEB) Leaderboard](https://huggingface.co/spaces/mteb/leaderboard).\n",
-        "\n",
-        "\n",
-        "To create the vector database, we'll use `FAISS`, a library developed by Facebook AI. This library offers efficient similarity search and clustering of dense vectors, which is what we need here. FAISS is currently one of the most used libraries for NN search in massive datasets.\n",
-        "\n",
-        "We'll access both the embeddings model and FAISS via LangChain API."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "ixmCdRzBQ5gu"
-      },
-      "outputs": [],
-      "source": [
-        "from langchain.vectorstores import FAISS\n",
-        "from langchain.embeddings import HuggingFaceEmbeddings\n",
-        "\n",
-        "db = FAISS.from_documents(chunked_docs,\n",
-        "                          HuggingFaceEmbeddings(model_name='BAAI/bge-base-en-v1.5'))"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "2iCgEPi0nnN6"
-      },
-      "source": [
-        "We need a way to return(retrieve) the documents given an unstructured query. For that, we'll use the `as_retriever` method using the `db` as a backbone:\n",
-        "- `search_type=\"similarity\"` means we want to perform similarity search between the query and documents\n",
-        "- `search_kwargs={'k': 4}` instructs the retriever to return top 4 results.\n"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 8,
-      "metadata": {
-        "id": "mBTreCQ9noHK"
-      },
-      "outputs": [],
-      "source": [
-        "retriever = db.as_retriever(\n",
-        "    search_type=\"similarity\",\n",
-        "    search_kwargs={'k': 4}\n",
-        ")"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "WgEhlISJpTgj"
-      },
-      "source": [
-        "The vector database and retriever are now set up, next we need to set up the next piece of the chain - the model."
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "tzQxx0HkXVFU"
-      },
-      "source": [
-        "## Load quantized model"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "9jy1cC65p_GD"
-      },
-      "source": [
-        "For this example, we chose [`HuggingFaceH4/zephyr-7b-beta`](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta), a small but powerful model.\n",
-        "\n",
-        "With many models being released every week, you may want to substitute this model to the latest and greatest. The best way to keep track of open source LLMs is to check the [Open-source LLM leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard).\n",
-        "\n",
-        "To make inference faster, we will load the quantized version of the model:"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "L-ggaa763VRo"
-      },
-      "outputs": [],
-      "source": [
-        "import torch\n",
-        "from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig\n",
-        "\n",
-        "model_name = 'HuggingFaceH4/zephyr-7b-beta'\n",
-        "\n",
-        "bnb_config = BitsAndBytesConfig(\n",
-        "    load_in_4bit=True,\n",
-        "    bnb_4bit_use_double_quant=True,\n",
-        "    bnb_4bit_quant_type=\"nf4\",\n",
-        "    bnb_4bit_compute_dtype=torch.bfloat16\n",
-        ")\n",
-        "\n",
-        "model = AutoModelForCausalLM.from_pretrained(model_name, quantization_config=bnb_config)\n",
-        "tokenizer = AutoTokenizer.from_pretrained(model_name)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "hVNRJALyXYHG"
-      },
-      "source": [
-        "## Setup the LLM chain"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "RUUNneJ1smhl"
-      },
-      "source": [
-        "Finally, we have all the pieces we need to set up the LLM chain.\n",
-        "\n",
-        "First, create a text_generation pipeline using the loaded model and its tokenizer.\n",
-        "\n",
-        "Next, create a prompt template - this should follow the format of the model, so if you substitute the model checkpoint, make sure to use the appropriate formatting."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 15,
-      "metadata": {
-        "id": "cR0k1cRWz8Pm"
-      },
-      "outputs": [],
-      "source": [
-        "from langchain.llms import HuggingFacePipeline\n",
-        "from langchain.prompts import PromptTemplate\n",
-        "from transformers import pipeline\n",
-        "from langchain_core.output_parsers import StrOutputParser\n",
-        "\n",
-        "text_generation_pipeline = pipeline(\n",
-        "    model=model,\n",
-        "    tokenizer=tokenizer,\n",
-        "    task=\"text-generation\",\n",
-        "    temperature=0.2,\n",
-        "    do_sample=True,\n",
-        "    repetition_penalty=1.1,\n",
-        "    return_full_text=True,\n",
-        "    max_new_tokens=400,\n",
-        ")\n",
-        "\n",
-        "llm = HuggingFacePipeline(pipeline=text_generation_pipeline)\n",
-        "\n",
-        "prompt_template = \"\"\"\n",
-        "<|system|>\n",
-        "Answer the question based on your knowledge. Use the following context to help:\n",
-        "\n",
-        "{context}\n",
-        "\n",
-        "</s>\n",
-        "<|user|>\n",
-        "{question}\n",
-        "</s>\n",
-        "<|assistant|>\n",
-        "\n",
-        " \"\"\"\n",
-        "\n",
-        "prompt = PromptTemplate(\n",
-        "    input_variables=[\"context\", \"question\"],\n",
-        "    template=prompt_template,\n",
-        ")\n",
-        "\n",
-        "llm_chain = prompt | llm | StrOutputParser()"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "l19UKq5HXfSp"
-      },
-      "source": [
-        "Note: _You can also use `tokenizer.apply_chat_template` to convert a list of messages (as dicts: `{'role': 'user', 'content': '(...)'}`) into a string with the appropriate chat format._\n",
-        "\n",
-        "\n",
-        "Finally, we need to combine the `llm_chain` with the retriever to create a RAG chain. We pass the original question through to the final generation step, as well as the retrieved context docs:"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 17,
-      "metadata": {
-        "id": "_rI3YNp9Xl4s"
-      },
-      "outputs": [],
-      "source": [
-        "from langchain_core.runnables import RunnablePassthrough\n",
-        "\n",
-        "retriever = db.as_retriever()\n",
-        "\n",
-        "rag_chain = (\n",
-        " {\"context\": retriever, \"question\": RunnablePassthrough()}\n",
-        "    | llm_chain\n",
-        ")\n"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "UsCOhfDDXpaS"
-      },
-      "source": [
-        "## Compare the results\n",
-        "\n",
-        "Let's see the difference RAG makes in generating answers to the library-specific questions."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 18,
-      "metadata": {
-        "id": "W7F07fQLXusU"
-      },
-      "outputs": [],
-      "source": [
-        "question = \"How do you combine multiple adapters?\""
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "KC0rJYU1x1ir"
-      },
-      "source": [
-        "First, let's see what kind of answer we can get with just the model itself, no context added:"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 20,
-      "metadata": {
-        "colab": {
-          "base_uri": "https://localhost:8080/",
-          "height": 125
-        },
-        "id": "GYh-HG1l0De5",
-        "outputId": "277d8e89-ce9b-4e04-c11b-639ad2645759"
-      },
-      "outputs": [
-        {
-          "data": {
-            "application/vnd.google.colaboratory.intrinsic+json": {
-              "type": "string"
-            },
-            "text/plain": [
-              "\" To combine multiple adapters, you need to ensure that they are compatible with each other and the devices you want to connect. Here's how you can do it:\\n\\n1. Identify the adapters you need: Determine which adapters you require to connect the devices you want to use together. For example, if you want to connect a USB-C device to an HDMI monitor, you may need a USB-C to HDMI adapter and a USB-C to USB-A adapter (if your computer only has USB-A ports).\\n\\n2. Connect the first adapter: Plug in the first adapter into the device you want to connect. For instance, if you're connecting a USB-C laptop to an HDMI monitor, plug the USB-C to HDMI adapter into the laptop's USB-C port.\\n\\n3. Connect the second adapter: Next, connect the second adapter to the first one. In this case, connect the USB-C to USB-A adapter to the USB-C port of the USB-C to HDMI adapter.\\n\\n4. Connect the final device: Finally, connect the device you want to use to the second adapter. For example, connect the HDMI cable from the monitor to the HDMI port on the USB-C to HDMI adapter.\\n\\n5. Test the connection: Turn on both devices and check whether everything is working correctly. If necessary, adjust the settings on your devices to ensure optimal performance.\\n\\nBy combining multiple adapters, you can connect a variety of devices together, even if they don't have the same type of connector. Just be sure to choose adapters that are compatible with all the devices you want to connect and test the connection thoroughly before relying on it for critical tasks.\""
-            ]
-          },
-          "execution_count": 20,
-          "metadata": {},
-          "output_type": "execute_result"
-        }
-      ],
-      "source": [
-        "llm_chain.invoke({\"context\":\"\", \"question\": question})"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "i-TIWr3wx9w8"
-      },
-      "source": [
-        "As you can see, the model interpreted the question as one about physical computer adapters, while in the context of PEFT, \"adapters\" refer to LoRA adapters.\n",
-        "Let's see if adding context from GitHub issues helps the model give a more relevant answer:"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": 21,
-      "metadata": {
-        "colab": {
-          "base_uri": "https://localhost:8080/",
-          "height": 125
-        },
-        "id": "FZpNA3o10H10",
-        "outputId": "31f9aed3-3dd7-4ff8-d1a8-866794fefe80"
-      },
-      "outputs": [
-        {
-          "data": {
-            "application/vnd.google.colaboratory.intrinsic+json": {
-              "type": "string"
-            },
-            "text/plain": [
-              "\" Based on the provided context, it seems that combining multiple adapters is still an open question in the community. Here are some possibilities:\\n\\n  1. Save the output from the base model and pass it to each adapter separately, as described in the first context snippet. This allows you to run multiple adapters simultaneously and reuse the output from the base model. However, this approach requires loading and running each adapter separately.\\n\\n  2. Export everything into a single PyTorch model, as suggested in the second context snippet. This would involve saving all the adapters and their weights into a single model, potentially making it larger and more complex. The advantage of this approach is that it would allow you to run all the adapters simultaneously without having to load and run them separately.\\n\\n  3. Merge multiple Lora adapters, as mentioned in the third context snippet. This involves adding multiple distinct, independent behaviors to a base model by merging multiple Lora adapters. It's not clear from the context how this would be done, but it suggests that there might be a recommended way of doing it.\\n\\n  4. Combine adapters through a specific architecture, as proposed in the fourth context snippet. This involves merging multiple adapters into a single architecture, potentially creating a more complex model with multiple behaviors. Again, it's not clear from the context how this would be done.\\n\\n   Overall, combining multiple adapters is still an active area of research, and there doesn't seem to be a widely accepted solution yet. If you're interested in exploring this further, it might be worth reaching out to the Hugging Face community or checking out their documentation for more information.\""
-            ]
-          },
-          "execution_count": 21,
-          "metadata": {},
-          "output_type": "execute_result"
-        }
-      ],
-      "source": [
-        "rag_chain.invoke(question)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "hZQedZKSyrwO"
-      },
-      "source": [
-        "As we can see, the added context, really helps the exact same model, provide a much more relevant and informed answer to the library-specific question.\n",
-        "\n",
-        "Notably, combining multiple adapters for inference has been added to the library, and one can find this information in the documentation, so for the next iteration of this RAG it may be worth including documentation embeddings."
-      ]
-    }
-  ],
-  "metadata": {
-    "accelerator": "GPU",
-    "colab": {
-      "gpuType": "T4",
-      "provenance": []
-    },
-    "kernelspec": {
-      "display_name": "Python 3",
-      "name": "python3"
-    },
-    "language_info": {
-      "codemirror_mode": {
-        "name": "ipython",
-        "version": 3
-      },
-      "file_extension": ".py",
-      "mimetype": "text/x-python",
-      "name": "python",
-      "nbconvert_exporter": "python",
-      "pygments_lexer": "ipython3",
-      "version": "3.11.3"
-    }
-  },
-  "nbformat": 4,
-  "nbformat_minor": 0
-}

src/notebooks/rag_zephyr_langchain.qmd

@@ -0,0 +1,232 @@
+---
+title: Simple RAG
+jupyter: python3
+eval: false
+code-annotations: hover
+
+---
+
+```{python}
+!pip install -q torch transformers accelerate bitsandbytes transformers sentence-transformers faiss-gpu
+```
+
+```{python}
+!pip install -q langchain
+```
+
+::: callout-note
+If running in Google Colab, you may need to run this cell to make sure you're using UTF-8 locale to install LangChain
+```{python}
+import locale
+locale.getpreferredencoding = lambda: "UTF-8"
+```
+:::
+
+
+## Prepare the data
+
+In this example, we'll load all of the issues (both open and closed) from [PEFT library's repo](https://github.com/huggingface/peft).
+
+First, you need to acquire a [GitHub personal access token](https://github.com/settings/tokens?type=beta) to access the GitHub API.
+
+```{python}
+from getpass import getpass
+
+ACCESS_TOKEN = getpass("YOUR_GITHUB_PERSONAL_TOKEN")  # <1>
+```
+1. You can also use an environment variable to store your token.
+
+Next, we'll load all of the issues in the [huggingface/peft](https://github.com/huggingface/peft) repo:
+- By default, pull requests are considered issues as well, here we chose to exclude them from data with by setting `include_prs=False`
+- Setting `state = "all"` means we will load both open and closed issues.
+
+```{python}
+from langchain.document_loaders import GitHubIssuesLoader
+
+loader = GitHubIssuesLoader(
+    repo="huggingface/peft",
+    access_token=ACCESS_TOKEN,
+    include_prs=False,
+    state="all"
+)
+
+docs = loader.load()
+```
+
+The content of individual GitHub issues may be longer than what an embedding model can take as input. If we want to embed all of the available content, we need to chunk the documents into appropriately sized pieces.
+
+The most common and straightforward approach to chunking is to define a fixed size of chunks and whether there should be any overlap between them. Keeping some overlap between chunks allows us to preserve some semantic context between the chunks.
+
+Other approaches are typically more involved and take into account the documents' structure and context. For example, one may want to split a document based on sentences or paragraphs, or create chunks based on the
+
+The fixed-size chunking, however, works well for most common cases, so that is what we'll do here.
+
+```{python}
+from langchain.text_splitter import CharacterTextSplitter
+
+splitter = CharacterTextSplitter(chunk_size=512, chunk_overlap=30)
+
+chunked_docs = splitter.split_documents(docs)
+```
+
+## Create the embeddings + retriever
+
+Now that the docs are all of the appropriate size, we can create a database with their embeddings.
+
+To create document chunk embeddings we'll use the `HuggingFaceEmbeddings` and the [`BAAI/bge-base-en-v1.5`](https://huggingface.co/BAAI/bge-base-en-v1.5) embeddings model. To create the vector database, we'll use `FAISS`, a library developed by Facebook AI. This library offers efficient similarity search and clustering of dense vectors, which is what we need here. FAISS is currently one of the most used libraries for NN search in massive datasets. 
+
+::: callout-tip
+There are many other embeddings models available on the Hub, and you can keep an eye on the best performing ones by checking the [Massive Text Embedding Benchmark (MTEB) Leaderboard](https://huggingface.co/spaces/mteb/leaderboard).
+:::
+
+We'll access both the embeddings model and FAISS via LangChain API.
+
+```{python}
+from langchain.vectorstores import FAISS
+from langchain.embeddings import HuggingFaceEmbeddings
+
+db = FAISS.from_documents(chunked_docs,
+                          HuggingFaceEmbeddings(model_name='BAAI/bge-base-en-v1.5'))
+```
+
+We need a way to return(retrieve) the documents given an unstructured query. For that, we'll use the `as_retriever` method using the `db` as a backbone:
+- `search_type="similarity"` means we want to perform similarity search between the query and documents
+- `search_kwargs={'k': 4}` instructs the retriever to return top 4 results.
+
+```{python}
+retriever = db.as_retriever(
+    search_type="similarity",            # <1>
+    search_kwargs={'k': 4}               # <1>
+)
+```
+1. The ideal search type is context dependent, and you should experiment to find the best one for your data.
+
+The vector database and retriever are now set up, next we need to set up the next piece of the chain - the model.
+
+## Load quantized model
+
+For this example, we chose [`HuggingFaceH4/zephyr-7b-beta`](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta), a small but powerful model.
+To make inference faster, we will load the quantized version of the model:
+
+:::::: {.callout-tip}
+With many models being released every week, you may want to substitute this model to the latest and greatest. The best way to keep track of open source LLMs is to check the [Open-source LLM leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard).
+:::
+
+```{python}
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
+
+model_name = 'HuggingFaceH4/zephyr-7b-beta'
+
+bnb_config = BitsAndBytesConfig(
+    load_in_4bit=True,
+    bnb_4bit_use_double_quant=True,
+    bnb_4bit_quant_type="nf4",
+    bnb_4bit_compute_dtype=torch.bfloat16
+)
+
+model = AutoModelForCausalLM.from_pretrained(model_name, quantization_config=bnb_config)
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+```
+
+## Setup the LLM chain
+
+Finally, we have all the pieces we need to set up the LLM chain.
+
+First, create a text_generation pipeline using the loaded model and its tokenizer.
+
+Next, create a prompt template - this should follow the format of the model, so if you substitute the model checkpoint, make sure to use the appropriate formatting.
+
+```{python}
+from langchain.llms import HuggingFacePipeline
+from langchain.prompts import PromptTemplate
+from transformers import pipeline
+from langchain_core.output_parsers import StrOutputParser
+
+text_generation_pipeline = pipeline(
+    model=model,                # <1> 
+    tokenizer=tokenizer,        # <2> 
+    task="text-generation",     # <3> 
+    temperature=0.2,            # <4> 
+    do_sample=True,             # <5> 
+    repetition_penalty=1.1,     # <6> 
+    return_full_text=True,      # <7> 
+    max_new_tokens=400,         # <8> 
+)
+
+llm = HuggingFacePipeline(pipeline=text_generation_pipeline)
+
+prompt_template = """
+<|system|>
+Answer the question based on your knowledge. Use the following context to help:
+
+{context}
+
+</s>
+<|user|>
+{question}
+</s>
+<|assistant|>
+
+ """
+
+prompt = PromptTemplate(
+    input_variables=["context", "question"],
+    template=prompt_template,
+)
+
+llm_chain = prompt | llm | StrOutputParser()
+```
+
+1. The pre-trained model for text generation.
+2. Tokenizer to preprocess input text and postprocess generated output.
+3. Specifies the task as text generation.
+4. Controls the randomness in the output generation. Lower values make the output more deterministic.
+5. Enables sampling to introduce randomness in the output generation.
+6. Penalizes repetition in the output to encourage diversity.
+7. Returns the full generated text including the input prompt.
+8. Limits the maximum number of new tokens generated.
+
+Note: _You can also use `tokenizer.apply_chat_template` to convert a list of messages (as dicts: `{'role': 'user', 'content': '(...)'}`) into a string with the appropriate chat format._
+
+
+Finally, we need to combine the `llm_chain` with the retriever to create a RAG chain. We pass the original question through to the final generation step, as well as the retrieved context docs:
+
+```{python}
+from langchain_core.runnables import RunnablePassthrough
+
+retriever = db.as_retriever()
+
+rag_chain = (
+ {"context": retriever, "question": RunnablePassthrough()}
+    | llm_chain
+)
+```
+
+## Compare the results
+
+Let's see the difference RAG makes in generating answers to the library-specific questions.
+
+```{python}
+question = "How do you combine multiple adapters?"
+```
+
+First, let's see what kind of answer we can get with just the model itself, no context added:
+
+```{python}
+#| colab: {base_uri: 'https://localhost:8080/', height: 125}
+llm_chain.invoke({"context":"", "question": question})
+```
+
+As you can see, the model interpreted the question as one about physical computer adapters, while in the context of PEFT, "adapters" refer to LoRA adapters.
+Let's see if adding context from GitHub issues helps the model give a more relevant answer:
+
+```{python}
+#| colab: {base_uri: 'https://localhost:8080/', height: 125}
+rag_chain.invoke(question)
+```
+
+As we can see, the added context, really helps the exact same model, provide a much more relevant and informed answer to the library-specific question.
+
+Notably, combining multiple adapters for inference has been added to the library, and one can find this information in the documentation, so for the next iteration of this RAG it may be worth including documentation embeddings.
+