
Building your own AI chatbot: Providing context with the RAG method (Part 4)
Learning project. This code is written to teach concepts. It is not hardened for production. Anything you put in front of real users needs proper security, testing, and monitoring first.
In this part Crumb learns about the bakery. We will give it four short documents and a way to find the right passage for each question, then hand that passage to the model along with the question. This is called retrieval-augmented generation, or RAG. By the end, you will have a chatbot that answers from the bakery's own documents, and a clear picture of where that approach still goes wrong.
This continues from Part 3. The complete code for this part is in the part-4 folder of the repo. We keep working in your my-chatbot folder.
This is the longest part of the series. Embeddings and retrieval are the biggest new ideas, and they are worth the extra room.
Crumb has no facts
At the end of Part 3, Crumb had a name, a voice, and four rules. What it did not have was any information about the bakery. Ask it something a customer would actually ask:
You: What time do you open on Saturday?
Crumb: We're open from 8am to 6pm on Saturdays. You can swing by then for a fresh loaf or some sweet treats!
[tokens: 224 in, 29 out]Those hours are invented. The model has never seen Bready or Not's hours, because nobody has shown them to it. It knows a lot about bakeries in general, so it produces something that sounds like a bakery. That is a guess with good manners.
The model cannot learn our hours by itself, so we have to put them in the request.
The obvious fix
Remember from Part 2 that the model reads the entire request every time: the system prompt, the history, and the new question. So the simplest way to give it facts is to paste every document into every request. The bot would always have everything in front of it.
For our bakery, that would honestly work. The four sample documents come to about 1,500 tokens. Add that to every request and Crumb can answer from real hours and real prices.
It stops working as the documents grow. Every token in the request is paid for on every turn, as the fruit example in Part 2 showed. A real business might have a hundred pages of policies, menus, and FAQs. Pasting all of it means paying for all of it on every "thanks", waiting while the model reads all of it, and eventually hitting the context window, the most tokens a model can read in one request. Somewhere along the way, the one paragraph that matters also gets lost in a pile of paragraphs that do not.
The RAG idea
So we send less. Before each question goes to the model, we look through the documents, pick the few passages most related to the question, and send only those. Then the model writes its answer from what we sent.
That is the whole idea. The first step is retrieval, finding the passages. The second step is generation, the model writing a reply. Retrieval-augmented generation is generation that has had some retrieval done for it first.
The model part is the same as in Part 3. All the new work is in retrieval, and the hard question there is how to tell which passages are "related" to a question. Matching words is the first thing that comes to mind, and it is not good enough. A customer asks "Where can I leave my car?" and the answer is in a section about parking that never uses the words "leave" or "car".
This is a simplified version of RAG. Real systems add more steps, such as combining keyword search with meaning search, re-ranking the results, or rewriting the question before searching. This series does not cover those.
Embeddings
An embedding is a way to turn text into numbers so that a computer can compare meanings. You send a piece of text to an embedding model, and it sends back a long list of numbers, called a vector. The model we use, openai/text-embedding-3-small, returns 1,536 numbers for any text you give it, whether that text is one word or a whole page.
The individual numbers mean nothing you can read. What matters is how vectors compare. The embedding model was trained so that texts with similar meanings get vectors that point in similar directions. "Where can I leave my car?" and "There are four free 30 minute spots directly in front of the shop" share almost no words, but they are about the same thing, so their vectors land close together.
Here is that question run against the bakery documents, using retrieve.py, which we will build shortly:
Question: Where can I leave my car?
1. score 0.340 [faq.md / Is there parking?]
2. score 0.218 [about.md / Where to find us]
3. score 0.205 [faq.md / Can I bring my dog?](The real output also prints each passage, trimmed here.) The parking section wins clearly, even though "car" appears nowhere in it. The second place goes to the section with the shop's address, which is a reasonable guess for someone driving over.
One thing to be clear about: none of this trains or changes the chat model. The embedding model is a separate model, and we only use it as a lookup tool. The chat model is exactly the same as in Part 3. It simply gets better information in its request.
Cosine similarity
To score how close two vectors are, we use cosine similarity. It measures the angle between two vectors and ignores their length. Two vectors that point the same way score 1, and two that have nothing to do with each other score close to 0. For finding related text, that angle is all we care about.
In code, it is one line with NumPy:
def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
"""How closely two vectors point the same way. Higher means more related."""
return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))That is the entire math lecture.
Chunking
We do not embed whole documents. One vector for the entire policies page would be a blur of opening hours, refunds, allergens, and discounts. It would be a little bit related to every question and very related to none. So we split each document into chunks, smaller passages that each cover one topic, and embed each chunk on its own.
Where to split is a real decision. Chunks that are too big are blurry in the way just described. Chunks that are too small lose their context: a line that says "Sunday: 7:00 am to 1:00 pm" means little once it is separated from the heading that says these are opening hours. Common approaches split by a fixed number of characters or tokens, often with some overlap between chunks.
Our documents are already organized for us. Each one is a markdown file with a `# ` title and a `## ` heading for each topic, so we split at the `## ` headings. Each chunk is one section. We also put the document title and the heading at the top of each chunk, so the chunk still says what it is about. Here is one real chunk:
Bready or Not Policies: Opening hours
- Monday: closed
- Tuesday: closed
- Wednesday and Thursday: 6:30 am to 2:00 pm
- Friday: 6:30 am to 7:00 pm. We bake a second batch at 4:00 pm for the evening ferry crowd.
- Saturday: 7:00 am to 3:00 pm
- Sunday: 7:00 am to 1:00 pmSetting up
Three small things before the code.
First, we need NumPy for the vector math. Your requirements.txt now looks like this:
openai==3.52.0
python-dotenv==1.2.4
numpy==2.5.5Install it from inside your virtual environment:
pip install -r requirements.txtSecond, add the embedding model to the end of your .env:
# The model that turns text into vectors for retrieval. It uses the same base URL and key.
EMBED_MODEL=openai/text-embedding-3-smallOpenRouter serves embedding models through the same OpenAI-compatible API as chat models, so no new key is needed.
Third, the bot needs something to read. Copy the content folder from the repo into your my-chatbot folder. It holds four markdown files about the bakery: about.md, faq.md, menu.md, and policies.md. Like the bakery itself, all of it is made up. Here is the top of policies.md, so you can see what Crumb will be reading:
# Bready or Not Policies
> Fictional sample content for a tutorial.
## Opening hours
- Monday: closed
- Tuesday: closed
- Wednesday and Thursday: 6:30 am to 2:00 pm
- Friday: 6:30 am to 7:00 pm. We bake a second batch at 4:00 pm for the evening ferry crowd.
- Saturday: 7:00 am to 3:00 pm
- Sunday: 7:00 am to 1:00 pmYou can use your own markdown files instead. Give each file a `# ` title and put each topic under its own `## ` heading. Text before the first `## ` heading is skipped, so put the facts in the sections.
Building the index: ingest.py
Embedding every chunk on every question would be slow and wasteful, since the documents do not change between questions. So we do it once, ahead of time, and save the results to a file. That file is the index, and the script that builds it is ingest.py.
Create ingest.py next to chat.py:
"""Part 4: read the documents, split them into chunks, embed each chunk, and save the index.
Run it once, and again whenever the files in content/ change:
python part-4/ingest.py
"""
import json
import sys
from pathlib import Path
from openai import APIConnectionError, APIStatusError, OpenAI
from retrieve import INDEX_FILE, load_embedding_settings
CONTENT_DIR = Path(__file__).parent / "content"
def chunk_markdown(text: str, source: str) -> list[dict[str, str]]:
"""Split a markdown document into one chunk per '## ' section."""
chunks: list[dict[str, str]] = []
doc_title = ""
heading = ""
body: list[str] = []
def flush() -> None:
content = "\n".join(body).strip()
if heading and content:
chunks.append(
{"source": source, "heading": heading, "text": f"{doc_title}: {heading}\n{content}"}
)
for line in text.splitlines():
if line.startswith("# "):
doc_title = line[2:].strip()
elif line.startswith("## "):
flush()
heading = line[3:].strip()
body = []
else:
body.append(line)
flush()
return chunks
def main() -> None:
settings = load_embedding_settings()
files = sorted(CONTENT_DIR.glob("*.md"))
chunks: list[dict] = []
for path in files:
found = chunk_markdown(path.read_text(encoding="utf-8"), path.name)
print(f"{path.name}: {len(found)} chunks")
chunks.extend(found)
if not chunks:
sys.exit(f"No chunks found. Put markdown files with '## ' headings in {CONTENT_DIR}.")
client = OpenAI(base_url=settings["LLM_BASE_URL"], api_key=settings["LLM_API_KEY"])
try:
response = client.embeddings.create(
model=settings["EMBED_MODEL"], input=[chunk["text"] for chunk in chunks]
)
except (APIConnectionError, APIStatusError) as error:
sys.exit(f"Could not embed the chunks: {type(error).__name__}. Check your .env settings.")
for chunk, item in zip(chunks, response.data, strict=True):
chunk["vector"] = item.embedding
index = {"embed_model": settings["EMBED_MODEL"], "chunks": chunks}
INDEX_FILE.write_text(json.dumps(index), encoding="utf-8")
print(f"Processed {len(files)} files and {len(chunks)} chunks with {settings['EMBED_MODEL']}.")
print(f"Embedding tokens used: {response.usage.total_tokens if response.usage else 'unknown'}.")
print(f"Wrote {INDEX_FILE.name}.")
if __name__ == "__main__":
main()chunk_markdown() walks the file line by line. A # ` line sets the document title. A `## ` line closes the previous section and starts a new one. Everything else is body text for the current section. The small `flush() function saves a section as a chunk, but only if it has both a heading and some text, which is why anything above the first `## ` is skipped. Each chunk keeps its file name and heading as well as its text, so we can show where an answer came from.
main() collects the chunks from every .md file in content/, then sends all of their texts to the embedding model in a single call. The API accepts a list and returns one vector per item, in the same order, so zip() can pair each chunk with its vector. The strict=True makes Python complain if the counts ever differ, instead of quietly pairing the wrong things.
The index records which embedding model built it. That matters because vectors from different embedding models are not comparable. A question embedded with one model and chunks embedded with another would produce scores, and the scores would be nonsense. We will check for that when the chat starts.
ingest.py imports two names from retrieve.py, which we write next. Do not run it yet.
Finding passages: retrieve.py
retrieve.py holds everything needed to answer one question: read the settings, embed some text, load the index, and score the chunks. It also runs on its own, which turns out to be the most useful debugging tool in this whole part.
Create retrieve.py next to chat.py:
"""Part 4: find the chunks that are most related to a question.
Run it by itself to see what retrieval returns:
python part-4/retrieve.py "What time do you close on Friday?"
"""
import json
import os
import sys
from pathlib import Path
import numpy as np
from dotenv import load_dotenv
from openai import APIConnectionError, APIStatusError, OpenAI
INDEX_FILE = Path(__file__).parent / "index.json"
# How many chunks go into each request. Always this many, whatever the scores are.
TOP_K = 3
Chunk = dict[str, str | list[float]]
def load_embedding_settings() -> dict[str, str]:
"""Read the embedding settings from .env and stop early if one is missing."""
load_dotenv()
names = ("LLM_BASE_URL", "LLM_API_KEY", "EMBED_MODEL")
missing = [name for name in names if not os.environ.get(name)]
if missing:
sys.exit(f"Missing settings in .env: {', '.join(missing)}")
return {name: os.environ[name] for name in names}
def load_index(embed_model: str, path: Path = INDEX_FILE) -> list[Chunk]:
"""Load the saved chunks. Stop with a clear message if the index cannot be used."""
if not path.exists():
sys.exit("No index.json found. Run 'python part-4/ingest.py' first.")
index = json.loads(path.read_text(encoding="utf-8"))
if index["embed_model"] != embed_model:
sys.exit(
f"index.json was built with {index['embed_model']}, but EMBED_MODEL is "
f"{embed_model}. Run 'python part-4/ingest.py' again."
)
return index["chunks"]
def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
"""How closely two vectors point the same way. Higher means more related."""
return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))
def top_chunks(
question_vector: list[float], chunks: list[Chunk], k: int = TOP_K
) -> list[tuple[float, Chunk]]:
"""Score every chunk against the question and return the best k."""
query = np.array(question_vector)
scored = [(cosine_similarity(query, np.array(chunk["vector"])), chunk) for chunk in chunks]
return sorted(scored, key=lambda pair: pair[0], reverse=True)[:k]
def retrieve(
client: OpenAI, embed_model: str, chunks: list[Chunk], question: str
) -> list[tuple[float, Chunk]]:
"""Embed the question and return the most related chunks with their scores."""
response = client.embeddings.create(model=embed_model, input=question)
return top_chunks(response.data[0].embedding, chunks)
def chunk_label(chunk: Chunk) -> str:
return f"{chunk['source']} / {chunk['heading']}"
def main() -> None:
if len(sys.argv) != 2:
sys.exit('Usage: python part-4/retrieve.py "your question"')
question = sys.argv[1]
settings = load_embedding_settings()
chunks = load_index(settings["EMBED_MODEL"])
client = OpenAI(base_url=settings["LLM_BASE_URL"], api_key=settings["LLM_API_KEY"])
try:
results = retrieve(client, settings["EMBED_MODEL"], chunks, question)
except (APIConnectionError, APIStatusError) as error:
sys.exit(f"Could not embed the question: {type(error).__name__}.")
print(f"Question: {question}\n")
for rank, (score, chunk) in enumerate(results, start=1):
print(f"{rank}. score {score:.3f} [{chunk_label(chunk)}]")
print(f"{chunk['text']}\n")
if __name__ == "__main__":
main()load_embedding_settings() reads only what the embedding side needs, so ingest.py and retrieve.py work without a chat model configured. load_index() stops with a clear message if there is no index yet, or if it was built with a different embedding model than the one in .env.
The search itself is top_chunks(). It scores the question against every chunk, sorts by score, and keeps the best TOP_K, which is 3. Scoring every chunk sounds expensive, and for 24 chunks it is nothing. Note the comment on TOP_K: we always send three chunks, whatever their scores. We will come back to that in a bit.
retrieve() ties it together: one API call to embed the question, then top_chunks() on the result. This is the function the chat will call.
Run ingest
Now build the index:
python ingest.pyabout.md: 5 chunks
faq.md: 8 chunks
menu.md: 4 chunks
policies.md: 7 chunks
Processed 4 files and 24 chunks with openai/text-embedding-3-small.
Embedding tokens used: 1481.
Wrote index.json.Embedding models are cheap. At the price OpenRouter lists for this model, those 1,481 tokens cost a small fraction of a cent.
Open index.json and you will find one entry per chunk. The file is one long line, so here is the first entry, reformatted, with the vector cut down to its first four numbers:
{
"source": "about.md",
"heading": "Our story",
"text": "About Bready or Not: Our story\nBready or Not opened in March 2019 on Lantern Row in the harbor town of Port Gallant. It was founded by Marguerite \"Mags\" Okonkwo-Reyes, a former tugboat engineer who started baking sourdough to stay awake on night shifts. When the crew began paying her for loaves, she took the hint.",
"vector": [
0.01006317138671875,
0.0222015380859375,
-0.01383209228515625,
0.0183258056640625
]
}The real vector has 1,536 numbers. Multiply that by 24 chunks and the file comes to about 750 KB, for less than 6 KB of text. Nobody said meaning was compact.
Try retrieval on its own
Before Crumb sees any of this, ask retrieve.py a question directly. No chat model is involved, so this is purely the search:
python retrieve.py "What time do you close on Friday?"Question: What time do you close on Friday?
1. score 0.392 [faq.md / When do things sell out?]
Bready or Not FAQ: When do things sell out?
Croissants are usually gone by 10:00 am on weekends. The second bake on Friday at 4:00 pm is the best time to find warm bread late in the day.
2. score 0.370 [policies.md / Opening hours]
Bready or Not Policies: Opening hours
- Monday: closed
- Tuesday: closed
- Wednesday and Thursday: 6:30 am to 2:00 pm
- Friday: 6:30 am to 7:00 pm. We bake a second batch at 4:00 pm for the evening ferry crowd.
- Saturday: 7:00 am to 3:00 pm
- Sunday: 7:00 am to 1:00 pm
3. score 0.355 [policies.md / Preorders and pickup]
Bready or Not Policies: Preorders and pickup
Bread and pastry preorders can be picked up from 8:00 am until one hour before closing on any day we are open. We hold preorders until closing. Anything not collected by then is donated, and it is not refunded. There is no pickup on Monday or Tuesday because nobody is here.Read the scores. The Opening hours chunk, which holds the answer, comes second at 0.370. First place went to the chunk about when things sell out, at 0.392, because it mentions Friday and a time of day. The third chunk is about pickup times, which is related but not helpful.
Your scores may differ from these in the third decimal place. The ranking is what matters.
This is how retrieval usually looks. The right chunk is in the results, but not always on top, and the scores are close together. If we sent only the best chunk, this question would get the wrong one. With three, the answer is in the request, and the model can find it.
A question about price lands more clearly:
python retrieve.py "How much is a croissant?"Question: How much is a croissant?
1. score 0.529 [menu.md / Pastries]
Bready or Not Menu: Pastries
- Croissant Examination: butter croissant. $4.75. Contains wheat and milk.
- Almond Joyride: almond croissant. $5.75. Contains wheat, milk, egg, and tree nuts (almonds).
- Sconed Thoughts: lemon and thyme scone. $4.25. Contains wheat, milk, and egg.
- Cinnamon Scroll of Truth: cinnamon roll with brown butter glaze. $5.50. Contains wheat, milk, and egg.
- Muffin to See Here: blueberry muffin. $4.00. Contains wheat, milk, and egg.
- Flour Child: flourless chocolate cake, by the slice. $6.50. Contains milk and egg. Made without wheat, but in a kitchen full of it.
2. score 0.488 [faq.md / When do things sell out?]
3. score 0.351 [menu.md / Breads](Trimmed. The real output prints the text of all three chunks.) The pastry menu wins, and the croissant price is in it. The sell-out chunk comes second because it talks about croissants, and the bread menu comes third, because a question about price is a little like every menu section.
Try your own questions here. Ask the same thing in different words and watch how the scores move. This is the quickest way to get a feel for what embeddings match well and what they miss.
Wiring it into the chat
Now Crumb gets to use it. Open chat.py. Four parts change.
First, replace everything from the top of the file down to and including the REQUIRED_SETTINGS line with this:
"""Part 4: a chatbot that answers from the bakery documents by retrieving passages first."""
import argparse
import os
import sys
from pathlib import Path
from dotenv import load_dotenv
from openai import APIConnectionError, APIStatusError, OpenAI
from retrieve import Chunk, chunk_label, load_index, retrieve
REQUIRED_SETTINGS = ("LLM_BASE_URL", "LLM_API_KEY", "LLM_MODEL", "EMBED_MODEL")There is a new import of argparse for a --debug flag, an import of the retrieval pieces from retrieve.py, and EMBED_MODEL is now a required setting.
Second, add a new function directly below build_messages(). Here it is with build_messages() above it, so you can see where it goes:
def build_messages(system_message: Message, history: list[Message]) -> list[Message]:
"""The system prompt always goes first. Only the conversation is trimmed."""
return [system_message] + trim_history(history)
def build_user_message(question: str, scored_chunks: list[tuple[float, Chunk]]) -> str:
"""Put the retrieved passages and the question into one user message."""
context = "\n\n".join(f"[{chunk_label(chunk)}]\n{chunk['text']}" for _, chunk in scored_chunks)
return f"Information from the bakery's documents:\n\n{context}\n\nCustomer question: {question}"build_user_message() is where the retrieved chunks meet the question. It puts each chunk under a label with its file and heading, then adds the customer's question at the end. For the Friday question, the user message the model receives looks like this:
Information from the bakery's documents:
[faq.md / When do things sell out?]
Bready or Not FAQ: When do things sell out?
Croissants are usually gone by 10:00 am on weekends. ...
[policies.md / Opening hours]
Bready or Not Policies: Opening hours
- Monday: closed
...
[policies.md / Preorders and pickup]
Bready or Not Policies: Preorders and pickup
...
Customer question: What time do you close on Friday?(Trimmed with the dots.) The chunks go in the user message, next to the question they belong to, not in the system prompt. The system prompt stays the same on every turn, and the chunks change with every question.
Third, replace take_turn() with this version:
def take_turn(
client: OpenAI,
model: str,
system_message: Message,
temperature: float,
history: list[Message],
question: str,
embed_model: str,
chunks: list[Chunk],
debug: bool = False,
) -> tuple[int, int]:
"""Retrieve passages, stream the reply, then remember the question and the reply.
The request carries the passages next to the question. The history keeps only the plain
question, so old passages do not pile up. If a call fails, the history is left unchanged.
"""
scored_chunks = retrieve(client, embed_model, chunks, question)
if debug:
for score, chunk in scored_chunks:
print(f"[retrieved {score:.3f} {chunk_label(chunk)}]")
user_message = {"role": "user", "content": build_user_message(question, scored_chunks)}
messages = build_messages(system_message, history) + [user_message]
print("Crumb: ", end="", flush=True)
reply, prompt_tokens, completion_tokens = stream_reply(client, model, messages, temperature)
history.append({"role": "user", "content": question})
history.append({"role": "assistant", "content": reply})
return prompt_tokens, completion_tokensEach turn now starts with retrieve(). With --debug, the chunks and their scores are printed before the reply. Then the user message is built from the chunks and the question, and the rest is the same as in Part 3.
Look at what goes into history at the end, though. It is the plain question, not the big message with the chunks in it. That is on purpose. The chunks were picked for this question. If they went into the history, every later request would carry the chunks from every earlier question too. The request would grow by about 300 tokens a turn, which is the Part 2 cost problem all over again, and old chunks about croissants would sit in the request competing with new chunks about cakes. Only the current question gets chunks.
Fourth, replace main() with this version:
def main() -> None:
parser = argparse.ArgumentParser(description="Chat with Crumb, the Bready or Not assistant.")
parser.add_argument("--debug", action="store_true", help="show the retrieved chunks and scores")
args = parser.parse_args()
settings = load_settings()
chunks = load_index(settings["EMBED_MODEL"])
client = OpenAI(base_url=settings["LLM_BASE_URL"], api_key=settings["LLM_API_KEY"])
model = settings["LLM_MODEL"]
embed_model = settings["EMBED_MODEL"]
temperature = read_temperature()
system_message = load_system_prompt()
history: list[Message] = []
print(f"Chatting with Crumb ({model}, temperature {temperature}, {len(chunks)} chunks loaded).")
print("Type 'reset' to forget the conversation, 'quit' to exit.")
while True:
try:
question = input("\nYou: ").strip()
except (EOFError, KeyboardInterrupt):
print()
break
if question.lower() in {"quit", "exit"}:
break
if question.lower() == "reset":
history.clear()
print("Conversation cleared.")
continue
if not question:
continue
try:
prompt_tokens, completion_tokens = take_turn(
client,
model,
system_message,
temperature,
history,
question,
embed_model,
chunks,
args.debug,
)
except (APIConnectionError, APIStatusError) as error:
print(f"\nError: {describe_error(error)}")
continue
print(f"[tokens: {prompt_tokens} in, {completion_tokens} out]")main() reads the --debug flag, loads the index once at startup, and hands the chunks and the embedding model to take_turn(). The banner now says how many chunks are loaded, which is a quick way to notice that you forgot to run ingest after adding a file.
Updating rule 3
Last, the system prompt. In Part 3, rule 3 told Crumb not to invent facts, with nothing to answer from. Now there is something to answer from, so the rule should point at it. Open system_prompt.md and replace rule 3 with this line:
3. Answer questions about the bakery using only the information from the bakery's documents that comes with each question. If the answer is not there, say "I do not know that yet" and suggest calling the shop. Never guess a price, a time, or a policy.It tells the model where the facts are, what to say when they are missing, and the three things it must never guess. The rest of the prompt is unchanged.
The payoff
Start the bot with the debug flag, so you can see what it retrieved:
python chat.py --debugAsk a question whose answer is split across two documents. The price of a quarter sheet cake is on the menu, and the notice it needs is in the policies:
Chatting with Crumb (meta-llama/llama-3.1-8b-instruct, temperature 0.3, 24 chunks loaded).
Type 'reset' to forget the conversation, 'quit' to exit.
You: How much is a quarter sheet cake, and how much notice do you need?
[retrieved 0.616 policies.md / Custom cake orders]
[retrieved 0.442 menu.md / Custom cakes]
[retrieved 0.372 policies.md / Refunds and replacements]
Crumb: A quarter sheet cake costs $95.00. According to our policies, we need at least 5 business days of notice for a quarter sheet cake order.
[tokens: 520 in, 33 out]Both chunks were retrieved, and Crumb used both. Here is the vegan question, in a fresh session:
You: Which breads are vegan?
[retrieved 0.664 faq.md / Do you have vegan options?]
[retrieved 0.607 menu.md / Breads]
[retrieved 0.475 menu.md / Pastries]
Crumb: We have three vegan bread options: The Loaf Actually, Rye Me a River, and Gluten for Punishment. All three are free from animal products.
[tokens: 596 in, 33 out]That is the right list. The second sentence is Crumb restating what "vegan" means, which is fine. And because every bakery needs one:
You: Who is Gerald?
[retrieved 0.559 about.md / Gerald]
[retrieved 0.308 faq.md / Do you teach classes?]
[retrieved 0.162 about.md / Where to find us]
Crumb: Gerald is our beloved sourdough starter, born in a jam jar aboard a tugboat in 2016. He's the foundation of every sourdough loaf we sell, and he's quite the celebrity around here!
[tokens: 445 in, 47 out]Compare that with the made-up Saturday hours at the top of this article. Same model, same temperature. The only difference is that the request now carries the right paragraph.
Look at the token counts as well. Part 3 requests went in at around 220 tokens. These go in at 450 to 600, because three chunks ride along with each question. That is the cost of context. It is still much less than the whole set of documents, and it stays about the same size however many documents you add.
Where this breaks
Retrieval gives the model facts. It does not make the model use them well, and it has its own blind spots.
The answer is not in the documents
There is no bagel anywhere in the bakery's documents, so the right answer is "I do not know that yet". Ask:
You: Do you sell bagels?
[retrieved 0.489 faq.md / Do you sell wholesale?]
[retrieved 0.424 faq.md / When do things sell out?]
[retrieved 0.399 faq.md / Do you teach classes?]
Crumb: We do not have a specific bagel product listed in our FAQ, but I can suggest checking our menu for the latest offerings. We're a small bakery, and our selection can vary from day to day. Would you like me to check if we have any bagels available today?
[tokens: 421 in, 58 out]It starts well. Then Crumb says the selection varies from day to day, which no document says, and offers to check today's stock, which it cannot do. Run it a few times and most replies add something invented, such as a website or a ciabatta the bakery does not make.
Retrieval did its job here. There is no bagel chunk to find, so it returned the three closest chunks it had. The model had a rule to use only the documents, and it filled the gap anyway, the same habit we saw in Part 3. Retrieval makes that less likely, not impossible. A larger LLM_MODEL usually does better.
The scores are not probabilities
The bagel question scored 0.489 against a chunk about wholesale. The Opening hours chunk scored only 0.370 for the Friday question, and that one was the right answer. A score only ranks the chunks for one question. It is not a percentage, and scores from different questions cannot be compared.
That is also why there is no cut-off. We always send the top three, related or not. Ask "What is the capital of France?" and all three chunks score below 0.18 and get sent anyway. Crumb declines because the system prompt says to, not because retrieval noticed. A threshold sounds easy, but the right value depends on the embedding model and the documents, so we leave it out.
Follow-up questions
Retrieval embeds only the newest question. The history goes to the chat model, but not to the search. Here is the second turn of a session where Crumb had just answered "How much is the Almond Joyride?" correctly:
You: How much is it?
[retrieved 0.278 faq.md / Is there parking?]
[retrieved 0.265 menu.md / Drinks]
[retrieved 0.255 policies.md / Discounts]
Crumb: The ferry lot charges $2.00 an hour. There are also four free 30 minute spots directly in front of the shop.
[tokens: 492 in, 28 out]On its own, "it" could mean anything. The pastry menu was not retrieved, the parking chunk was, and Crumb gave parking prices with complete confidence.
Follow-ups that name something survive better. "And on Sunday?" after a Saturday question pulled the Opening hours chunk, but only in third place, behind a sourdough class. It made it in because we send three.
The right chunk, the wrong answer
Even with the answer in front of it, a small model can fumble. Here is a follow-up after "What time do you open on Saturday?":
You: And what about the other days?
[retrieved 0.264 policies.md / Opening hours]
[retrieved 0.229 policies.md / Preorders and pickup]
[retrieved 0.227 faq.md / When do things sell out?]
Crumb: I do not know that yet. It seems we only have information about weekend croissant sales. If you're looking for more specific information, I recommend calling the shop for the most up-to-date details.
[tokens: 559 in, 42 out]The Opening hours chunk came first, with every day of the week in it. Crumb read the sell-out chunk instead. Retrieval did its job, and the model did not.
The possible fix for follow-ups is to rewrite the question with the conversation before searching, so "How much is it?" becomes "How much is the Almond Joyride?". We do not build that here.
Bringing your own content
To use your own documents, put markdown files in content/ with a # ` title and `## ` sections, and run `python ingest.py again. Delete the sample files if you only want yours.
Run ingest again whenever the content changes, since the index only knows what was there when you built it. Also run it again if you change EMBED_MODEL. Vectors from different embedding models cannot be compared, and the chat will refuse to start with a mismatched index rather than give you meaningless scores.
Our index is a JSON file, and every question is scored against every chunk. That is perfectly fine for hundreds of chunks. When people have hundreds of thousands, they usually reach for a vector database, which stores the vectors and finds the nearest ones quickly without checking each one.
What comes next
Crumb now reads documents and answers from them. Documents can contain instructions as well as facts, though, and in Part 5 we look at what happens when someone puts the wrong ones there.