
Building your own AI chatbot: Adding chat history (Part 2)
Learning project. This code is written to teach concepts. It is not hardened for production. Anything you put in front of real users needs proper security, testing, and monitoring first.
At the end of Part 1, you had a bot running, but it had no awareness of other messages in the conversation. There was no memory or context. In this part we fix that. By the end, the bot will remember the conversation, and you will start to see how the added context raises token usage and cost.
The complete code for this part is in the part-2 folder of the repo. We continue from your Part 1 my-chatbot folder. The packages and the .env file stay the same. All the changes are in chat.py.
The model does not remember you
Here is the part that surprises most people. The model never remembers anything. Not your name, not the previous message, not the fact that you are mid-conversation. Every request starts from nothing.
So how do chat apps remember? They cheat, honestly and in plain sight. The app keeps a list of every message in the conversation, yours and the bot's, and sends the entire list with every new question. The model reads the whole transcript from the top, then writes the next reply. It looks like memory from the outside. On the inside, it is the model rereading the whole conversation every time you press Enter.
That means memory is our job, not the model's. In Part 1, the messages list we sent always held exactly one message. In Part 2 it will hold the conversation so far, plus the new question.
The code change
Open chat.py. Three things change: the top of the file, the ask() function, and main(). Everything else (STATUS_HINTS, load_settings(), and describe_error()) stays exactly as it was.
First, replace everything above STATUS_HINTS with this:
"""Part 2: a chatbot that remembers the conversation and streams its replies."""
import os
import sys
from dotenv import load_dotenv
from openai import APIConnectionError, APIStatusError, OpenAI
REQUIRED_SETTINGS = ("LLM_BASE_URL", "LLM_API_KEY", "LLM_MODEL")
# How many saved messages are sent with each request, plus the new question. Keep this even:
# saved history is whole question and answer pairs, so the trimmed list starts with a user message.
MAX_HISTORY_MESSAGES = 20
Message = dict[str, str]The imports are the same as before. The new pieces are MAX_HISTORY_MESSAGES, which we will get to in the trimming section, and Message, a short name for the {"role": ..., "content": ...} dictionaries so the function signatures stay readable.
Next, delete the whole ask() function and put these three functions in its place, between load_settings() and describe_error():
def trim_history(history: list[Message], max_messages: int = MAX_HISTORY_MESSAGES) -> list[Message]:
"""Return only the most recent saved messages, so the request stays a sensible size."""
return history[-max_messages:]
def stream_reply(client: OpenAI, model: str, messages: list[Message]) -> tuple[str, int, int]:
"""Print the reply as it arrives. Return the full text and the token counts."""
stream = client.chat.completions.create(model=model, messages=messages, stream=True)
parts: list[str] = []
prompt_tokens = 0
completion_tokens = 0
for chunk in stream:
# The token counts arrive on the last chunk, so look at every chunk for them.
if chunk.usage:
prompt_tokens = chunk.usage.prompt_tokens
completion_tokens = chunk.usage.completion_tokens
if not chunk.choices:
continue
text = chunk.choices[0].delta.content or ""
print(text, end="", flush=True)
parts.append(text)
print()
return "".join(parts), prompt_tokens, completion_tokens
def take_turn(client: OpenAI, model: str, history: list[Message], question: str) -> tuple[int, int]:
"""Stream the reply, then remember the question and the reply.
If the model call fails, the history is left unchanged.
"""
messages = trim_history(history) + [{"role": "user", "content": question}]
print("Bot: ", end="", flush=True)
reply, prompt_tokens, completion_tokens = stream_reply(client, model, messages)
history.append({"role": "user", "content": question})
history.append({"role": "assistant", "content": reply})
return prompt_tokens, completion_tokensFinally, replace main() with this version:
def main() -> None:
settings = load_settings()
client = OpenAI(base_url=settings["LLM_BASE_URL"], api_key=settings["LLM_API_KEY"])
model = settings["LLM_MODEL"]
history: list[Message] = []
print(f"Chatting with {model}. Type 'reset' to forget the conversation, 'quit' to exit.")
while True:
try:
question = input("\nYou: ").strip()
except (EOFError, KeyboardInterrupt):
print()
break
if question.lower() in {"quit", "exit"}:
break
if question.lower() == "reset":
history.clear()
print("Conversation cleared.")
continue
if not question:
continue
try:
prompt_tokens, completion_tokens = take_turn(client, model, history, question)
except (APIConnectionError, APIStatusError) as error:
print(f"\nError: {describe_error(error)}")
continue
print(f"[tokens: {prompt_tokens} in, {completion_tokens} out]")The memory itself is the history list that main() creates before the loop. It starts empty and lives for as long as the program runs.
take_turn() does the remembering. It builds the outgoing messages from the saved history plus your new question, sends them, and waits for the reply. Only after the reply has arrived does it append both your question and the reply to history. If the call fails partway, nothing is saved, so a network hiccup cannot leave a question without an answer sitting in the conversation.
There is also a new reset command, which empties the list. It is three lines, and it turns out to be the most satisfying way to watch the bot lose its memory on purpose.
Run it and try the test from Part 1 again:
python chat.pyChatting with meta-llama/llama-3.1-8b-instruct. Type 'reset' to forget the conversation, 'quit' to exit.
You: My name is Ben.
Bot: Nice to meet you, Ben! Is there anything I can help you with or would you like to chat about something in particular?
[tokens: 15 in, 27 out]
You: What is my name?
Bot: Your name is Ben.
[tokens: 56 in, 6 out]Ben is remembered. Here is the messages list the second request sent:
[
{"role": "user", "content": "My name is Ben."},
{"role": "assistant", "content": "Nice to meet you, Ben! Is there anything I can help you with or would you like to chat about something in particular?"},
{"role": "user", "content": "What is my name?"}
]In Part 1, that list held only the last line. Now it carries the first question, the bot's first reply (saved with the role assistant, exactly as it came back), and the new question. The model did not remember Ben. We handed it the conversation, and it read the answer off the page.
That is also why the second token count went up. In Part 1 the same question went in at 15 tokens. Now it is 56, because the whole list is counted. We will come back to that in a bit.
Streaming
You probably noticed the reply did not arrive in one piece. It appeared a few words at a time, the way the chat apps do it. That is streaming.
Without streaming, the service generates the whole reply and then sends it, and you stare at a blank prompt until it is done. With stream=True, the service sends the reply in small chunks as the model produces them. The total time is about the same, but the first words show up almost immediately, and people are far more patient with text that is visibly being written than with a cursor that is visibly doing nothing.
The change is small. In stream_reply(), create() gets stream=True and returns something you loop over instead of a finished response. Each chunk carries a little piece of the reply in chunk.choices[0].delta.content, often a single word or part of one. When I asked a test question, the reply "Hello, how are you?" arrived as Hell, then o, then , how are you?. The loop prints each piece the moment it arrives (flush=True makes sure the terminal shows it right away) and collects the pieces so the full reply can be saved to the history.
The token counts arrive differently when streaming. There is no finished response to read usage from, so the service sends the counts on the last chunk. That is why the loop checks chunk.usage on every chunk and keeps whatever it finds. OpenRouter sends these counts on a streamed reply without being asked. Other OpenAI-compatible services may only send them if the request includes stream_options={"include_usage": True}, and without it the token line will show zeros. OpenAI's own API works that way. Its final usage chunk also has an empty choices list, which is what the if not chunk.choices: continue line is there for.
What memory costs
That jump from 15 to 56 tokens was not a one-off. Here are five questions in a row, from a fresh start:
You: Name one fruit.
Bot: Apple.
[tokens: 14 in, 3 out]
You: Name another one.
Bot: Orange.
[tokens: 30 in, 3 out]
You: Name a third one.
Bot: Banana.
[tokens: 47 in, 4 out]
You: Name a fourth one.
Bot: Mango.
[tokens: 65 in, 4 out]
You: Which fruits have you named so far?
Bot: The fruits I've named so far are:
1. Apple
2. Orange
3. Banana
4. Mango
[tokens: 86 in, 25 out]The replies are tiny, three or four tokens each, but the input count climbs every turn: 14, 30, 47, 65, 86. Every question pays again for every message before it. You are billed for input tokens, so the cost of a single message keeps rising for as long as the conversation runs. Fruit is cheap. A long technical conversation with pasted code is not.
There is also a hard ceiling. Every model has a context window, which is the most tokens it can read in one request. The whole request, the entire history plus the new question, has to fit inside it. Context windows are large these days, and you can find each model's on its page at OpenRouter, but a list that only ever grows will reach the ceiling eventually. When it does, the service will either refuse the request or cut it down for you, and you will not enjoy either.
So an endless history costs more on every turn and eventually stops fitting. We need to take care as it grows.
Trimming
The simplest fix is to send only the most recent messages. That is all trim_history() does:
def trim_history(history: list[Message], max_messages: int = MAX_HISTORY_MESSAGES) -> list[Message]:
"""Return only the most recent saved messages, so the request stays a sensible size."""
return history[-max_messages:]It is one slice. take_turn() calls it on the saved history, then adds the new question on the end:
messages = trim_history(history) + [{"role": "user", "content": question}]With the default of 20, each request carries the last ten questions and answers, plus the new question. The full history is still kept in history. Trimming only decides how much of it gets sent.
Why must the limit be even? Messages are saved in pairs, a question and then its answer, so the saved history always alternates user, assistant, user, assistant. Take an even number off the end and the slice starts on a user message. Take an odd number and it starts on an assistant reply to a question the model never sees, which is confusing for the model and pointless for you.
This is, of course, a very simplified introduction to this topic. There are several sophisticated methods of handling chat history in practice that we will not get into in this series.
If you cloned the repo, part-2/tests/test_history.py checks the trimming and the history handling: that the slice keeps the right messages, that it starts with a user message, that a successful turn saves the question and the reply in order, and that a failed call saves nothing. Install part-2/requirements-dev.txt and run pytest part-2. The tests need no network and no API key. You do not need them to follow along.
Where this breaks
Trimming keeps the cost under control by forgetting things. To see it happen, set MAX_HISTORY_MESSAGES to 4 in chat.py, so the bot sees only the last two questions and answers. Then introduce yourself and make some small talk:
Chatting with meta-llama/llama-3.1-8b-instruct. Type 'reset' to forget the conversation, 'quit' to exit.
You: My name is Ben.
Bot: Nice to meet you, Ben! Is there anything I can help you with or would you like to chat about something in particular?
[tokens: 15 in, 27 out]
You: Name one fruit.
Bot: Apple.
[tokens: 55 in, 3 out]
You: Name another fruit.
Bot: Banana.
[tokens: 71 in, 4 out]
You: Name a third fruit.
Bot: Orange.
[tokens: 48 in, 3 out]
You: What is my name?
Bot: I don't know your name. This conversation just started, and I don't have any information about you. Would you like to tell me your name?
[tokens: 49 in, 32 out]Ben has fallen off the front of the list. Follow the input counts and you can see the exact moment. They grow from 15 to 55 to 71 while everything still fits, then drop to 48 on the fourth turn. That is the request where your name was trimmed away. From then on, the bot is as clueless as it was in Part 1, and it says so with the same friendly confidence.
The bot is also wrong about the details. It says the conversation just started, when we are five questions in. From where it sits, that is a fair reading: the oldest thing it can see is a question about fruit.
A real limit of 20 only delays this. Any fixed window forgets whatever is older than the window, and it does not care whether that was small talk or the one fact that mattered. Set the limit back to 20 when you are done.
There are better ways to decide what to drop, and we will not build them here. One is to count tokens instead of messages, so a short "thanks" and a pasted error log are not treated as the same size. Another is to ask the model to summarize the oldest messages and send the short summary in their place, so the bot keeps the gist of what it would otherwise forget.
In Part 3, the bot gets a personality: a system prompt that tells it who it is and how to behave, before you say a word.