Ben Lobaugh
  • Growth & Leadership
  • Giving Back
  • Sailing
  • VW Bus
  • About
Building your own AI chatbot: Basic bot setup (Part 1)

Building your own AI chatbot: Basic bot setup (Part 1)

Learning project. This code is written to teach concepts. It is not hardened for production. Anything you put in front of real users needs proper security, testing, and monitoring first.

Over five parts, this series teaches you how to build a chatbot from scratch in plain Python, in the console, with no frameworks. In Part 4 RAG is introduced and you will learn how to teach the bot with your own training data. The tutorial will train the bot to use a small fictional bakery using the bakery's own documents.

Part 1 is the foundation: a bot that answers you, but does not retain details about a conversation.

The complete code for this part is in the part-1 folder of the repo.

What you need

You need Python 3.11 or newer. Check with python3 --version before anything else.

I will use OpenRouter for this series. Any OpenAI-compatible service can be used. OpenRouter puts models from many different companies behind one API and one bill, which makes it a comfortable place to experiment.

You will also need an API key. To create one, sign in to OpenRouter and open the keys page at openrouter.ai/keys.

Set up the project

Make an empty folder, step into it, and create a virtual environment so this project's packages stay out of everything else on your machine:

mkdir my-chatbot
cd my-chatbot
python3 -m venv .venv
source .venv/bin/activate

On Windows, the activate step is .venv\Scripts\activate instead. Every file from here on goes directly in my-chatbot.

Create requirements.txt with the two packages we need:

openai==3.52.0
python-dotenv==1.2.4

The openai package is the official client for OpenAI's API, and can communicate with any OpenAI-compatible API.

Install them:

pip install -r requirements.txt

Now the settings. Create .env:

# Any OpenAI-compatible service works. These values are for OpenRouter.
LLM_BASE_URL=https://openrouter.ai/api/v1
LLM_API_KEY=your-key-here
# Pick any chat model from https://openrouter.ai/models
LLM_MODEL=meta-llama/llama-3.1-8b-instruct

Replace your-key-here with the key you just created.

Text in, text out

Before the code, a quick word on what we are actually calling.

A large language model (LLM) is a program trained on an enormous amount of text to predict what text should come next. You send it text, it sends text back. That is the whole interface. There is no session and no login state on the model's side, and nothing about you carries over from one request to the next. Remember that last part, because it is a key principle to creating a usable chatbot.

Chat services wrap that text in a little structure. Instead of one string, you send a list of messages, and each message is a small dictionary with two fields. The role field says who is speaking, and content holds what they said. Your messages have the role user. The model's replies come back with the role assistant. There is also a system role for standing instructions to the model, which we will put to work in Part 3.

Along with the messages, you send a model id that says which model should answer. On OpenRouter, ids look like meta-llama/llama-3.1-8b-instruct: the organization that made the model, a slash, and the model name.

Put together, the body of a request is a small piece of JSON. This is the exact request our bot sends when you ask it about France:

{
  "model": "meta-llama/llama-3.1-8b-instruct",
  "messages": [
    {"role": "user", "content": "What is the capital of France?"}
  ]
}

That is the whole thing. Your API key travels separately in an HTTP header, so the service knows who to bill, but the model never sees it. What the model sees is the messages list, and only the messages list. If something is not in that list, as far as the model is concerned it does not exist.

The request shape we are using was popularized by OpenAI's API, and so many services copied it that it became a de facto standard. When a service calls itself "OpenAI-compatible", it means it accepts that same request at its own address. OpenRouter does, and so do local model servers like Ollama. That is why our .env has a base URL in it: the official openai package works with any of them once you point it somewhere else.

The code

Create chat.py:

"""Part 1: the simplest possible chatbot. One message in, one reply out."""

import os
import sys

from dotenv import load_dotenv
from openai import APIConnectionError, APIStatusError, OpenAI

REQUIRED_SETTINGS = ("LLM_BASE_URL", "LLM_API_KEY", "LLM_MODEL")

STATUS_HINTS = {
    400: "The service rejected the request. Check that LLM_MODEL is a valid model name.",
    401: "The API key was rejected. Check LLM_API_KEY in .env.",
    402: "The account is out of credit.",
    404: "The model was not found. Check LLM_MODEL in .env.",
    429: "Rate limited. Wait a moment and try again.",
}


def load_settings() -> dict[str, str]:
    """Read settings from .env and stop early if one is missing."""
    load_dotenv()
    missing = [name for name in REQUIRED_SETTINGS if not os.environ.get(name)]
    if missing:
        sys.exit(f"Missing settings in .env: {', '.join(missing)}")
    return {name: os.environ[name] for name in REQUIRED_SETTINGS}


def ask(client: OpenAI, model: str, question: str) -> tuple[str, int, int]:
    """Send one message to the model. Return the reply and the token counts."""
    response = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": question}],
    )
    reply = response.choices[0].message.content or ""
    usage = response.usage
    prompt_tokens = usage.prompt_tokens if usage else 0
    completion_tokens = usage.completion_tokens if usage else 0
    return reply, prompt_tokens, completion_tokens


def describe_error(error: Exception) -> str:
    """Turn an API failure into one short line. Never include the key."""
    if isinstance(error, APIConnectionError):
        return "Could not reach the service. Check your network and LLM_BASE_URL."
    if isinstance(error, APIStatusError):
        default = f"The service returned an error (status {error.status_code})."
        return STATUS_HINTS.get(error.status_code, default)
    return "Something went wrong talking to the service."


def main() -> None:
    settings = load_settings()
    client = OpenAI(base_url=settings["LLM_BASE_URL"], api_key=settings["LLM_API_KEY"])
    model = settings["LLM_MODEL"]

    print(f"Chatting with {model}. Type 'quit' to exit.")
    while True:
        try:
            question = input("\nYou: ").strip()
        except (EOFError, KeyboardInterrupt):
            print()
            break
        if question.lower() in {"quit", "exit"}:
            break
        if not question:
            continue
        try:
            reply, prompt_tokens, completion_tokens = ask(client, model, question)
        except (APIConnectionError, APIStatusError) as error:
            print(f"Error: {describe_error(error)}")
            continue
        print(f"Bot: {reply}")
        print(f"[tokens: {prompt_tokens} in, {completion_tokens} out]")


if __name__ == "__main__":
    main()

Under eighty lines, and a good chunk of that is error handling. Let us walk through it in the order you are probably wondering about it.

The interesting part is ask(), because that is where the model gets called:

    response = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": question}],
    )

There is the messages list from the previous section, with exactly one message in it: whatever you just typed, with the role user. The client turns this into an HTTP POST to the chat completions endpoint at your base URL and hands back the parsed response.

Here is the response to the France question. I trimmed it to the fields that matter here. The full response also carries an id, the provider that served the request, the cost, and a few other details.

{
  "model": "meta-llama/llama-3.1-8b-instruct",
  "choices": [
    {
      "index": 0,
      "finish_reason": "stop",
      "message": {
        "role": "assistant",
        "content": "The capital of France is Paris."
      }
    }
  ],
  "usage": {
    "prompt_tokens": 17,
    "completion_tokens": 8,
    "total_tokens": 25
  }
}

The reply is itself a message, with the role assistant. Hold on to that, because it matters in Part 2. The client turns this JSON into Python objects with the same shape, which is why the code reads it the way it does.

The reply lives at response.choices[0].message.content. It is a list because the API can return several alternative replies to one request. We ask for one, so we take the first. The or "" is there because content is allowed to be empty in some responses, and printing None at someone is rude. Next to the choices, response.usage holds the number of tokens the service counted on the way in and on the way out. Some services leave it out, so the code falls back to zero instead of crashing.

Finally, main() builds one OpenAI client with your base URL and key, then loops. It reads a line, skips blank ones, leaves on quit or exit, sends everything else through ask(), and prints the reply followed by the token counts.

Run it

Make sure the virtual environment is still active, then start the bot:

python chat.py

Here is my first run:

Chatting with meta-llama/llama-3.1-8b-instruct. Type 'quit' to exit.

You: What is the capital of France?
Bot: The capital of France is Paris.
[tokens: 17 in, 8 out]

You: quit

The capital of France is Paris. Somewhere, a data center spun up to tell us something every geography quiz already knew.

Tokens, and what this costs

That last line is the program reporting how much text went each way, measured in tokens. Models do not read characters or words. They read tokens, which are chunks of text from a fixed vocabulary, often a whole short word or a piece of a longer one. Services count tokens, limit by tokens, and bill by tokens.

My question was six words but 17 tokens in. The extra tokens are the wrapping the service adds around each message to mark who is speaking and where the message ends. You pay for that wrapping too.

Prices on OpenRouter are listed per million tokens, separately for input and output, and the same model can cost different amounts depending on which provider ends up serving it. On the day I wrote this, the model page for meta-llama/llama-3.1-8b-instruct listed input from $0.02 to $0.22 per million tokens and output from $0.04 to $0.287. Bigger models can cost a great deal more. Real use needs proper budgeting and monitoring.

You can compare models and prices at openrouter.ai/models. Switching models is one line: change LLM_MODEL in .env to another model id and restart the bot. No code changes.

Where this breaks

Try introducing yourself:

Chatting with meta-llama/llama-3.1-8b-instruct. Type 'quit' to exit.

You: My name is Ben.
Bot: Nice to meet you, Ben! Is there something I can help you with or would you like to chat?
[tokens: 15 in, 23 out]

You: What is my name?
Bot: I'm happy to chat with you, but I don't have the ability to know your name unless you tell me. This is a new conversation, and I don't have any prior information about you. If you'd like to introduce yourself, I'd be happy to learn your name!
[tokens: 15 in, 59 out]

You: ^D

The bot greets Ben warmly and then, one message later, has no idea who Ben is. It even gives a reasonably honest explanation of why.

Look at the token counts. Both questions went in at 15 tokens. If the first exchange had been part of the second request, the second number would be bigger. It was not, because ask() only ever sends one message: the newest one. The service keeps nothing between calls. As far as the model is concerned, every message you send is the first thing anyone has ever said to it.

Switching services

Because the code only speaks the OpenAI-compatible format, moving it to a different service means changing the three lines in .env. Nothing in chat.py changes.

As an example, Ollama runs models locally on your own machine and serves an OpenAI-compatible API. According to its documentation, a local server listens at http://localhost:11434/v1, the API key is required by the client but ignored by the server, and models are named by name and tag, like llama3.1:8b. After installing Ollama and downloading a model with ollama pull llama3.1:8b, set these values in .env:

  • LLM_BASE_URL to http://localhost:11434/v1
  • LLM_API_KEY to any non-empty text, such as ollama
  • LLM_MODEL to llama3.1:8b

I ran exactly that against Ollama 0.40.3 on my laptop:

Chatting with llama3.1:8b. Type 'quit' to exit.

You: What is the capital of France?
Bot: The capital of France is Paris.
[tokens: 17 in, 8 out]

You: quit

The same program, a different service, no API bill. The answer and the token counts match the OpenRouter run exactly, which makes sense: it is the same Llama 3.1 8B model family, just running on my laptop instead of someone else's server. The token line works because Ollama reports usage too. Local models are slower on a laptop and smaller than what you can rent, but they are great for experimenting without watching the meter.

In Part 2, the bot gets a memory!

Build an AI Chatbot
RAG Tutorial
OpenAI API
Chatbot
AI
Artificial Intelligence
benlobaugh.com