Running a Local LLM on the Raspberry Pi
Today the Pi gets a mind of its own: a large language model running entirely on the device, giving ChatGPT-style answers with no internet and full privacy. You will learn what an LLM is, the real trade-offs of running one locally, and how to load a small quantized model and prompt it, then wrap it so your assistant can speak its answers aloud.
What a local LLM is
A large language model is the kind of AI behind ChatGPT: it takes text in and generates a text response. Normally it runs on a company's servers, but a local LLM runs directly on your device. You will not match a cloud model's quality, since those run on enormous, expensive hardware, but you will be genuinely surprised how capable a small model on a Raspberry Pi can be, and it answers with no internet connection at all.
Why run it locally
Running locally has real advantages. There is no cloud and no internet requirement, so it works in a rural area, on a boat, or anywhere offline. There is no subscription. Above all there is privacy: your prompts never leave the device, so nothing you type can be logged, trained on, or intercepted, which matters when the input is sensitive. You also get control: you can swap in models tuned for coding or math, and a local model is not bound by a cloud service's content restrictions. The single trade-off is power: local models are smaller and less capable than the biggest cloud ones.
Quantized models for the edge
Full LLMs are heavy, demanding lots of memory and computation, more than a small board can spare. The solution is a quantized model, a version compressed to use less memory and run faster, at a small cost to accuracy. That is why you use a small quantized model like TinyLlama here: it fits the Pi's 8GB of RAM, leaves room for your other tasks, and still gives good answers. Choosing the right size is the familiar edge trade-off between quality and available resources.
Prompts, templates, and temperature
You interact by sending a prompt, your question or instruction. Wrapping it in a template improves the answers: a system instruction like you are a helpful assistant, answer precisely, optionally followed by an example, then the user's question. Several parameters shape the output. Temperature controls randomness: higher values make responses more varied and creative, lower values make them more focused and deterministic. There are also limits like the maximum number of tokens to generate and how much prior context to keep, which is how you could give the assistant memory of the conversation.
Working through it
Install the model and libraries. Run the setup script to download the quantized model (around 1GB, so have a good connection) and install the Python library that loads and runs it. Activate the virtual environment.
Load the model in Python. Point the code at the model file and initialize the LLM. The model path is required; a wrong or missing path is the most common first error.
Send a prompt. Wrap your question in the assistant template and pass it to the model, then print the response. Try prompts like a fun fact about space or explain what a variable is.
Tune the temperature. Run the same prompt at a low and a high temperature and compare: low gives steadier answers, high gives more varied ones.
Speak the answer. Pass the model's response to the text-to-speech function from last lesson so the Pi reads it aloud, forming the core of your assistant.
Prompt a local LLM
from llama_cpp import Llama
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--model", required=True) # path to the quantized model
parser.add_argument("--prompt", required=True) # your question
parser.add_argument("--temperature", type=float, default=0.3)
args = parser.parse_args()
llm = Llama(model_path=args.model, n_ctx=2048)
# a template guides the model to answer helpfully
text = (
"You are a helpful assistant. Answer clearly and precisely.\n"
f"Question: {args.prompt}\nAnswer:"
)
result = llm(text, max_tokens=200, temperature=args.temperature)
print(result["choices"][0]["text"].strip())The model loads from a file and answers a prompt with no internet. The template steers it toward a helpful reply, and temperature (lower here for steadier answers) controls how varied the output is. Run it with --model and --prompt on the command line.