Designing the AI Assistant Framework
This is the planning session for the capstone: your own AI assistant that sees with the camera, hears with the microphone, thinks with the local LLM, and responds through speech or the screen. You will not write the full program yet; you will design its architecture as clean modules and plan the flow in pseudo code, so building it next lesson is straightforward.
What a good assistant does
The familiar assistants listen and respond, understand basic commands, keep some context, and take actions like playing music or answering a question. Yours will be simpler but more capable in one key way: it will also see, using the camera and your AI models. So it will listen to you, look at what is in front of it, respond with text-to-speech, and trigger actions such as lights, alerts, or answers. The goal is something that reacts to both your voice and its surroundings, not just canned replies to fixed phrases.
Modular architecture
The key design idea is modularity: break the assistant into separate components, each responsible for one job. A vision module handles person and object detection from the camera. An audio input module captures voice and transcribes it to text. A language model module (the local LLM) processes that text and decides the response or action. An output module speaks through text-to-speech and triggers hardware. Each lives in its own Python file with clear functions, and a main program wires them together. This keeps the system understandable and lets you improve one part without breaking the others.
Why separate modules pays off
Modularity is not just tidiness. When each capability is isolated, you can test the microphone code alone, swap the local LLM for a cloud one without touching the vision code, or add a new action without rewriting everything. It also makes the flow readable: the main loop reads like a sentence, capture, detect, listen, think, speak, while the messy details stay inside each module. This is exactly the functions-and-modularity lesson from Month 1, applied at the scale of a whole system.
Planning with pseudo code
Before real code, you sketch the logic in pseudo code, plain-language steps that describe what should happen without worrying about syntax. A simple assistant loop runs forever: capture a frame, and if a person is in the chosen zone, greet them and ask how you can help; then listen, convert the speech to text, generate a response with the LLM, and speak it aloud. Writing this out first means the coding step is just filling in each line with the module call that does it. It also forces you to decide the trigger, a person entering a zone, a wake word, or a button.
Working through it
Decide the behavior and features. Choose what your assistant should do (greet on entry, answer questions, control lights, send alerts) and give it a personality. This decision drives the rest of the design.
List the modules. Define the vision, audio input, language model, and output modules, and write down the responsibility and main functions of each.
Choose the trigger. Pick how the assistant activates: a person entering a region of interest, a spoken wake word, or a button press.
Write the pseudo code flow. Sketch the main loop in plain language: capture, detect person, greet, listen, transcribe, generate a response, speak. Keep the module calls as single steps.
See the integration in action. Run the provided example that combines the month's pieces: a person entering the zone triggers a greeting, records your question, transcribes it, and passes it to the local LLM for a spoken answer. Tune the record and prep timings to your liking.
The assistant loop in pseudo code
while True:
frame = capture_camera() # vision module
if person_in_zone(frame, ROI): # vision module
speak("Hi, how can I help you today?") # output module
audio = listen_for_speech() # audio input module
text = speech_to_text(audio) # audio input module
reply = ask_llm(text) # language model module
speak(reply) # output moduleEach line is one clear step handled by one module. This plain-language flow is the whole assistant; building it becomes a matter of writing each module's function behind these calls.