Intro to the XIAO ESP32-S3 Sense and Smart Glasses Setup

This lesson turns your kit into wearable, AI-ready smart glasses built on the XIAO ESP32-S3 Sense, a tiny board with a camera and microphone. You will mount it to a pair of glasses, flash its firmware, and set up the openglass software so the camera streams to a program on your computer that can process it with a language model. Because this is a wearable camera, use it only where recording is legal and only with the consent of anyone in view.

Privacy first with a wearable camera

Before anything technical: a camera you wear is a recording device, and recording people has legal and ethical limits. Use these glasses only on yourself, in your own space, and where recording is legal, and get the consent of anyone who might be captured. This is the ethics of Month 5 applied to hardware. The project is a great way to learn about vision on a tiny device; it is not a license to record others.

What the board can do

The XIAO ESP32-S3 Sense is a small but capable board: a dual-core ESP32-S3 that can run lightweight AI, with a built-in camera and microphone, and Wi-Fi and Bluetooth for connectivity. That combination, tiny size, a camera, a mic, and wireless, is what makes it ideal for wearable vision and audio projects. Its size keeps the glasses light enough to wear comfortably.

How the smart glasses work

The board is not doing the heavy AI itself. It captures images and sends them, over Bluetooth or Wi-Fi, to a small server program running on your computer. That program does the intelligent processing with a language model, either a local one like Ollama with a vision model, or a cloud one like OpenAI, and sends results back. So the glasses are the eyes, and your computer is the brain. You will run this software from the openglass open-source project.

Version pinning: why the newest is not the best

A crucial gotcha: this project was last updated many months ago, and the ESP32 board support package has changed since. The newest version will likely fail to compile because files were restructured. So in the editor's board manager, you deliberately install an older, known-good ESP32 version (around 2.0.x rather than the latest) that matches the code. Pinning to a compatible version is a real and common part of working with open-source hardware projects, not a workaround to skip.

Working through it

Mount and power the board. Attach the module to the arm of the glasses with the camera facing forward and unobstructed, keeping it light. Power it from a USB battery pack (recommended) or your computer's USB port, and confirm the onboard LED lights.

Get the project and set the board package. Download the openglass repository, open the firmware sketch in the editor, add the ESP32 board manager URL, and install a compatible older ESP32 version (around 2.0.x) rather than the newest.

Configure and upload the firmware. Select the XIAO ESP32-S3 board and its port, set PSRAM to OPI and a working upload speed, then upload. If a cable or port misbehaves, try a different USB cable.

Install the server software. In the project folder run the install, install Node.js first if needed, then put your OpenAI key or Ollama setup into the keys file.

Run it and connect. Start the server, open the local web page, and connect to the device over Bluetooth using a browser that allows it (Chrome or Edge). If using a local model, install Ollama and pull a vision model like Moondream, which is free.

Editor board settings that work

# ESP32 board package: install an OLDER version (around 2.0.x),
# NOT the newest, or the firmware will not compile.

# Board:          XIAO ESP32-S3
# PSRAM:          OPI PSRAM
# CPU Frequency:  80 MHz
# Upload Speed:   115200
# Port:           select the one that appears when the board is plugged in

# then: Upload the firmware.ino sketch

These are the settings that let the firmware flash successfully. The single most common failure is using the newest ESP32 package; pin to the older compatible version the project expects.

Set up and run the server

# in the openglass project folder
npm install               # install Node dependencies (install Node.js first)

# put your key in sources/keys.ts (OpenAI), OR use a local model:
#   download Ollama, then:
ollama pull moondream      # free local vision model

npm start                 # starts the local server (e.g. localhost:8081)
# open that URL in Chrome or Edge and connect the device over Bluetooth

The server relays images from the glasses to a vision model and back. Chrome or Edge are needed because some browsers disable the Bluetooth pairing the connection requires. A local model like Moondream keeps it free and private.