Month 5 Box - Cyber Security

Lesson 3 - Spy Glasses and AI

Lesson 3: AI-Powered Smart Glasses (OpenGlass Pt. 1)


In this lesson, you’ll transform your XIAO ESP32 S3 Sense into AI-powered smart glasses using the OpenGlass open-source project. This is your first major step into wearable, real-time computer vision—where your device can “see” text in the world and extract it instantly using onboard OCR (optical character recognition).


🎯 What You’ll Accomplish Today:

  • Install and run the OpenGlass code on your ESP32
  • Use the onboard camera to extract and display live text data
  • View that data in real time via the Arduino IDE Serial Monitor
  • Begin exploring how low-power AI devices can sense and interpret their environment


🔍 What Is OpenGlass?


OpenGlass is an open-source project designed to:

  • Use the ESP32’s camera to capture images
  • Run OCR to extract readable text
  • Output that data through serial communication
  • (Optionally) integrate with AI tools in future lessons


You’re not modifying or extending the project today—just getting it installed, running, and tested successfully on your own device.


📎 GitHub repo: https://github.com/BasedHardware/OpenGlass


🛠️ Setup Instructions:


Step 1 – Download and Open the Code


  • Download the OpenGlass project as a ZIP
  • Unzip it and open the Arduino sketch in your IDE
  • Set your board to XIAO_ESP32S3 and select the correct COM port


Step 2 – Board Package Compatibility



⚠️ OpenGlass may not compile correctly on the latest ESP32 board versions.


  • Go to Tools > Board > Board Manager
  • Search for "ESP32" by Espressif Systems
  • Downgrade to version 2.0.17 for best compatibility
  • Install any missing libraries listed in the repo’s README


Once ready, hit Upload to flash the code to your board.


👓 Step 3 – Test the OCR Smart Glasses


  • Open the Serial Monitor in Arduino IDE
  • Hold up printed or digital text in front of the ESP32’s camera
  • Watch as the OCR engine extracts the text and prints it to the monitor in real time


Try scanning:


  • Book pages
  • Text on your phone screen
  • Labels, signs, or printouts


This is a completely local system—no cloud, no Wi-Fi required—just smart wearable vision.


🧪 Hands-On Activity:

  • Test the glasses on three types of text:
  1. Printed book
  2. Screen display
  3. Handwritten note
  • Note which conditions (lighting, distance, font) give the best results


🧾 The video includes side-by-side comparisons of clean vs. blurry reads to help you calibrate.


🛠️ Troubleshooting Tips:


If you’re not seeing readable text:


  • Make sure the camera is not blocked or misaligned
  • Ensure the Serial Monitor is set to the correct baud rate (check the repo)
  • Re-upload the code if you saw compile errors
  • Confirm you're on ESP32 board version 2.0.7 or 2.0.9


📚 Homework Assignment:


  • Capture at least 5 readable OCR results
  • Post one interesting or funny result in the #ai-glasses-log Discord thread
  • (Optional) Jot down two ideas for how you might extend this project (e.g., speech output, text-triggered alerts)



✅ By the End of This Lesson, You’ll Have:


  • Installed and run the OpenGlass OCR engine
  • Converted your ESP32 glasses into a real-time text-reading AI device
  • Started thinking about real-world applications for low-power computer vision


In Lesson 4, we’ll explore how to connect this visual data to external actions—like triggering smart home automations or sending alerts.