AI Object Recognition: See & Speak - Seeed Studio XIAO ESP32S3 - dumpling—li - CodeCraft Hub
Scene: Smart Home. Edge AI application on CodeCraft Hub for Smart Home use cases.
Devices / BOM: Seeed Studio XIAO ESP32S3, Grove Vision AI V2, Grove Shield for Seeed Studio XIAO, Grove - Universal 4 Pin Buckled 20cm Cable (5pcs), MP3
Smart Home application on CodeCraft Hub. This project transforms a smart edge vision device into an offline, interactive vocabulary builder. Inspired by the desire to learn languages through real-world objects, we have created a "smart eye" that can recognize five everyday items .
Documentation
This project transforms a smart edge vision device into an offline, interactive vocabulary builder. Inspired by the desire to learn languages through real-world objects, we have created a "smart eye" that can recognize five everyday items (cup, phone, tape, Coke, mouse). When the camera identifies an object (e.g., a cup), the device instantly speaks the English word for that object (e.g., "Cup"). It recognizes five distinct items and plays their corresponding English names. No cloud connection or data upload is required, ensuring instant response and complete privacy. This turns any desk or room into an engaging, low-tech English learning environment.
Deploy steps
- Train your vision recognition model.
This is an image classification vision model trained using the Vision AI V2, with five categories: Object A, Object B, Object C, Object D, Object E, and No Object.
If you want to customize your own visual recognition model, you need to retrain it.
Deploy an AI model to your device (in-app tool on CodeCraft Hub).
- Hardware Connection
Insert the XIAO ESP32-S3 into the Grove Shield: Ensure the USB port aligns with the shield's orientation and press the pins firmly into place.
Connect the XIAO ESP32-S3 to your computer via a USB Type-C cable. Note: The Type-C cable must be a data cable, not a charging-only cable.
Connect the Vision AI V2 module (visual input) to the XIAO-ESP32's SDA/SCL pins.
GND → XIAO GND
VCC → XIAO 3.3V
SDA → XIAO D4
SCL → XIAO D5
Insert a FAT32‑formatted memory card, connect a speaker, then connect to the XIAO board's serial port. TX and RX are crossed (not side-by-side)!!!!
GND → XIAO GND
VCC → XIAO 3.3V
RX → XIAO D6
TX → XIAO D7
- Firmware Flashing
This application supports theXIAO ESP32‑S3 . Select the corresponding firmware based on your board.
Click the "Flash Firmware" button below, follow the prompts to select the appropriate serial port, and connect the device.
If the device is not recognized, double‑click the Reset button on the XIAO to enter Download Mode, then try again.
Flash firmware to your device (in-app tool on CodeCraft Hub).
- Powering On and Usage
After flashing is complete, press the reset button on the XIAO‑ESP32‑S3.
When a cup is recognized, the MP3 player speaks the English word for "cup".
When a phone is recognized, the MP3 player speaks the English word for "phone".
When tape is recognized, the MP3 player speaks the English word for "tape".
When a mouse is recognized, the MP3 player speaks the English word for "mouse".
When Coke is recognized, the MP3 player speaks the English word for "Coke".
- Future Expandable Directions
Sentence Mode ： When an object is recognized, the device not only plays the vocabulary word but also plays a simple sentence containing that word (e.g., when a "cup" is recognized, it plays "This is a cup.").
Spelling Challenge ：Play the letter-by-letter spelling of the word (e.g., "C-U-P, Cup!") and utilize the microphone functionality of the XIAO ESP32-S3 to allow the user to practice repeating or spelling along, enabling interactive learning.
Multi-language Switch： Use a physical button or a specific gesture to switch between different audio file packages, allowing the same object to play vocabulary words in different languages such as English, Chinese, French, etc.
- File
The file must be placed in the root directory of the memory card, not inside any folder.
Online AI Dubbing Tool https://ttsmaker.cn/
0001.mp3
0002.mp3
0003.mp3
0004.mp3
0005.mp3