Sound source localization + Camera - SenseCraft_Rick - CodeCraft Hub
Scene: Safety. Edge AI application on CodeCraft Hub for Safety use cases.
Safety application on CodeCraft Hub. Did somebody say "listen and track"? 👂👀 We gave our reCamera Gimbal a pair of ears! By integrating it with the reSpeaker Mic Array, the gimbal can now calculate the Direction of Arrival (DOA) and physically rotate to track the sound source in real-time. 🤖🔊
Documentation
we connected the Seeed Studio reSpeaker USB Mic Array to our RISC-V powered reCamera Gimbal. By utilizing Sound Source Localization (DOA), the system enables the gimbal's brushless motors to automatically rotate and face the direction of the speaker in real-time. It’s a super fun exploration into how our multi-modal edge devices can seamlessly interact!
reCamera Gimbal: https://lnkd.in/eejXKj86
reSpeaker USB 4-Mic Array: https://lnkd.in/gfyTBJaE
Deploy steps
- Download the complete project
Enter the project https://github.com/xr686/reCamera-Gimbal-reSpeaker-Tracker.git, download the Compressed Packet, and decompress it.
- Python Environment (Host PC)
To communicate with the reSpeaker via USB, install the dependencies on your PC:
# Install required Python packages from the provided requirements file
Code snippet included — open the application page to copy it.
# Install libusb via Conda (Crucial for backend driver support)
Code snippet included — open the application page to copy it.
- Deploy the Node-RED of this project to reCamera
Prepare your reCamera, connect it to your computer using a Type-C cable, select local deployment, and connect to 192.168.42.1
After deployment, remember to click on the top right corner to check if it is running.
After the final deployment is completed, the workflow you see should be like this:
- Run the reSpeaker program
Please replace "192.168.31.198" in the code reSpeaker.py with the real IP Address of your reCamera Gimbal after it connects to WiFi.
Run the Python script on your PC by executing:
Code snippet included — open the application page to copy it.
Speak into the microphone—you should see the Python terminal continuously printing SPEECH_DETECTED: 1 , while the reCamera rapidly turns towards the angle of your voice.