This article explains how to use the basic AI conversation function (AI conversation through the integration of LLM, STT, and TTS Web APIs) that inherits the mechanism of robo8080's AI Stack-chan.
This shows the support status of various AI services required for conversation.
You can select which AI service to use by configuring the YAML file on the SD card (you will need to obtain an API key separately).
| Local execution | Japanese | English | Remarks | |
|---|---|---|---|---|
| OpenAI ChatGPT | × | 〇 | 〇 | ・You need to get an API key ・Supports Function Calling. Details page ・Supports MCP. Details page ・CoreS3 camera images can be input. Details page |
| ModuleLLM | 〇 | 〇 | 〇 | Please check How to set up ModuleLLM. Function calling is also supported (see Appendix B on the same page). |
| 🆕OpenAI Compatible API | - | - | - | Please check How to use the OpenAI compatible endpoint |
| Local execution | Japanese | English | Remarks | |
|---|---|---|---|---|
| Google Cloud STT | × | 〇 | 〇 | You need to get an API key |
| OpenAI Whisper | × | 〇 | 〇 | You need to get an API key (You can use a common API key with OpenAI ChatGPT) |
| ModuleLLM ASR | 〇 | × | 〇 | Please check How to set up ModuleLLM |
| Local execution | Japanese | English | Remarks | |
|---|---|---|---|---|
| Web版VoiceVox | × | 〇 | × | You need to get an API key |
| ElevenLabs | × | 〇 | 〇 | You need to get an API key |
| OpenAI TTS | × | 〇 | 〇 | You need to get an API key (You can use a common API key with OpenAI ChatGPT) |
| AquesTalk | 〇 | 〇 | × | Library and dictionary data must be downloaded separately.Details page |
| ModuleLLM TTS | 〇 | × | 〇 | Please check How to set up ModuleLLM |
| Local execution | Japanese | English | Remarks | |
|---|---|---|---|---|
| SimpleVox | × | 〇 | 〇 | Details page |
| ModuleLLM KWS | 〇 | × | 〇 | ・Please check How to set up ModuleLLM |
Various settings are made using the YAML file saved on the SD card.
There are three types of YAML files:
- SC_SecConfig.yaml
Wi-Fi password, API key settings. (Sensitive information) - SC_BasicConfig.yaml
Servo related settings. - SC_ExConfig.yaml
Other app-specific settings.
SD card folder：/yaml
File name：SC_SecConfig.yaml
Set the Wi-Fi password and API keys for various AI services.
wifi:
ssid: "********"
password: "********"
apikey:
stt: "********" # ApiKey of SpeechToText Service (OpenAI Whisper/ Google Cloud STT )
aiservice: "********" # ApiKey of AIService (OpenAI ChatGPT)
tts: "********" # ApiKey of TextToSpeech Service (VoiceVox / ElevenLabs/ OpenAI )SD card folder：/yaml
File name：SC_BasicConfig.yaml
Configure the servo settings.
servo:
pin:
# ServoPin
# Core2 PortA X:33,Y:32 PortC X:13,Y:14
# CoreS3 PortA X:2, Y:1 PortB X:9, Y:8 PortC X:17, Y:18
# When using SCS0009 or Dynamixel XL330, x:RX, y:TX (not used)
# RT Version (Dynamixel): x:6 y:7
# M5StackChan (SCS0009) : x:7 y:6
x: 7
y: 6
offset:
# Specified by +- from 90 degree during servo initialization
x: 0
y: 0
center:
# サーボの初期位置
# SG90: x:90 y:90
# SCS0009: x:150, y:150
# Dynamixel XL330: x:180, y:270
# RT Version X:180 Y:5
# M5StackChan: x:150, y:90
x: 150
y: 90
lower_limit:
# 可動範囲の下限（下限と言っても取り付け方により逆の場合あり, 値の小さい方を指定。）
# SG90: x:0, y:60
# SCS0009: x:0, y:120
# Dynamixel XL330: x:0, y:220
# RT Version X:90 Y:-5
# M5StackChan: x:0 y:0
x: 0
y: 0
upper_limit:
# 可動範囲の上限（上限と言っても取り付け方により逆の場合もあり, 値の大きい方を指定。）
# SG90: x:180, y:90
# SCS0009: x:300, y:150
# Dynamixel XL330: x:360, y:270
# Dynamixel RTVersion X:270 Y:15
# M5StackChan: X:300 y:90
x: 300
y: 90
servo_type: "M5_SCS" # "PWM": SG90PWMServo
# "SCS": Feetech SCS0009
# "DYN_XL330": Dynamixel XL330
# "RT_DYN_XL330": RTVersion
# "M5_SCS": M5StackChan Servo
takao_base: false # Whether to use takaobase to feed power from the rear connector.(Stack-chan_Takao_Base https://ssci.to/8905)If you are using M5StackChan (StackChan, a product officially released by M5Stack), please ensure that the version of the stackchan-arduino library is v0.0.7 or higher.
SD card folder：/app/AiStackChanEx
File name：SC_ExConfig.yaml
Select an AI service and set parameters for each service.
llm:
type: 0 # 0:ChatGPT 1:ModuleLLM
tts:
type: 0 # 0:VOICEVOX 1:ElevenLabs 2:OpenAI TTS 3:AquesTalk 4:ModuleLLM
model: "" # VOICEVOX (model is not supported)
#model: "eleven_multilingual_v2" # ElevenLabs
#model: "tts-1" # OpenAI TTS
#model: "" # AquesTalk (model is not supported)
voice: "3" # VOICEVOX (Zundamon)
#voice: "AZnzlk1XvdvUeBnXmlld" # ElevenLabs
#voice: "alloy" # OpenAI TTS
#voice: "" # AquesTalk (model is not supported)
stt:
type: 0 # 0:Google STT 1:OpenAI Whisper 2:ModuleLLM(ASR)
wakeword:
type: 0 # 0:SimpleVox 1:ModuleLLM(KWS)
keyword: "" # SimpleVox (Initial setting is not possible. Press and hold Button B to register.)
#keyword: "HI STUCK" # ModuleLLM(KWS)
# ModuleLLM
moduleLLM:
# Serial Pin
# Core2 Rx:13,Tx:14
# CoreS3 Rx:18,Tx:17
rxPin: 13
txPin: 14
Please make sure to install VSCode, the PlatformIO extension for VSCode, and the necessary USB drivers in advance.
You can download the USB drivers from the M5Stack website. The required driver depends on whether your M5Stack uses the CP210x or CH9102 USB-serial conversion IC, but installing both drivers is not a problem.
- Clone this repository into an appropriate directory.
git clone https://github.com/ronron-gh/AI_StackChan_Ex.git
If the path is too deep, the library include path may not work. Please clone as close to the root of the C drive as possible (e.g., C:\Git).
- In the PlatformIO Home screen, click "Open Project".
- Select the firmware folder (the folder containing platformio.ini) of the cloned project and click "Open".
The required libraries will start installing, and you will see progress at the bottom right of the VSCode window. Please wait until it completes.
-
Connect your PC and M5Stack with a USB cable.
-
Follow the steps shown below to select the build environment (env), then build and flash the firmware.
The default env is m5stack-core2(s3), but for example, if you want to use the OpenAI Realtime API, select m5stack-core2(s3)-realtime (see the explanation for each feature). When you select an env, library installation may start again as in step 3, so please wait for it to complete before building and flashing.
After launching M5Core and your avatar is displayed, touch the area around the avatar's forehead to start recording, then speak. The recording time is approximately 7 seconds.
Custom instructions (so-called "roles") and memory (long-term memory) allow you to customize the AI conversation function according to the user's attributes.
You can open the settings screen by accessing http://(Stack-chan's IP address) from a PC or smartphone web browser. (A QR code for access will be displayed by pressing the C button or touching the right edge of the LCD.)
〇 About memory (Long-term Memory)
To enable memory, set enableMemory to true in /app/AiStackChanEx/SC_ExConfig.yaml on the SD card.
Currently, ChatGPT (including the Realtime API) and Gemini Live supports memory.
SC_ExConfig.yaml
llm:
type: 0 # 0:ChatGPT 1:ModuleLLM 2:ModuleLLM(Function Calling) 3:Gemini
enableMemory: true # Enable memory with true (default is false)When memory is enabled, if there are user attributes (such as hobbies or work) or impressive episodes in the conversation, they will be summarized and saved to SPIFFS. The contents of SPIFFS are retained even when the power is turned off and will be read as memory information at the next startup.
Whether or not something is remembered is determined by the LLM during the conversation, so it may not always be remembered as the user expects (it is also possible to explicitly instruct "Save the current conversation content to memory"). Also, information that has been remembered may be lost in the process of repeated summarization.
- When a face is detected, voice recognition is activated.
- Touching the left center of the LCD will put the device into silent mode, and it will not wake up even if a face is detected. (Instead, Stack-chan will smile while a face is being detected.)
- The camera image is displayed in the upper left corner of the LCD. Touch the image area to turn the display on/off.
※Face detection is disabled by default by commenting out the following in platformio.ini. To enable it, please enable DENABLE_CAMERA and DENABLE_FACE_DETECT.
build_flags=
-DBOARD_HAS_PSRAM
-DARDUINO_M5STACK_CORES3
;-DENABLE_CAMERA
;-DENABLE_FACE_DETECT
-DENABLE_WAKEWORD