Skip to content

README_en.md

Latest commit

 

History

History
101 lines (80 loc) · 6.64 KB

README_en.md

File metadata and controls

101 lines (80 loc) · 6.64 KB
Raw

日本語

AI Stack-chan Ex

Based on robo8080's AI Stack-chan, we have added the following functions.

  • Expanding AI capabilities
    • Full localization of AI conversation functions using Module LLM (an expansion module for M5Stack)
    • Latency-free conversations with Realtime API
  • Compatible with stackchan-arduino library
    • This makes it possible to use YAML for initial configuration and support for serial servos.
  • Class design that makes it easy to create additional user applications

Stack-chan is a super-kawaii palm-sized communication robot developed and released by Shinya Ishikawa-san.


Table of Contents

Development environment

  • Target device：M5Stack Core2 / CoreS3
  • Development PC：
    • OS: Windows11
    • IDE：VSCode + PlatformIO

Basic Usage

To use the basic AI conversation features (LLM, STT, TTS via Web APIs) inherited from robo8080's AI Stack-chan, please follow the instructions and setup in the Basic Usage page.

How to Use Module LLM

By replacing LLM, STT, and TTS with Module LLM APIs, you can run all AI conversation features completely locally.
After reviewing the Basic Usage, please refer to the How to Configure Module LLM page for setup instructions.

How to Use Realtime API

With the conventional AI conversation flow, each API (STT → LLM → TTS) introduced latency, often resulting in responses taking over 10 seconds. By using the OpenAI Realtime API, you can input audio data directly to the LLM and receive audio responses, minimizing latency for near real-time conversations.
To use the Realtime API, please follow the instructions in the Realtime API page.

Other features

About creating a user application

By referring to the moddable version of Stack-chan's MOD, we have made it possible to create user applications. (The moddable version of Stack-chan is called the original, and is a repository published by Shishikawa-san.)

The source code for user applications is stored in the mod folder, which already contains the applications shown in the table below. You can also use these as a reference to create and add new applications.

No. App Name Explanation (How to use) Supplement
1 AI Stack-chan This is the main app of this repository.
2 Pomodoro Timer This app alternates between a 25 minute alarm and a 5 minute alarm.
Button A: Start/Stop
Button C: Silent mode off/on		 The default setting is silent mode.
3 Digital Photo Frame The JPEG file saved in the folder "/app/AiStackChanEx/photo" on the SD card will be displayed on the LCD.
Button A: Show next photo
Button C: Start slideshow		 ・JPEG files saved to the SD card must be 320x240 in size.
・During development, there were some cases where the SD card could not be mounted and could not be restored unless it was reformatted. We believe this has been improved, but please back up the data on the SD card just in case.
4 Status Monitor Displays various system information.
5 ESP-NOW Remote Control Controls the servos using pose data received from the optional joystick controller for the official M5Stack Stack-chan.
The receive channel is fixed to 1.		 Wi-Fi is disconnected while ESP-NOW is in use. When switching to another Mod, the firmware attempts to reconnect Wi-Fi.

You can register multiple applications using the code below. You can switch between them by flicking the LCD left and right while the application is running.（Twitter Videos）。

[main.cpp]
ModBase* init_mod(void)
{
  ModBase* mod;
  add_mod(new AiStackChanMod());      // AI Stack-chan
  add_mod(new PomodoroMod());         // Pomodoro Timer
  add_mod(new StatusMonitorMod());    // Status Monitor
  mod = get_current_mod();
  mod->init();
  return mod;
}

About contributions

Issues and pull requests are welcome. If you have any problems or suggestions for improvement, please contact us via an issue first.
Additionally, if you would like to support us as a Sponsor, it would be greatly appreciated and would help us with daily troubleshooting, responding to inquiries, and supporting new hardware.
We would also appreciate it if you could understand the repository policy described below.

Repository Policy

Currently, this project is developed as a hobby, but it is based on the following policy:

  • Pursuing the performance of an AI voice assistant
    With the rapid development of generative AI, Stack-chan is becoming increasingly practical as an AI voice assistant. Through development so far, we have achieved real-time voice conversations and feature expansion via MCP, and in the future, we aim to implement long-term memory (so that recent conversations are remembered even after a power cycle).
  • Open as a base for derivative development
    We aim for a simple and general-purpose design so that the results of this repository can serve as a base for many people's AI Stack-chan development.
  • Support for new hardware
    As long as it does not conflict with the above policies, we would like to cover new hardware with future potential, such as ModuleLLM.
  • Contributing to the original Stack-chan repository
    This repository respects and is influenced by the so-called "original" or "TypeScript version" repository by Shinya Ishikawa-san (repository). We hope to contribute to the development of the original Stack-chan using the results obtained here (although, due to my lack of TypeScript skills, this has not been realized much yet. If you have the skills, please consider contributing).

Notes

  • Because the folder name is long, the library include path may not work depending on the workspace location. Please make the workspace as close to the C drive as possible. (Example: C:\Git)