Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions AI_RULES.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,9 +119,11 @@ menu of short prompts; the screen is edge-to-edge text (no borders or decoration
touch, detected by the `espressif/m5stack_tab5` BSP), rotated to 1280x720 landscape by the
PPA (`LV_DISPLAY_ROTATION_90`, keyboard at the bottom); LVGL buffers in PSRAM. Backlight
GPIO 22. System I2C on GPIO 31/32 (BSP): IO expanders, touch, codecs, INA226 battery monitor.
- IO expanders (PI4IOE5V6408), reset by the BSP at init: 0x43 P1 speaker amp, P2 5 V to the
side port (the keyboard needs it), P4 LCD reset, P5 touch reset; 0x44 P0 ESP32-C6 power,
P5 fast-charge (low = on), P7 charger enable. `board::init_power()` sets them: the battery
- IO expanders (PI4IOE5V6408), reset by the BSP at init: 0x43 P0 antenna (low = internal),
P1 speaker amp, P2 5 V to the side port (the keyboard needs it), P4 LCD reset, P5 touch
reset; 0x44 P0 ESP32-C6 power, P3 USB-A 5 V, P4 power-off pulse, P5 quick charge (low = on),
P7 charger enable. The reset leaves every pin high-impedance: an output only drives once set
push-pull (`esp_io_expander_set_output_mode()`). `board::init_power()` sets them: the battery
charges only while firmware enables the charger.
- Keyboard: STM32 at I2C 0x6D on its own bus (G0 SDA, G1 SCL), INT G50 low while events are
queued, raw mode (press/release by row/column; reading 0x20 pops an event, so never retry
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ OpenAI, and only while you use ChatGPT.
the Tab5 keyboard just press them: Ctrl+letter, Tab and Esc go straight to the server.
- **History**: roll the trackball left (older) or right (newer). Tab5: Up / Down.
- **Disconnect**: type `exit` to return to the menu.
- **ChatGPT**: type to chat, or hold the trackball (Tab5: hold Ctrl+Space) to speak and let
- **ChatGPT**: type to chat, or hold the trackball (Tab5: hold Ctrl) to speak and let
go to send. Start talking
once `* Recording` shows. Your speech is transcribed by OpenAI's speech-to-text model, then
sent as your message to the chat model you picked. If the microphone picks up nothing, the
Expand All @@ -118,7 +118,7 @@ at a time. Use one-shot commands instead, for example `top -n 1`.
| Aa | Hold for capitals; tap for one capital; tap twice for caps lock |
| Sym + key | The key's second symbol (`?` `/` `<` `>` `{` `}` `\|` `~` `:` `"` `=` `,`) |
| Ctrl + letter | Control code to the server (Ctrl+C, Ctrl+D, Ctrl+Z, Ctrl+L ...) |
| Ctrl+Space (hold) | Talk to ChatGPT; held for a second elsewhere, deletes the shown history entry |
| Ctrl (hold on its own) | Talk to ChatGPT while held; held for a second elsewhere, deletes the shown history entry |

Held keys repeat. The keyboard can be attached or removed while SimplrSSH runs, and the Tab5
charges its battery over USB-C while SimplrSSH runs.
Expand Down
30 changes: 20 additions & 10 deletions main/board_tab5.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -21,16 +21,19 @@
static const char *TAG = "board";

const char* const board::NAME = "Tab5";
const char* const board::HINT_TALK = "hold Ctrl+Space";
const char* const board::HINT_TALK = "hold Ctrl";
const char* const board::HINT_SCROLL = "Sym+Up/Down or drag: scroll.";
const char* const board::HINT_HISTORY = "Up/Down: history.";
const char* const board::SCREEN_DESC = "a 5-inch tablet with a 1280x720 text screen";

// IO expander pins (M5Stack Tab5 pin map)
static const uint32_t IO_RF_EXTERNAL = IO_EXPANDER_PIN_NUM_0; // 0x43 P0: antenna, low = internal
static const uint32_t IO_SPEAKER_EN = IO_EXPANDER_PIN_NUM_1; // 0x43 P1: NS4150B amplifier
static const uint32_t IO_EXT5V_EN = IO_EXPANDER_PIN_NUM_2; // 0x43 P2: 5 V to the side port (keyboard)
static const uint32_t IO1_WLAN_PWR_EN = IO_EXPANDER_PIN_NUM_0; // 0x44 P0: ESP32-C6 power
static const uint32_t IO1_QC_DISABLE = IO_EXPANDER_PIN_NUM_5; // 0x44 P5: low = fast charge allowed
static const uint32_t IO1_USB5V_EN = IO_EXPANDER_PIN_NUM_3; // 0x44 P3: 5 V out of the USB-A port
static const uint32_t IO1_PWROFF = IO_EXPANDER_PIN_NUM_4; // 0x44 P4: power-off pulse
static const uint32_t IO1_NQC_EN = IO_EXPANDER_PIN_NUM_5; // 0x44 P5: low = quick charge
static const uint32_t IO1_CHARGE_EN = IO_EXPANDER_PIN_NUM_7; // 0x44 P7: IP2326 charger on

static bool s_wifi_ready = false;
Expand All @@ -50,23 +53,30 @@ const board::Ui& board::ui()

void board::init_power()
{
// Both expanders are reset by their driver on first use (all inputs): set what SimplrSSH needs
// Both expanders are reset by their driver on first use: set what SimplrSSH needs
esp_io_expander_handle_t io = bsp_io_expander_init();
esp_io_expander_handle_t io1 = bsp_io_expander1_init();
if (!io || !io1) {
ESP_LOGE(TAG, "IO expanders not reachable");
return;
}
// Speaker amplifier off until audio_tab5.cpp plays something; side-port 5 V on for the keyboard
esp_io_expander_set_dir(io, IO_SPEAKER_EN | IO_EXT5V_EN, IO_EXPANDER_OUTPUT);
esp_io_expander_set_level(io, IO_SPEAKER_EN, 0);
// The driver's reset leaves every pin high-impedance (pulled low): each output must be set
// push-pull, or it never drives. Levels as M5Stack's own firmware sets them.
// Internal antenna; speaker amplifier off until audio_tab5.cpp plays; side-port 5 V on for the keyboard
const uint32_t outs = IO_RF_EXTERNAL | IO_SPEAKER_EN | IO_EXT5V_EN;
esp_io_expander_set_dir(io, outs, IO_EXPANDER_OUTPUT);
esp_io_expander_set_level(io, IO_RF_EXTERNAL | IO_SPEAKER_EN, 0);
esp_io_expander_set_level(io, IO_EXT5V_EN, 1);
esp_io_expander_set_output_mode(io, outs, IO_EXPANDER_OUTPUT_MODE_PUSH_PULL);

// Power the WiFi co-processor now, so it has booted by the time WiFi is first used.
// The battery only charges while the firmware enables the charger (as the factory firmware does).
esp_io_expander_set_dir(io1, IO1_WLAN_PWR_EN | IO1_QC_DISABLE | IO1_CHARGE_EN, IO_EXPANDER_OUTPUT);
esp_io_expander_set_level(io1, IO1_WLAN_PWR_EN, 1);
esp_io_expander_set_level(io1, IO1_QC_DISABLE, 0);
// The battery only charges while the firmware enables the charger (standard charging, as
// the factory firmware does). USB-A 5 V off, no power-off pulse.
const uint32_t outs1 = IO1_WLAN_PWR_EN | IO1_USB5V_EN | IO1_PWROFF | IO1_NQC_EN | IO1_CHARGE_EN;
esp_io_expander_set_dir(io1, outs1, IO_EXPANDER_OUTPUT);
esp_io_expander_set_level(io1, IO1_USB5V_EN | IO1_PWROFF | IO1_CHARGE_EN, 0);
esp_io_expander_set_level(io1, IO1_WLAN_PWR_EN | IO1_NQC_EN, 1);
esp_io_expander_set_output_mode(io1, outs1, IO_EXPANDER_OUTPUT_MODE_PUSH_PULL);
vTaskDelay(pdMS_TO_TICKS(50));
esp_io_expander_set_level(io1, IO1_CHARGE_EN, 1);
}
Expand Down
2 changes: 1 addition & 1 deletion main/chat_app.cpp
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
/*
* ChatGPT App
* One chat session with an OpenAI model: your messages right-aligned, replies
* left-aligned, the input line at the bottom. Hold the trackball (Tab5: Ctrl+Space) to talk
* left-aligned, the input line at the bottom. Hold the trackball (Tab5: Ctrl) to talk
* (Whisper), optionally hear the reply (TTS). Network and audio work runs in a
* short-lived worker task, so the screen and keyboard stay responsive.
*
Expand Down Expand Up @@ -72,7 +72,7 @@
return key;
}
for (char c : key) {
if (!isalnum((unsigned char)c) && c != '-' && c != '_') {

Check warning on line 75 in main/chat_app.cpp

View workflow job for this annotation

GitHub Actions / Static analysis (cppcheck)

cppcheck useStlAlgorithm

style: Consider using std::any_of algorithm instead of a raw loop.
problem = "The key contains a character OpenAI keys don't use (only letters, digits, - and _).";
key.clear();
break;
Expand All @@ -88,10 +88,10 @@
void SSHTerminal::chat_load_settings()
{
nvs_handle_t h;
if (settings_nvs::open("chat", NVS_READONLY, &h) != ESP_OK) {

Check warning on line 91 in main/chat_app.cpp

View workflow job for this annotation

GitHub Actions / Security lint (flawfinder)

flawfinder

[2] (misc) open:Check when opening files - can an attacker redirect it (via symlinks), force the opening of special file type (e.g., device files), move things around to create a race condition, control its ancestors, or change its contents? (CWE-362).
return;
}
char model[64];

Check warning on line 94 in main/chat_app.cpp

View workflow job for this annotation

GitHub Actions / Security lint (flawfinder)

flawfinder

[2] (buffer) char:Statically-sized arrays can be improperly restricted, leading to potential overflows or other issues (CWE-119!/CWE-120). Perform bounds checking, use functions that limit length, or ensure that the size is larger than the maximum possible length.
size_t len = sizeof(model);
bool saved_model = nvs_get_str(h, "model", model, &len) == ESP_OK && model[0];
if (saved_model) {
Expand Down Expand Up @@ -119,7 +119,7 @@
void SSHTerminal::chat_save_settings()
{
nvs_handle_t h;
esp_err_t err = settings_nvs::open("chat", NVS_READWRITE, &h);

Check warning on line 122 in main/chat_app.cpp

View workflow job for this annotation

GitHub Actions / Security lint (flawfinder)

flawfinder

[2] (misc) open:Check when opening files - can an attacker redirect it (via symlinks), force the opening of special file type (e.g., device files), move things around to create a race condition, control its ancestors, or change its contents? (CWE-362).
if (err == ESP_OK) {
err = nvs_set_str(h, "model", chat_model.c_str());
if (err == ESP_OK) {
Expand Down Expand Up @@ -213,7 +213,7 @@

void SSHTerminal::chat_models_worker(void* param)
{
SSHTerminal* t = (SSHTerminal*)param;

Check warning on line 216 in main/chat_app.cpp

View workflow job for this annotation

GitHub Actions / Static analysis (cppcheck)

cppcheck cstyleCast

style: C-style pointer casting
std::string key, error;
openai::Models models;
esp_err_t err = vault::get(KEY_ID, key);
Expand Down Expand Up @@ -383,7 +383,7 @@
lv_obj_set_style_width(chat_view, 2, LV_PART_SCROLLBAR);
lv_obj_add_event_cb(chat_view, scroll_end_cb, LV_EVENT_SCROLL_END, this);
chat_follow = true;
lv_obj_add_flag(terminal_output, LV_OBJ_FLAG_HIDDEN);

Check warning on line 386 in main/chat_app.cpp

View workflow job for this annotation

GitHub Actions / Compiler warnings

Compiler -Wdeprecated-declarations

'void lv_obj_add_flag(lv_obj_t*, lv_obj_flag_t)' is deprecated: Use the dedicated lv_obj_set_<flag>() setters instead, e.g. lv_obj_set_hidden(obj, true).

Check warning on line 386 in main/chat_app.cpp

View workflow job for this annotation

GitHub Actions / build

Compiler -Wdeprecated-declarations

'void lv_obj_add_flag(lv_obj_t*, lv_obj_flag_t)' is deprecated: Use the dedicated lv_obj_set_<flag>() setters instead, e.g. lv_obj_set_hidden(obj, true).

Check warning on line 386 in main/chat_app.cpp

View workflow job for this annotation

GitHub Actions / build

Compiler -Wdeprecated-declarations

'void lv_obj_add_flag(lv_obj_t*, lv_obj_flag_t)' is deprecated: Use the dedicated lv_obj_set_<flag>() setters instead, e.g. lv_obj_set_hidden(obj, true).
lv_obj_move_foreground(side_panel);

chat_add_bubble(BUBBLE_INFO, "ChatGPT (" + chat_model + ") - type, or " + board::HINT_TALK + " to talk.\n"
Expand All @@ -404,7 +404,7 @@
lv_obj_delete(chat_view);
chat_view = NULL;
}
lv_obj_clear_flag(terminal_output, LV_OBJ_FLAG_HIDDEN);

Check warning on line 407 in main/chat_app.cpp

View workflow job for this annotation

GitHub Actions / Compiler warnings

Compiler -Wdeprecated-declarations

'void lv_obj_remove_flag(lv_obj_t*, lv_obj_flag_t)' is deprecated: Use the dedicated lv_obj_set_<flag>() setters instead, e.g. lv_obj_set_hidden(obj, false).

Check warning on line 407 in main/chat_app.cpp

View workflow job for this annotation

GitHub Actions / build

Compiler -Wdeprecated-declarations

'void lv_obj_remove_flag(lv_obj_t*, lv_obj_flag_t)' is deprecated: Use the dedicated lv_obj_set_<flag>() setters instead, e.g. lv_obj_set_hidden(obj, false).

Check warning on line 407 in main/chat_app.cpp

View workflow job for this annotation

GitHub Actions / build

Compiler -Wdeprecated-declarations

'void lv_obj_remove_flag(lv_obj_t*, lv_obj_flag_t)' is deprecated: Use the dedicated lv_obj_set_<flag>() setters instead, e.g. lv_obj_set_hidden(obj, false).
update_input_display();
}

Expand Down Expand Up @@ -465,7 +465,7 @@

void SSHTerminal::chat_worker(void* param)
{
SSHTerminal* t = (SSHTerminal*)param;

Check warning on line 468 in main/chat_app.cpp

View workflow job for this annotation

GitHub Actions / Static analysis (cppcheck)

cppcheck cstyleCast

style: C-style pointer casting
t->chat_job();
t->chat_ui_banner("");
t->chat_busy = false;
Expand Down Expand Up @@ -509,7 +509,7 @@
}

if (chat_job_voice) {
int16_t* pcm = (int16_t*)heap_caps_malloc(MAX_RECORD_SAMPLES * 2, MALLOC_CAP_SPIRAM);

Check warning on line 512 in main/chat_app.cpp

View workflow job for this annotation

GitHub Actions / Static analysis (cppcheck)

cppcheck cstyleCast

style: C-style pointer casting
if (!pcm || audio::mic_start() != ESP_OK) {
chat_recording = false;
heap_caps_free(pcm);
Expand Down Expand Up @@ -620,7 +620,7 @@
openai::speak(key, tts_model, reply, [&](const uint8_t* data, size_t len) {
// Keep whole 16-bit samples across chunk boundaries
if (has_carry && len > 0) {
uint8_t pair[2] = {carry, data[0]};

Check warning on line 623 in main/chat_app.cpp

View workflow job for this annotation

GitHub Actions / Static analysis (cppcheck)

cppcheck constVariable

style: Variable 'pair' can be declared as const array
audio::speaker_write(pair, 2);
data++;
len--;
Expand Down Expand Up @@ -771,9 +771,9 @@
std::string problem = "No openai.key file at the top of the SD card.";
for (const char* name : {"openai.key", "openai.key.txt", "openai.txt"}) {
std::string path = std::string(sdcard::MOUNT_POINT) + "/" + name;
FILE* f = fopen(path.c_str(), "rb");

Check warning on line 774 in main/chat_app.cpp

View workflow job for this annotation

GitHub Actions / Security lint (flawfinder)

flawfinder

[2] (misc) fopen:Check when opening files - can an attacker redirect it (via symlinks), force the opening of special file type (e.g., device files), move things around to create a race condition, control its ancestors, or change its contents? (CWE-362).
if (f) {
char buf[1024];

Check warning on line 776 in main/chat_app.cpp

View workflow job for this annotation

GitHub Actions / Security lint (flawfinder)

flawfinder

[2] (buffer) char:Statically-sized arrays can be improperly restricted, leading to potential overflows or other issues (CWE-119!/CWE-120). Perform bounds checking, use functions that limit length, or ensure that the size is larger than the maximum possible length.
size_t n = fread(buf, 1, sizeof(buf), f);
fclose(f);
std::string raw(buf, n);
Expand Down Expand Up @@ -841,7 +841,7 @@

case WizardStep::ChatModel: {
int n = !input.empty() && input.size() <= 2 && std::all_of(input.begin(), input.end(), ::isdigit)
? atoi(input.c_str()) : -1;

Check warning on line 844 in main/chat_app.cpp

View workflow job for this annotation

GitHub Actions / Security lint (flawfinder)

flawfinder

[2] (integer) atoi:Unless checked, the resulting number can exceed the expected range (CWE-190). If source untrusted, check both minimum and maximum, even if the input had no minus sign (large numbers can roll over into negative number; consider saving to an unsigned value if that is intended).
size_t shown = std::min(chat_models.size(), MODELS_SHOWN);
if (n >= 1 && (size_t)n <= shown) {
chat_model = chat_models[n - 1];
Expand Down
7 changes: 4 additions & 3 deletions main/include/tab5_keymap.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,14 @@
* Turns the Tab5 Keyboard's raw key presses and releases (row 0-4, column 0-13) into
* SimplrSSH keys (input_events.hpp). The keyboard firmware reports only positions; this does
* the legends, the Sym layer, Aa (hold = shift, tap = one capital, double tap = caps lock),
* Ctrl codes, auto-repeat and Ctrl+Space push-to-talk. No ESP-IDF headers: tested on the host.
* Ctrl codes, auto-repeat and Ctrl push-to-talk. No ESP-IDF headers: tested on the host.
*
* Up / Down older / newer command, previous / next menu choice
* Left / Right move the cursor; Sym+Left / Sym+Right: start / end of the line
* Sym+Up / Sym+Down scroll the screen
* Del delete at the cursor
* Ctrl+letter control code (Ctrl+C = 3 ...), also Ctrl+[ \ ] ^ _ /
* Ctrl+Space (hold) push-to-talk in ChatGPT; held 1 s elsewhere: delete history entry
* Ctrl (hold alone) push-to-talk in ChatGPT; held 1 s elsewhere: delete history entry
*/

#ifndef TAB5_KEYMAP_HPP
Expand Down Expand Up @@ -54,7 +54,8 @@ class Tab5Keymap
uint32_t repeat_value = 0;
uint32_t repeat_next_ms = 0;

bool talk_down = false; // Ctrl+Space held
bool talk_down = false; // Ctrl held, no other key pressed before HOLD_MS
bool talk_used = false; // Another key pressed while talking
bool talk_holding = false; // HOLD_START sent
uint32_t talk_pressed_ms = 0;

Expand Down
47 changes: 28 additions & 19 deletions main/tab5_keymap.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,6 @@ static bool is_sym(int r, int c) { return at(r, c, 3, 0); }
static bool is_aa(int r, int c) { return at(r, c, 3, 1); }
static bool is_ctrl(int r, int c) { return at(r, c, 4, 0); }
static bool is_alt(int r, int c) { return at(r, c, 4, 1); }
static bool is_space(int r, int c) { return at(r, c, 4, 13); }

// a - b >= 0 for wrapping millisecond counters
static bool reached(uint32_t now, uint32_t when)
Expand Down Expand Up @@ -101,12 +100,35 @@ void Tab5Keymap::key_event(bool pressed, int row, int col, uint32_t now_ms)
if (row < 0 || row >= ROWS || col < 0 || col >= COLS) {
return;
}
if (is_sym(row, col)) {
sym = pressed;
return;
}
if (is_ctrl(row, col)) {
ctrl = pressed;
if (pressed) {
// Ctrl on its own is push-to-talk once held for HOLD_MS (tick)
talk_down = true;
talk_holding = false;
talk_used = false;
talk_pressed_ms = now_ms;
} else if (talk_down) {
talk_down = false;
if (talk_holding) {
talk_holding = false;
// A long hold deletes a history entry outside ChatGPT: only when Ctrl was alone
bool long_hold = !talk_used && now_ms - talk_pressed_ms >= LONG_HOLD_MS;
push(KEY_EVENT | (long_hold ? INPUT_HOLD_END_LONG : INPUT_HOLD_END));
}
}
return;
}
if (pressed && talk_down) {
// Another key with Ctrl: a control code (Ctrl+C ...), not talking
if (talk_holding) {
talk_used = true;
} else {
talk_down = false;
}
}
if (is_sym(row, col)) {
sym = pressed;
return;
}
if (is_alt(row, col)) {
Expand Down Expand Up @@ -141,25 +163,12 @@ void Tab5Keymap::key_event(bool pressed, int row, int col, uint32_t now_ms)
if (id == repeat_key) {
repeat_key = -1;
}
if (is_space(row, col) && talk_down) {
talk_down = false;
if (talk_holding) {
talk_holding = false;
push(KEY_EVENT | (now_ms - talk_pressed_ms >= LONG_HOLD_MS ? INPUT_HOLD_END_LONG : INPUT_HOLD_END));
}
}
return;
}

if (aa_down) {
aa_used = true;
}
if (is_space(row, col) && ctrl) {
talk_down = true;
talk_holding = false;
talk_pressed_ms = now_ms;
return;
}
bool repeats = false;
uint32_t value = translate(row, col, &repeats);
one_shot = false;
Expand Down Expand Up @@ -191,6 +200,6 @@ void Tab5Keymap::reset()
push(KEY_EVENT | INPUT_HOLD_END); // Stop a recording in progress
}
sym = ctrl = aa_down = aa_used = one_shot = caps_lock = false;
talk_down = talk_holding = false;
talk_down = talk_holding = talk_used = false;
repeat_key = -1;
}
10 changes: 10 additions & 0 deletions sdkconfig.defaults
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,16 @@ CONFIG_LWIP_IPV6=n
# libssh2 protocol tracing is never used in production and could log session data
CONFIG_LIBSSH2_DEBUG_ENABLE=n

# --- Build and boot (both boards) ---------------------------------------------
# app_main mounts the SD card, starts the display and touch and builds the terminal screen
# before handing over to keypad_task; the 3.5 KB default overflows. Freed when it returns.
CONFIG_ESP_MAIN_TASK_STACK_SIZE=12288
# Optimize for speed (TLS, SSH crypto, rendering); assertions compiled out
CONFIG_COMPILER_OPTIMIZATION_PERF=y
CONFIG_COMPILER_OPTIMIZATION_ASSERTIONS_DISABLE=y
# Problems only in the log (AI_RULES.md: ESP_LOGI sparingly)
CONFIG_LOG_DEFAULT_LEVEL_WARN=y

# --- Internal RAM: move bulk buffers to PSRAM --------------------------------
# WiFi/LWIP buffers in PSRAM when possible
CONFIG_SPIRAM_TRY_ALLOCATE_WIFI_LWIP=y
Expand Down
Loading
Loading