Supporting 5000 MIT 8T - Rebase of #2962 on master and extras - #3161
Supporting 5000 MIT 8T - Rebase of #2962 on master and extras#3161smoe wants to merge 22 commits into
Conversation
The CMT2300A receive loop previously used an either/or structure: when a packet interrupt fired, it drained the hardware FIFO but did no processing; when no interrupt was pending, it processed only one buffered packet. This caused FIFO overflows with MIT inverters that send 6 response fragments in rapid succession — the 64-byte hardware FIFO can hold ~2 packets, and the old code couldn't drain fast enough. Changes: - Remove either/or: drain FIFO then process all buffered packets in the same loop() iteration - Process entire software buffer (while loop) instead of one packet - Fix available() to check only PKT_OK flag instead of OR'ing PREAM_OK|SYNC_OK|CRC_OK|PKT_OK, which could trigger reads before a packet was fully received
Add passive RF capture mode to the CMT2300A radio for protocol reverse-engineering. When enabled via the web API, the radio hops across all legal EU channels (50ms dwell) and streams decoded packets over a WebSocket endpoint. Key changes: - CMT radio starts in RX mode on init for passive listening - Channel hopping across legal frequency range in capture mode - FIFO drain loop processes all queued packets per iteration - WebSocket endpoint for real-time packet streaming - Web API endpoints to enable/disable capture mode - Documentation in docs/CaptureMode.md Also fixes FIFO burst reception: the hardware FIFO drain now runs unconditionally and uses break (not continue) on buffer full, ensuring back-to-back MIT response fragments are not lost. Increases MAX_RETRANSMIT_COUNT to 20 for MIT inverters which respond with 6 rapid fragments per poll.
Add initial support for the Hoymiles MIT-5000-8T microinverter. The MIT-5000-8T presents as a 4-channel device at the RF level (each MPPT has dual panel inputs aggregated internally). - New MIT_8CH class extending HMT_Abstract - 4-channel AC/DC data parsing from validated RF captures - Auto-detection by serial number prefix in Hoymiles.cpp
MIT-5000-8T sends SystemConfigPara across 3 fragments (vs 2 for HM/HMT), requiring a larger buffer. Increase expected size to 48 bytes and add validation for the MIT response format.
The MIT-5000-8T sends response fragments at ~835ms intervals instead of the ~50ms typical for HMS/HMT inverters. The default 500ms RX timeout causes premature retransmit requests after receiving only 1-2 of 6 fragments, resulting in 13-15x retransmit ratios and ~55% success. Discovery: capture mode showed all 6 MIT fragments arriving correctly at ~835ms spacing. The DTU-Pro (which works) likely uses longer timeouts. Changes: - RealTimeRunDataCommand: 6000ms timeout for 0x1520 (MIT) serials - AlarmDataCommand: 12000ms timeout for 0x1520 serials - SingleDataCommand: 2000ms retransmit timeout for 0x1520 serials - Clean up debug logging from HoymilesRadio_CMT Note: OTA updates via /api/firmware/update appear to silently fail (device reports old git hash). Flash via USB/serial to test.
The CMT2300A radio automatically exits RX mode (goes to STBY) after receiving a packet. The read() function was reading the FIFO and clearing interrupts, but never putting the radio back into RX mode. This caused subsequent fragments in a burst to be missed — the radio was in STBY when they arrived. This explains why only 2 of 6 MIT-5000-8T fragments were received: frag 1 arrives, radio exits RX, re-enters RX only after the drain loop completes, missing frags 2-3. Then catches frag 4, misses 5-6. The HMS-4CH (Shed) worked because its fragments arrive at ~50ms intervals which is fast enough that the drain loop + next poll caught them. The MIT's ~835ms (capture mode) / ~50ms (normal mode) timing hit a window where the radio was in STBY. Fix: after reading each packet, immediately re-enter RX mode by clearing the FIFO and calling GoRx(). This keeps the radio continuously listening during multi-fragment bursts.
…ch read The previous fix (GoRx after ReadFifo) improved reception from 2/6 to 3/6 fragments, but the GoRx state transition takes too long — fragments arriving during the STBY→RFS→RX transition are still missed. Better approach: set the RX_AUTO_EXIT_DIS bit (0x20) in MODE_CTL when entering RX mode. This tells the CMT2300A to stay in RX after receiving a packet, eliminating the state transition gap entirely. The radio remains continuously listening throughout the entire fragment burst.
The previous commit set RX_AUTO_EXIT_DIS in startListening() before calling GoRx(), but GoRx() writes the bare GO_RX command (0x08) to MODE_CTL, overwriting the 0x20 bit we just set. Fix: OR the RX_AUTO_EXIT_DIS bit into the GoRx command itself within CMT2300A_AutoSwitchStatus(), so both bits are written atomically. Also fix the TX/RX status check comparisons to mask out the new bit.
…instead RX_AUTO_EXIT_DIS caused zero packets to be received — the radio likely needs the RX→STBY transition to properly latch FIFO data. New approach: after reading each packet, write GO_RX directly to MODE_CTL with a single register write (no polling, no FIFO clear). This is the fastest possible RX re-entry — just one SPI transaction instead of the full startListening() sequence.
Instead of draining only what's immediately available and returning, keep polling for 80ms after the last received packet. This catches burst fragments arriving at ~50ms intervals — the radio re-enters RX after each read (fast GoRx write), and the polling loop checks for the next fragment before the loop() cycle completes. Previously, the drain loop would exit after reading 1 packet, set _packetReceived=false, and not check again until the next loop() iteration — by which time the radio had been in STBY and missed the next fragment.
The bare GoRx write (single register write to MODE_CTL) corrupted FIFO reads — 19 'Frame kaputt' CRC failures per cycle. The radio needs the full GoStby → EnableReadFifo → ClearRxFifo → GoRx sequence to properly reset its internal state between packets. Combined with the 80ms polling drain loop, this should catch burst fragments: read packet → full RX re-entry → poll for next packet within 80ms window.
Every approach to re-enter RX after reading a packet has failed: - Bare GoRx write: corrupts FIFO reads (Frame kaputt flood) - Full startListening (GoStby+EnableReadFifo+ClearRxFifo+GoRx): also corrupts because ClearRxFifo destroys in-flight packet data - RX_AUTO_EXIT_DIS bit: radio receives nothing at all The CMT2300A's single-packet FIFO design means it can only hold one packet at a time. After receiving a packet, the radio must exit RX, have the FIFO read, then re-enter RX for the next packet. This inherently creates a gap where packets are missed. Falling back to the extended timeout approach: the radio naturally receives 2-3 of 6 fragments per burst, and retransmit requests (with 6000ms/2000ms timeouts for MIT) eventually collect all fragments. This gives ~75-83% first-try success rate.
|
InverterAbstract.h now got additional attributes to consistently provide parameters for HM[ST] | MIT. |
|
|
||
| uint8_t InverterAbstract::getMaxRetransmitCount() const | ||
| { | ||
| return MAX_DEFAULT_RETRANSMIT_COUNT; |
There was a problem hiding this comment.
| return MAX_DEFAULT_RETRANSMIT_COUNT; | |
| return 5; |
|
|
||
| uint8_t MIT_8CH::getMaxRetransmitCount() const | ||
| { | ||
| return MAX_MIT_RETRANSMIT_COUNT; |
There was a problem hiding this comment.
| return MAX_MIT_RETRANSMIT_COUNT; | |
| return 20; |
| #define MAX_DEFAULT_RETRANSMIT_COUNT 5 // Used to send the retransmit package | ||
| #define MAX_MIT_RETRANSMIT_COUNT 20 // Used to send the retransmit package to MIT inverters |
There was a problem hiding this comment.
no longer needed when values are substituted in InverterAbstract.cpp and MIT_8CH.cpp
| * 70: value matches PAC — possibly apparent power (S in VA) | ||
| */ | ||
| #include "MIT_8CH.h" | ||
| #include "commands/CommandAbstract.h" |
There was a problem hiding this comment.
Can be removed when MAX_MIT_RETRANSMIT_COUNT is substitited.
| * Copyright (C) 2022-2026 Thomas Basler and others | ||
| */ | ||
| #include "InverterAbstract.h" | ||
| #include "commands/CommandAbstract.h" |
There was a problem hiding this comment.
Can be removed when MAX_DEFAULT_RETRANSMIT_COUNT is substituted
|
@ms49434, I kind of like the #defines as these have all parameters nicely together and it is consistent with how everything is organised in the master branch. Also, to get access to the parameters you need an instance of those classes while you can literally grep in the code for those #defines (so have them while nothing is running) and at any point of the program's execution without any overhead. To have them or not to have them, that to decide is a matter of upstream. The whole purpose of this branch is to render the MIT work by @Geoffn-Hub easier for upstream to adopt. Any extra changes of mine (or yours :-) ) would be detrimental to that cause, I fear. That is - unless if you are upstream. Since I do not have immediate access to a functional OpenDTU (only the ones my neighbour failed to solder properly) or an MIT-8T - do you already have a success story? |
|
My OpenDTU is running perfectly with a stripped version of this PR (RF capture code removed). I wanted to ensure that the modifications are 100% backward compatible with my existing inverters, mainly HMS-2000-4T. I own a MIT-5000-8T as well but haven't fired it up yet. Now that the code is looking more mature, I'll give it a try in the next days. |
|
It would be lovely if you could confirm that the RF capture code is now no longer of any harm for older models. |
The full PR is running stable as well with capture mode enabled. No packet loss and no performance issues. |
|
Tested PR #3161 on OpenDTU-OnBattery with an HMS-2000-4T, serial prefix 0x1164, using an ESP32-S3 and CMT2300A. Current radio statistics:
Before applying the PR, polling could stall for much longer and partial receptions occurred. With the PR, requests now complete reliably within approximately six seconds. There are still many fragment retransmission requests, but every request currently completes successfully and the CMT queue remains small. |
|
Thank you for the report. The many requests are in line with what other installations have reported. I have no immediate suggestions and for the foreseeable future lacks some combination of opportunity, equipment and skills to chase this up myself. |
|
If you Need some Support Tell me so i can try to do my best. |
Update after field testingI have updated the Validate fragment lengths before calculating checksumsCommit Without these checks, subtracting the checksum length from a malformed short fragment could underflow and make the checksum code read outside the received data. This is the integration-branch version of #3163. Avoid advancing beyond an empty command queueCommit For an empty Behaviour for queues containing one or more commands remains unchanged. This is the integration-branch version of #3164. Reject oversized CMT2300A packets instead of truncating themCommit A packet larger than the supported RF packet size is still removed from the radio FIFO, but it is discarded instead of being passed to the parser as an apparently valid truncated packet. Select the MIT retransmission channel before receivingCommit An actual fragment request is identified by both its command byte and its exact packet length. The observed channel pattern is:
The channel is selected before receive mode is started. Logging therefore cannot delay the short response. Invalid fragment numbers and channels outside the supported range are rejected. Before every ordinary transmission, the radio is returned to the inverter's base channel. A previous MIT receive offset therefore cannot affect the following MIT or HMS command. Accept the SystemConfig response returned by the tested MITCommit The tested inverter repeatedly returned a complete, CRC-valid, one-fragment response containing 16 bytes. Requiring 38 bytes rejected that valid response every time. The normal 16-byte validity requirement is therefore retained. Field-test resultThe latest continuous recording produced:
MIT communication still requires several fragment retransmissions and therefore occupies the shared CMT radio for considerably longer than HMS communication. However, no interruption of an active HMS transaction, starvation, or HMS terminal failure was observed. Capture mode remained disabled during these tests. The read-only polling path is well tested; MIT power-limit and other control commands have not yet been field-tested. A clean
|
4 participants
As described on #2962.