TINYRLAM

A Resource-Efficient Reinforcement Learning Agent for Adaptive Energy Management in Energy-Harvesting Wireless Sensor Networks

Sub-Kilobyte Tabular Reinforcement Learning for Solar-Powered Sensor Nodes

---

Overview

Energy management in resource-constrained Wireless Sensor Networks (WSNs) is a fundamental trade-off: static rule-based heuristics cannot adapt dynamically to unpredictable solar harvesting and bursty traffic, while modern Deep Reinforcement Learning (DRL) algorithms vastly exceed the strict kilobyte-scale SRAM and computational limits of embedded microcontrollers.

TINYRLAM (Tiny Reinforcement Learning-based Adaptive Manager) bridges this critical gap. It is a fully on-device, tabular Q-learning framework specifically engineered for ultra-low-power microcontrollers (e.g., 8-bit AVR, ARM Cortex-M0+/M4, ESP32). With a fixed 24-state space discretization ($4 \times 3 \times 2 = 24$) and constant-time $\mathcal{O}(1)$ updates, its policy representation requires only 384 bytes of raw SRAM storage—less than 5% of a standard 8 KB microcontroller memory budget.

Figure: Closed-loop TINYRLAM agent–environment interaction architecture across physical harvesting and embedded microcontroller domains.

---

Key Highlights

• Sub-Kilobyte Memory Footprint: 24 states $\times$ 4 actions = 96 single-precision float values (384 bytes total), fitting easily into constrained 8 KB SRAM devices.
• Constant-Time Execution: $\mathcal{O}(1)$ greedy action inference and scalar Bellman updates executing in sub-microsecond cycles on a 10–16 MHz CPU clock.
• Autonomous Energy Hoarding: Learns proactive energy storage during peak daylight to guarantee survival and low Packet Loss Rates (PLR $\approx 2%$) through extended night cycles without hand-tuned threshold heuristics.
• Comprehensive Benchmarks: Validated against 6 baseline paradigms (Expert Heuristic, Double Q-Learning, SARSA(0), Fixed Duty Cycling, Uniform Random, and Always-Sleep).
• Embedded C / Arduino Ready: Includes a zero-heap, header-only C library (include/tinyrlam.h) and ready-to-flash Arduino firmware (firmware/arduino/tinyrlam_node/).
• Deterministic & Reproducible: Fully reproducible with single-command scripts and multi-seed statistical validation ($n=20$ Monte Carlo runs).

---

Benchmark Results

TINYRLAM was evaluated across 5,000 simulated operational episodes (200 timesteps per episode, 80/20 Day/Night solar cycle, Poisson traffic $\lambda = 0.3$).

Policy / Algorithm	Memory Type	SRAM Storage	Mean Lifetime (Steps)	Cumulative Reward	Packet Loss Rate (PLR)	Adaptation Capability
TINYRLAM (Q-Learning)	Tabular Array	384 Bytes	90.4 ± 1.2	+3,152.4	1.94%	Autonomous / Online
Double Q-Learning	Dual Tabular	768 Bytes	89.8 ± 1.4	+3,118.0	2.10%	Autonomous / Online
SARSA(0)	Tabular Array	384 Bytes	88.6 ± 1.8	+3,040.2	2.45%	Autonomous / Online
Expert Heuristic	Hardcoded Rules	0 Bytes	89.1 ± 0.0	+3,410.0	0.00%	Static (Non-Adaptive)
Fixed Duty Cycle (20%)	Periodic Timer	0 Bytes	62.4 ± 3.1	+1,420.5	14.80%	Static (Non-Adaptive)
Uniform Random Policy	Stochastic	0 Bytes	14.2 ± 0.8	-420.0	38.20%	None
Always-Sleep (Upper Bound)	Passive	0 Bytes	200.0 ± 0.0	+600.0	100.00%	Zero Throughput

---

Mathematical Formulation

1. Markov Decision Process (MDP) Definition

TINYRLAM models node energy management as a discrete-time Markov Decision Process tuple $\langle \mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{R}, \gamma \rangle$:

• State Space $\mathcal{S}$ ($|\mathcal{S}| = 24$): $$s_t = \langle B_t, q_t, D_t \rangle$$

  • Battery Level $B_t \in {0, 1, 2, 3}$: Critical ($0..25%$), Low ($25..50%$), Medium ($50..75%$), High ($75..100%$).
  • Packet Buffer Queue $q_t \in {0, 1, 2}$: Empty ($0$), Partial ($1$), Full ($2$).
  • Solar Day/Night Phase $D_t \in {0, 1}$: Night ($0$, no harvesting), Day ($1$, solar harvesting active).

• Action Space $\mathcal{A}$ ($|\mathcal{A}| = 4$): $$\mathcal{A} = {\text{transmit}, \text{delay}, \text{sleep}, \text{wake}}$$

2. Multi-Objective Reward Function

$$R(s_t, a_t, s_{t+1}) = r_{\text{survival}} + r_{\text{tx}} - p_{\text{loss}} - p_{\text{overflow}} + r_{\text{sleep}} + r_{\text{hoard}} - p_{\text{death}}$$

• Packet Transmission: $+10$ if battery is healthy ($B_t \ge 2$), $+5$ otherwise.
• Penalties: $-50$ for node energy depletion (absorbing terminal state), $-7$ for dropped packets, $-5$ for buffer overflow.
• Energy Hoarding Incentives: $+3$ for sleeping during daylight with non-full battery, $+5$ bonus for maintaining $B_t \ge 3$ before night onset.

3. Tabular Q-Learning Bellman Update

$$Q(s_t, a_t) \leftarrow Q(s_t, a_t) + \alpha \left[ R_{t+1} + \gamma \max_{a' \in \mathcal{A}} Q(s_{t+1}, a') - Q(s_t, a_t) \right]$$ where $\alpha = 0.1$ (learning rate) and $\gamma = 0.9$ (discount factor).

---

Repository Structure

TinyRLAM/
├── .github/                      # GitHub Actions CI/CD and issue templates
│   ├── workflows/ci.yml
│   └── ISSUE_TEMPLATE/
├── src/
│   └── tinyrlam/                 # Core Python library package
│       ├── __init__.py           # Package exports and version metadata
│       ├── agent.py              # TinyRLAM Q-learning tabular agent
│       ├── environment.py        # WSN Energy-Harvesting MDP simulation environment
│       ├── baselines.py          # Benchmark heuristics and alternative RL algorithms
│       ├── evaluator.py          # Multi-seed Monte Carlo evaluation harness
│       ├── c_export.py           # Embedded C header generation utility
│       ├── visualization.py      # Publication-grade plotting utilities
│       └── cli.py                # Command-line interface tool
├── include/                      # Embedded C header files
│   ├── tinyrlam.h                # Standalone, header-only C inference & online learning library
│   └── tinyrl_qtable.h           # Auto-generated 24-state pre-trained Q-table C array
├── firmware/
│   └── arduino/
│       └── tinyrlam_node/
│           └── tinyrlam_node.ino # Complete Arduino/ESP32 sensor node sketch
├── scripts/                      # Utility and reproduction scripts
│   ├── reproduce_paper_results.py# Master reproduction script (all benchmarks + figures)
│   ├── generate_all_figures.py   # Publication figure generator (15 figures in PDF & PNG)
│   ├── export_qtable.py          # Standalone Q-table exporter script
│   ├── verify_checklist.py       # Algorithmic convergence & baseline sanity check
│   └── generate_report.py        # Automated PDF summary generator
├── paper/                        # Research manuscript artifacts
│   ├── latex/main.tex            # IEEE Conference LaTeX source
│   ├── sections/                 # Modular manuscript markdown source (Sections 0..8)
│   ├── reviews/                  # Reviewer response letters & revision notes
│   ├── TINYRLAM_Final_Manuscript.docx
│   └── TINYRLAM_Manuscript.pdf   # Compiled paper PDF
├── docs/                         # Detailed guides & API reference
│   ├── api_reference.md          # Python & C API documentation
│   ├── TinyRL_Agent_WSN_Design_Guide.md
│   └── verification_report.md
├── figures/                      # High-resolution publication figures (300 DPI PNG & vector PDF)
├── tests/                        # Comprehensive Pytest / Unittest suite
│   ├── test_environment.py
│   ├── test_agent.py
│   ├── test_baselines.py
│   ├── test_c_export.py
│   └── test_reproducibility.py
├── pyproject.toml                # Build system metadata (PEP 517/518)
├── setup.py                      # Setup installation script
├── requirements.txt              # Production dependencies
├── requirements-dev.txt          # Development dependencies
├── CITATION.cff                  # Citation metadata
├── LICENSE                       # MIT License
└── README.md

---

Quickstart & Installation

1. Clone and Install

# Clone the repository
git clone https://github.com/vansh7nvc/TinyRLAM.git
cd TinyRLAM

# Install package in editable mode
pip install -e .

# Or install with development dependencies (pytest, flake8)
pip install -e ".[dev]"

2. Python API Example

from tinyrlam import WSNEnvironment, TinyRLAMAgent

# 1. Initialize environment and agent
env = WSNEnvironment(packet_arrival_rate=0.3, day_ratio=0.8)
agent = TinyRLAMAgent(alpha=0.1, gamma=0.9, epsilon=1.0)

# 2. Train agent
print("Training TINYRLAM agent...")
results = agent.train(env, num_episodes=5000, max_timesteps=200)

# 3. Query learned policy
state = (3, 1, 1) # High battery, 1 packet in buffer, Day phase
action = agent.get_best_action(state)
print(f"Optimal action for state {state}: {action}") # -> 'transmit'

# 4. Export trained model to C header for microcontroller deployment
from tinyrlam.c_export import export_qtable_to_c_header
export_qtable_to_c_header(agent.Q, output_path="include/tinyrl_qtable.h")

---

Command Line Interface (CLI)

TINYRLAM provides an intuitive CLI for training, benchmarking, and exporting:

# Train an agent with custom hyperparameters
tinyrlam train --episodes 5000 --alpha 0.1 --gamma 0.9 --export-c include/tinyrl_qtable.h

# Benchmark against all 6 baseline policies
tinyrlam eval --episodes 5000 --seed 42

# Export embedded C headers
tinyrlam export-c --output include/tinyrl_qtable.h

# Generate all 15 publication figures
tinyrlam generate-figures

---

Reproducing Paper Results

To reproduce all numerical tables, baselines, multi-seed statistical confidence intervals, and 15 publication figures:

python scripts/reproduce_paper_results.py

To run individual verification scripts:

# Baseline sanity check & reward validation
python scripts/verify_checklist.py

# Re-generate all high-resolution figures into figures/
python scripts/generate_all_figures.py

---

Embedded C & Microcontroller Deployment

TINYRLAM is designed to run directly on bare-metal C/C++ or Arduino-compatible microcontrollers.

1. Flash to Arduino / ESP32

Open firmware/arduino/tinyrlam_node/tinyrlam_node.ino in the Arduino IDE, connect your microcontroller via USB, and click Upload.

2. Standalone C Library Integration (include/tinyrlam.h)

#include "tinyrlam.h"

// Instantiate agent (384 bytes in SRAM)
tinyrlam_agent_t agent;

void setup_node() {
    tinyrlam_init(&agent, 0.1f, 0.9f);
}

void execute_duty_cycle() {
    // 1. Read sensors & discretize into 1-byte state index
    uint8_t batt_bin  = read_battery_bin(); // 0..3
    uint8_t queue_bin = read_queue_bin();   // 0..2
    uint8_t is_day    = read_solar_bin();   // 0..1
    uint8_t state     = tinyrlam_encode_state(batt_bin, queue_bin, is_day);

    // 2. Select optimal action (Constant time O(1))
    tinyrlam_action_t action = tinyrlam_predict(&agent, state);

    // 3. Execute hardware action
    float reward = execute_hardware_action(action);

    // 4. (Optional) Learn on-device continuously
    uint8_t next_state = read_next_state();
    tinyrlam_learn(&agent, state, action, reward, next_state);
}

---

Running Unit Tests

To run the full unit test suite:

# Standard Python unittest
python -m unittest discover tests -v

# Or using pytest
pytest tests -v

---

Citation

If you use TINYRLAM in your research or application, please cite the manuscript:

@inproceedings{malik2026tinyrlam,
  title={TINYRLAM: A Resource-Efficient Reinforcement Learning Agent for Adaptive Energy Management in Energy-Harvesting Wireless Sensor Networks},
  author={Malik, Seema and Gupta, Kirti and Gupta, Yashita and Sharma, Vansh and Kansal, Nishtha},
  booktitle={IEEE Conference on Cyber-Physical Systems and IoT},
  year={2026},
  publisher={IEEE}
}

---

Authors & Acknowledgments

• Dr. Seema Malik (Corresponding Author) — simadahiya@gmail.com
• Kirti Gupta — 01417711724IIOT@vipstc.edu
• Yashita Gupta — 00417711724IIOT@vipstc.edu
• Vansh Sharma — 02717711724IIOT@vipstc.edu
• Dr. Nishtha Kansal — nishtha.kansal@vips.edu

School of Engineering & Technology, Vivekananda Institute of Professional Studies – Technical Campus (VIPS-TC), Delhi, India.

---

License

This project is licensed under the MIT License.