Skip to content

LWIP.md

Latest commit

 

History

History
264 lines (198 loc) · 8.78 KB

LWIP.md

File metadata and controls

264 lines (198 loc) · 8.78 KB
Raw

Tailscale lwIP Integration

Status: ✅ IMPLEMENTED

TCP/UDP connectivity over Tailscale now works via lwIP integration. Standard BSD sockets are fully functional on the Tailscale interface.

Verified working:

  • TCP connections (echo server on port 6666/7777)
  • ICMP ping responses
  • Full TCP handshake (SYN → SYN-ACK → ACK → data → FIN)

Goal

Create a virtual lwIP network interface (TUN-like) for Tailscale, allowing standard BSD sockets to work transparently over the Tailscale network. ESPHome's web server and other services will be accessible on both WiFi and Tailscale IPs.

User Requirements

  • Dual-stack: Services accessible on both WiFi (192.168.x.x) and Tailscale (100.x.x.x) IPs
  • Replace TailscaleSocket API: Remove custom socket API, use only standard BSD sockets
  • Memory budget: ~20-30KB additional

Architecture Overview

Previous Flow (Synthetic TCP Stack)

UDP/DERP → WireGuard decrypt → Manual IP/TCP parsing → TailscaleSocket callbacks

Current Flow (lwIP Integration)

UDP/DERP → WireGuard decrypt → TailscaleNetif → lwIP TCP/UDP stack → Standard sockets

Implementation Plan

Phase 1: Create TailscaleNetif Class

New files:

  • components/tailscale/tailscale_netif.h
  • components/tailscale/tailscale_netif.cpp

Class design:

class TailscaleNetif {
public:
  bool init(const std::string& tailscale_ip);  // Create esp_netif with IP
  void start();                                  // Bring interface up
  void stop();                                   // Bring interface down
  void receive(const uint8_t* ip_packet, size_t len);  // Inject into lwIP

  using TransmitCallback = std::function<void(const uint8_t*, size_t)>;
  void set_transmit_callback(TransmitCallback cb);  // Called when lwIP sends

private:
  esp_netif_t* netif_;
  esp_netif_driver_base_t driver_base_;
  TransmitCallback transmit_cb_;
  bool running_;
};

Key implementation details:

  1. Custom esp_netif_netstack_config with lwIP init and input functions
  2. Interface type: Point-to-point (no ARP), similar to PPP/SLIP
  3. MTU: 1280 bytes (safe for WireGuard overhead)
  4. Route priority: 50 (lower than WiFi's 100, so WiFi is default for non-Tailscale IPs)
  5. Subnet: 100.0.0.0/8 for Tailscale address space

Phase 2: Integrate with TailscaleComponent

Modify tailscale.h:

// Add member
std::unique_ptr<TailscaleNetif> tailscale_netif_;

Modify tailscale.cpp:

  1. Initialization (in handle_fetching_map_state_() after receiving IP):
tailscale_netif_ = make_unique<TailscaleNetif>();
tailscale_netif_->init(node_config_.ipv4_address);
tailscale_netif_->set_transmit_callback([this](const uint8_t* pkt, size_t len) {
  route_outgoing_packet_(pkt, len);  // Send via WireGuard
});
tailscale_netif_->start();
  1. Receive path (modify WireGuard decrypt callback):
wg_device_manager_->set_decrypt_callback(
  [this](const std::string& peer_ip, const uint8_t* ip_packet, size_t len) {
    if (tailscale_netif_) {
      tailscale_netif_->receive(ip_packet, len);  // Inject into lwIP
    }
  });
  1. Transmit path (new method route_outgoing_packet_()):
void route_outgoing_packet_(const uint8_t* ip_packet, size_t len) {
  // Extract destination IP from packet header
  uint32_t dst_ip = extract_dst_ip(ip_packet);
  std::string dst_str = ip_to_string(dst_ip);

  // Send via WireGuard to the peer
  wg_device_manager_->send_ip_packet(dst_str, ip_packet, len);
}

Phase 3: Remove Synthetic TCP Stack

Files to delete:

  • components/tailscale/tailscale_socket.h
  • components/tailscale/tailscale_socket.cpp
  • components/tailscale/echo_socket.h

Code to remove from tailscale.h:

  • enum class TcpState
  • struct TcpConnection
  • class TailscaleSocket forward declaration
  • TCP-related methods: bind_socket(), send_tcp_data(), close_tcp_connection(), handle_tcp_packet_(), send_tcp_packet_(), checksum functions
  • TCP-related members: tcp_connections_, bound_sockets_

Code to remove from tailscale.cpp:

  • handle_tcp_packet_() implementation (~200 lines)
  • send_tcp_packet_() implementation (~120 lines)
  • TCP socket API implementations
  • Manual ICMP echo handling in decrypt callback
  • Echo server socket setup

Phase 4: Update HTTP Server (if needed)

Modify http_server.h/cpp:

  • Replace TailscaleServerSocket base class with standard socket implementation
  • Use socket(), bind(), listen(), accept() APIs
  • Bind to INADDR_ANY (0.0.0.0) to listen on all interfaces
void HTTPServer::start(uint16_t port) {
  int sock = socket(AF_INET, SOCK_STREAM, 0);
  sockaddr_in addr = { .sin_family = AF_INET, .sin_port = htons(port), .sin_addr.s_addr = INADDR_ANY };
  bind(sock, (sockaddr*)&addr, sizeof(addr));
  listen(sock, 4);
  // Accept loop in FreeRTOS task
}

Critical Files to Modify

File Changes
tailscale.h Add TailscaleNetif member, remove TCP stack declarations
tailscale.cpp Init netif, modify decrypt callback, remove TCP handling
tailscale_netif.h NEW - TailscaleNetif class declaration
tailscale_netif.cpp NEW - esp_netif driver implementation
tailscale_socket.h DELETE
tailscale_socket.cpp DELETE
echo_socket.h DELETE
http_server.h/cpp Rewrite to use BSD sockets

Reference Files (Read Only)

  • .esphome/idf_components/.../esp_modem/src/esp_modem_netif.cpp - esp_netif driver pattern
  • .esphome/idf_components/.../slip_modem/library/slip_modem_netif.c - Custom netstack pattern

Technical Considerations

lwIP Thread Safety

  • Use LOCK_TCPIP_CORE() / UNLOCK_TCPIP_CORE() for thread safety
  • Or use tcpip_callback() for deferred execution from non-lwIP threads

Point-to-Point Interface (No ARP)

  • Custom output function bypasses ARP (like PPP/SLIP)
  • Set netif->flags without NETIF_FLAG_ETHARP

Routing

  • Packets to 100.x.x.x → Tailscale interface (via subnet match)
  • All other traffic → WiFi interface (default route)
  • Services on INADDR_ANY accessible on both

Memory Impact

  • Remove synthetic TCP: saves ~2.5KB
  • Add TailscaleNetif: ~1KB
  • Net savings: ~1.5KB

Testing Checklist

  1. ICMP ping - ping 100.64.0.x from another Tailscale node
  2. TCP connection - nc 100.64.0.x 80 then send HTTP request
  3. Dual-interface - Access web server on both WiFi and Tailscale IPs
  4. Multiple connections - 4+ concurrent TCP connections
  5. WireGuard handshake - Connection survives peer handshake refresh
  6. DERP fallback - Works when direct path unavailable

Potential Challenges

Challenge Mitigation
Thread safety Use lwIP locking macros
Routing conflicts Set correct route_prio (50 for Tailscale, 100 for WiFi)
ARP-less interface Use P2P output function, not ethernet output
MTU fragmentation Set MTU=1280 to avoid fragmentation

Critical Fixes Discovered During Implementation

1. esp_netif_new() Does NOT Add netif to netif_list

Problem: esp_netif_new() creates the ESP-IDF wrapper and underlying lwIP netif, but does NOT insert it into lwIP's netif_list. This caused TCP SYN-ACK packets to be routed to the WiFi interface instead of the Tailscale interface.

Symptom: ICMP ping worked (uses incoming interface), TCP RST worked, but TCP SYN-ACK was sent via WiFi (ip4_output_if: st1 instead of ts0).

Root Cause: ip4_route() searches netif_list to find the output interface. Our netif wasn't in the list.

Fix: Manually insert the netif at the head of netif_list after configuration:

LOCK_TCPIP_CORE();
struct netif* lwip_netif = esp_netif_get_netif_impl(esp_netif_);
if (lwip_netif not in netif_list) {
  lwip_netif->next = netif_list;
  netif_list = lwip_netif;
}
UNLOCK_TCPIP_CORE();

2. TCP Checksum Must Exclude IP Header

Problem: TCP checksum was calculated over the entire pbuf, which included the IP header.

Fix: Use pbuf_header(p, -(s16_t)IP_HLEN) to hide the IP header before checksum calculation, then restore with pbuf_header(p, IP_HLEN).

3. ESP-IDF Memory Allocation with Exceptions Disabled

Problem: new (std::nothrow) still aborts on allocation failure when ESP-IDF has C++ exceptions disabled.

Fix: Use malloc()/free() instead of new/delete for buffer pools.

4. TX Buffer Pool Size for ESP32-C3

Problem: TX_POOL_SIZE=32 × 1536 bytes = ~49KB, exceeding available RAM on ESP32-C3.

Fix: Reduced TX_POOL_SIZE to 4 (~6KB total), sufficient for typical traffic patterns.

Files Implemented

File Status
tailscale_netif.h ✅ Created
tailscale_netif.cpp ✅ Created
tailscale.cpp ✅ Modified (netif integration)
tailscale_socket.h/cpp Kept for reference (can be deleted)