Merge pull request 'Reconnect tuning: 1 min idle timeout + 5 min periodic autoreconnect (v0.21.3)' (#46) from fix/reconnect-tuning into main

The v0.21.2 frame-based watchdog works in production (it fired today: 'no
decoded bridge frames for 628474 ms — forcing reconnect', and the reconnect
delivered queued packets), but the bridge/radio side at 192.168.1.4 keeps
killing inbound RX within ~20 min of a fresh sync, repeatedly. Two tuning
changes requested by the owner:

- MESH_IDLE_TIMEOUT_MS default lowered from 600000 (10 min) to 60000 (1 min).
- New lazy periodic autoreconnect: MESH_RECONNECT_INTERVAL_MS (new env,
  default 300000 = 5 min, read at construction like the others). An interval
  started in onModuleInit unconditionally logs and destroys the socket so the
  existing close -> scheduleReconnect loop re-establishes with a fresh sync on
  schedule; cleared in onModuleDestroy. When not connected at tick time,
  destroy is a no-op and the retry loop owns recovery — no special-casing.

Tests: 171 -> 172 — periodic reconnect observed end-to-end with an overridden
short interval (fake bridge disconnect + re-establish); existing watchdog tests
pass unchanged, comments updated to the new 60 s default.

AGENTS.md

@@ -12,7 +12,7 @@
 ## Verification
 
 1. **Local checks** — `npm run build` (nest build) and `npm test` (vitest; no e2e suite yet). Both must pass before a PR is opened.
-2. **Docker** — `docker compose build`, then after `docker compose up -d`: `curl -s -H "Host: bot.dev.kkhost.pl" http://localhost/api/status` returns JSON (`{ connected, myNodeNum, host, port }`). `"connected":false` is normal while the bridge/device is down — the service retries every 5 s.
+2. **Docker** — `docker compose build`, then after `docker compose up -d`: `curl -s -H "Host: bot.dev.kkhost.pl" http://localhost/api/status` returns JSON (`{ connected, myNodeNum, host, port }`). `"connected":false` is normal while the bridge/device is down — the service retries every 5 s. An idle watchdog also force-reconnects an ESTABLISHED-but-silent connection (zombie bridge: outbound works, inbound dead) when no DECODED FRAME arrives from the bridge for `MESH_IDLE_TIMEOUT_MS` ms (default 60000 = 1 min; checked every 30 s — raw non-frame bytes do NOT count), an undecoded frame buffer over `MESH_MAX_FRAME_BUFFER` bytes (default 256000) forces a reconnect as a stream-desync backstop, and the socket has TCP keepalive enabled (30 s). A lazy periodic autoreconnect additionally unconditionally re-establishes every `MESH_RECONNECT_INTERVAL_MS` ms (default 300000 = 5 min) for a fresh sync.
 
 ## API
 

CHANGELOG.md

@@ -1,5 +1,23 @@
 # Changelog
 
+## v0.21.3 — 2026-09-26 09:31
+
+- Idle timeout default lowered from 10 min to **1 min** (`MESH_IDLE_TIMEOUT_MS` fallback `600000` → `60000`) — the v0.21.2 frame-based watchdog works in production (it fired today and the reconnect delivered queued packets), but the bridge/radio side at 192.168.1.4 keeps killing inbound RX within ~20 min of a fresh sync, so a shorter detection window is wanted.
+- New **lazy periodic autoreconnect**: `MESH_RECONNECT_INTERVAL_MS` (new env, default **300000** = 5 min, read at construction) — an interval that unconditionally logs ("periodic reconnect — MESH_RECONNECT_INTERVAL_MS elapsed") and destroys the socket so the existing close→scheduleReconnect loop re-establishes with a fresh sync on schedule. When not connected at tick time, destroy is a no-op and the retry loop owns recovery (no special-casing). Started in `onModuleInit`, cleared in `onModuleDestroy`.
+- Tests: 171 → 172 — periodic reconnect observed end-to-end with an overridden short interval (fake bridge disconnect + re-establish); existing watchdog tests pass unchanged, comments updated to the new 60 s default.
+
+## v0.21.2 — 2026-09-26 08:47
+
+- Idle watchdog now tracks DECODED FRAMES, not raw bytes (fixes a second production zombie that slipped past v0.21.1): text+telemetry RX was dead for >1 h while TX worked and the watchdog never fired — /proc/net/snmp showed InSegs delta = 2 segments per 30 s, i.e. the bridge peer keeps dribbling small NON-frame bytes (~every 15 s) that kept resetting `lastPacketAt` on every socket chunk even though zero valid Meshtastic frames decoded. The `'data'` handler no longer updates `lastPacketAt` per chunk; it is updated only when `extractFrames` yields ≥1 valid packet (the reset in the `'connect'` handler is unchanged), so a stream that decodes nothing trips the watchdog after `MESH_IDLE_TIMEOUT_MS` even while raw bytes keep arriving.
+- Buffer-overflow backstop: if the undecoded frame buffer exceeds `MESH_MAX_FRAME_BUFFER` (new env, default **256000** = 250 KB) the service logs ERROR ("frame buffer overflow — stream desync, forcing reconnect") and destroys the socket → existing close→scheduleReconnect self-heals a desynced stream. The undecoded buffer is also reset on every (re)connect so leftover garbage from a dead session can't re-trigger the backstop or corrupt the first frames of a healthy new stream.
+- Tests: 169 → 171 — raw non-frame bytes do NOT keep the watchdog alive (garbage-only stream forces disconnect + reconnect, exactly today's incident shape), and an undecoded chunk over `MESH_MAX_FRAME_BUFFER` triggers destroy + a fresh session that decodes normally. The existing valid-frames-reset-the-clock test now exercises frame-based tracking unchanged.
+
+## v0.21.1 — 2026-09-26 06:47
+
+- Bridge zombie-connection fix (production incident: outbound sends kept working ~10 h while inbound packets stopped entirely and `/api/status` stayed `connected:true`). New **idle watchdog**: the service tracks `lastPacketAt`, updated on EVERY chunk received from the bridge in the lowest-level socket `'data'` path (any frame kind, decodable or not) and reset when a connection successfully starts. A 30 s interval checks: if connected AND no inbound traffic for `MESH_IDLE_TIMEOUT_MS` (new env, default **600000** = 10 min), it logs WARN ("no bridge traffic for X ms — forcing reconnect") and destroys the socket — the EXISTING connect/retry loop (`MESH_RECONNECT_MS`) then re-establishes with a fresh sync. No changes to connect-error retry/backoff behavior; this only adds detection for established-but-silent connections.
+- TCP keepalive on the bridge socket: `socket.setKeepAlive(true, 30_000)` — cheap insurance against dead peers at the TCP level, complementing the application-level watchdog.
+- Tests: 165 → 169 — silence past an overridden timeout forces disconnect + reconnect (fake bridge), inbound traffic keeps resetting the idle clock so no false trigger, default timeout does not fire in a test window, and `setKeepAlive(true, 30_000)` is asserted on the socket.
+
 ## v0.21.0 — 2026-09-25 19:59
 
 - `#rss get <name|#tag>` now replies with **3–5 SEPARATE mesh messages** instead of one small summary (owner request: the single-message summary was too small). Each item is its own packet, so no fragmentation — and each rides the trigger's channel threaded as a quote of the command packet.

package.json

@@ -1,6 +1,6 @@
 {
   "name": "mesh-bot",
-  "version": "0.21.0",
+  "version": "0.21.3",
   "description": "",
   "author": "",
   "private": true,

src/mesh/mesh.service.spec.ts

@@ -1,6 +1,7 @@
 import { create, fromBinary, toBinary } from '@bufbuild/protobuf';
 import * as P from '@meshtastic/protobufs';
 import { BadRequestException, NotFoundException } from '@nestjs/common';
+import * as net from 'node:net';
 import { DatabaseSync } from 'node:sqlite';
 import { beforeEach, describe, expect, it, vi } from 'vitest';
 import { BROADCAST_ADDR, extractFrames, frame } from './codec.js';
@@ -249,6 +250,195 @@ describe('MeshService connect', () => {
   });
 });
 
+const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
+
+describe('MeshService idle watchdog', () => {
+  it('forces a reconnect when an established connection goes silent past MESH_IDLE_TIMEOUT_MS', async () => {
+    const bridge = await startBridge();
+    process.env.MESH_HOST = '127.0.0.1';
+    process.env.MESH_PORT = String(bridge.port);
+    process.env.MESH_RECONNECT_MS = '50'; // fast retry so the re-establishment is observable
+    process.env.MESH_IDLE_TIMEOUT_MS = '100'; // override — the default (60000) would never fire in a test
+    const svc = new MeshService();
+    try {
+      svc.onModuleInit();
+      await waitFor(
+        () => bridge.sockets.length === 1 && svc.status().connected,
+      );
+
+      // zombie case: connected but no inbound traffic at all — wait past the timeout, then drive a watchdog tick
+      await sleep(250);
+      svc.checkIdle();
+
+      // socket torn down → existing connect/retry loop re-establishes with a fresh session
+      await waitFor(() => bridge.sockets.length >= 2 && svc.status().connected);
+    } finally {
+      delete process.env.MESH_RECONNECT_MS;
+      delete process.env.MESH_IDLE_TIMEOUT_MS;
+      svc.onModuleDestroy();
+      bridge.close();
+    }
+  });
+
+  it('raw non-frame bytes do NOT keep the watchdog alive — a garbage-only stream forces a reconnect', async () => {
+    const bridge = await startBridge();
+    process.env.MESH_HOST = '127.0.0.1';
+    process.env.MESH_PORT = String(bridge.port);
+    process.env.MESH_RECONNECT_MS = '50'; // fast retry so the re-establishment is observable
+    process.env.MESH_IDLE_TIMEOUT_MS = '100'; // override — the default (60000) would never fire in a test
+    const svc = new MeshService();
+    try {
+      svc.onModuleInit();
+      await waitFor(
+        () => bridge.sockets.length === 1 && svc.status().connected,
+      );
+
+      // today's incident: the peer keeps dribbling small non-frame bytes (keepalive noise / desynced stream) — raw chunks must not reset the idle clock
+      const garbage = Buffer.from('zz-no-frames-here'); // pure ASCII — never forms a 0x94 0xc3 frame header
+      for (let i = 0; i < 6; i++) {
+        await sleep(50); // ~300 ms wall time, well past the timeout
+        bridge.sockets[0].write(garbage);
+      }
+
+      svc.checkIdle();
+      await waitFor(() => bridge.sockets.length >= 2 && svc.status().connected);
+    } finally {
+      delete process.env.MESH_RECONNECT_MS;
+      delete process.env.MESH_IDLE_TIMEOUT_MS;
+      svc.onModuleDestroy();
+      bridge.close();
+    }
+  });
+
+  it('forces a reconnect when the undecoded frame buffer exceeds MESH_MAX_FRAME_BUFFER', async () => {
+    const bridge = await startBridge();
+    process.env.MESH_HOST = '127.0.0.1';
+    process.env.MESH_PORT = String(bridge.port);
+    process.env.MESH_RECONNECT_MS = '50'; // fast retry so the re-establishment is observable
+    process.env.MESH_MAX_FRAME_BUFFER = '100'; // tiny cap — Node delivers socket data in ~64 KB events, far below the 256_000 default
+    const svc = new MeshService();
+    try {
+      svc.onModuleInit();
+      await waitFor(
+        () => bridge.sockets.length === 1 && svc.status().connected,
+      );
+
+      // non-frame bytes (0x42 is not part of any frame header) — a desynced stream that never forms frames; the first ~64 KB data event alone exceeds the cap
+      bridge.sockets[0].write(Buffer.alloc(300_000, 0x42));
+
+      // overflow forced a reconnect — fresh session
+      await waitFor(() => bridge.sockets.length >= 2 && svc.status().connected);
+
+      // the new socket starts with a clean buffer and decodes valid frames normally (the myInfo frame is well under the cap)
+      const fr = create(P.Mesh.FromRadioSchema, {
+        payloadVariant: { case: 'myInfo', value: { myNodeNum: 777 } },
+      });
+      bridge.sockets[1].write(frame(toBinary(P.Mesh.FromRadioSchema, fr)));
+      await waitFor(() => svc.status().myNodeNum === 777);
+    } finally {
+      delete process.env.MESH_RECONNECT_MS;
+      delete process.env.MESH_MAX_FRAME_BUFFER;
+      svc.onModuleDestroy();
+      bridge.close();
+    }
+  });
+
+  it('inbound traffic keeps resetting the idle clock — no false trigger while packets arrive', async () => {
+    const bridge = await startBridge();
+    process.env.MESH_HOST = '127.0.0.1';
+    process.env.MESH_PORT = String(bridge.port);
+    process.env.MESH_IDLE_TIMEOUT_MS = '100';
+    const svc = new MeshService();
+    try {
+      svc.onModuleInit();
+      await waitFor(
+        () => bridge.sockets.length === 1 && svc.status().connected,
+      );
+
+      // steady drip of inbound frames for well over the timeout in wall time — each chunk resets lastPacketAt
+      for (let i = 0; i < 5; i++) {
+        await sleep(60);
+        bridge.sockets[0].write(nodeInfoFrame(42, kamilNodeInfo()));
+      }
+
+      svc.checkIdle(); // wall time is past the timeout, but traffic kept resetting it
+      await sleep(100); // enough for a destroy to surface as 'close' (connected would flip false)
+      expect(svc.status().connected).toBe(true);
+      expect(bridge.sockets.length).toBe(1);
+    } finally {
+      delete process.env.MESH_IDLE_TIMEOUT_MS;
+      svc.onModuleDestroy();
+      bridge.close();
+    }
+  });
+
+  it('does not trigger under the default timeout (MESH_IDLE_TIMEOUT_MS unset)', async () => {
+    const bridge = await startBridge();
+    process.env.MESH_HOST = '127.0.0.1';
+    process.env.MESH_PORT = String(bridge.port);
+    delete process.env.MESH_IDLE_TIMEOUT_MS; // default 60000 — a test can't wait that long, so silence must NOT fire
+    const svc = new MeshService();
+    try {
+      svc.onModuleInit();
+      await waitFor(
+        () => bridge.sockets.length === 1 && svc.status().connected,
+      );
+
+      await sleep(250); // silent, but far under the default budget
+      svc.checkIdle();
+
+      expect(svc.status().connected).toBe(true);
+      expect(bridge.sockets.length).toBe(1);
+    } finally {
+      delete process.env.MESH_IDLE_TIMEOUT_MS;
+      svc.onModuleDestroy();
+      bridge.close();
+    }
+  });
+
+  it('enables TCP keepalive on the bridge socket', async () => {
+    const setKeepAlive = vi.spyOn(net.Socket.prototype, 'setKeepAlive');
+    const bridge = await startBridge();
+    process.env.MESH_HOST = '127.0.0.1';
+    process.env.MESH_PORT = String(bridge.port);
+    const svc = new MeshService();
+    try {
+      svc.onModuleInit();
+      await waitFor(() => svc.status().connected);
+      expect(setKeepAlive).toHaveBeenCalledWith(true, 30_000);
+    } finally {
+      setKeepAlive.mockRestore();
+      svc.onModuleDestroy();
+      bridge.close();
+    }
+  });
+});
+
+describe('MeshService periodic autoreconnect', () => {
+  it('unconditionally reconnects when MESH_RECONNECT_INTERVAL_MS elapses', async () => {
+    const bridge = await startBridge();
+    process.env.MESH_HOST = '127.0.0.1';
+    process.env.MESH_PORT = String(bridge.port);
+    process.env.MESH_RECONNECT_MS = '50'; // fast retry so the re-establishment is observable
+    process.env.MESH_RECONNECT_INTERVAL_MS = '100'; // override — the default (300000) would never elapse in a test
+    const svc = new MeshService();
+    try {
+      svc.onModuleInit();
+      await waitFor(
+        () => bridge.sockets.length === 1 && svc.status().connected,
+      );
+
+      // interval fires → destroy → existing close/retry loop re-establishes with a fresh session
+      await waitFor(() => bridge.sockets.length >= 2 && svc.status().connected);
+    } finally {
+      delete process.env.MESH_RECONNECT_MS;
+      delete process.env.MESH_RECONNECT_INTERVAL_MS;
+      svc.onModuleDestroy();
+      bridge.close();
+    }
+  });
+});
+
 describe('MeshService send', () => {
   it('sends on the test channel (index 3) by default, never primary', async () => {
     const bridge = await startBridge();

src/mesh/mesh.service.ts

@@ -81,6 +81,8 @@ export interface TelemetryRecord {
 const HISTORY_LIMIT = 100;
 /** Rows kept in SQLite across restarts; older rows are pruned on every poller tick. */
 const DB_RETENTION = 1000;
+// how often the idle watchdog checks for a silent-but-connected bridge (ms)
+const IDLE_CHECK_MS = 30_000;
 
 // one row per MessageRecord, keyed by packet id (tx ids and injected ids are random uint32, rx uses the wire id)
 export const MESSAGES_DDL = `CREATE TABLE IF NOT EXISTS messages(
@@ -208,6 +210,21 @@ export class MeshService implements OnModuleInit, OnModuleDestroy {
   );
   // how often the poller re-upserts the history buffer into SQLite (ms)
   private readonly pollMs = parseInt(process.env.MESH_POLL_MS ?? '1000', 10);
+  // idle watchdog: force a reconnect when an ESTABLISHED connection stops decoding frames — the zombie-bridge case where outbound sends keep working but inbound frame flow is dead (raw non-frame bytes do NOT reset the clock). Read at construction like the other env config.
+  private readonly idleTimeoutMs = parseInt(
+    process.env.MESH_IDLE_TIMEOUT_MS ?? '60000',
+    10,
+  );
+  // lazy periodic autoreconnect: unconditionally destroy + re-establish every N ms so a fresh sync happens on schedule — the bridge/radio side tends to kill inbound RX within ~20 min of a fresh sync. Read at construction like the other env config.
+  private readonly reconnectIntervalMs = parseInt(
+    process.env.MESH_RECONNECT_INTERVAL_MS ?? '300000',
+    10,
+  );
+  // cap on undecoded frame buffer: a desynced stream that never forms frames would grow it without bound — force a reconnect instead. Env-overridable so tests can trigger it (Node caps socket 'data' events at ~64 KB).
+  private readonly maxFrameBuffer = parseInt(
+    process.env.MESH_MAX_FRAME_BUFFER ?? '256000',
+    10,
+  );
 
   /** SQLite database backing the message history; null = persistence disabled (tests construct without one). */
   private db: DatabaseSync | null = null;
@@ -255,6 +272,10 @@ export class MeshService implements OnModuleInit, OnModuleDestroy {
   // telemetry history (newest kept, capped at HISTORY_LIMIT)
   private telemetryHistory: TelemetryRecord[] = [];
   private reconnectTimer: NodeJS.Timeout | null = null;
+  /** epoch ms when the last valid frame was decoded from the bridge; reset on connect — idle watchdog baseline */
+  private lastPacketAt = Date.now();
+  private idleTimer: NodeJS.Timeout | null = null;
+  private periodicReconnectTimer: NodeJS.Timeout | null = null;
   private shuttingDown = false;
 
   onModuleInit(): void {
@@ -264,6 +285,15 @@ export class MeshService implements OnModuleInit, OnModuleDestroy {
       // ponytail: full re-upsert of the ≤100-row buffer every tick — trivial load, zero bookkeeping
       this.pollTimer = setInterval(() => this.flushHistory(), this.pollMs);
     }
+    // idle watchdog: catches established-but-silent connections (zombie bridge) that the connect/retry loop never sees
+    this.idleTimer = setInterval(() => this.checkIdle(), IDLE_CHECK_MS);
+    // lazy periodic autoreconnect: unconditional fresh sync on a fixed schedule; when not connected, destroy is a no-op and the retry loop owns recovery
+    this.periodicReconnectTimer = setInterval(() => {
+      this.logger.log(
+        'periodic reconnect — MESH_RECONNECT_INTERVAL_MS elapsed',
+      );
+      this.socket?.destroy();
+    }, this.reconnectIntervalMs);
     this.connect();
   }
 
@@ -271,6 +301,8 @@ export class MeshService implements OnModuleInit, OnModuleDestroy {
     this.shuttingDown = true;
     if (this.reconnectTimer) clearTimeout(this.reconnectTimer);
     if (this.pollTimer) clearInterval(this.pollTimer);
+    if (this.idleTimer) clearInterval(this.idleTimer);
+    if (this.periodicReconnectTimer) clearInterval(this.periodicReconnectTimer);
     this.socket?.destroy();
     this.db?.close();
   }
@@ -641,11 +673,17 @@ export class MeshService implements OnModuleInit, OnModuleDestroy {
   private connect(): void {
     if (this.shuttingDown) return;
     const socket = net.createConnection({ host: this.host, port: this.port });
+    // TCP-level keepalive: cheap insurance against a dead peer that the idle watchdog (application level) complements
+    socket.setKeepAlive(true, 30_000);
     this.socket = socket;
     socket.on('connect', () => {
       this.connected = true;
+      // fresh connection — reset the idle clock so it can't self-trigger before any traffic arrives
+      this.lastPacketAt = Date.now();
       // fresh sync on every (re)connect — drop stale channels/nodes/device info/telemetry from the old session;
-      // message history is NOT reset: it's an append-only log backed by SQLite and survives reconnects
+      // message history is NOT reset: it's an append-only log backed by SQLite and survives reconnects.
+      // The undecoded frame buffer goes too: leftover desync garbage must not re-trigger the overflow backstop on a healthy new stream.
+      this.buffer = new Uint8Array(0);
       this.channelMap = new Map();
       this.nodeMap = new Map();
       this.metadata = null;
@@ -666,9 +704,19 @@ export class MeshService implements OnModuleInit, OnModuleDestroy {
       const merged = new Uint8Array(this.buffer.length + chunk.length);
       merged.set(this.buffer, 0);
       merged.set(chunk, this.buffer.length);
+      // buffer-overflow backstop: a desynced stream that never forms frames would grow the buffer without bound — cap it and self-heal via reconnect
+      if (merged.length > this.maxFrameBuffer) {
+        this.logger.error(
+          'frame buffer overflow — stream desync, forcing reconnect',
+        );
+        socket.destroy();
+        return;
+      }
       this.buffer = merged;
       const { packets, rest } = extractFrames(this.buffer);
       this.buffer = rest;
+      // the watchdog tracks DECODED FRAMES, not raw bytes: non-frame noise (keepalives, desynced streams) must not keep a dead connection alive
+      if (packets.length) this.lastPacketAt = Date.now();
       for (const p of packets) {
         decodeFromRadio(p)
           .then((fr) => this.handleFromRadio(fr))
@@ -694,6 +742,25 @@ export class MeshService implements OnModuleInit, OnModuleDestroy {
     this.reconnectTimer = setTimeout(() => this.connect(), this.reconnectMs);
   }
 
+  /**
+   * Idle watchdog tick: force a reconnect when an ESTABLISHED connection has decoded no frames from
+   * the bridge for MESH_IDLE_TIMEOUT_MS — the zombie case where outbound sends keep working but inbound
+   * frame flow is dead (raw non-frame bytes arriving on the socket do not count). Destroying the socket
+   * routes through 'close' → onDown → scheduleReconnect, so the existing connect/retry loop re-establishes
+   * with a fresh sync; no new retry machinery here. Public so tests can drive a tick without waiting for
+   * the real interval (like flushHistory()).
+   */
+  checkIdle(): void {
+    if (!this.connected) return; // disconnected state is owned by the connect/retry loop
+    const idle = Date.now() - this.lastPacketAt;
+    if (idle > this.idleTimeoutMs) {
+      this.logger.warn(
+        `no decoded bridge frames for ${idle} ms — forcing reconnect`,
+      );
+      this.socket?.destroy();
+    }
+  }
+
   private async handleFromRadio(
     fr: import('@meshtastic/protobufs').Mesh.FromRadio,
   ): Promise<void> {