Reconnect tuning: 1 min idle timeout + 5 min periodic autoreconnect (v0.21.3) #46

Merged
kamilkrzywda merged 1 commits from fix/reconnect-tuning into main 2026-09-26 11:37:41 +02:00
5 changed files with 51 additions and 6 deletions
+1 -1
View File
@@ -12,7 +12,7 @@
## Verification
1. **Local checks** — `npm run build` (nest build) and `npm test` (vitest; no e2e suite yet). Both must pass before a PR is opened.
2. **Docker** — `docker compose build`, then after `docker compose up -d`: `curl -s -H "Host: bot.dev.kkhost.pl" http://localhost/api/status` returns JSON (`{ connected, myNodeNum, host, port }`). `"connected":false` is normal while the bridge/device is down — the service retries every 5 s. An idle watchdog also force-reconnects an ESTABLISHED-but-silent connection (zombie bridge: outbound works, inbound dead) when no DECODED FRAME arrives from the bridge for `MESH_IDLE_TIMEOUT_MS` ms (default 600000 = 10 min; checked every 30 s — raw non-frame bytes do NOT count), an undecoded frame buffer over `MESH_MAX_FRAME_BUFFER` bytes (default 256000) forces a reconnect as a stream-desync backstop, and the socket has TCP keepalive enabled (30 s).
2. **Docker** — `docker compose build`, then after `docker compose up -d`: `curl -s -H "Host: bot.dev.kkhost.pl" http://localhost/api/status` returns JSON (`{ connected, myNodeNum, host, port }`). `"connected":false` is normal while the bridge/device is down — the service retries every 5 s. An idle watchdog also force-reconnects an ESTABLISHED-but-silent connection (zombie bridge: outbound works, inbound dead) when no DECODED FRAME arrives from the bridge for `MESH_IDLE_TIMEOUT_MS` ms (default 60000 = 1 min; checked every 30 s — raw non-frame bytes do NOT count), an undecoded frame buffer over `MESH_MAX_FRAME_BUFFER` bytes (default 256000) forces a reconnect as a stream-desync backstop, and the socket has TCP keepalive enabled (30 s). A lazy periodic autoreconnect additionally unconditionally re-establishes every `MESH_RECONNECT_INTERVAL_MS` ms (default 300000 = 5 min) for a fresh sync.
## API
+6
View File
@@ -1,5 +1,11 @@
# Changelog
## v0.21.3 — 2026-09-26 09:31
- Idle timeout default lowered from 10 min to **1 min** (`MESH_IDLE_TIMEOUT_MS` fallback `600000` → `60000`) — the v0.21.2 frame-based watchdog works in production (it fired today and the reconnect delivered queued packets), but the bridge/radio side at 192.168.1.4 keeps killing inbound RX within ~20 min of a fresh sync, so a shorter detection window is wanted.
- New **lazy periodic autoreconnect**: `MESH_RECONNECT_INTERVAL_MS` (new env, default **300000** = 5 min, read at construction) — an interval that unconditionally logs ("periodic reconnect — MESH_RECONNECT_INTERVAL_MS elapsed") and destroys the socket so the existing close→scheduleReconnect loop re-establishes with a fresh sync on schedule. When not connected at tick time, destroy is a no-op and the retry loop owns recovery (no special-casing). Started in `onModuleInit`, cleared in `onModuleDestroy`.
- Tests: 171 → 172 — periodic reconnect observed end-to-end with an overridden short interval (fake bridge disconnect + re-establish); existing watchdog tests pass unchanged, comments updated to the new 60 s default.
## v0.21.2 — 2026-09-26 08:47
- Idle watchdog now tracks DECODED FRAMES, not raw bytes (fixes a second production zombie that slipped past v0.21.1): text+telemetry RX was dead for >1 h while TX worked and the watchdog never fired — /proc/net/snmp showed InSegs delta = 2 segments per 30 s, i.e. the bridge peer keeps dribbling small NON-frame bytes (~every 15 s) that kept resetting `lastPacketAt` on every socket chunk even though zero valid Meshtastic frames decoded. The `'data'` handler no longer updates `lastPacketAt` per chunk; it is updated only when `extractFrames` yields ≥1 valid packet (the reset in the `'connect'` handler is unchanged), so a stream that decodes nothing trips the watchdog after `MESH_IDLE_TIMEOUT_MS` even while raw bytes keep arriving.
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "mesh-bot",
"version": "0.21.2",
"version": "0.21.3",
"description": "",
"author": "",
"private": true,
+28 -3
View File
@@ -258,7 +258,7 @@ describe('MeshService idle watchdog', () => {
process.env.MESH_HOST = '127.0.0.1';
process.env.MESH_PORT = String(bridge.port);
process.env.MESH_RECONNECT_MS = '50'; // fast retry so the re-establishment is observable
process.env.MESH_IDLE_TIMEOUT_MS = '100'; // override — the default (600000) would never fire in a test
process.env.MESH_IDLE_TIMEOUT_MS = '100'; // override — the default (60000) would never fire in a test
const svc = new MeshService();
try {
svc.onModuleInit();
@@ -285,7 +285,7 @@ describe('MeshService idle watchdog', () => {
process.env.MESH_HOST = '127.0.0.1';
process.env.MESH_PORT = String(bridge.port);
process.env.MESH_RECONNECT_MS = '50'; // fast retry so the re-establishment is observable
process.env.MESH_IDLE_TIMEOUT_MS = '100'; // override — the default (600000) would never fire in a test
process.env.MESH_IDLE_TIMEOUT_MS = '100'; // override — the default (60000) would never fire in a test
const svc = new MeshService();
try {
svc.onModuleInit();
@@ -376,7 +376,7 @@ describe('MeshService idle watchdog', () => {
const bridge = await startBridge();
process.env.MESH_HOST = '127.0.0.1';
process.env.MESH_PORT = String(bridge.port);
delete process.env.MESH_IDLE_TIMEOUT_MS; // default 600000 — a test can't wait that long, so silence must NOT fire
delete process.env.MESH_IDLE_TIMEOUT_MS; // default 60000 — a test can't wait that long, so silence must NOT fire
const svc = new MeshService();
try {
svc.onModuleInit();
@@ -414,6 +414,31 @@ describe('MeshService idle watchdog', () => {
});
});
describe('MeshService periodic autoreconnect', () => {
it('unconditionally reconnects when MESH_RECONNECT_INTERVAL_MS elapses', async () => {
const bridge = await startBridge();
process.env.MESH_HOST = '127.0.0.1';
process.env.MESH_PORT = String(bridge.port);
process.env.MESH_RECONNECT_MS = '50'; // fast retry so the re-establishment is observable
process.env.MESH_RECONNECT_INTERVAL_MS = '100'; // override — the default (300000) would never elapse in a test
const svc = new MeshService();
try {
svc.onModuleInit();
await waitFor(
() => bridge.sockets.length === 1 && svc.status().connected,
);
// interval fires → destroy → existing close/retry loop re-establishes with a fresh session
await waitFor(() => bridge.sockets.length >= 2 && svc.status().connected);
} finally {
delete process.env.MESH_RECONNECT_MS;
delete process.env.MESH_RECONNECT_INTERVAL_MS;
svc.onModuleDestroy();
bridge.close();
}
});
});
describe('MeshService send', () => {
it('sends on the test channel (index 3) by default, never primary', async () => {
const bridge = await startBridge();
+15 -1
View File
@@ -212,7 +212,12 @@ export class MeshService implements OnModuleInit, OnModuleDestroy {
private readonly pollMs = parseInt(process.env.MESH_POLL_MS ?? '1000', 10);
// idle watchdog: force a reconnect when an ESTABLISHED connection stops decoding frames — the zombie-bridge case where outbound sends keep working but inbound frame flow is dead (raw non-frame bytes do NOT reset the clock). Read at construction like the other env config.
private readonly idleTimeoutMs = parseInt(
process.env.MESH_IDLE_TIMEOUT_MS ?? '600000',
process.env.MESH_IDLE_TIMEOUT_MS ?? '60000',
10,
);
// lazy periodic autoreconnect: unconditionally destroy + re-establish every N ms so a fresh sync happens on schedule — the bridge/radio side tends to kill inbound RX within ~20 min of a fresh sync. Read at construction like the other env config.
private readonly reconnectIntervalMs = parseInt(
process.env.MESH_RECONNECT_INTERVAL_MS ?? '300000',
10,
);
// cap on undecoded frame buffer: a desynced stream that never forms frames would grow it without bound — force a reconnect instead. Env-overridable so tests can trigger it (Node caps socket 'data' events at ~64 KB).
@@ -270,6 +275,7 @@ export class MeshService implements OnModuleInit, OnModuleDestroy {
/** epoch ms when the last valid frame was decoded from the bridge; reset on connect — idle watchdog baseline */
private lastPacketAt = Date.now();
private idleTimer: NodeJS.Timeout | null = null;
private periodicReconnectTimer: NodeJS.Timeout | null = null;
private shuttingDown = false;
onModuleInit(): void {
@@ -281,6 +287,13 @@ export class MeshService implements OnModuleInit, OnModuleDestroy {
}
// idle watchdog: catches established-but-silent connections (zombie bridge) that the connect/retry loop never sees
this.idleTimer = setInterval(() => this.checkIdle(), IDLE_CHECK_MS);
// lazy periodic autoreconnect: unconditional fresh sync on a fixed schedule; when not connected, destroy is a no-op and the retry loop owns recovery
this.periodicReconnectTimer = setInterval(() => {
this.logger.log(
'periodic reconnect — MESH_RECONNECT_INTERVAL_MS elapsed',
);
this.socket?.destroy();
}, this.reconnectIntervalMs);
this.connect();
}
@@ -289,6 +302,7 @@ export class MeshService implements OnModuleInit, OnModuleDestroy {
if (this.reconnectTimer) clearTimeout(this.reconnectTimer);
if (this.pollTimer) clearInterval(this.pollTimer);
if (this.idleTimer) clearInterval(this.idleTimer);
if (this.periodicReconnectTimer) clearInterval(this.periodicReconnectTimer);
this.socket?.destroy();
this.db?.close();
}