Reconnect tuning: 1 min idle timeout + 5 min periodic autoreconnect (v0.21.3) #46
@@ -12,7 +12,7 @@
|
|||||||
## Verification
|
## Verification
|
||||||
|
|
||||||
1. **Local checks** — `npm run build` (nest build) and `npm test` (vitest; no e2e suite yet). Both must pass before a PR is opened.
|
1. **Local checks** — `npm run build` (nest build) and `npm test` (vitest; no e2e suite yet). Both must pass before a PR is opened.
|
||||||
2. **Docker** — `docker compose build`, then after `docker compose up -d`: `curl -s -H "Host: bot.dev.kkhost.pl" http://localhost/api/status` returns JSON (`{ connected, myNodeNum, host, port }`). `"connected":false` is normal while the bridge/device is down — the service retries every 5 s. An idle watchdog also force-reconnects an ESTABLISHED-but-silent connection (zombie bridge: outbound works, inbound dead) when no DECODED FRAME arrives from the bridge for `MESH_IDLE_TIMEOUT_MS` ms (default 600000 = 10 min; checked every 30 s — raw non-frame bytes do NOT count), an undecoded frame buffer over `MESH_MAX_FRAME_BUFFER` bytes (default 256000) forces a reconnect as a stream-desync backstop, and the socket has TCP keepalive enabled (30 s).
|
2. **Docker** — `docker compose build`, then after `docker compose up -d`: `curl -s -H "Host: bot.dev.kkhost.pl" http://localhost/api/status` returns JSON (`{ connected, myNodeNum, host, port }`). `"connected":false` is normal while the bridge/device is down — the service retries every 5 s. An idle watchdog also force-reconnects an ESTABLISHED-but-silent connection (zombie bridge: outbound works, inbound dead) when no DECODED FRAME arrives from the bridge for `MESH_IDLE_TIMEOUT_MS` ms (default 60000 = 1 min; checked every 30 s — raw non-frame bytes do NOT count), an undecoded frame buffer over `MESH_MAX_FRAME_BUFFER` bytes (default 256000) forces a reconnect as a stream-desync backstop, and the socket has TCP keepalive enabled (30 s). A lazy periodic autoreconnect additionally unconditionally re-establishes every `MESH_RECONNECT_INTERVAL_MS` ms (default 300000 = 5 min) for a fresh sync.
|
||||||
|
|
||||||
## API
|
## API
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,11 @@
|
|||||||
# Changelog
|
# Changelog
|
||||||
|
|
||||||
|
## v0.21.3 — 2026-09-26 09:31
|
||||||
|
|
||||||
|
- Idle timeout default lowered from 10 min to **1 min** (`MESH_IDLE_TIMEOUT_MS` fallback `600000` → `60000`) — the v0.21.2 frame-based watchdog works in production (it fired today and the reconnect delivered queued packets), but the bridge/radio side at 192.168.1.4 keeps killing inbound RX within ~20 min of a fresh sync, so a shorter detection window is wanted.
|
||||||
|
- New **lazy periodic autoreconnect**: `MESH_RECONNECT_INTERVAL_MS` (new env, default **300000** = 5 min, read at construction) — an interval that unconditionally logs ("periodic reconnect — MESH_RECONNECT_INTERVAL_MS elapsed") and destroys the socket so the existing close→scheduleReconnect loop re-establishes with a fresh sync on schedule. When not connected at tick time, destroy is a no-op and the retry loop owns recovery (no special-casing). Started in `onModuleInit`, cleared in `onModuleDestroy`.
|
||||||
|
- Tests: 171 → 172 — periodic reconnect observed end-to-end with an overridden short interval (fake bridge disconnect + re-establish); existing watchdog tests pass unchanged, comments updated to the new 60 s default.
|
||||||
|
|
||||||
## v0.21.2 — 2026-09-26 08:47
|
## v0.21.2 — 2026-09-26 08:47
|
||||||
|
|
||||||
- Idle watchdog now tracks DECODED FRAMES, not raw bytes (fixes a second production zombie that slipped past v0.21.1): text+telemetry RX was dead for >1 h while TX worked and the watchdog never fired — /proc/net/snmp showed InSegs delta = 2 segments per 30 s, i.e. the bridge peer keeps dribbling small NON-frame bytes (~every 15 s) that kept resetting `lastPacketAt` on every socket chunk even though zero valid Meshtastic frames decoded. The `'data'` handler no longer updates `lastPacketAt` per chunk; it is updated only when `extractFrames` yields ≥1 valid packet (the reset in the `'connect'` handler is unchanged), so a stream that decodes nothing trips the watchdog after `MESH_IDLE_TIMEOUT_MS` even while raw bytes keep arriving.
|
- Idle watchdog now tracks DECODED FRAMES, not raw bytes (fixes a second production zombie that slipped past v0.21.1): text+telemetry RX was dead for >1 h while TX worked and the watchdog never fired — /proc/net/snmp showed InSegs delta = 2 segments per 30 s, i.e. the bridge peer keeps dribbling small NON-frame bytes (~every 15 s) that kept resetting `lastPacketAt` on every socket chunk even though zero valid Meshtastic frames decoded. The `'data'` handler no longer updates `lastPacketAt` per chunk; it is updated only when `extractFrames` yields ≥1 valid packet (the reset in the `'connect'` handler is unchanged), so a stream that decodes nothing trips the watchdog after `MESH_IDLE_TIMEOUT_MS` even while raw bytes keep arriving.
|
||||||
|
|||||||
+1
-1
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "mesh-bot",
|
"name": "mesh-bot",
|
||||||
"version": "0.21.2",
|
"version": "0.21.3",
|
||||||
"description": "",
|
"description": "",
|
||||||
"author": "",
|
"author": "",
|
||||||
"private": true,
|
"private": true,
|
||||||
|
|||||||
@@ -258,7 +258,7 @@ describe('MeshService idle watchdog', () => {
|
|||||||
process.env.MESH_HOST = '127.0.0.1';
|
process.env.MESH_HOST = '127.0.0.1';
|
||||||
process.env.MESH_PORT = String(bridge.port);
|
process.env.MESH_PORT = String(bridge.port);
|
||||||
process.env.MESH_RECONNECT_MS = '50'; // fast retry so the re-establishment is observable
|
process.env.MESH_RECONNECT_MS = '50'; // fast retry so the re-establishment is observable
|
||||||
process.env.MESH_IDLE_TIMEOUT_MS = '100'; // override — the default (600000) would never fire in a test
|
process.env.MESH_IDLE_TIMEOUT_MS = '100'; // override — the default (60000) would never fire in a test
|
||||||
const svc = new MeshService();
|
const svc = new MeshService();
|
||||||
try {
|
try {
|
||||||
svc.onModuleInit();
|
svc.onModuleInit();
|
||||||
@@ -285,7 +285,7 @@ describe('MeshService idle watchdog', () => {
|
|||||||
process.env.MESH_HOST = '127.0.0.1';
|
process.env.MESH_HOST = '127.0.0.1';
|
||||||
process.env.MESH_PORT = String(bridge.port);
|
process.env.MESH_PORT = String(bridge.port);
|
||||||
process.env.MESH_RECONNECT_MS = '50'; // fast retry so the re-establishment is observable
|
process.env.MESH_RECONNECT_MS = '50'; // fast retry so the re-establishment is observable
|
||||||
process.env.MESH_IDLE_TIMEOUT_MS = '100'; // override — the default (600000) would never fire in a test
|
process.env.MESH_IDLE_TIMEOUT_MS = '100'; // override — the default (60000) would never fire in a test
|
||||||
const svc = new MeshService();
|
const svc = new MeshService();
|
||||||
try {
|
try {
|
||||||
svc.onModuleInit();
|
svc.onModuleInit();
|
||||||
@@ -376,7 +376,7 @@ describe('MeshService idle watchdog', () => {
|
|||||||
const bridge = await startBridge();
|
const bridge = await startBridge();
|
||||||
process.env.MESH_HOST = '127.0.0.1';
|
process.env.MESH_HOST = '127.0.0.1';
|
||||||
process.env.MESH_PORT = String(bridge.port);
|
process.env.MESH_PORT = String(bridge.port);
|
||||||
delete process.env.MESH_IDLE_TIMEOUT_MS; // default 600000 — a test can't wait that long, so silence must NOT fire
|
delete process.env.MESH_IDLE_TIMEOUT_MS; // default 60000 — a test can't wait that long, so silence must NOT fire
|
||||||
const svc = new MeshService();
|
const svc = new MeshService();
|
||||||
try {
|
try {
|
||||||
svc.onModuleInit();
|
svc.onModuleInit();
|
||||||
@@ -414,6 +414,31 @@ describe('MeshService idle watchdog', () => {
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
describe('MeshService periodic autoreconnect', () => {
|
||||||
|
it('unconditionally reconnects when MESH_RECONNECT_INTERVAL_MS elapses', async () => {
|
||||||
|
const bridge = await startBridge();
|
||||||
|
process.env.MESH_HOST = '127.0.0.1';
|
||||||
|
process.env.MESH_PORT = String(bridge.port);
|
||||||
|
process.env.MESH_RECONNECT_MS = '50'; // fast retry so the re-establishment is observable
|
||||||
|
process.env.MESH_RECONNECT_INTERVAL_MS = '100'; // override — the default (300000) would never elapse in a test
|
||||||
|
const svc = new MeshService();
|
||||||
|
try {
|
||||||
|
svc.onModuleInit();
|
||||||
|
await waitFor(
|
||||||
|
() => bridge.sockets.length === 1 && svc.status().connected,
|
||||||
|
);
|
||||||
|
|
||||||
|
// interval fires → destroy → existing close/retry loop re-establishes with a fresh session
|
||||||
|
await waitFor(() => bridge.sockets.length >= 2 && svc.status().connected);
|
||||||
|
} finally {
|
||||||
|
delete process.env.MESH_RECONNECT_MS;
|
||||||
|
delete process.env.MESH_RECONNECT_INTERVAL_MS;
|
||||||
|
svc.onModuleDestroy();
|
||||||
|
bridge.close();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
describe('MeshService send', () => {
|
describe('MeshService send', () => {
|
||||||
it('sends on the test channel (index 3) by default, never primary', async () => {
|
it('sends on the test channel (index 3) by default, never primary', async () => {
|
||||||
const bridge = await startBridge();
|
const bridge = await startBridge();
|
||||||
|
|||||||
@@ -212,7 +212,12 @@ export class MeshService implements OnModuleInit, OnModuleDestroy {
|
|||||||
private readonly pollMs = parseInt(process.env.MESH_POLL_MS ?? '1000', 10);
|
private readonly pollMs = parseInt(process.env.MESH_POLL_MS ?? '1000', 10);
|
||||||
// idle watchdog: force a reconnect when an ESTABLISHED connection stops decoding frames — the zombie-bridge case where outbound sends keep working but inbound frame flow is dead (raw non-frame bytes do NOT reset the clock). Read at construction like the other env config.
|
// idle watchdog: force a reconnect when an ESTABLISHED connection stops decoding frames — the zombie-bridge case where outbound sends keep working but inbound frame flow is dead (raw non-frame bytes do NOT reset the clock). Read at construction like the other env config.
|
||||||
private readonly idleTimeoutMs = parseInt(
|
private readonly idleTimeoutMs = parseInt(
|
||||||
process.env.MESH_IDLE_TIMEOUT_MS ?? '600000',
|
process.env.MESH_IDLE_TIMEOUT_MS ?? '60000',
|
||||||
|
10,
|
||||||
|
);
|
||||||
|
// lazy periodic autoreconnect: unconditionally destroy + re-establish every N ms so a fresh sync happens on schedule — the bridge/radio side tends to kill inbound RX within ~20 min of a fresh sync. Read at construction like the other env config.
|
||||||
|
private readonly reconnectIntervalMs = parseInt(
|
||||||
|
process.env.MESH_RECONNECT_INTERVAL_MS ?? '300000',
|
||||||
10,
|
10,
|
||||||
);
|
);
|
||||||
// cap on undecoded frame buffer: a desynced stream that never forms frames would grow it without bound — force a reconnect instead. Env-overridable so tests can trigger it (Node caps socket 'data' events at ~64 KB).
|
// cap on undecoded frame buffer: a desynced stream that never forms frames would grow it without bound — force a reconnect instead. Env-overridable so tests can trigger it (Node caps socket 'data' events at ~64 KB).
|
||||||
@@ -270,6 +275,7 @@ export class MeshService implements OnModuleInit, OnModuleDestroy {
|
|||||||
/** epoch ms when the last valid frame was decoded from the bridge; reset on connect — idle watchdog baseline */
|
/** epoch ms when the last valid frame was decoded from the bridge; reset on connect — idle watchdog baseline */
|
||||||
private lastPacketAt = Date.now();
|
private lastPacketAt = Date.now();
|
||||||
private idleTimer: NodeJS.Timeout | null = null;
|
private idleTimer: NodeJS.Timeout | null = null;
|
||||||
|
private periodicReconnectTimer: NodeJS.Timeout | null = null;
|
||||||
private shuttingDown = false;
|
private shuttingDown = false;
|
||||||
|
|
||||||
onModuleInit(): void {
|
onModuleInit(): void {
|
||||||
@@ -281,6 +287,13 @@ export class MeshService implements OnModuleInit, OnModuleDestroy {
|
|||||||
}
|
}
|
||||||
// idle watchdog: catches established-but-silent connections (zombie bridge) that the connect/retry loop never sees
|
// idle watchdog: catches established-but-silent connections (zombie bridge) that the connect/retry loop never sees
|
||||||
this.idleTimer = setInterval(() => this.checkIdle(), IDLE_CHECK_MS);
|
this.idleTimer = setInterval(() => this.checkIdle(), IDLE_CHECK_MS);
|
||||||
|
// lazy periodic autoreconnect: unconditional fresh sync on a fixed schedule; when not connected, destroy is a no-op and the retry loop owns recovery
|
||||||
|
this.periodicReconnectTimer = setInterval(() => {
|
||||||
|
this.logger.log(
|
||||||
|
'periodic reconnect — MESH_RECONNECT_INTERVAL_MS elapsed',
|
||||||
|
);
|
||||||
|
this.socket?.destroy();
|
||||||
|
}, this.reconnectIntervalMs);
|
||||||
this.connect();
|
this.connect();
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -289,6 +302,7 @@ export class MeshService implements OnModuleInit, OnModuleDestroy {
|
|||||||
if (this.reconnectTimer) clearTimeout(this.reconnectTimer);
|
if (this.reconnectTimer) clearTimeout(this.reconnectTimer);
|
||||||
if (this.pollTimer) clearInterval(this.pollTimer);
|
if (this.pollTimer) clearInterval(this.pollTimer);
|
||||||
if (this.idleTimer) clearInterval(this.idleTimer);
|
if (this.idleTimer) clearInterval(this.idleTimer);
|
||||||
|
if (this.periodicReconnectTimer) clearInterval(this.periodicReconnectTimer);
|
||||||
this.socket?.destroy();
|
this.socket?.destroy();
|
||||||
this.db?.close();
|
this.db?.close();
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user