Compare commits

...
8 Commits
Author SHA1 Message Date
kamilkrzywda 77459adc13 Merge pull request 'Watchdog: track decoded frames, not raw bytes (v0.21.2)' (#45) from fix/watchdog-frame-tracking into main
CI / build + unit (push) Failing after 21s
2026-09-26 10:53:28 +02:00
kamilkrzywda 33a88f3d83 Watchdog tracks decoded frames, not raw bytes; buffer-overflow backstop (v0.21.2)
CI / build + unit (pull_request) Failing after 39s
Second production zombie slipped past v0.21.1's watchdog: text+telemetry RX
dead for >1 h while TX worked and the watchdog never fired. Root cause proven
at runtime via /proc/net/snmp — InSegs delta = 2 segments per 30 s (OutSegs 0):
the bridge peer keeps dribbling small NON-frame bytes (~every 15 s, keepalive
noise or a desynced stream) and v0.21.1 updated lastPacketAt on EVERY socket
chunk, so 'no traffic for MESH_IDLE_TIMEOUT_MS' was never true even though zero
valid Meshtastic frames decoded.

- The 'data' handler no longer updates lastPacketAt per chunk; it is updated
  only when extractFrames yields >=1 valid packet (the reset in the 'connect'
  handler is unchanged), so a stream that decodes nothing trips the watchdog
  after MESH_IDLE_TIMEOUT_MS even while raw bytes keep arriving.
- Buffer-overflow backstop: if the undecoded frame buffer exceeds
  MESH_MAX_FRAME_BUFFER (new env, default 256000) — e.g. one data event of a
  desynced stream that never forms frames — log ERROR and destroy the socket;
  the existing close -> scheduleReconnect loop self-heals it. The undecoded
  buffer is also reset on every (re)connect so leftover garbage from a dead
  session can't re-trigger the backstop or corrupt the first frames of a
  healthy new stream.

Tests: 169 -> 171 — raw non-frame bytes do NOT keep the watchdog alive
(garbage-only stream forces disconnect + reconnect, today's incident shape),
and an undecoded chunk over MESH_MAX_FRAME_BUFFER triggers destroy + a fresh
session that decodes normally (cap env-overridden to 100 since Node delivers
socket data in ~64 KB events, far below the 256_000 default).
2026-09-26 08:50:27 +00:00
kamilkrzywda 306756c63b Merge pull request 'Bridge idle watchdog + forced reconnect (v0.21.1)' (#44) from fix/bridge-idle-reconnect into main
CI / build + unit (push) Successful in 45s
2026-09-26 08:53:06 +02:00
kamilkrzywda af1c77fd54 Force reconnect when the bridge connection goes silent (v0.21.1)
CI / build + unit (pull_request) Successful in 1m8s
Production incident: the bridge TCP connection went zombie — outbound sends
kept working for ~10 h while inbound packets stopped entirely, and
/api/status stayed connected:true because nothing detects inbound silence on
an established connection.

- Track lastPacketAt: updated on every chunk received from the bridge in the
  lowest-level socket 'data' path (any frame kind, decodable or not) and reset
  when a connection successfully starts, so a fresh connection can't
  self-trigger.
- Idle watchdog: a 30 s interval checks connected AND now - lastPacketAt >
  MESH_IDLE_TIMEOUT_MS (new env, default 600000 = 10 min) — logs WARN with the
  idle duration and destroys the socket so the EXISTING connect/retry loop
  re-establishes it with a fresh sync. No changes to connect-error retry or
  backoff; this only adds detection for established-but-silent connections.
- TCP keepalive: socket.setKeepAlive(true, 30_000) on the bridge socket —
  cheap insurance against dead peers at the TCP level.

Tests (fake-bridge based): silence past an overridden timeout forces a
disconnect + reconnect; inbound traffic keeps resetting the idle clock so no
false trigger; the default timeout does not fire in a test window; and
setKeepAlive(true, 30_000) is asserted on the socket. checkIdle() is public so
tests drive a tick without waiting for the real interval (like flushHistory()).
2026-09-26 06:50:01 +00:00
kamilkrzywda 73ce49c741 Merge pull request '#rss get: 3-5 separate messages with short headers (v0.21.0)' (#43) from feat/rss-multi-message into main
CI / build + unit (push) Successful in 41s
2026-09-25 22:12:05 +02:00
kamilkrzywda cdd52a3ddb Send #rss get summaries as 3-5 separate mesh messages (v0.21.0)
CI / build + unit (pull_request) Successful in 1m20s
The single-message summary was too small (owner request). The summarizer
prompt now asks for exactly 3-5 lines, one item per line, in the format
'<short header>: <one or two short sentences>' with a <=20-char header,
no intro/outro/numbering. maxTokens stays 2500 (reasoning-token headroom).

The reply is parsed into items: drop empty lines and lines without a ':'
header separator, cap at 5; fewer than 3 valid lines sends what we got
(no failure/retry), zero valid lines sends nothing. Each item goes out as
its own mesh message via respond(id, line) - all threaded to the trigger
packet id on its channel. Per-item backstop fitOneLine() hard-cuts any
line >185 chars at a word boundary near 180 so every item fits one packet;
the old whole-reply fitOnePacket() cut is gone.

CommandService sends array replies line by line through the same routing
as single replies (respond per line, or legacy sendText fallback per line
when there is no packet id). LLM failure path unchanged: single
'summary failed' reply.
2026-09-25 20:09:16 +00:00
kamilkrzywda 172e7bed9c Merge pull request 'Cap #rss get summaries to one mesh packet (v0.20.1)' (#42) from fix/rss-summary-length into main
CI / build + unit (push) Successful in 1m10s
2026-09-25 21:25:02 +02:00
kamilkrzywda e2d712a26c Cap #rss get summaries to one mesh packet (v0.20.1)
CI / build + unit (pull_request) Successful in 42s
Replies >~190 chars fragment into fire-and-forget packets recipients
often never receive; old prompt produced ~326-334 char replies.

Prompt now asks for at most 3 short bullet lines, each under 45 chars,
bullets only (verified live: exactly 3 compliant bullets, ~110 chars).
fitOnePacket() backstop cuts any over-length reply at complete-bullet
boundaries so delivery is one packet even if the model ignores the prompt.

maxTokens 200 -> 2500 on the summary chat(): empirically verified that
this server's reasoning tokens count against max_tokens (cap 150 ->
completion 150 of which 105 were reasoning; cap 500 fully consumed by
reasoning, zero visible content) and thinking is uncontrollable
server-side (/no_think, chat_template_kwargs.enable_thinking,
thinking_budget all ignored), observed 32..1776 tokens for this task.
2500 covers every observed run + ~80 content tokens, worst case ~42 s
under the 60 s LLM timeout.
2026-09-25 19:22:13 +00:00
9 changed files with 454 additions and 37 deletions
+2 -2
View File
@@ -12,7 +12,7 @@
## Verification
1. **Local checks** — `npm run build` (nest build) and `npm test` (vitest; no e2e suite yet). Both must pass before a PR is opened.
2. **Docker** — `docker compose build`, then after `docker compose up -d`: `curl -s -H "Host: bot.dev.kkhost.pl" http://localhost/api/status` returns JSON (`{ connected, myNodeNum, host, port }`). `"connected":false` is normal while the bridge/device is down — the service retries every 5 s.
2. **Docker** — `docker compose build`, then after `docker compose up -d`: `curl -s -H "Host: bot.dev.kkhost.pl" http://localhost/api/status` returns JSON (`{ connected, myNodeNum, host, port }`). `"connected":false` is normal while the bridge/device is down — the service retries every 5 s. An idle watchdog also force-reconnects an ESTABLISHED-but-silent connection (zombie bridge: outbound works, inbound dead) when no DECODED FRAME arrives from the bridge for `MESH_IDLE_TIMEOUT_MS` ms (default 600000 = 10 min; checked every 30 s — raw non-frame bytes do NOT count), an undecoded frame buffer over `MESH_MAX_FRAME_BUFFER` bytes (default 256000) forces a reconnect as a stream-desync backstop, and the socket has TCP keepalive enabled (30 s).
## API
@@ -27,5 +27,5 @@
- MESHTASTIC.md — rules for Meshtastic communication (owner authority, language, response protocol); read before acting on mesh messages.
- Hourly cron weather report job (`src/weather`): fetches Open-Meteo current conditions and broadcasts a one-line report to the mesh on the pinned send channel; env `WEATHER_CRON`/`WEATHER_LAT`/`WEATHER_LON`/`WEATHER_CITY`.
- Channel-2 auto-responder (`src/commands`): event-driven — `CommandService.onRx()` is called by `MeshService` for every decoded rx text (right after the notify gate) and dispatches exact, case-sensitive `#command` matches on `COMMAND_CHANNEL` (env, default 2) through a single-registration-point registry (handlers registered in the constructor via `register(name, fn)`; handler gets `{ text, from?, id? }`, returns reply string or void = no reply). First command: `#ping` → replies `#pong!`. Replies go through `MeshService.respond(id, reply)`: auto-routed (rx → DM back to the original sender unless the original was a broadcast; tx → its destination), riding the original record's channel with `replyId = id` so the recipient app renders it as a threaded quote — `COMMAND_CHANNEL` is only the trigger/listen gate. Only when the trigger has no packet id does it fall back to legacy sendText routing on `COMMAND_CHANNEL`. A failing handler/respond only logs a warn. DI: provided by `CommandModule`'s factory (injects `MeshService`, attaches itself via `attachCommands()`) — no constructor-level cycle; `forwardRef` was tried and left one side null in NestJS 12's injector. Prefix dispatch (`registerPrefix(name, fn)`) lets one handler own a whole family: the `#rss` subcommands live in `src/rss` (`RssService.handle`, registered as prefix `'rss'` from the CommandModule factory — exact matches still win over prefixes). Subcommands: `#rss help`; `#rss add <url> <name>` (validates URL, fetches with 15 s timeout, requires ≥1 titled item, rejects duplicate names case-insensitively; LLM generates 2-4 lowercase tags from up to 8 latest titles — on LLM failure the source is added with empty tags and the reply says so); `#rss remove <name>`; `#rss tags` (all distinct tags); `#rss sources`; `#rss get <name|#tag>` (LLM headline summary, maxTokens 200; `#tag` collects items across ALL sources carrying that tag — one dead feed does not sink the summary). Storage: `rss_sources(name TEXT PRIMARY KEY, url, tags, created_at)` table in the EXISTING MESH_DB_PATH file via a second node:sqlite connection (names matched case-insensitively); feed parsing via `fast-xml-parser` (RSS 2.0 + Atom; malformed/no-items → "not a valid feed"); LLM via injected `LlmService.chat()` with explicit maxTokens (50 tags / 200 summary) to keep mesh replies short.
- Channel-2 auto-responder (`src/commands`): event-driven — `CommandService.onRx()` is called by `MeshService` for every decoded rx text (right after the notify gate) and dispatches exact, case-sensitive `#command` matches on `COMMAND_CHANNEL` (env, default 2) through a single-registration-point registry (handlers registered in the constructor via `register(name, fn)`; handler gets `{ text, from?, id? }`, returns a reply string — or an array of lines, sent as one message per line through the same routing — or void = no reply). First command: `#ping` → replies `#pong!`. Replies go through `MeshService.respond(id, reply)`: auto-routed (rx → DM back to the original sender unless the original was a broadcast; tx → its destination), riding the original record's channel with `replyId = id` so the recipient app renders it as a threaded quote — `COMMAND_CHANNEL` is only the trigger/listen gate. Only when the trigger has no packet id does it fall back to legacy sendText routing on `COMMAND_CHANNEL`. A failing handler/respond only logs a warn. DI: provided by `CommandModule`'s factory (injects `MeshService`, attaches itself via `attachCommands()`) — no constructor-level cycle; `forwardRef` was tried and left one side null in NestJS 12's injector. Prefix dispatch (`registerPrefix(name, fn)`) lets one handler own a whole family: the `#rss` subcommands live in `src/rss` (`RssService.handle`, registered as prefix `'rss'` from the CommandModule factory — exact matches still win over prefixes). Subcommands: `#rss help`; `#rss add <url> <name>` (validates URL, fetches with 15 s timeout, requires ≥1 titled item, rejects duplicate names case-insensitively; LLM generates 2-4 lowercase tags from up to 8 latest titles — on LLM failure the source is added with empty tags and the reply says so); `#rss remove <name>`; `#rss tags` (all distinct tags); `#rss sources`; `#rss get <name|#tag>` (LLM headline summary as 3-5 SEPARATE mesh messages: prompt asks for exactly 3-5 lines, one item per line, `<short header>: <one or two short sentences>` with ≤20-char header, no intro/outro/numbering; reply parsed into items — drop empty/separator-less lines, cap at 5, fewer than 3 → send what we got (no retry), zero → no message; each item sent via its own respond() threaded to the trigger packet id, per-line backstop `fitOneLine()` hard-cuts >185-char lines at a word boundary near 180 so every item fits one packet; maxTokens 2500 is headroom because the LLM server's reasoning tokens count against it — see rss.service.ts comment; `#tag` collects items across ALL sources carrying that tag — one dead feed does not sink the summary). Storage: `rss_sources(name TEXT PRIMARY KEY, url, tags, created_at)` table in the EXISTING MESH_DB_PATH file via a second node:sqlite connection (names matched case-insensitively); feed parsing via `fast-xml-parser` (RSS 2.0 + Atom; malformed/no-items → "not a valid feed"); LLM via injected `LlmService.chat()` with explicit maxTokens (50 tags / 2500 summary headroom).
- LLM client (`src/llm`): `LlmService` wraps the local OpenAI-compatible server (default `http://192.168.1.70:1111/v1`, no auth) with native fetch — plain feature module, no Mesh/Schedule dependency. Methods: `listModels()` → `data[].id`; `chat(messages, { maxTokens?, temperature? })` → `choices[0].message.content` (non-2xx throws with status + body snippet); `summarize(text)` = chat() with system prompt "Summarize the following text concisely." Env: `LLM_API_BASE`, `LLM_MODEL` (default `qwen3.8-27b`), `LLM_TIMEOUT_MS` (default 60000, via `AbortSignal.timeout`).
+29
View File
@@ -1,5 +1,34 @@
# Changelog
## v0.21.2 — 2026-09-26 08:47
- Idle watchdog now tracks DECODED FRAMES, not raw bytes (fixes a second production zombie that slipped past v0.21.1): text+telemetry RX was dead for >1 h while TX worked and the watchdog never fired — /proc/net/snmp showed InSegs delta = 2 segments per 30 s, i.e. the bridge peer keeps dribbling small NON-frame bytes (~every 15 s) that kept resetting `lastPacketAt` on every socket chunk even though zero valid Meshtastic frames decoded. The `'data'` handler no longer updates `lastPacketAt` per chunk; it is updated only when `extractFrames` yields ≥1 valid packet (the reset in the `'connect'` handler is unchanged), so a stream that decodes nothing trips the watchdog after `MESH_IDLE_TIMEOUT_MS` even while raw bytes keep arriving.
- Buffer-overflow backstop: if the undecoded frame buffer exceeds `MESH_MAX_FRAME_BUFFER` (new env, default **256000** = 250 KB) the service logs ERROR ("frame buffer overflow — stream desync, forcing reconnect") and destroys the socket → existing close→scheduleReconnect self-heals a desynced stream. The undecoded buffer is also reset on every (re)connect so leftover garbage from a dead session can't re-trigger the backstop or corrupt the first frames of a healthy new stream.
- Tests: 169 → 171 — raw non-frame bytes do NOT keep the watchdog alive (garbage-only stream forces disconnect + reconnect, exactly today's incident shape), and an undecoded chunk over `MESH_MAX_FRAME_BUFFER` triggers destroy + a fresh session that decodes normally. The existing valid-frames-reset-the-clock test now exercises frame-based tracking unchanged.
## v0.21.1 — 2026-09-26 06:47
- Bridge zombie-connection fix (production incident: outbound sends kept working ~10 h while inbound packets stopped entirely and `/api/status` stayed `connected:true`). New **idle watchdog**: the service tracks `lastPacketAt`, updated on EVERY chunk received from the bridge in the lowest-level socket `'data'` path (any frame kind, decodable or not) and reset when a connection successfully starts. A 30 s interval checks: if connected AND no inbound traffic for `MESH_IDLE_TIMEOUT_MS` (new env, default **600000** = 10 min), it logs WARN ("no bridge traffic for X ms — forcing reconnect") and destroys the socket — the EXISTING connect/retry loop (`MESH_RECONNECT_MS`) then re-establishes with a fresh sync. No changes to connect-error retry/backoff behavior; this only adds detection for established-but-silent connections.
- TCP keepalive on the bridge socket: `socket.setKeepAlive(true, 30_000)` — cheap insurance against dead peers at the TCP level, complementing the application-level watchdog.
- Tests: 165 → 169 — silence past an overridden timeout forces disconnect + reconnect (fake bridge), inbound traffic keeps resetting the idle clock so no false trigger, default timeout does not fire in a test window, and `setKeepAlive(true, 30_000)` is asserted on the socket.
## v0.21.0 — 2026-09-25 19:59
- `#rss get <name|#tag>` now replies with **3–5 SEPARATE mesh messages** instead of one small summary (owner request: the single-message summary was too small). Each item is its own packet, so no fragmentation — and each rides the trigger's channel threaded as a quote of the command packet.
- New summarizer prompt asks for exactly 3-5 lines, one item per line, format `<short header>: <one or two short sentences>` with a ≤20-char header (e.g. `Culture: Museum opens new wing`), no intro/outro/numbering. maxTokens stays 2500 (reasoning-token headroom — see v0.20.1).
- The LLM reply is parsed into items: split on newlines, trim, drop empty lines and lines without a `:` header separator, cap at 5. Fewer than 3 valid lines → send what we got (no failure, no retry — the model is non-deterministic); zero valid lines → no message sent.
- Per-item backstop replaces the old whole-reply cut: any single line >185 chars is hard-cut at a word boundary near 180 (`fitOneLine`), so every item fits one packet even if the model ignores the prompt. `fitOnePacket()` (whole-reply ≤190) is gone — it no longer applies to joined text.
- CommandService now sends array replies line by line: each element goes through the same routing as a single reply (`respond(id, line)` threaded to the trigger packet id; legacy sendText fallback per line when there is no packet id). A failing send stops the remaining lines (warn log), like any failed command.
- LLM failure path unchanged: single `summary failed` reply via respond(). `#rss get #tag` gets the same multi-message treatment automatically (shared summary path — covered by tests). All other subcommands unchanged.
- Tests: 158 → 165 — prompt shape + maxTokens asserted on the new call, 5-item cap, invalid-line filtering, fewer-than-3 fallback, per-line over-length cut through `handle()`, direct `parseSummaryItems`/`fitOneLine` specs, and CommandService array-reply routing (threaded respond ×N to one packet id; legacy sendText fallback).
## v0.20.1 — 2026-09-25 19:20
- `#rss get` summaries now fit ONE mesh packet (≤190 chars). Mesh fragments longer replies into fire-and-forget packets that recipients frequently never receive, and the old prompt ("a few short one-line highlights") produced ~326-334 char replies.
- The summary prompt now asks for "at most 3 short bullet lines, each under 45 characters … bullets only - no intro, no other text" (verified against the live LLM: exactly 3 compliant bullets, ~110 chars). A `fitOnePacket()` backstop additionally cuts any over-length reply at complete-bullet boundaries so delivery is one packet even if the model ignores the prompt.
- The summary `chat()` call now uses an explicit `maxTokens = 2500` (was 200). Empirically verified on this server: qwen3.8-27b emits `reasoning_content` and those tokens COUNT against max_tokens (cap 150 → completion_tokens 150, of which 105 were reasoning, finish_reason 'length'; a cap of 500 was fully consumed by reasoning with zero visible content). Thinking length for this task is unbounded and uncontrollable server-side (`/no_think`, `chat_template_kwargs.enable_thinking` and `thinking_budget` are all ignored) — observed 32..1776 tokens across runs. 2500 covers every observed run plus ~80 content tokens, and keeps the worst case (~42 s at ~60 tok/s) under the 60 s LLM timeout. The old cap of 200 guaranteed empty replies whenever reasoning exceeded it.
- Tests: 154 → 158 — prompt shape + maxTokens asserted on the summary call, over-length reply cut through `handle()`, and direct `fitOnePacket` specs (pass-through at/under 190, bullet-boundary cut, hard-cut fallback for non-bulleted paragraphs).
## v0.20.0 — 2026-09-25 18:40
- New `#rss` command family on the command channel (ch2): `#rss help`, `#rss add <url> <name>` (validates URL + fetches with a 15 s timeout, requires ≥1 item with a non-empty title, rejects duplicate names case-insensitively; tags are generated by the LLM from up to 8 latest titles — on LLM failure the source is still added with empty tags and the reply says so), `#rss remove <name>`, `#rss tags` (all distinct tags), `#rss sources`, `#rss get <name>` and `#rss get #<tag>` (LLM headline summary, maxTokens 200; tag form collects items across ALL sources carrying the tag — one dead feed does not sink the summary). Bare `#rss` / unknown subcommand → help. All replies ride the existing respond() path like `#ping`.
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "mesh-bot",
"version": "0.20.0",
"version": "0.21.2",
"description": "",
"author": "",
"private": true,
+24
View File
@@ -114,6 +114,30 @@ describe('CommandService.onRx', () => {
expect(mesh.respond).toHaveBeenCalledWith(5, 'up from 99');
});
it('sends each line of an array reply as its own threaded respond() to the same packet id', async () => {
const [svc, mesh] = makeService();
svc.register('multi', () => ['one: first item', 'two: second item']);
await svc.onRx(2, '#multi', 42, 9);
expect(mesh.respond).toHaveBeenCalledTimes(2);
expect(mesh.respond.mock.calls[0]).toEqual([9, 'one: first item']);
expect(mesh.respond.mock.calls[1]).toEqual([9, 'two: second item']);
expect(mesh.sendText).not.toHaveBeenCalled();
});
it('sends each line of an array reply via the legacy sendText fallback when there is no packet id', async () => {
const [svc, mesh] = makeService();
svc.register('multi', () => ['one: first item', 'two: second item']);
await svc.onRx(2, '#multi', 42); // no id → legacy path on COMMAND_CHANNEL, DM back to the sender
expect(mesh.sendText).toHaveBeenCalledTimes(2);
expect(mesh.sendText.mock.calls[0]).toEqual([
42,
'one: first item',
undefined,
2,
]);
expect(mesh.respond).not.toHaveBeenCalled();
});
it('swallows a failing handler (a bad command must not crash the app)', async () => {
const [svc, mesh] = makeService();
svc.register('boom', () => {
+8 -5
View File
@@ -14,10 +14,10 @@ export interface CommandContext {
id?: number;
}
/** A command handler: returns the reply text to send back, or void/undefined for no reply. */
/** A command handler: returns the reply text(s) to send back — an array sends one message per line — or void/undefined for no reply. */
export type CommandHandler = (
ctx: CommandContext,
) => string | undefined | Promise<string | undefined>;
) => string | string[] | undefined | Promise<string | string[] | undefined>;
@Injectable()
export class CommandService {
@@ -83,18 +83,21 @@ export class CommandService {
try {
const reply = await handler({ text: trimmed, from, id });
if (reply === undefined || reply === null) return; // void = no reply
// an array reply sends one message per line, all threaded to the same trigger packet id
for (const line of Array.isArray(reply) ? reply : [reply]) {
if (id !== undefined && this.mesh) {
// respond() auto-routes the reply: rides the original record's channel with replyId = id (threaded quote)
await this.mesh.respond(id, reply);
// respond() auto-routes each line: rides the original record's channel with replyId = id (threaded quote)
await this.mesh.respond(id, line);
} else {
// defensive fallback when the trigger has no packet id: legacy routing on COMMAND_CHANNEL
await this.mesh?.sendText(
from ?? BROADCAST_ADDR,
reply,
line,
undefined,
this.commandChannel,
);
}
}
} catch (err) {
this.logger.warn(
`[commands] '${trimmed}' failed: ${(err as Error).message}`,
+165
View File
@@ -1,6 +1,7 @@
import { create, fromBinary, toBinary } from '@bufbuild/protobuf';
import * as P from '@meshtastic/protobufs';
import { BadRequestException, NotFoundException } from '@nestjs/common';
import * as net from 'node:net';
import { DatabaseSync } from 'node:sqlite';
import { beforeEach, describe, expect, it, vi } from 'vitest';
import { BROADCAST_ADDR, extractFrames, frame } from './codec.js';
@@ -249,6 +250,170 @@ describe('MeshService connect', () => {
});
});
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
describe('MeshService idle watchdog', () => {
it('forces a reconnect when an established connection goes silent past MESH_IDLE_TIMEOUT_MS', async () => {
const bridge = await startBridge();
process.env.MESH_HOST = '127.0.0.1';
process.env.MESH_PORT = String(bridge.port);
process.env.MESH_RECONNECT_MS = '50'; // fast retry so the re-establishment is observable
process.env.MESH_IDLE_TIMEOUT_MS = '100'; // override — the default (600000) would never fire in a test
const svc = new MeshService();
try {
svc.onModuleInit();
await waitFor(
() => bridge.sockets.length === 1 && svc.status().connected,
);
// zombie case: connected but no inbound traffic at all — wait past the timeout, then drive a watchdog tick
await sleep(250);
svc.checkIdle();
// socket torn down → existing connect/retry loop re-establishes with a fresh session
await waitFor(() => bridge.sockets.length >= 2 && svc.status().connected);
} finally {
delete process.env.MESH_RECONNECT_MS;
delete process.env.MESH_IDLE_TIMEOUT_MS;
svc.onModuleDestroy();
bridge.close();
}
});
it('raw non-frame bytes do NOT keep the watchdog alive — a garbage-only stream forces a reconnect', async () => {
const bridge = await startBridge();
process.env.MESH_HOST = '127.0.0.1';
process.env.MESH_PORT = String(bridge.port);
process.env.MESH_RECONNECT_MS = '50'; // fast retry so the re-establishment is observable
process.env.MESH_IDLE_TIMEOUT_MS = '100'; // override — the default (600000) would never fire in a test
const svc = new MeshService();
try {
svc.onModuleInit();
await waitFor(
() => bridge.sockets.length === 1 && svc.status().connected,
);
// today's incident: the peer keeps dribbling small non-frame bytes (keepalive noise / desynced stream) — raw chunks must not reset the idle clock
const garbage = Buffer.from('zz-no-frames-here'); // pure ASCII — never forms a 0x94 0xc3 frame header
for (let i = 0; i < 6; i++) {
await sleep(50); // ~300 ms wall time, well past the timeout
bridge.sockets[0].write(garbage);
}
svc.checkIdle();
await waitFor(() => bridge.sockets.length >= 2 && svc.status().connected);
} finally {
delete process.env.MESH_RECONNECT_MS;
delete process.env.MESH_IDLE_TIMEOUT_MS;
svc.onModuleDestroy();
bridge.close();
}
});
it('forces a reconnect when the undecoded frame buffer exceeds MESH_MAX_FRAME_BUFFER', async () => {
const bridge = await startBridge();
process.env.MESH_HOST = '127.0.0.1';
process.env.MESH_PORT = String(bridge.port);
process.env.MESH_RECONNECT_MS = '50'; // fast retry so the re-establishment is observable
process.env.MESH_MAX_FRAME_BUFFER = '100'; // tiny cap — Node delivers socket data in ~64 KB events, far below the 256_000 default
const svc = new MeshService();
try {
svc.onModuleInit();
await waitFor(
() => bridge.sockets.length === 1 && svc.status().connected,
);
// non-frame bytes (0x42 is not part of any frame header) — a desynced stream that never forms frames; the first ~64 KB data event alone exceeds the cap
bridge.sockets[0].write(Buffer.alloc(300_000, 0x42));
// overflow forced a reconnect — fresh session
await waitFor(() => bridge.sockets.length >= 2 && svc.status().connected);
// the new socket starts with a clean buffer and decodes valid frames normally (the myInfo frame is well under the cap)
const fr = create(P.Mesh.FromRadioSchema, {
payloadVariant: { case: 'myInfo', value: { myNodeNum: 777 } },
});
bridge.sockets[1].write(frame(toBinary(P.Mesh.FromRadioSchema, fr)));
await waitFor(() => svc.status().myNodeNum === 777);
} finally {
delete process.env.MESH_RECONNECT_MS;
delete process.env.MESH_MAX_FRAME_BUFFER;
svc.onModuleDestroy();
bridge.close();
}
});
it('inbound traffic keeps resetting the idle clock — no false trigger while packets arrive', async () => {
const bridge = await startBridge();
process.env.MESH_HOST = '127.0.0.1';
process.env.MESH_PORT = String(bridge.port);
process.env.MESH_IDLE_TIMEOUT_MS = '100';
const svc = new MeshService();
try {
svc.onModuleInit();
await waitFor(
() => bridge.sockets.length === 1 && svc.status().connected,
);
// steady drip of inbound frames for well over the timeout in wall time — each chunk resets lastPacketAt
for (let i = 0; i < 5; i++) {
await sleep(60);
bridge.sockets[0].write(nodeInfoFrame(42, kamilNodeInfo()));
}
svc.checkIdle(); // wall time is past the timeout, but traffic kept resetting it
await sleep(100); // enough for a destroy to surface as 'close' (connected would flip false)
expect(svc.status().connected).toBe(true);
expect(bridge.sockets.length).toBe(1);
} finally {
delete process.env.MESH_IDLE_TIMEOUT_MS;
svc.onModuleDestroy();
bridge.close();
}
});
it('does not trigger under the default timeout (MESH_IDLE_TIMEOUT_MS unset)', async () => {
const bridge = await startBridge();
process.env.MESH_HOST = '127.0.0.1';
process.env.MESH_PORT = String(bridge.port);
delete process.env.MESH_IDLE_TIMEOUT_MS; // default 600000 — a test can't wait that long, so silence must NOT fire
const svc = new MeshService();
try {
svc.onModuleInit();
await waitFor(
() => bridge.sockets.length === 1 && svc.status().connected,
);
await sleep(250); // silent, but far under the default budget
svc.checkIdle();
expect(svc.status().connected).toBe(true);
expect(bridge.sockets.length).toBe(1);
} finally {
delete process.env.MESH_IDLE_TIMEOUT_MS;
svc.onModuleDestroy();
bridge.close();
}
});
it('enables TCP keepalive on the bridge socket', async () => {
const setKeepAlive = vi.spyOn(net.Socket.prototype, 'setKeepAlive');
const bridge = await startBridge();
process.env.MESH_HOST = '127.0.0.1';
process.env.MESH_PORT = String(bridge.port);
const svc = new MeshService();
try {
svc.onModuleInit();
await waitFor(() => svc.status().connected);
expect(setKeepAlive).toHaveBeenCalledWith(true, 30_000);
} finally {
setKeepAlive.mockRestore();
svc.onModuleDestroy();
bridge.close();
}
});
});
describe('MeshService send', () => {
it('sends on the test channel (index 3) by default, never primary', async () => {
const bridge = await startBridge();
+54 -1
View File
@@ -81,6 +81,8 @@ export interface TelemetryRecord {
const HISTORY_LIMIT = 100;
/** Rows kept in SQLite across restarts; older rows are pruned on every poller tick. */
const DB_RETENTION = 1000;
// how often the idle watchdog checks for a silent-but-connected bridge (ms)
const IDLE_CHECK_MS = 30_000;
// one row per MessageRecord, keyed by packet id (tx ids and injected ids are random uint32, rx uses the wire id)
export const MESSAGES_DDL = `CREATE TABLE IF NOT EXISTS messages(
@@ -208,6 +210,16 @@ export class MeshService implements OnModuleInit, OnModuleDestroy {
);
// how often the poller re-upserts the history buffer into SQLite (ms)
private readonly pollMs = parseInt(process.env.MESH_POLL_MS ?? '1000', 10);
// idle watchdog: force a reconnect when an ESTABLISHED connection stops decoding frames — the zombie-bridge case where outbound sends keep working but inbound frame flow is dead (raw non-frame bytes do NOT reset the clock). Read at construction like the other env config.
private readonly idleTimeoutMs = parseInt(
process.env.MESH_IDLE_TIMEOUT_MS ?? '600000',
10,
);
// cap on undecoded frame buffer: a desynced stream that never forms frames would grow it without bound — force a reconnect instead. Env-overridable so tests can trigger it (Node caps socket 'data' events at ~64 KB).
private readonly maxFrameBuffer = parseInt(
process.env.MESH_MAX_FRAME_BUFFER ?? '256000',
10,
);
/** SQLite database backing the message history; null = persistence disabled (tests construct without one). */
private db: DatabaseSync | null = null;
@@ -255,6 +267,9 @@ export class MeshService implements OnModuleInit, OnModuleDestroy {
// telemetry history (newest kept, capped at HISTORY_LIMIT)
private telemetryHistory: TelemetryRecord[] = [];
private reconnectTimer: NodeJS.Timeout | null = null;
/** epoch ms when the last valid frame was decoded from the bridge; reset on connect — idle watchdog baseline */
private lastPacketAt = Date.now();
private idleTimer: NodeJS.Timeout | null = null;
private shuttingDown = false;
onModuleInit(): void {
@@ -264,6 +279,8 @@ export class MeshService implements OnModuleInit, OnModuleDestroy {
// ponytail: full re-upsert of the ≤100-row buffer every tick — trivial load, zero bookkeeping
this.pollTimer = setInterval(() => this.flushHistory(), this.pollMs);
}
// idle watchdog: catches established-but-silent connections (zombie bridge) that the connect/retry loop never sees
this.idleTimer = setInterval(() => this.checkIdle(), IDLE_CHECK_MS);
this.connect();
}
@@ -271,6 +288,7 @@ export class MeshService implements OnModuleInit, OnModuleDestroy {
this.shuttingDown = true;
if (this.reconnectTimer) clearTimeout(this.reconnectTimer);
if (this.pollTimer) clearInterval(this.pollTimer);
if (this.idleTimer) clearInterval(this.idleTimer);
this.socket?.destroy();
this.db?.close();
}
@@ -641,11 +659,17 @@ export class MeshService implements OnModuleInit, OnModuleDestroy {
private connect(): void {
if (this.shuttingDown) return;
const socket = net.createConnection({ host: this.host, port: this.port });
// TCP-level keepalive: cheap insurance against a dead peer that the idle watchdog (application level) complements
socket.setKeepAlive(true, 30_000);
this.socket = socket;
socket.on('connect', () => {
this.connected = true;
// fresh connection — reset the idle clock so it can't self-trigger before any traffic arrives
this.lastPacketAt = Date.now();
// fresh sync on every (re)connect — drop stale channels/nodes/device info/telemetry from the old session;
// message history is NOT reset: it's an append-only log backed by SQLite and survives reconnects
// message history is NOT reset: it's an append-only log backed by SQLite and survives reconnects.
// The undecoded frame buffer goes too: leftover desync garbage must not re-trigger the overflow backstop on a healthy new stream.
this.buffer = new Uint8Array(0);
this.channelMap = new Map();
this.nodeMap = new Map();
this.metadata = null;
@@ -666,9 +690,19 @@ export class MeshService implements OnModuleInit, OnModuleDestroy {
const merged = new Uint8Array(this.buffer.length + chunk.length);
merged.set(this.buffer, 0);
merged.set(chunk, this.buffer.length);
// buffer-overflow backstop: a desynced stream that never forms frames would grow the buffer without bound — cap it and self-heal via reconnect
if (merged.length > this.maxFrameBuffer) {
this.logger.error(
'frame buffer overflow — stream desync, forcing reconnect',
);
socket.destroy();
return;
}
this.buffer = merged;
const { packets, rest } = extractFrames(this.buffer);
this.buffer = rest;
// the watchdog tracks DECODED FRAMES, not raw bytes: non-frame noise (keepalives, desynced streams) must not keep a dead connection alive
if (packets.length) this.lastPacketAt = Date.now();
for (const p of packets) {
decodeFromRadio(p)
.then((fr) => this.handleFromRadio(fr))
@@ -694,6 +728,25 @@ export class MeshService implements OnModuleInit, OnModuleDestroy {
this.reconnectTimer = setTimeout(() => this.connect(), this.reconnectMs);
}
/**
* Idle watchdog tick: force a reconnect when an ESTABLISHED connection has decoded no frames from
* the bridge for MESH_IDLE_TIMEOUT_MS — the zombie case where outbound sends keep working but inbound
* frame flow is dead (raw non-frame bytes arriving on the socket do not count). Destroying the socket
* routes through 'close' → onDown → scheduleReconnect, so the existing connect/retry loop re-establishes
* with a fresh sync; no new retry machinery here. Public so tests can drive a tick without waiting for
* the real interval (like flushHistory()).
*/
checkIdle(): void {
if (!this.connected) return; // disconnected state is owned by the connect/retry loop
const idle = Date.now() - this.lastPacketAt;
if (idle > this.idleTimeoutMs) {
this.logger.warn(
`no decoded bridge frames for ${idle} ms — forcing reconnect`,
);
this.socket?.destroy();
}
}
private async handleFromRadio(
fr: import('@meshtastic/protobufs').Mesh.FromRadio,
): Promise<void> {
+122 -11
View File
@@ -5,7 +5,7 @@ import { DatabaseSync } from 'node:sqlite';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import type { LlmService } from '../llm/llm.service.js';
import { RssStore } from './rss.store.js';
import { RssService } from './rss.service.js';
import { fitOneLine, parseSummaryItems, RssService } from './rss.service.js';
const FEED_A = `<rss version="2.0"><channel>
<item><title>Alpha headline</title></item>
@@ -149,18 +149,79 @@ describe('RssService.handle', () => {
expect(await svc.handle(ctx('#rss sources'))).toBe('Aye, Bee');
});
it('get <name>: summarizes the latest titles via chat with a bounded maxTokens', async () => {
it('get <name>: summarizes via chat with a bounded maxTokens and replies with headered lines', async () => {
stored().add({ name: 'myfeed', url: 'https://a.example/rss', tags: [] });
fetchMock.mockResolvedValueOnce(feedResponse(FEED_A));
llm.chat.mockResolvedValueOnce('Alpha and beta happened.');
llm.chat.mockResolvedValueOnce(
'Chips: EU funds foundries.\nPark: Riverfront expansion approved.',
);
const reply = await svc.handle(ctx('#rss get myfeed'));
expect(reply).toBe('Alpha and beta happened.');
expect(reply).toEqual([
'Chips: EU funds foundries.',
'Park: Riverfront expansion approved.',
]);
const [messages, opts] = llm.chat.mock.calls[0];
expect(opts).toEqual({ maxTokens: 200 });
expect(messages[0].content).toContain('Alpha headline');
expect(messages[0].content).toContain('Beta headline');
// headroom for the server's reasoning tokens (they count against max_tokens) — see rss.service.ts
expect(opts).toEqual({ maxTokens: 2500 });
const prompt = messages[0].content;
expect(prompt).toContain('exactly 3-5 lines');
expect(prompt).toContain('<short header>: <one or two short sentences>');
expect(prompt).toContain('at most 20 characters');
expect(prompt).toContain('No intro, no outro, no numbering');
expect(prompt).toContain('Alpha headline');
expect(prompt).toContain('Beta headline');
});
it('get <name>: caps items at 5', async () => {
stored().add({ name: 'myfeed', url: 'https://a.example/rss', tags: [] });
fetchMock.mockResolvedValueOnce(feedResponse(FEED_A));
const seven = ['a', 'b', 'c', 'd', 'e', 'f', 'g']
.map((c) => `${c}: item ${c}`)
.join('\n');
llm.chat.mockResolvedValueOnce(seven);
const reply = await svc.handle(ctx('#rss get myfeed'));
expect(reply).toEqual([
'a: item a',
'b: item b',
'c: item c',
'd: item d',
'e: item e',
]);
});
it('get <name>: drops empty and separator-less lines, sends what remains when fewer than 3 are valid', async () => {
stored().add({ name: 'myfeed', url: 'https://a.example/rss', tags: [] });
fetchMock.mockResolvedValueOnce(feedResponse(FEED_A));
llm.chat.mockResolvedValueOnce(
'Chips: EU funds foundries.\n\nJust a sentence without a header\n \nPark: expansion approved.',
);
const reply = await svc.handle(ctx('#rss get myfeed'));
expect(reply).toEqual([
'Chips: EU funds foundries.',
'Park: expansion approved.',
]);
});
it('get <name>: hard-cuts an over-length line at a word boundary so each item fits one packet', async () => {
stored().add({ name: 'myfeed', url: 'https://a.example/rss', tags: [] });
fetchMock.mockResolvedValueOnce(feedResponse(FEED_A));
const longLine = `Chips: ${'word '.repeat(40)}`; // 207 chars — over the per-line cap
llm.chat.mockResolvedValueOnce(`${longLine}\nPark: short line.`);
const reply = await svc.handle(ctx('#rss get myfeed'));
expect(Array.isArray(reply)).toBe(true);
const [cut, ok] = reply as string[];
expect(cut.length).toBeLessThanOrEqual(180);
expect(longLine.startsWith(cut)).toBe(true); // prefix — cut at a word boundary, not mid-word
expect(cut.endsWith(' ')).toBe(false);
expect(ok).toBe('Park: short line.'); // other lines untouched
});
it('get #tag: collects items across all sources with the tag; one dead feed does not sink it', async () => {
@@ -171,11 +232,11 @@ describe('RssService.handle', () => {
.mockResolvedValueOnce(feedResponse(FEED_A)) // A answers (Alpha, Beta)
.mockRejectedValueOnce(new Error('down')); // B is dead
llm.chat.mockResolvedValueOnce('Only alpha survived.');
llm.chat.mockResolvedValueOnce('Chips: only alpha survived.');
const reply = await svc.handle(ctx('#rss get #tech'));
// B's fetch rejected, yet the summary still went out from A's titles alone
expect(reply).toBe('Only alpha survived.');
expect(reply).toEqual(['Chips: only alpha survived.']);
expect(llm.chat).toHaveBeenCalledTimes(1);
const content = llm.chat.mock.calls[0][0][0].content;
expect(content).toContain('Alpha headline');
@@ -188,9 +249,15 @@ describe('RssService.handle', () => {
.mockResolvedValueOnce(feedResponse(FEED_A)) // A → Alpha, Beta
.mockResolvedValueOnce(feedResponse(FEED_B)); // B → Gamma
llm.chat.mockResolvedValueOnce('all three.');
await svc.handle(ctx('#rss get #tech'));
llm.chat.mockResolvedValueOnce(
'Alpha: first headline.\nGamma: second headline.',
);
const reply = await svc.handle(ctx('#rss get #tech'));
expect(reply).toEqual([
'Alpha: first headline.',
'Gamma: second headline.',
]);
const content = llm.chat.mock.calls[0][0][0].content;
expect(content).toContain('Alpha headline');
expect(content).toContain('Gamma headline'); // proves B's feed was collected too
@@ -204,3 +271,47 @@ describe('RssService.handle', () => {
);
});
});
describe('parseSummaryItems (per-line items for multi-message summaries)', () => {
it('keeps lines with a header separator, drops empty and separator-less ones', () => {
expect(
parseSummaryItems(
'Chips: EU funds foundries.\n\nNo header here\n \nPark: expansion approved.',
),
).toEqual(['Chips: EU funds foundries.', 'Park: expansion approved.']);
});
it('caps at 5 items', () => {
const seven = ['a', 'b', 'c', 'd', 'e', 'f', 'g']
.map((c) => `${c}: item`)
.join('\n');
expect(parseSummaryItems(seven)).toHaveLength(5);
expect(parseSummaryItems(seven)[4]).toBe('e: item');
});
it('returns [] when nothing is valid', () => {
expect(parseSummaryItems('no headers at all\n')).toEqual([]);
});
});
describe('fitOneLine (one mesh packet per line)', () => {
it('passes through lines that already fit, including exactly 185 chars', () => {
const short = 'Chips: EU funds foundries.';
expect(fitOneLine(short)).toBe(short);
const exact = 'x'.repeat(185);
expect(fitOneLine(exact)).toBe(exact);
});
it('cuts over-long lines at a word boundary near 180', () => {
const long = `Chips: ${'word '.repeat(40)}`; // 207 chars
const cut = fitOneLine(long);
expect(cut.length).toBeLessThanOrEqual(180);
expect(long.startsWith(cut)).toBe(true); // prefix — not mid-word
expect(cut.endsWith(' ')).toBe(false);
});
it('hard-cuts at 180 when there is no word boundary in reach', () => {
const long = 'w'.repeat(250);
expect(fitOneLine(long)).toBe('w'.repeat(180));
});
});
+41 -9
View File
@@ -10,12 +10,24 @@ import { RssStore } from './rss.store.js';
const FETCH_TIMEOUT_MS = 15000; // feed fetches — same budget as the opencode notifier's POSTs
const MAX_TITLES = 8; // "up to 8 latest item titles" per the design
const TAG_MAX_TOKENS = 50;
const SUMMARY_MAX_TOKENS = 200;
// The LLM server (qwen3.8-27b) emits reasoning_content and those tokens COUNT against max_tokens
// (verified: cap 150 → completion_tokens 150 of which 105 were reasoning, finish_reason 'length').
// Thinking length for this task is unbounded and uncontrollable server-side (/no_think,
// chat_template_kwargs.enable_thinking and thinking_budget are all ignored) — observed 32..1776
// tokens across runs. So the cap must be headroom for reasoning + ~80 content tokens:
// 2500 covers every observed run (worst case ≈42 s at ~60 tok/s, under LLM_TIMEOUT_MS=60 s).
const SUMMARY_MAX_TOKENS = 2500;
// Mesh fragments replies longer than ~190 chars into fire-and-forget packets that recipients
// often never get — each summary line must fit ONE packet on its own. The prompt asks for short
// headered lines; this is the hard backstop if the model ignores it (cut at a word boundary).
const MAX_LINE_CHARS = 185; // per-line one-packet threshold
const LINE_CUT_AT = 180; // where over-long lines get cut, snapped back to a word boundary
const MAX_ITEMS = 5; // cap on summary items sent as separate messages
/**
* The `#rss` command family: stored RSS sources (SQLite, shared MESH_DB_PATH file) with
* LLM-generated tags and on-demand headline summaries. All replies are short strings —
* CommandService routes them through MeshService.respond() like any other command reply.
* LLM-generated tags and on-demand headline summaries. Replies are short strings — or an array
* of lines for multi-message summaries, which CommandService sends one respond() per line.
*/
@Injectable()
export class RssService {
@@ -29,8 +41,8 @@ export class RssService {
this.store = new RssStore(db);
}
/** Entry point for the `#rss` prefix — always replies (errors included). */
async handle(ctx: CommandContext): Promise<string> {
/** Entry point for the `#rss` prefix — always replies (errors included); `get` may reply with several lines. */
async handle(ctx: CommandContext): Promise<string | string[]> {
const words = ctx.text.trim().split(/\s+/);
switch (words[1]) {
case undefined: // bare `#rss`
@@ -131,8 +143,8 @@ export class RssService {
return names.length ? names.join(', ') : 'no sources yet';
}
/** `#rss get <name>` or `#rss get #<tag>`. */
private async get(target: string): Promise<string> {
/** `#rss get <name>` or `#rss get #<tag>` — replies with 3-5 separate one-packet lines. */
private async get(target: string): Promise<string | string[]> {
if (!target) return 'usage: #rss get <name|#tag>';
let urls: string[];
@@ -163,15 +175,18 @@ export class RssService {
if (!titles.length) return 'no items found';
try {
return await this.llm.chat(
const reply = await this.llm.chat(
[
{
role: 'user',
content: `Summarize these RSS headlines into a few short one-line highlights. Be concise:\n${titles.join('\n')}`,
content: `Summarize these RSS headlines into exactly 3-5 lines, one item per line, each in the format "<short header>: <one or two short sentences>" where the header is at most 20 characters (e.g. "Culture: Museum opens new wing"). No intro, no outro, no numbering - items only:\n${titles.join('\n')}`,
},
],
{ maxTokens: SUMMARY_MAX_TOKENS },
);
// each item goes out as its own mesh message — CommandService sends the array line by line;
// fewer than 3 valid lines is fine (the model is non-deterministic), we send what we got
return parseSummaryItems(reply).map(fitOneLine);
} catch (err) {
this.logger.warn(`[rss] summary failed: ${(err as Error).message}`);
return 'summary failed';
@@ -190,6 +205,23 @@ export class RssService {
}
}
/** Parse an LLM summary into per-line items: drop empty lines and lines without a `:` header separator, cap at MAX_ITEMS. */
export function parseSummaryItems(reply: string): string[] {
return reply
.split('\n')
.map((l) => l.trim())
.filter((l) => l && l.includes(':'))
.slice(0, MAX_ITEMS);
}
/** Hard-cut a single summary line at a word boundary near LINE_CUT_AT when it exceeds MAX_LINE_CHARS. */
export function fitOneLine(line: string): string {
if (line.length <= MAX_LINE_CHARS) return line;
const slice = line.slice(0, LINE_CUT_AT);
const at = slice.lastIndexOf(' ');
return (at > 0 ? slice.slice(0, at) : slice).trimEnd();
}
/** Normalize LLM tag output: lowercase, no #, whitespace → '-', deduped. */
export function sanitizeTags(raw: string): string[] {
const seen = new Set<string>();