Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
84915fe314 | ||
|
|
5506fb9330 | ||
|
|
beebcd5851 | ||
|
|
5a1493fb4a | ||
|
|
8fb16217f7 | ||
|
|
3b1f5de101 | ||
|
|
46d32b3553 | ||
|
|
e20e11e2d5 | ||
|
|
0b20df242c | ||
|
|
a16b6c19d4 | ||
|
|
9e558fd95e | ||
|
|
70be8bdaa7 | ||
|
|
847c3d47d0 | ||
|
|
1b768b14a7 | ||
|
|
3f6c02d71e | ||
|
|
98b9ec5185 | ||
|
|
0722c82359 | ||
|
|
1ae8d7a926 | ||
|
|
77459adc13 | ||
|
|
33a88f3d83 |
@@ -1,5 +1,53 @@
|
|||||||
# Changelog
|
# Changelog
|
||||||
|
|
||||||
|
## v0.21.11 — 2026-10-05 14:47
|
||||||
|
|
||||||
|
- Migrated the opencode notifier to the v2 API routes: discovery is now `GET {api}{proxyPath}/api/session` (a `{ data, cursor }` envelope; sessions filtered by `location.directory`, most-recent `time.updated` wins) and send is now `POST {api}{proxyPath}/api/session/{sid}/prompt` with a `{text}` body, instead of the v1 `GET /session` + `POST /session/{id}/message` (`{parts:[{type:"text",text}]}`) routes. CodeNomad's per-workspace instance proxy was upgraded and now only serves opencode v2 `/api/*` — every v1 `/session/…` call 403s, which was breaking all notifications. v2 returns 2xx immediately (no longer holds the response until the turn finishes), so the 15 s read-timeout-as-delivered branch is kept only as a harmless safety net.
|
||||||
|
|
||||||
|
## v0.21.10 — 2026-09-30 05:46
|
||||||
|
|
||||||
|
- Container now runs with `TZ=Europe/Warsaw` (new line in the bot service's `environment:` block, hardcoded like MESH_HOST/MESH_PORT): scheduled jobs and log timestamps use local Polish time instead of UTC — the weather cron (`WEATHER_CRON=0 6,12,18 * * *`) previously fired at 06/12/18 UTC = 08/14/20 CEST. DST-aware via the IANA zone (CEST +2 / CET +1). Deployment-config only — no application code changed; takes effect on the next `docker compose up -d --build`.
|
||||||
|
|
||||||
|
## v0.21.9 — 2026-09-29 20:56
|
||||||
|
|
||||||
|
- New `#loser` command (owner request): replies with which user/node has the worst packet delivery record, over directed sends only. Meshtastic shares no mesh-wide loss stats, so it groups our own tx history by destination and counts delivered vs failed per node; broadcasts are excluded (firmware strips want_ack pre-TX, they stay pending — and the broadcast addr is never a real destination even if a NACK lands on it). Loser = most failures, tie-break worse failure ratio. Node resolved to `longName`/`shortName`, falling back to the `@!xxxxxxxx` mention when unknown. Three reply shapes: worst delivery line (`worst delivery: <name> (@!hex) - N failed / M delivered`), a "no failed sends yet" line, and a "not enough data yet — only directed sends count (broadcasts have no acks)" line. Registered in the CommandService constructor next to `#ping`; listed in `#help`.
|
||||||
|
- Tests: 175 → 180 — new `#loser` specs cover all three reply cases, broadcast/pending exclusion, the unknown-node `@!hex` fallback, and the failure-ratio tie-break; the `#help` spec now also asserts `#loser` is listed.
|
||||||
|
|
||||||
|
## v0.21.8 — 2026-09-27 07:13
|
||||||
|
|
||||||
|
- Fixed `npm test` exiting 1 with "Errors 2" even though all tests pass (every CI run red since v0.21.2): the two watchdog specs that force a reconnect destroy the service's socket while data is still pending in its receive buffer (the overflow spec floods 300 KB and destroys on the first ~64 KB chunk; the garbage spec destroys right after its last write), so the kernel sends RST instead of FIN and the fake bridge's accepted server-side sockets emit `read ECONNRESET`. Those test-owned sockets had no `'error'` handler, so Node threw it as an uncaught exception that vitest counts as a run-level failure. `startBridge()` now attaches a no-op `'error'` handler to every accepted socket (single registration point, covers all four spec files using the fake bridge). Test-only change — `fake-bridge.ts` is imported by specs only; no runtime diff, no deploy needed.
|
||||||
|
|
||||||
|
## v0.21.7 — 2026-09-27 06:50
|
||||||
|
|
||||||
|
- Fixed double registration of scheduled jobs: `ScheduleModule.forRoot()` was called from both `WeatherModule` and `BatteryAlertModule`, so Nest created two ScheduleExplorers, each scanning all providers — every @Cron/@Interval job (hourly weather broadcast, per-minute battery check) fired twice. Removed the call from `battery.module.ts`; the single registration in `weather.module.ts` remains.
|
||||||
|
- Tests: 174 → 175 — new regression spec (`src/schedule-singleton.spec.ts`) asserts `ScheduleModule.forRoot(` appears exactly once across all `src/**/*.module.ts`.
|
||||||
|
|
||||||
|
## v0.21.6 — 2026-09-26 10:37
|
||||||
|
|
||||||
|
- New `#help` command (owner request): replies with a short list of all available commands as ONE multi-line message (single packet, not one spaced packet per line) — `#ping`, the `#rss help/add/remove/get/sources/tags` family. Registered in the CommandService constructor next to `#ping`.
|
||||||
|
- Tests: 173 → 174 — `#help` dispatches through respond() exactly once with text containing both `#ping` and `#rss`. (Also hardened the v0.21.5 multi-send-spacing spec, which flaked under full-suite load: it now uses fake timers to assert deterministically that line 1 goes out before any time elapses and lines 2-3 land exactly one delay apart.)
|
||||||
|
|
||||||
|
## v0.21.5 — 2026-09-26 10:26
|
||||||
|
|
||||||
|
- Multi-line command replies are now spaced apart: `CommandService.onRx` waits `MESH_MULTI_SEND_DELAY_MS` (new env, default **5000** = 5 s, read at construction) between consecutive lines of an array reply — no delay before the first line. Production incident: `#rss get lublin` sent its items back-to-back within the same second and 3 of 4 were dropped in RF (the owner saw only the last one); each packet now gets time to land.
|
||||||
|
- Tests: 172 → 173 — a 3-line reply with an overridden short delay asserts all lines are sent, the first goes out immediately (no pre-delay), and each subsequent line lands ≥ the configured gap after the previous one; existing specs pin `MESH_MULTI_SEND_DELAY_MS=0` in `beforeEach` so they stay fast.
|
||||||
|
|
||||||
|
## v0.21.4 — 2026-09-26 09:51
|
||||||
|
|
||||||
|
- `LLM_TIMEOUT_MS` default raised from 60000 (60 s) to **180000** (3 min): production incident — `#rss get lublin` at 9:41 AM failed with "The operation was aborted due to timeout" because the local OpenAI-compatible server (qwen3.8-27b, uncontrollable reasoning tokens) was slow/busy on that call; the same endpoint answers a trivial prompt in ~1 s when idle, so this is latency variance, not a dead endpoint. Still env-overridable (`LLM_TIMEOUT_MS`), only the fallback changed.
|
||||||
|
|
||||||
|
## v0.21.3 — 2026-09-26 09:31
|
||||||
|
|
||||||
|
- Idle timeout default lowered from 10 min to **1 min** (`MESH_IDLE_TIMEOUT_MS` fallback `600000` → `60000`) — the v0.21.2 frame-based watchdog works in production (it fired today and the reconnect delivered queued packets), but the bridge/radio side at 192.168.1.4 keeps killing inbound RX within ~20 min of a fresh sync, so a shorter detection window is wanted.
|
||||||
|
- New **lazy periodic autoreconnect**: `MESH_RECONNECT_INTERVAL_MS` (new env, default **300000** = 5 min, read at construction) — an interval that unconditionally logs ("periodic reconnect — MESH_RECONNECT_INTERVAL_MS elapsed") and destroys the socket so the existing close→scheduleReconnect loop re-establishes with a fresh sync on schedule. When not connected at tick time, destroy is a no-op and the retry loop owns recovery (no special-casing). Started in `onModuleInit`, cleared in `onModuleDestroy`.
|
||||||
|
- Tests: 171 → 172 — periodic reconnect observed end-to-end with an overridden short interval (fake bridge disconnect + re-establish); existing watchdog tests pass unchanged, comments updated to the new 60 s default.
|
||||||
|
|
||||||
|
## v0.21.2 — 2026-09-26 08:47
|
||||||
|
|
||||||
|
- Idle watchdog now tracks DECODED FRAMES, not raw bytes (fixes a second production zombie that slipped past v0.21.1): text+telemetry RX was dead for >1 h while TX worked and the watchdog never fired — /proc/net/snmp showed InSegs delta = 2 segments per 30 s, i.e. the bridge peer keeps dribbling small NON-frame bytes (~every 15 s) that kept resetting `lastPacketAt` on every socket chunk even though zero valid Meshtastic frames decoded. The `'data'` handler no longer updates `lastPacketAt` per chunk; it is updated only when `extractFrames` yields ≥1 valid packet (the reset in the `'connect'` handler is unchanged), so a stream that decodes nothing trips the watchdog after `MESH_IDLE_TIMEOUT_MS` even while raw bytes keep arriving.
|
||||||
|
- Buffer-overflow backstop: if the undecoded frame buffer exceeds `MESH_MAX_FRAME_BUFFER` (new env, default **256000** = 250 KB) the service logs ERROR ("frame buffer overflow — stream desync, forcing reconnect") and destroys the socket → existing close→scheduleReconnect self-heals a desynced stream. The undecoded buffer is also reset on every (re)connect so leftover garbage from a dead session can't re-trigger the backstop or corrupt the first frames of a healthy new stream.
|
||||||
|
- Tests: 169 → 171 — raw non-frame bytes do NOT keep the watchdog alive (garbage-only stream forces disconnect + reconnect, exactly today's incident shape), and an undecoded chunk over `MESH_MAX_FRAME_BUFFER` triggers destroy + a fresh session that decodes normally. The existing valid-frames-reset-the-clock test now exercises frame-based tracking unchanged.
|
||||||
|
|
||||||
## v0.21.1 — 2026-09-26 06:47
|
## v0.21.1 — 2026-09-26 06:47
|
||||||
|
|
||||||
- Bridge zombie-connection fix (production incident: outbound sends kept working ~10 h while inbound packets stopped entirely and `/api/status` stayed `connected:true`). New **idle watchdog**: the service tracks `lastPacketAt`, updated on EVERY chunk received from the bridge in the lowest-level socket `'data'` path (any frame kind, decodable or not) and reset when a connection successfully starts. A 30 s interval checks: if connected AND no inbound traffic for `MESH_IDLE_TIMEOUT_MS` (new env, default **600000** = 10 min), it logs WARN ("no bridge traffic for X ms — forcing reconnect") and destroys the socket — the EXISTING connect/retry loop (`MESH_RECONNECT_MS`) then re-establishes with a fresh sync. No changes to connect-error retry/backoff behavior; this only adds detection for established-but-silent connections.
|
- Bridge zombie-connection fix (production incident: outbound sends kept working ~10 h while inbound packets stopped entirely and `/api/status` stayed `connected:true`). New **idle watchdog**: the service tracks `lastPacketAt`, updated on EVERY chunk received from the bridge in the lowest-level socket `'data'` path (any frame kind, decodable or not) and reset when a connection successfully starts. A 30 s interval checks: if connected AND no inbound traffic for `MESH_IDLE_TIMEOUT_MS` (new env, default **600000** = 10 min), it logs WARN ("no bridge traffic for X ms — forcing reconnect") and destroys the socket — the EXISTING connect/retry loop (`MESH_RECONNECT_MS`) then re-establishes with a fresh sync. No changes to connect-error retry/backoff behavior; this only adds detection for established-but-silent connections.
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ services:
|
|||||||
bot:
|
bot:
|
||||||
build: .
|
build: .
|
||||||
environment:
|
environment:
|
||||||
|
TZ: Europe/Warsaw
|
||||||
MESH_HOST: 192.168.1.4
|
MESH_HOST: 192.168.1.4
|
||||||
MESH_PORT: '4403'
|
MESH_PORT: '4403'
|
||||||
MESH_DB_PATH: ./data/mesh-bot.db
|
MESH_DB_PATH: ./data/mesh-bot.db
|
||||||
|
|||||||
+1
-1
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "mesh-bot",
|
"name": "mesh-bot",
|
||||||
"version": "0.21.1",
|
"version": "0.21.11",
|
||||||
"description": "",
|
"description": "",
|
||||||
"author": "",
|
"author": "",
|
||||||
"private": true,
|
"private": true,
|
||||||
|
|||||||
@@ -1,10 +1,9 @@
|
|||||||
import { Module } from '@nestjs/common';
|
import { Module } from '@nestjs/common';
|
||||||
import { ScheduleModule } from '@nestjs/schedule';
|
|
||||||
import { MeshModule } from '../mesh/mesh.module.js';
|
import { MeshModule } from '../mesh/mesh.module.js';
|
||||||
import { BatteryAlertService } from './battery.service.js';
|
import { BatteryAlertService } from './battery.service.js';
|
||||||
|
|
||||||
@Module({
|
@Module({
|
||||||
imports: [ScheduleModule.forRoot(), MeshModule],
|
imports: [MeshModule],
|
||||||
providers: [BatteryAlertService],
|
providers: [BatteryAlertService],
|
||||||
exports: [BatteryAlertService],
|
exports: [BatteryAlertService],
|
||||||
})
|
})
|
||||||
|
|||||||
@@ -13,7 +13,11 @@ import {
|
|||||||
KAMIL_NUM,
|
KAMIL_NUM,
|
||||||
} from '../mesh/fake-bridge.js';
|
} from '../mesh/fake-bridge.js';
|
||||||
import { BROADCAST_ADDR } from '../mesh/codec.js';
|
import { BROADCAST_ADDR } from '../mesh/codec.js';
|
||||||
import type { MeshService } from '../mesh/mesh.service.js';
|
import type {
|
||||||
|
MessageRecord,
|
||||||
|
MeshService,
|
||||||
|
NodeSummary,
|
||||||
|
} from '../mesh/mesh.service.js';
|
||||||
import { MeshService as RealMeshService } from '../mesh/mesh.service.js';
|
import { MeshService as RealMeshService } from '../mesh/mesh.service.js';
|
||||||
import { CommandService } from './command.service.js';
|
import { CommandService } from './command.service.js';
|
||||||
|
|
||||||
@@ -30,6 +34,7 @@ function makeService(
|
|||||||
|
|
||||||
beforeEach(() => {
|
beforeEach(() => {
|
||||||
delete process.env.COMMAND_CHANNEL; // default channel 2 under test
|
delete process.env.COMMAND_CHANNEL; // default channel 2 under test
|
||||||
|
process.env.MESH_MULTI_SEND_DELAY_MS = '0'; // no spacing under test unless a test opts in — the real default (5000) would slow the suite down
|
||||||
});
|
});
|
||||||
|
|
||||||
describe('CommandService.onRx', () => {
|
describe('CommandService.onRx', () => {
|
||||||
@@ -40,6 +45,16 @@ describe('CommandService.onRx', () => {
|
|||||||
expect(mesh.sendText).not.toHaveBeenCalled();
|
expect(mesh.sendText).not.toHaveBeenCalled();
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it('replies #help with a short single-message list of all commands', async () => {
|
||||||
|
const [svc, mesh] = makeService();
|
||||||
|
await svc.onRx(2, '#help', 42, 7);
|
||||||
|
expect(mesh.respond).toHaveBeenCalledTimes(1); // one packet — not one per line
|
||||||
|
const text = mesh.respond.mock.calls[0][1];
|
||||||
|
expect(text).toContain('#ping');
|
||||||
|
expect(text).toContain('#loser');
|
||||||
|
expect(text).toContain('#rss');
|
||||||
|
});
|
||||||
|
|
||||||
it('falls back to a directed sendText on COMMAND_CHANNEL when the trigger has no packet id (DM)', async () => {
|
it('falls back to a directed sendText on COMMAND_CHANNEL when the trigger has no packet id (DM)', async () => {
|
||||||
const [svc, mesh] = makeService();
|
const [svc, mesh] = makeService();
|
||||||
await svc.onRx(2, '#ping', 42); // from present but no id — defensive path
|
await svc.onRx(2, '#ping', 42); // from present but no id — defensive path
|
||||||
@@ -124,6 +139,37 @@ describe('CommandService.onRx', () => {
|
|||||||
expect(mesh.sendText).not.toHaveBeenCalled();
|
expect(mesh.sendText).not.toHaveBeenCalled();
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it('spaces consecutive multi-line sends by MESH_MULTI_SEND_DELAY_MS — no delay before the first line', async () => {
|
||||||
|
process.env.MESH_MULTI_SEND_DELAY_MS = '50'; // short so the test stays fast
|
||||||
|
let sends = 0;
|
||||||
|
const respond = vi.fn(async () => {
|
||||||
|
sends++;
|
||||||
|
return { id: 1 };
|
||||||
|
});
|
||||||
|
vi.useFakeTimers();
|
||||||
|
try {
|
||||||
|
const [svc, mesh] = makeService(undefined, respond);
|
||||||
|
svc.register('multi3', () => ['a: one', 'b: two', 'c: three']);
|
||||||
|
|
||||||
|
const p = svc.onRx(2, '#multi3', 42, 9);
|
||||||
|
// flush microtasks without advancing time — the first line must already be out (no delay before it)
|
||||||
|
await vi.advanceTimersByTimeAsync(0);
|
||||||
|
expect(sends).toBe(1);
|
||||||
|
|
||||||
|
// each subsequent line waits exactly MESH_MULTI_SEND_DELAY_MS after the previous one landed
|
||||||
|
await vi.advanceTimersByTimeAsync(50);
|
||||||
|
expect(sends).toBe(2);
|
||||||
|
await vi.advanceTimersByTimeAsync(50);
|
||||||
|
expect(sends).toBe(3);
|
||||||
|
|
||||||
|
await p;
|
||||||
|
expect(mesh.respond).toHaveBeenCalledTimes(3);
|
||||||
|
} finally {
|
||||||
|
vi.useRealTimers();
|
||||||
|
delete process.env.MESH_MULTI_SEND_DELAY_MS;
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
it('sends each line of an array reply via the legacy sendText fallback when there is no packet id', async () => {
|
it('sends each line of an array reply via the legacy sendText fallback when there is no packet id', async () => {
|
||||||
const [svc, mesh] = makeService();
|
const [svc, mesh] = makeService();
|
||||||
svc.register('multi', () => ['one: first item', 'two: second item']);
|
svc.register('multi', () => ['one: first item', 'two: second item']);
|
||||||
@@ -164,6 +210,125 @@ describe('CommandService.onRx', () => {
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
describe('#loser', () => {
|
||||||
|
const tx = (
|
||||||
|
id: number,
|
||||||
|
to: number,
|
||||||
|
status: MessageRecord['status'],
|
||||||
|
): MessageRecord => ({
|
||||||
|
id,
|
||||||
|
dir: 'tx',
|
||||||
|
to,
|
||||||
|
text: 'hi',
|
||||||
|
channel: 3,
|
||||||
|
at: Date.now(),
|
||||||
|
status,
|
||||||
|
});
|
||||||
|
|
||||||
|
const makeLoserMesh = (
|
||||||
|
messages: MessageRecord[],
|
||||||
|
nodes: NodeSummary[],
|
||||||
|
): [CommandService, MeshService] => {
|
||||||
|
const mesh = {
|
||||||
|
sendText: vi.fn(),
|
||||||
|
respond: vi.fn().mockResolvedValue({ id: 1 }),
|
||||||
|
messages: () => messages,
|
||||||
|
nodes: async () => nodes,
|
||||||
|
} as unknown as MeshService;
|
||||||
|
return [new CommandService(mesh), mesh];
|
||||||
|
};
|
||||||
|
|
||||||
|
it('reports the node with the most failures — named via longName, broadcasts and pending sends excluded', async () => {
|
||||||
|
const messages = [
|
||||||
|
tx(10, BROADCAST_ADDR, 'failed'), // firmware quirk: a broadcast NACK is never a real destination
|
||||||
|
tx(9, 222, 'failed'), // Alice: 3 failed / 0 delivered → loser
|
||||||
|
tx(8, 111, 'delivered'), // Bob: 1 failed / 2 delivered
|
||||||
|
tx(7, 222, 'failed'),
|
||||||
|
tx(6, 111, 'failed'),
|
||||||
|
tx(5, 333, 'pending'), // no final status yet — ignored
|
||||||
|
tx(4, 222, 'failed'),
|
||||||
|
{
|
||||||
|
id: 3,
|
||||||
|
dir: 'rx',
|
||||||
|
from: 999,
|
||||||
|
text: 'hello',
|
||||||
|
channel: 0,
|
||||||
|
at: Date.now(),
|
||||||
|
},
|
||||||
|
];
|
||||||
|
const nodes = [
|
||||||
|
{ num: 111, shortName: 'Bob', lastHeard: null },
|
||||||
|
{ num: 222, longName: 'Alice', lastHeard: null },
|
||||||
|
];
|
||||||
|
const [svc, mesh] = makeLoserMesh(messages, nodes);
|
||||||
|
|
||||||
|
await svc.onRx(2, '#loser', 42, 7);
|
||||||
|
|
||||||
|
expect(mesh.respond).toHaveBeenCalledWith(
|
||||||
|
7,
|
||||||
|
'worst delivery: Alice (@!000000de) - 3 failed / 0 delivered',
|
||||||
|
);
|
||||||
|
expect(mesh.sendText).not.toHaveBeenCalled();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('says no failed sends when the directed history is all delivered', async () => {
|
||||||
|
const messages = [tx(2, 5, 'delivered'), tx(1, 6, 'delivered')];
|
||||||
|
const [svc, mesh] = makeLoserMesh(messages, []);
|
||||||
|
|
||||||
|
await svc.onRx(2, '#loser', 42, 7);
|
||||||
|
|
||||||
|
expect(mesh.respond).toHaveBeenCalledWith(
|
||||||
|
7,
|
||||||
|
'no failed sends yet - everyone delivers :)',
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('explains the directed-only rule when there is no final-status data at all', async () => {
|
||||||
|
const messages = [tx(2, BROADCAST_ADDR, 'pending'), tx(1, 5, 'pending')];
|
||||||
|
const [svc, mesh] = makeLoserMesh(messages, []);
|
||||||
|
|
||||||
|
await svc.onRx(2, '#loser', 42, 7);
|
||||||
|
|
||||||
|
expect(mesh.respond).toHaveBeenCalledWith(
|
||||||
|
7,
|
||||||
|
'not enough data yet - only directed sends count (broadcasts have no acks)',
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('falls back to the @!hex mention when the worst node has no known name', async () => {
|
||||||
|
const messages = [tx(1, 600, 'failed')];
|
||||||
|
const nodes = [{ num: 5, longName: 'Someone Else', lastHeard: null }];
|
||||||
|
const [svc, mesh] = makeLoserMesh(messages, nodes);
|
||||||
|
|
||||||
|
await svc.onRx(2, '#loser', 42, 7);
|
||||||
|
|
||||||
|
expect(mesh.respond).toHaveBeenCalledWith(
|
||||||
|
7,
|
||||||
|
'worst delivery: @!00000258 - 1 failed / 0 delivered',
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('tie-breaks equal failure counts by the worse failure ratio', async () => {
|
||||||
|
const messages = [
|
||||||
|
// node 7: 1 failed / 9 delivered (ratio 0.1) — not the loser
|
||||||
|
tx(16, 8, 'failed'), // node 8: 1 failed / 1 delivered (ratio 0.5) → loser
|
||||||
|
tx(15, 8, 'delivered'),
|
||||||
|
tx(14, 7, 'failed'),
|
||||||
|
...Array.from({ length: 9 }, (_, i) =>
|
||||||
|
tx(i + 1, 7, 'delivered' as const),
|
||||||
|
),
|
||||||
|
];
|
||||||
|
const [svc, mesh] = makeLoserMesh(messages, []);
|
||||||
|
|
||||||
|
await svc.onRx(2, '#loser', 42, 7);
|
||||||
|
|
||||||
|
expect(mesh.respond).toHaveBeenCalledWith(
|
||||||
|
7,
|
||||||
|
'worst delivery: @!00000008 - 1 failed / 1 delivered',
|
||||||
|
);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
describe('#rss prefix dispatch', () => {
|
describe('#rss prefix dispatch', () => {
|
||||||
let dir: string | undefined;
|
let dir: string | undefined;
|
||||||
|
|
||||||
|
|||||||
@@ -1,4 +1,6 @@
|
|||||||
import { Injectable, Logger } from '@nestjs/common';
|
import { Injectable, Logger } from '@nestjs/common';
|
||||||
|
// pure formatting helper (no cycle: battery → mesh only)
|
||||||
|
import { mention } from '../battery/battery.service.js';
|
||||||
import { BROADCAST_ADDR } from '../mesh/codec.js';
|
import { BROADCAST_ADDR } from '../mesh/codec.js';
|
||||||
// type-only: the instances come from DI (command.module.ts factory) or direct construction in tests — no runtime import cycles with src/mesh / src/rss
|
// type-only: the instances come from DI (command.module.ts factory) or direct construction in tests — no runtime import cycles with src/mesh / src/rss
|
||||||
import type { MeshService } from '../mesh/mesh.service.js';
|
import type { MeshService } from '../mesh/mesh.service.js';
|
||||||
@@ -27,6 +29,11 @@ export class CommandService {
|
|||||||
process.env.COMMAND_CHANNEL ?? '2',
|
process.env.COMMAND_CHANNEL ?? '2',
|
||||||
10,
|
10,
|
||||||
);
|
);
|
||||||
|
// spacing between consecutive multi-line reply messages — back-to-back packets get dropped in RF on a real mesh; each packet needs time to land. Read at construction like the other env config.
|
||||||
|
private readonly multiSendDelayMs = parseInt(
|
||||||
|
process.env.MESH_MULTI_SEND_DELAY_MS ?? '5000',
|
||||||
|
10,
|
||||||
|
);
|
||||||
// single registration point: the constructor registers every built-in command — add the next one there (or via register())
|
// single registration point: the constructor registers every built-in command — add the next one there (or via register())
|
||||||
private readonly commands = new Map<string, CommandHandler>();
|
private readonly commands = new Map<string, CommandHandler>();
|
||||||
// prefix handlers own a whole family (`#rss`, `#rss add ...`) instead of one exact name
|
// prefix handlers own a whole family (`#rss`, `#rss add ...`) instead of one exact name
|
||||||
@@ -38,9 +45,68 @@ export class CommandService {
|
|||||||
rss?: RssService,
|
rss?: RssService,
|
||||||
) {
|
) {
|
||||||
this.register('ping', () => '#pong!');
|
this.register('ping', () => '#pong!');
|
||||||
|
this.register('loser', () => this.loserReply());
|
||||||
|
// single multi-line string = ONE packet — not an array, which would drip-feed one spaced packet per line
|
||||||
|
this.register('help', () =>
|
||||||
|
[
|
||||||
|
'#ping - alive check',
|
||||||
|
'#loser - worst delivery record',
|
||||||
|
'#rss help - rss subcommands',
|
||||||
|
'#rss add <url> <name> - add feed with AI tags',
|
||||||
|
'#rss remove <name> - drop feed',
|
||||||
|
'#rss get <name|#tag> - news summary',
|
||||||
|
'#rss sources / #rss tags - list feeds/tags',
|
||||||
|
].join('\n'),
|
||||||
|
);
|
||||||
if (rss) this.registerPrefix('rss', (ctx) => rss.handle(ctx));
|
if (rss) this.registerPrefix('rss', (ctx) => rss.handle(ctx));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* #loser — worst packet delivery record, over directed sends only. Meshtastic shares no mesh-wide
|
||||||
|
* loss stats, so the only real data is our own tx history: group by destination, count delivered vs
|
||||||
|
* failed per node; broadcasts are excluded (firmware strips want_ack pre-TX, they stay pending — and
|
||||||
|
* the broadcast addr is never a real destination even if a NACK ever lands on it). Loser = most
|
||||||
|
* failures, tie-break worse failure ratio. One reply line, ≤ ~180 chars.
|
||||||
|
*/
|
||||||
|
private async loserReply(): Promise<string> {
|
||||||
|
const mesh = this.mesh;
|
||||||
|
if (!mesh)
|
||||||
|
return 'not enough data yet - only directed sends count (broadcasts have no acks)';
|
||||||
|
const stats = new Map<number, { failed: number; delivered: number }>();
|
||||||
|
for (const m of mesh.messages()) {
|
||||||
|
if (m.dir !== 'tx' || m.to === undefined || m.to === BROADCAST_ADDR)
|
||||||
|
continue; // directed sends only
|
||||||
|
if (m.status !== 'delivered' && m.status !== 'failed') continue; // pending = no final status yet
|
||||||
|
const s = stats.get(m.to) ?? { failed: 0, delivered: 0 };
|
||||||
|
if (m.status === 'failed') s.failed += 1;
|
||||||
|
else s.delivered += 1;
|
||||||
|
stats.set(m.to, s);
|
||||||
|
}
|
||||||
|
if (stats.size === 0)
|
||||||
|
return 'not enough data yet - only directed sends count (broadcasts have no acks)';
|
||||||
|
let worst: { num: number; failed: number; total: number } | undefined;
|
||||||
|
for (const [num, s] of stats) {
|
||||||
|
if (s.failed === 0) continue; // zero-failure nodes can't be the loser
|
||||||
|
const total = s.failed + s.delivered;
|
||||||
|
// most failures first, then worse failure ratio on a tie
|
||||||
|
if (
|
||||||
|
!worst ||
|
||||||
|
s.failed > worst.failed ||
|
||||||
|
(s.failed === worst.failed &&
|
||||||
|
s.failed / total > worst.failed / worst.total)
|
||||||
|
) {
|
||||||
|
worst = { num, failed: s.failed, total };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (!worst) return 'no failed sends yet - everyone delivers :)';
|
||||||
|
const names = new Map(
|
||||||
|
(await mesh.nodes()).map((n) => [n.num, n.longName ?? n.shortName]),
|
||||||
|
);
|
||||||
|
const name = names.get(worst.num);
|
||||||
|
const who = name ? `${name} (${mention(worst.num)})` : mention(worst.num);
|
||||||
|
return `worst delivery: ${who} - ${worst.failed} failed / ${worst.total - worst.failed} delivered`;
|
||||||
|
}
|
||||||
|
|
||||||
/** Register a command handler (names are case-sensitive; the leading `#` is stripped before lookup). */
|
/** Register a command handler (names are case-sensitive; the leading `#` is stripped before lookup). */
|
||||||
register(name: string, fn: CommandHandler): void {
|
register(name: string, fn: CommandHandler): void {
|
||||||
this.commands.set(name, fn);
|
this.commands.set(name, fn);
|
||||||
@@ -83,8 +149,15 @@ export class CommandService {
|
|||||||
try {
|
try {
|
||||||
const reply = await handler({ text: trimmed, from, id });
|
const reply = await handler({ text: trimmed, from, id });
|
||||||
if (reply === undefined || reply === null) return; // void = no reply
|
if (reply === undefined || reply === null) return; // void = no reply
|
||||||
// an array reply sends one message per line, all threaded to the same trigger packet id
|
// an array reply sends one message per line, all threaded to the same trigger packet id — spaced so each packet has time to land in RF (no delay before the first line)
|
||||||
for (const line of Array.isArray(reply) ? reply : [reply]) {
|
const lines = Array.isArray(reply) ? reply : [reply];
|
||||||
|
for (let i = 0; i < lines.length; i++) {
|
||||||
|
if (i > 0 && this.multiSendDelayMs > 0) {
|
||||||
|
await new Promise((resolve) =>
|
||||||
|
setTimeout(resolve, this.multiSendDelayMs),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
const line = lines[i];
|
||||||
if (id !== undefined && this.mesh) {
|
if (id !== undefined && this.mesh) {
|
||||||
// respond() auto-routes each line: rides the original record's channel with replyId = id (threaded quote)
|
// respond() auto-routes each line: rides the original record's channel with replyId = id (threaded quote)
|
||||||
await this.mesh.respond(id, line);
|
await this.mesh.respond(id, line);
|
||||||
|
|||||||
@@ -12,9 +12,9 @@ export class LlmService {
|
|||||||
private readonly apiBase =
|
private readonly apiBase =
|
||||||
process.env.LLM_API_BASE ?? 'http://192.168.1.70:1111/v1';
|
process.env.LLM_API_BASE ?? 'http://192.168.1.70:1111/v1';
|
||||||
private readonly model = process.env.LLM_MODEL ?? 'qwen3.8-27b';
|
private readonly model = process.env.LLM_MODEL ?? 'qwen3.8-27b';
|
||||||
// local 27B model can be slow — generous default
|
// local 27B model can be slow (uncontrollable reasoning tokens) — generous default: 3 min headroom for slow reasoning periods
|
||||||
private readonly timeoutMs = parseInt(
|
private readonly timeoutMs = parseInt(
|
||||||
process.env.LLM_TIMEOUT_MS ?? '60000',
|
process.env.LLM_TIMEOUT_MS ?? '180000',
|
||||||
10,
|
10,
|
||||||
);
|
);
|
||||||
|
|
||||||
|
|||||||
@@ -10,6 +10,8 @@ export function startBridge(
|
|||||||
): Promise<{ port: number; sockets: net.Socket[]; close: () => void }> {
|
): Promise<{ port: number; sockets: net.Socket[]; close: () => void }> {
|
||||||
const sockets: net.Socket[] = [];
|
const sockets: net.Socket[] = [];
|
||||||
const server = net.createServer((socket) => {
|
const server = net.createServer((socket) => {
|
||||||
|
// Test-owned socket: the service destroys its end mid-test with unread data pending (overflow/garbage tests), which makes the kernel send RST and this side emit ECONNRESET. Expected teardown noise — without a handler Node throws it as an uncaught exception and vitest fails the whole run.
|
||||||
|
socket.on('error', () => {});
|
||||||
onSocket?.(socket); // attached at accept time, before any data can arrive
|
onSocket?.(socket); // attached at accept time, before any data can arrive
|
||||||
sockets.push(socket);
|
sockets.push(socket);
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -258,7 +258,7 @@ describe('MeshService idle watchdog', () => {
|
|||||||
process.env.MESH_HOST = '127.0.0.1';
|
process.env.MESH_HOST = '127.0.0.1';
|
||||||
process.env.MESH_PORT = String(bridge.port);
|
process.env.MESH_PORT = String(bridge.port);
|
||||||
process.env.MESH_RECONNECT_MS = '50'; // fast retry so the re-establishment is observable
|
process.env.MESH_RECONNECT_MS = '50'; // fast retry so the re-establishment is observable
|
||||||
process.env.MESH_IDLE_TIMEOUT_MS = '100'; // override — the default (600000) would never fire in a test
|
process.env.MESH_IDLE_TIMEOUT_MS = '100'; // override — the default (60000) would never fire in a test
|
||||||
const svc = new MeshService();
|
const svc = new MeshService();
|
||||||
try {
|
try {
|
||||||
svc.onModuleInit();
|
svc.onModuleInit();
|
||||||
@@ -280,6 +280,69 @@ describe('MeshService idle watchdog', () => {
|
|||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it('raw non-frame bytes do NOT keep the watchdog alive — a garbage-only stream forces a reconnect', async () => {
|
||||||
|
const bridge = await startBridge();
|
||||||
|
process.env.MESH_HOST = '127.0.0.1';
|
||||||
|
process.env.MESH_PORT = String(bridge.port);
|
||||||
|
process.env.MESH_RECONNECT_MS = '50'; // fast retry so the re-establishment is observable
|
||||||
|
process.env.MESH_IDLE_TIMEOUT_MS = '100'; // override — the default (60000) would never fire in a test
|
||||||
|
const svc = new MeshService();
|
||||||
|
try {
|
||||||
|
svc.onModuleInit();
|
||||||
|
await waitFor(
|
||||||
|
() => bridge.sockets.length === 1 && svc.status().connected,
|
||||||
|
);
|
||||||
|
|
||||||
|
// today's incident: the peer keeps dribbling small non-frame bytes (keepalive noise / desynced stream) — raw chunks must not reset the idle clock
|
||||||
|
const garbage = Buffer.from('zz-no-frames-here'); // pure ASCII — never forms a 0x94 0xc3 frame header
|
||||||
|
for (let i = 0; i < 6; i++) {
|
||||||
|
await sleep(50); // ~300 ms wall time, well past the timeout
|
||||||
|
bridge.sockets[0].write(garbage);
|
||||||
|
}
|
||||||
|
|
||||||
|
svc.checkIdle();
|
||||||
|
await waitFor(() => bridge.sockets.length >= 2 && svc.status().connected);
|
||||||
|
} finally {
|
||||||
|
delete process.env.MESH_RECONNECT_MS;
|
||||||
|
delete process.env.MESH_IDLE_TIMEOUT_MS;
|
||||||
|
svc.onModuleDestroy();
|
||||||
|
bridge.close();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
it('forces a reconnect when the undecoded frame buffer exceeds MESH_MAX_FRAME_BUFFER', async () => {
|
||||||
|
const bridge = await startBridge();
|
||||||
|
process.env.MESH_HOST = '127.0.0.1';
|
||||||
|
process.env.MESH_PORT = String(bridge.port);
|
||||||
|
process.env.MESH_RECONNECT_MS = '50'; // fast retry so the re-establishment is observable
|
||||||
|
process.env.MESH_MAX_FRAME_BUFFER = '100'; // tiny cap — Node delivers socket data in ~64 KB events, far below the 256_000 default
|
||||||
|
const svc = new MeshService();
|
||||||
|
try {
|
||||||
|
svc.onModuleInit();
|
||||||
|
await waitFor(
|
||||||
|
() => bridge.sockets.length === 1 && svc.status().connected,
|
||||||
|
);
|
||||||
|
|
||||||
|
// non-frame bytes (0x42 is not part of any frame header) — a desynced stream that never forms frames; the first ~64 KB data event alone exceeds the cap
|
||||||
|
bridge.sockets[0].write(Buffer.alloc(300_000, 0x42));
|
||||||
|
|
||||||
|
// overflow forced a reconnect — fresh session
|
||||||
|
await waitFor(() => bridge.sockets.length >= 2 && svc.status().connected);
|
||||||
|
|
||||||
|
// the new socket starts with a clean buffer and decodes valid frames normally (the myInfo frame is well under the cap)
|
||||||
|
const fr = create(P.Mesh.FromRadioSchema, {
|
||||||
|
payloadVariant: { case: 'myInfo', value: { myNodeNum: 777 } },
|
||||||
|
});
|
||||||
|
bridge.sockets[1].write(frame(toBinary(P.Mesh.FromRadioSchema, fr)));
|
||||||
|
await waitFor(() => svc.status().myNodeNum === 777);
|
||||||
|
} finally {
|
||||||
|
delete process.env.MESH_RECONNECT_MS;
|
||||||
|
delete process.env.MESH_MAX_FRAME_BUFFER;
|
||||||
|
svc.onModuleDestroy();
|
||||||
|
bridge.close();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
it('inbound traffic keeps resetting the idle clock — no false trigger while packets arrive', async () => {
|
it('inbound traffic keeps resetting the idle clock — no false trigger while packets arrive', async () => {
|
||||||
const bridge = await startBridge();
|
const bridge = await startBridge();
|
||||||
process.env.MESH_HOST = '127.0.0.1';
|
process.env.MESH_HOST = '127.0.0.1';
|
||||||
@@ -313,7 +376,7 @@ describe('MeshService idle watchdog', () => {
|
|||||||
const bridge = await startBridge();
|
const bridge = await startBridge();
|
||||||
process.env.MESH_HOST = '127.0.0.1';
|
process.env.MESH_HOST = '127.0.0.1';
|
||||||
process.env.MESH_PORT = String(bridge.port);
|
process.env.MESH_PORT = String(bridge.port);
|
||||||
delete process.env.MESH_IDLE_TIMEOUT_MS; // default 600000 — a test can't wait that long, so silence must NOT fire
|
delete process.env.MESH_IDLE_TIMEOUT_MS; // default 60000 — a test can't wait that long, so silence must NOT fire
|
||||||
const svc = new MeshService();
|
const svc = new MeshService();
|
||||||
try {
|
try {
|
||||||
svc.onModuleInit();
|
svc.onModuleInit();
|
||||||
@@ -351,6 +414,31 @@ describe('MeshService idle watchdog', () => {
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
describe('MeshService periodic autoreconnect', () => {
|
||||||
|
it('unconditionally reconnects when MESH_RECONNECT_INTERVAL_MS elapses', async () => {
|
||||||
|
const bridge = await startBridge();
|
||||||
|
process.env.MESH_HOST = '127.0.0.1';
|
||||||
|
process.env.MESH_PORT = String(bridge.port);
|
||||||
|
process.env.MESH_RECONNECT_MS = '50'; // fast retry so the re-establishment is observable
|
||||||
|
process.env.MESH_RECONNECT_INTERVAL_MS = '100'; // override — the default (300000) would never elapse in a test
|
||||||
|
const svc = new MeshService();
|
||||||
|
try {
|
||||||
|
svc.onModuleInit();
|
||||||
|
await waitFor(
|
||||||
|
() => bridge.sockets.length === 1 && svc.status().connected,
|
||||||
|
);
|
||||||
|
|
||||||
|
// interval fires → destroy → existing close/retry loop re-establishes with a fresh session
|
||||||
|
await waitFor(() => bridge.sockets.length >= 2 && svc.status().connected);
|
||||||
|
} finally {
|
||||||
|
delete process.env.MESH_RECONNECT_MS;
|
||||||
|
delete process.env.MESH_RECONNECT_INTERVAL_MS;
|
||||||
|
svc.onModuleDestroy();
|
||||||
|
bridge.close();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
describe('MeshService send', () => {
|
describe('MeshService send', () => {
|
||||||
it('sends on the test channel (index 3) by default, never primary', async () => {
|
it('sends on the test channel (index 3) by default, never primary', async () => {
|
||||||
const bridge = await startBridge();
|
const bridge = await startBridge();
|
||||||
|
|||||||
+44
-12
@@ -210,9 +210,19 @@ export class MeshService implements OnModuleInit, OnModuleDestroy {
|
|||||||
);
|
);
|
||||||
// how often the poller re-upserts the history buffer into SQLite (ms)
|
// how often the poller re-upserts the history buffer into SQLite (ms)
|
||||||
private readonly pollMs = parseInt(process.env.MESH_POLL_MS ?? '1000', 10);
|
private readonly pollMs = parseInt(process.env.MESH_POLL_MS ?? '1000', 10);
|
||||||
// idle watchdog: force a reconnect when an ESTABLISHED connection goes silent — the zombie-bridge case where outbound sends keep working but inbound packets stop entirely. Read at construction like the other env config.
|
// idle watchdog: force a reconnect when an ESTABLISHED connection stops decoding frames — the zombie-bridge case where outbound sends keep working but inbound frame flow is dead (raw non-frame bytes do NOT reset the clock). Read at construction like the other env config.
|
||||||
private readonly idleTimeoutMs = parseInt(
|
private readonly idleTimeoutMs = parseInt(
|
||||||
process.env.MESH_IDLE_TIMEOUT_MS ?? '600000',
|
process.env.MESH_IDLE_TIMEOUT_MS ?? '60000',
|
||||||
|
10,
|
||||||
|
);
|
||||||
|
// lazy periodic autoreconnect: unconditionally destroy + re-establish every N ms so a fresh sync happens on schedule — the bridge/radio side tends to kill inbound RX within ~20 min of a fresh sync. Read at construction like the other env config.
|
||||||
|
private readonly reconnectIntervalMs = parseInt(
|
||||||
|
process.env.MESH_RECONNECT_INTERVAL_MS ?? '300000',
|
||||||
|
10,
|
||||||
|
);
|
||||||
|
// cap on undecoded frame buffer: a desynced stream that never forms frames would grow it without bound — force a reconnect instead. Env-overridable so tests can trigger it (Node caps socket 'data' events at ~64 KB).
|
||||||
|
private readonly maxFrameBuffer = parseInt(
|
||||||
|
process.env.MESH_MAX_FRAME_BUFFER ?? '256000',
|
||||||
10,
|
10,
|
||||||
);
|
);
|
||||||
|
|
||||||
@@ -262,9 +272,10 @@ export class MeshService implements OnModuleInit, OnModuleDestroy {
|
|||||||
// telemetry history (newest kept, capped at HISTORY_LIMIT)
|
// telemetry history (newest kept, capped at HISTORY_LIMIT)
|
||||||
private telemetryHistory: TelemetryRecord[] = [];
|
private telemetryHistory: TelemetryRecord[] = [];
|
||||||
private reconnectTimer: NodeJS.Timeout | null = null;
|
private reconnectTimer: NodeJS.Timeout | null = null;
|
||||||
/** epoch ms of the last inbound chunk from the bridge (any frame kind); reset on connect — idle watchdog baseline */
|
/** epoch ms when the last valid frame was decoded from the bridge; reset on connect — idle watchdog baseline */
|
||||||
private lastPacketAt = Date.now();
|
private lastPacketAt = Date.now();
|
||||||
private idleTimer: NodeJS.Timeout | null = null;
|
private idleTimer: NodeJS.Timeout | null = null;
|
||||||
|
private periodicReconnectTimer: NodeJS.Timeout | null = null;
|
||||||
private shuttingDown = false;
|
private shuttingDown = false;
|
||||||
|
|
||||||
onModuleInit(): void {
|
onModuleInit(): void {
|
||||||
@@ -276,6 +287,13 @@ export class MeshService implements OnModuleInit, OnModuleDestroy {
|
|||||||
}
|
}
|
||||||
// idle watchdog: catches established-but-silent connections (zombie bridge) that the connect/retry loop never sees
|
// idle watchdog: catches established-but-silent connections (zombie bridge) that the connect/retry loop never sees
|
||||||
this.idleTimer = setInterval(() => this.checkIdle(), IDLE_CHECK_MS);
|
this.idleTimer = setInterval(() => this.checkIdle(), IDLE_CHECK_MS);
|
||||||
|
// lazy periodic autoreconnect: unconditional fresh sync on a fixed schedule; when not connected, destroy is a no-op and the retry loop owns recovery
|
||||||
|
this.periodicReconnectTimer = setInterval(() => {
|
||||||
|
this.logger.log(
|
||||||
|
'periodic reconnect — MESH_RECONNECT_INTERVAL_MS elapsed',
|
||||||
|
);
|
||||||
|
this.socket?.destroy();
|
||||||
|
}, this.reconnectIntervalMs);
|
||||||
this.connect();
|
this.connect();
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -284,6 +302,7 @@ export class MeshService implements OnModuleInit, OnModuleDestroy {
|
|||||||
if (this.reconnectTimer) clearTimeout(this.reconnectTimer);
|
if (this.reconnectTimer) clearTimeout(this.reconnectTimer);
|
||||||
if (this.pollTimer) clearInterval(this.pollTimer);
|
if (this.pollTimer) clearInterval(this.pollTimer);
|
||||||
if (this.idleTimer) clearInterval(this.idleTimer);
|
if (this.idleTimer) clearInterval(this.idleTimer);
|
||||||
|
if (this.periodicReconnectTimer) clearInterval(this.periodicReconnectTimer);
|
||||||
this.socket?.destroy();
|
this.socket?.destroy();
|
||||||
this.db?.close();
|
this.db?.close();
|
||||||
}
|
}
|
||||||
@@ -662,7 +681,9 @@ export class MeshService implements OnModuleInit, OnModuleDestroy {
|
|||||||
// fresh connection — reset the idle clock so it can't self-trigger before any traffic arrives
|
// fresh connection — reset the idle clock so it can't self-trigger before any traffic arrives
|
||||||
this.lastPacketAt = Date.now();
|
this.lastPacketAt = Date.now();
|
||||||
// fresh sync on every (re)connect — drop stale channels/nodes/device info/telemetry from the old session;
|
// fresh sync on every (re)connect — drop stale channels/nodes/device info/telemetry from the old session;
|
||||||
// message history is NOT reset: it's an append-only log backed by SQLite and survives reconnects
|
// message history is NOT reset: it's an append-only log backed by SQLite and survives reconnects.
|
||||||
|
// The undecoded frame buffer goes too: leftover desync garbage must not re-trigger the overflow backstop on a healthy new stream.
|
||||||
|
this.buffer = new Uint8Array(0);
|
||||||
this.channelMap = new Map();
|
this.channelMap = new Map();
|
||||||
this.nodeMap = new Map();
|
this.nodeMap = new Map();
|
||||||
this.metadata = null;
|
this.metadata = null;
|
||||||
@@ -680,14 +701,22 @@ export class MeshService implements OnModuleInit, OnModuleDestroy {
|
|||||||
this.logger.log(`requested initial sync (want_config_id=${nonce})`);
|
this.logger.log(`requested initial sync (want_config_id=${nonce})`);
|
||||||
});
|
});
|
||||||
socket.on('data', (chunk) => {
|
socket.on('data', (chunk) => {
|
||||||
// lowest-level receive path: every inbound chunk counts as bridge traffic, decodable or not — this is what the idle watchdog watches
|
|
||||||
this.lastPacketAt = Date.now();
|
|
||||||
const merged = new Uint8Array(this.buffer.length + chunk.length);
|
const merged = new Uint8Array(this.buffer.length + chunk.length);
|
||||||
merged.set(this.buffer, 0);
|
merged.set(this.buffer, 0);
|
||||||
merged.set(chunk, this.buffer.length);
|
merged.set(chunk, this.buffer.length);
|
||||||
|
// buffer-overflow backstop: a desynced stream that never forms frames would grow the buffer without bound — cap it and self-heal via reconnect
|
||||||
|
if (merged.length > this.maxFrameBuffer) {
|
||||||
|
this.logger.error(
|
||||||
|
'frame buffer overflow — stream desync, forcing reconnect',
|
||||||
|
);
|
||||||
|
socket.destroy();
|
||||||
|
return;
|
||||||
|
}
|
||||||
this.buffer = merged;
|
this.buffer = merged;
|
||||||
const { packets, rest } = extractFrames(this.buffer);
|
const { packets, rest } = extractFrames(this.buffer);
|
||||||
this.buffer = rest;
|
this.buffer = rest;
|
||||||
|
// the watchdog tracks DECODED FRAMES, not raw bytes: non-frame noise (keepalives, desynced streams) must not keep a dead connection alive
|
||||||
|
if (packets.length) this.lastPacketAt = Date.now();
|
||||||
for (const p of packets) {
|
for (const p of packets) {
|
||||||
decodeFromRadio(p)
|
decodeFromRadio(p)
|
||||||
.then((fr) => this.handleFromRadio(fr))
|
.then((fr) => this.handleFromRadio(fr))
|
||||||
@@ -714,17 +743,20 @@ export class MeshService implements OnModuleInit, OnModuleDestroy {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Idle watchdog tick: force a reconnect when an ESTABLISHED connection has seen no inbound bridge
|
* Idle watchdog tick: force a reconnect when an ESTABLISHED connection has decoded no frames from
|
||||||
* traffic for MESH_IDLE_TIMEOUT_MS — the zombie case where outbound sends keep working but inbound
|
* the bridge for MESH_IDLE_TIMEOUT_MS — the zombie case where outbound sends keep working but inbound
|
||||||
* packets stop. Destroying the socket routes through 'close' → onDown → scheduleReconnect, so the
|
* frame flow is dead (raw non-frame bytes arriving on the socket do not count). Destroying the socket
|
||||||
* existing connect/retry loop re-establishes with a fresh sync; no new retry machinery here.
|
* routes through 'close' → onDown → scheduleReconnect, so the existing connect/retry loop re-establishes
|
||||||
* Public so tests can drive a tick without waiting for the real interval (like flushHistory()).
|
* with a fresh sync; no new retry machinery here. Public so tests can drive a tick without waiting for
|
||||||
|
* the real interval (like flushHistory()).
|
||||||
*/
|
*/
|
||||||
checkIdle(): void {
|
checkIdle(): void {
|
||||||
if (!this.connected) return; // disconnected state is owned by the connect/retry loop
|
if (!this.connected) return; // disconnected state is owned by the connect/retry loop
|
||||||
const idle = Date.now() - this.lastPacketAt;
|
const idle = Date.now() - this.lastPacketAt;
|
||||||
if (idle > this.idleTimeoutMs) {
|
if (idle > this.idleTimeoutMs) {
|
||||||
this.logger.warn(`no bridge traffic for ${idle} ms — forcing reconnect`);
|
this.logger.warn(
|
||||||
|
`no decoded bridge frames for ${idle} ms — forcing reconnect`,
|
||||||
|
);
|
||||||
this.socket?.destroy();
|
this.socket?.destroy();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -72,7 +72,7 @@ describe('OpencodeService.notify', () => {
|
|||||||
|
|
||||||
expect(fetchMock).toHaveBeenCalledTimes(2);
|
expect(fetchMock).toHaveBeenCalledTimes(2);
|
||||||
const [url, init] = fetchMock.mock.calls[1];
|
const [url, init] = fetchMock.mock.calls[1];
|
||||||
expect(url).toBe(`${API}/proxy/abc/session/ses_pinned/message`);
|
expect(url).toBe(`${API}/proxy/abc/api/session/ses_pinned/prompt`);
|
||||||
const opts = init as {
|
const opts = init as {
|
||||||
method?: string;
|
method?: string;
|
||||||
headers?: Record<string, string>;
|
headers?: Record<string, string>;
|
||||||
@@ -83,15 +83,10 @@ describe('OpencodeService.notify', () => {
|
|||||||
expect(opts.headers?.authorization).toBe(
|
expect(opts.headers?.authorization).toBe(
|
||||||
'Basic ' + Buffer.from('u:p').toString('base64'),
|
'Basic ' + Buffer.from('u:p').toString('base64'),
|
||||||
);
|
);
|
||||||
// body is EXACTLY {"parts":[{"type":"text","text":msg}]} — em-dash, not a hyphen
|
// body is EXACTLY {"text": msg} — em-dash, not a hyphen — and a 2xx resolves notify as delivered
|
||||||
expect(String(opts.body)).toBe(
|
expect(String(opts.body)).toBe(
|
||||||
JSON.stringify({
|
JSON.stringify({
|
||||||
parts: [
|
text: 'New mesh message (id 7) from node 42: "hi" — please react to it.',
|
||||||
{
|
|
||||||
type: 'text',
|
|
||||||
text: 'New mesh message (id 7) from node 42: "hi" — please react to it.',
|
|
||||||
},
|
|
||||||
],
|
|
||||||
}),
|
}),
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
@@ -111,12 +106,7 @@ describe('OpencodeService.notify', () => {
|
|||||||
const [, init] = fetchMock.mock.calls[1];
|
const [, init] = fetchMock.mock.calls[1];
|
||||||
expect(String((init as { body?: string }).body)).toBe(
|
expect(String((init as { body?: string }).body)).toBe(
|
||||||
JSON.stringify({
|
JSON.stringify({
|
||||||
parts: [
|
text: 'New mesh message (id 7): "hi" — please react to it.',
|
||||||
{
|
|
||||||
type: 'text',
|
|
||||||
text: 'New mesh message (id 7): "hi" — please react to it.',
|
|
||||||
},
|
|
||||||
],
|
|
||||||
}),
|
}),
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
@@ -149,7 +139,7 @@ describe('OpencodeService.notify', () => {
|
|||||||
await svc.notify(7, 'hi');
|
await svc.notify(7, 'hi');
|
||||||
|
|
||||||
expect(fetchMock.mock.calls[1][0]).toBe(
|
expect(fetchMock.mock.calls[1][0]).toBe(
|
||||||
`${API}/proxy/abc/session/ses_pinned/message`,
|
`${API}/proxy/abc/api/session/ses_pinned/prompt`,
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -182,7 +172,7 @@ describe('OpencodeService.notify', () => {
|
|||||||
await svc.notify(7, 'hi');
|
await svc.notify(7, 'hi');
|
||||||
|
|
||||||
expect(fetchMock.mock.calls[1][0]).toBe(
|
expect(fetchMock.mock.calls[1][0]).toBe(
|
||||||
`${API}/proxy/b/session/ses_pinned/message`,
|
`${API}/proxy/b/api/session/ses_pinned/prompt`,
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -191,18 +181,37 @@ describe('OpencodeService.notify', () => {
|
|||||||
fetchMock
|
fetchMock
|
||||||
.mockResolvedValueOnce(json([WORKSPACE]))
|
.mockResolvedValueOnce(json([WORKSPACE]))
|
||||||
.mockResolvedValueOnce(
|
.mockResolvedValueOnce(
|
||||||
json([
|
// v2 envelope {data, cursor}; the /elsewhere session is more recent but a different
|
||||||
{ id: 'ses_old', time: { updated: 100 } },
|
// directory, so the location.directory filter must exclude it
|
||||||
{ id: 'ses_new', time: { updated: 200 } },
|
json({
|
||||||
]),
|
data: [
|
||||||
|
{
|
||||||
|
id: 'ses_old',
|
||||||
|
location: { directory: WS_PATH },
|
||||||
|
time: { updated: 100 },
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: 'ses_other_ws',
|
||||||
|
location: { directory: '/elsewhere' },
|
||||||
|
time: { updated: 300 },
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: 'ses_new',
|
||||||
|
location: { directory: WS_PATH },
|
||||||
|
time: { updated: 200 },
|
||||||
|
},
|
||||||
|
],
|
||||||
|
cursor: { previous: null, next: null },
|
||||||
|
}),
|
||||||
)
|
)
|
||||||
.mockResolvedValueOnce(json({}));
|
.mockResolvedValueOnce(json({}));
|
||||||
|
|
||||||
await svc.notify(7, 'hi');
|
await svc.notify(7, 'hi');
|
||||||
|
|
||||||
expect(fetchMock.mock.calls[1][0]).toBe(`${API}/proxy/abc/session`);
|
// discovery GET hits the v2 route; the prompt POST goes to the most recently updated session of THIS directory
|
||||||
|
expect(fetchMock.mock.calls[1][0]).toBe(`${API}/proxy/abc/api/session`);
|
||||||
expect(fetchMock.mock.calls[2][0]).toBe(
|
expect(fetchMock.mock.calls[2][0]).toBe(
|
||||||
`${API}/proxy/abc/session/ses_new/message`,
|
`${API}/proxy/abc/api/session/ses_new/prompt`,
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -210,13 +219,22 @@ describe('OpencodeService.notify', () => {
|
|||||||
const svc = makeService({ OPENCODE_USER: 'u', OPENCODE_PASS: 'p' });
|
const svc = makeService({ OPENCODE_USER: 'u', OPENCODE_PASS: 'p' });
|
||||||
fetchMock
|
fetchMock
|
||||||
.mockResolvedValueOnce(json([WORKSPACE]))
|
.mockResolvedValueOnce(json([WORKSPACE]))
|
||||||
.mockResolvedValueOnce(json([{ id: 'ses_a' }, { id: 'ses_b', time: {} }]))
|
.mockResolvedValueOnce(
|
||||||
|
json({
|
||||||
|
data: [
|
||||||
|
{ id: 'ses_a', location: { directory: WS_PATH } },
|
||||||
|
{ id: 'ses_b', location: { directory: WS_PATH }, time: {} },
|
||||||
|
],
|
||||||
|
cursor: { previous: null, next: null },
|
||||||
|
}),
|
||||||
|
)
|
||||||
.mockResolvedValueOnce(json({}));
|
.mockResolvedValueOnce(json({}));
|
||||||
|
|
||||||
await svc.notify(7, 'hi');
|
await svc.notify(7, 'hi');
|
||||||
|
|
||||||
|
expect(fetchMock.mock.calls[1][0]).toBe(`${API}/proxy/abc/api/session`);
|
||||||
expect(fetchMock.mock.calls[2][0]).toBe(
|
expect(fetchMock.mock.calls[2][0]).toBe(
|
||||||
`${API}/proxy/abc/session/ses_a/message`,
|
`${API}/proxy/abc/api/session/ses_a/prompt`,
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -232,11 +250,11 @@ describe('OpencodeService.notify', () => {
|
|||||||
|
|
||||||
await svc.notify(7, 'hi');
|
await svc.notify(7, 'hi');
|
||||||
|
|
||||||
// workspaces GET + message POST only — no /session discovery GET
|
// workspaces GET + prompt POST only — no /api/session discovery GET
|
||||||
expect(fetchMock).toHaveBeenCalledTimes(2);
|
expect(fetchMock).toHaveBeenCalledTimes(2);
|
||||||
expect(fetchMock.mock.calls[0][0]).toBe(`${API}/api/workspaces`);
|
expect(fetchMock.mock.calls[0][0]).toBe(`${API}/api/workspaces`);
|
||||||
expect(fetchMock.mock.calls[1][0]).toBe(
|
expect(fetchMock.mock.calls[1][0]).toBe(
|
||||||
`${API}/proxy/abc/session/ses_pinned/message`,
|
`${API}/proxy/abc/api/session/ses_pinned/prompt`,
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -248,7 +266,7 @@ describe('OpencodeService.notify', () => {
|
|||||||
});
|
});
|
||||||
fetchMock
|
fetchMock
|
||||||
.mockResolvedValueOnce(json([WORKSPACE]))
|
.mockResolvedValueOnce(json([WORKSPACE]))
|
||||||
.mockRejectedValueOnce(timeoutError()); // the message POST times out — opencode holds the response until the turn finishes
|
.mockRejectedValueOnce(timeoutError()); // the prompt POST times out — a read timeout is still treated as delivered (safety net)
|
||||||
|
|
||||||
const logSpy = vi.spyOn(Logger.prototype, 'log');
|
const logSpy = vi.spyOn(Logger.prototype, 'log');
|
||||||
await expect(svc.notify('hi')).resolves.toBeUndefined();
|
await expect(svc.notify('hi')).resolves.toBeUndefined();
|
||||||
|
|||||||
@@ -2,7 +2,8 @@ import { Injectable, Logger } from '@nestjs/common';
|
|||||||
|
|
||||||
// ponytail: hardcoded backoff [2s, 10s, 30s] (~45 s window) — no env knob; tune here if outages run longer
|
// ponytail: hardcoded backoff [2s, 10s, 30s] (~45 s window) — no env knob; tune here if outages run longer
|
||||||
const RETRY_DELAYS_MS = [2_000, 10_000, 30_000];
|
const RETRY_DELAYS_MS = [2_000, 10_000, 30_000];
|
||||||
// opencode accepts a session message immediately but holds the HTTP response until the agent turn finishes
|
// opencode v2 returns 2xx immediately (it no longer holds the response until the agent turn finishes);
|
||||||
|
// the 15 s timeout only guards a stuck connection — a read timeout is still treated as delivered
|
||||||
const TIMEOUT_MS = 15_000;
|
const TIMEOUT_MS = 15_000;
|
||||||
|
|
||||||
/** Entry of GET {api}/api/workspaces (only the fields we use). */
|
/** Entry of GET {api}/api/workspaces (only the fields we use). */
|
||||||
@@ -14,9 +15,11 @@ interface WorkspaceEntry {
|
|||||||
proxyPath?: string;
|
proxyPath?: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Entry of GET {base}/session (only the fields we use). */
|
/** Entry of GET {base}/api/session (v2; only the fields we use — directory lives under location, time is epoch ms). */
|
||||||
interface SessionEntry {
|
interface SessionEntry {
|
||||||
id: string;
|
id: string;
|
||||||
|
title?: string;
|
||||||
|
location?: { directory?: string };
|
||||||
time?: { updated?: number };
|
time?: { updated?: number };
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -43,11 +46,11 @@ export class OpencodeService {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Push an rx mesh message into the opencode session as a text part. Resolves when delivered —
|
* Push an rx mesh message into the opencode session (v2: POST {base}/api/session/{id}/prompt with a
|
||||||
* including on a final-POST read timeout, which means DELIVERED (opencode holds the response until
|
* `{text}` body). Resolves when delivered — including on a final-POST read timeout, which still means
|
||||||
* the agent turn finishes) and is never retried. Workspace/session resolution failures are retried
|
* DELIVERED (kept as a harmless safety net; v2 normally returns 2xx immediately) and is never retried.
|
||||||
* up to RETRY_DELAYS_MS.length + 1 attempts total; the last failure logs `opencode notify dropped
|
* Workspace/session resolution failures are retried up to RETRY_DELAYS_MS.length + 1 attempts total;
|
||||||
* after N attempts`. No-op when disabled (no credentials).
|
* the last failure logs `opencode notify dropped after N attempts`. No-op when disabled (no credentials).
|
||||||
*/
|
*/
|
||||||
async notify(id: number, text: string, from?: number): Promise<void> {
|
async notify(id: number, text: string, from?: number): Promise<void> {
|
||||||
if (!this.enabled) return;
|
if (!this.enabled) return;
|
||||||
@@ -112,12 +115,19 @@ export class OpencodeService {
|
|||||||
const base = this.api + (ws.proxyPath ?? '');
|
const base = this.api + (ws.proxyPath ?? '');
|
||||||
let sid = this.sessionOverride;
|
let sid = this.sessionOverride;
|
||||||
if (!sid) {
|
if (!sid) {
|
||||||
const sessions = await this.getJson<SessionEntry[]>(`${base}/session`);
|
// v2 discovery: GET {base}/api/session returns an envelope { data, cursor } — NOT a bare array
|
||||||
|
const envelope = await this.getJson<{ data?: SessionEntry[] }>(
|
||||||
|
`${base}/api/session`,
|
||||||
|
);
|
||||||
|
// only sessions of THIS workspace directory — the instance proxy may serve other sessions too
|
||||||
|
const sessions = (envelope.data ?? []).filter(
|
||||||
|
(s) => s.location?.directory === this.workspacePath,
|
||||||
|
);
|
||||||
if (sessions.length === 0)
|
if (sessions.length === 0)
|
||||||
throw new Error(
|
throw new Error(
|
||||||
`no sessions in workspace ${ws.id} (${this.workspacePath})`,
|
`no sessions in workspace ${ws.id} (${this.workspacePath})`,
|
||||||
);
|
);
|
||||||
// most recently updated session; first entry when none carry time.updated
|
// most recently updated session (epoch ms); first entry when none carry time.updated
|
||||||
const updated = (s: SessionEntry): number => s.time?.updated ?? 0;
|
const updated = (s: SessionEntry): number => s.time?.updated ?? 0;
|
||||||
sid = sessions.some((s) => updated(s))
|
sid = sessions.some((s) => updated(s))
|
||||||
? sessions.reduce((a, b) => (updated(b) > updated(a) ? b : a)).id
|
? sessions.reduce((a, b) => (updated(b) > updated(a) ? b : a)).id
|
||||||
@@ -148,16 +158,17 @@ export class OpencodeService {
|
|||||||
msg: string,
|
msg: string,
|
||||||
): Promise<void> {
|
): Promise<void> {
|
||||||
try {
|
try {
|
||||||
const res = await fetch(`${base}/session/${sid}/message`, {
|
const res = await fetch(`${base}/api/session/${sid}/prompt`, {
|
||||||
method: 'POST',
|
method: 'POST',
|
||||||
headers: { ...this.authHeaders(), 'content-type': 'application/json' },
|
headers: { ...this.authHeaders(), 'content-type': 'application/json' },
|
||||||
body: JSON.stringify({ parts: [{ type: 'text', text: msg }] }),
|
body: JSON.stringify({ text: msg }),
|
||||||
signal: AbortSignal.timeout(TIMEOUT_MS),
|
signal: AbortSignal.timeout(TIMEOUT_MS),
|
||||||
});
|
});
|
||||||
if (!res.ok) throw new Error(`HTTP ${res.status}`); // HTTP errors retry like network failures
|
if (!res.ok) throw new Error(`HTTP ${res.status}`); // HTTP errors retry like network failures
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
// a read timeout here means DELIVERED, not failed — opencode accepted the message and is holding
|
// a read timeout here means DELIVERED, not failed — kept as a harmless safety net: v2 normally
|
||||||
// the response until the agent turn finishes; never retried
|
// returns 2xx immediately, but if the response is still pending past the timeout we treat the
|
||||||
|
// message as accepted and never retry
|
||||||
if ((err as Error).name === 'TimeoutError') {
|
if ((err as Error).name === 'TimeoutError') {
|
||||||
this.logger.log(
|
this.logger.log(
|
||||||
`opencode notify accepted (response still pending after ${TIMEOUT_MS / 1000} s)`,
|
`opencode notify accepted (response still pending after ${TIMEOUT_MS / 1000} s)`,
|
||||||
|
|||||||
@@ -0,0 +1,28 @@
|
|||||||
|
import { readdirSync, readFileSync } from 'node:fs';
|
||||||
|
import { join } from 'node:path';
|
||||||
|
import { fileURLToPath } from 'node:url';
|
||||||
|
import { describe, expect, it } from 'vitest';
|
||||||
|
|
||||||
|
const srcDir = fileURLToPath(new URL('.', import.meta.url));
|
||||||
|
|
||||||
|
function moduleFiles(dir: string): string[] {
|
||||||
|
return readdirSync(dir, { withFileTypes: true }).flatMap((entry) =>
|
||||||
|
entry.isDirectory()
|
||||||
|
? moduleFiles(join(dir, entry.name))
|
||||||
|
: entry.name.endsWith('.module.ts')
|
||||||
|
? [join(dir, entry.name)]
|
||||||
|
: [],
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
describe('ScheduleModule.forRoot single registration', () => {
|
||||||
|
// Regression guard (v0.21.7): each forRoot() call creates its own ScheduleExplorer that scans all providers, so a second call made every @Cron/@Interval job fire twice.
|
||||||
|
it('appears exactly once across all src/**/*.module.ts', () => {
|
||||||
|
const hits = moduleFiles(srcDir).flatMap((file) =>
|
||||||
|
(
|
||||||
|
readFileSync(file, 'utf8').match(/ScheduleModule\.forRoot\(/g) ?? []
|
||||||
|
).map(() => file.slice(srcDir.length)),
|
||||||
|
);
|
||||||
|
expect(hits).toHaveLength(1);
|
||||||
|
});
|
||||||
|
});
|
||||||
Reference in New Issue
Block a user