Smooth jagged sub-national region boundaries (v0.16.0) #30

Merged
kamilkrzywda merged 1 commits from feat/smooth-regions into master 2026-10-13 11:53:59 +02:00
6 changed files with 313 additions and 4 deletions
Showing only changes of commit f1cc4fbc10 - Show all commits
+9
View File
@@ -2,6 +2,15 @@
All notable changes to **History** are listed here, newest first.
## v0.16.0 — 2026-10-13 09:37
- **Smoothed the jagged sub-national region boundaries with Douglas–Peucker vertex-thinning (Part of #16).** The v0.14 coarsening (boolean dissolve) and Natural Earth's 1:10m digitization left some boundaries overly jagged — the worst being regions the dissolve split into many small zig-zag fragments, e.g. **SI-048 Kocevje (Slovenia, 57 parts)** and **MK-36 Kavadartsi (N. Macedonia, 8 parts)**. The new `scripts/smooth-regions.mjs` (`npm run smooth:regions`) runs `@turf/simplify` (Ramer–Douglas–Peucker, tolerance ~**50 m**) over **every ring of every polygon**, dropping the redundant "in-between" vertices of a shallow zig-zag while keeping the overall shape.
- **Vertex-thinning only — no topology change.** DP only removes points: it never moves a border, never merges/dissolves a part, and never creates or removes a ring. So the dataset contract is fully preserved — still **1,796 regions** with identical `id` / `properties.name` / `properties.parent`, all **177 countries** covered, the same **5,026 parts / 5,027 rings**, the single **KZ-KZY** hole, and all **83 islands**. The 18 degenerate 3-point "micro-dot" rings (Ragged Island, the Paracels, …) are kept verbatim (`@turf/simplify` requires ≥4 pts and would otherwise throw).
- **A gentle pass, reported honestly.** At 50 m the dataset is already fairly smooth (the v0.14 dissolve produced low-vertex, widely-spaced boundaries — median consecutive-vertex spacing ~18 km), so this is a light cleanup: **111,891 → 111,175 vertices (716 removed, 0.64%)** across **542 of 1,796** regions. **Kocelje 612 → 599** and **Kavadartsi 169 → 163** verts; both keep their exact part count (the "mess" there is fragmentation, which thinning — under the no-topology-change constraint — does not fix). The file shrinks **2,302,473 → 2,289,528 bytes (2.29 MB)**. Bump `TOLERANCE_M` to 100 / 200 m and re-run for a stronger pass (100 m → 974 removed; 200 m → 1,312 removed) — the pass stays idempotent either way.
- **The "messy" regions are really fragmentation artifacts, not fine jaggedness.** Ranking by **part density** (parts packed per unit area) rather than raw part count — which is dominated by legitimate *dispersed* archipelagos (Nunavut 213, Alaska 159, Antarctica 135, British Columbia 84) — surfaces the genuinely messy **coherent regions the dissolve left fragmented**: **SI-048 Kocevje (SVN, 57 parts in ~16,600 km²)**, **GR-H Stereá Elláda / Central Greece (GRC, 53 parts)**, **KR-46 South Jeolla (KOR, 28 parts)**, **SE-C Uppsala (SWE, 19 parts)**, **HR-13 Zadarska (HRV, 17 parts)**, plus the milder Balkan **MK-36 Kavadartsi (MKD, 8 parts)**. These real land regions (the Balkan ones in particular) ended up as many scattered fragments — the "similar shape problems".
- **Deterministic + idempotent.** One `@turf/simplify` pass at a fixed tolerance, then 4-dp rounding (the same convention as the other scripts). The output already sits at the pass's fixed point, so re-running on the smoothed file is byte-identical (sha256 `70a43541…`, verified across repeated runs); `public/regions.geo.json` and the built `dist/regions.geo.json` are sha256-identical.
- **Tests.** `scripts/regions.test.ts` gains an eighth test pinning the v0.16 fixed point: 1,796 regions, 5,026 parts, 5,027 rings, the single KZ-KZY hole, and exactly **111,175** vertices (8 tests total). All prior assertions (well-formed, unique ids, 177-country coverage, area floor, holes, coverage band) pass unchanged on the smoothed dataset.
- Part of #16
## v0.15.0 — 2026-10-12 13:49
- **Filled the small holes left by the v0.14 coarsening (Part of #16).** The coarsened `public/regions.geo.json` still carried **2,245 tiny enclosed holes** (all < 500 km² in practice) — unfilled patches left by the boolean dissolves and Natural Earth's own digitization, which showed up as "holes in regions". The new `scripts/cleanup-regions.mjs` (`npm run cleanup:regions`) drops every inner ring smaller than the 1,000 km² threshold so each region becomes solid: across **498 features, 56,484 km²** of artifact area was filled, removing the visible unfilled dots.
- **Real water is kept, not erased.** Any enclosed hole ≥ 1,000 km² survives as a hole. In this dataset the genuinely large lakes (the Great Lakes, the Caspian, the Aral, Lake Superior) had already been dissolved by the v0.14 coarsening, so none remain to preserve; the only surviving hole is a single medium one — **KZ-KZY (Kyzylorda, Kazakhstan) at 5,858 km²**, a real water body in the Aral basin — which is kept. The 1,000 km² threshold sits comfortably between the tiny artifacts (all < 500) and any genuine lake, so it fills exactly the artifacts and nothing else.
+84 -2
View File
@@ -1,12 +1,12 @@
{
"name": "history",
"version": "0.15.0",
"version": "0.16.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "history",
"version": "0.15.0",
"version": "0.16.0",
"dependencies": {
"@radix-ui/react-dialog": "^1.1.6",
"@radix-ui/react-label": "^2.1.2",
@@ -32,6 +32,7 @@
"@eslint/js": "^9.20.0",
"@playwright/test": "^1.62.1",
"@tailwindcss/vite": "^4.1.4",
"@turf/simplify": "^7.4.0",
"@turf/union": "^7.4.0",
"@types/better-sqlite3": "^7.6.13",
"@types/express": "^4.17.25",
@@ -3335,6 +3336,54 @@
"vite": "^5.2.0 || ^6 || ^7 || ^8"
}
},
"node_modules/@turf/boolean-point-on-line": {
"version": "7.4.0",
"resolved": "https://registry.npmjs.org/@turf/boolean-point-on-line/-/boolean-point-on-line-7.4.0.tgz",
"integrity": "sha512-5i5UBeZf/LuzgavkIj8L8L5/8SYWItWzQQcE9ymFZr60lV5lM8eFvQDfzAz2R3WQFCrxXExPqNvAVZOKGUyvoA==",
"dev": true,
"license": "MIT",
"dependencies": {
"@turf/helpers": "7.4.0",
"@turf/invariant": "7.4.0",
"@types/geojson": "^7946.0.10",
"tslib": "^2.8.1"
},
"funding": {
"url": "https://opencollective.com/turf"
}
},
"node_modules/@turf/clean-coords": {
"version": "7.4.0",
"resolved": "https://registry.npmjs.org/@turf/clean-coords/-/clean-coords-7.4.0.tgz",
"integrity": "sha512-xl3lVPey/FyVc3Z+drAWtVuVtvt737RHKGUMGyuskmKYW0JUeFDgqSR0OnrDfxXWb58Im/65BgPd1B+TUA9WfA==",
"dev": true,
"license": "MIT",
"dependencies": {
"@turf/boolean-point-on-line": "7.4.0",
"@turf/helpers": "7.4.0",
"@turf/invariant": "7.4.0",
"@types/geojson": "^7946.0.10",
"tslib": "^2.8.1"
},
"funding": {
"url": "https://opencollective.com/turf"
}
},
"node_modules/@turf/clone": {
"version": "7.4.0",
"resolved": "https://registry.npmjs.org/@turf/clone/-/clone-7.4.0.tgz",
"integrity": "sha512-IfYnuil7XYJauy3crzIYEr26QkmBiTgFdGfYwUUe3S6dawX+lyb3vhuaYyssEPcCJq83g0G7hwxZy7141Mmzjg==",
"dev": true,
"license": "MIT",
"dependencies": {
"@turf/helpers": "7.4.0",
"@types/geojson": "^7946.0.10",
"tslib": "^2.8.1"
},
"funding": {
"url": "https://opencollective.com/turf"
}
},
"node_modules/@turf/helpers": {
"version": "7.4.0",
"resolved": "https://registry.npmjs.org/@turf/helpers/-/helpers-7.4.0.tgz",
@@ -3349,6 +3398,21 @@
"url": "https://opencollective.com/turf"
}
},
"node_modules/@turf/invariant": {
"version": "7.4.0",
"resolved": "https://registry.npmjs.org/@turf/invariant/-/invariant-7.4.0.tgz",
"integrity": "sha512-OAsc3qdNx+tRqzWmMNFMnVlWWACIqnYqIejZKvb2oKkKPYJJrURPXZ6OdrGD58ljJMolLoqwIZGSx3VndaEjxg==",
"dev": true,
"license": "MIT",
"dependencies": {
"@turf/helpers": "7.4.0",
"@types/geojson": "^7946.0.10",
"tslib": "^2.8.1"
},
"funding": {
"url": "https://opencollective.com/turf"
}
},
"node_modules/@turf/meta": {
"version": "7.4.0",
"resolved": "https://registry.npmjs.org/@turf/meta/-/meta-7.4.0.tgz",
@@ -3364,6 +3428,24 @@
"url": "https://opencollective.com/turf"
}
},
"node_modules/@turf/simplify": {
"version": "7.4.0",
"resolved": "https://registry.npmjs.org/@turf/simplify/-/simplify-7.4.0.tgz",
"integrity": "sha512-HTolHFuXLReKHJoh3iz92+WpF5EsuCmmZR7fNv3hWnA4hZfkuEzbGB/WZqRdQpvDs99IUS8QIc6BAp6FHzQbBQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"@turf/clean-coords": "7.4.0",
"@turf/clone": "7.4.0",
"@turf/helpers": "7.4.0",
"@turf/meta": "7.4.0",
"@types/geojson": "^7946.0.10",
"tslib": "^2.8.1"
},
"funding": {
"url": "https://opencollective.com/turf"
}
},
"node_modules/@turf/union": {
"version": "7.4.0",
"resolved": "https://registry.npmjs.org/@turf/union/-/union-7.4.0.tgz",
+3 -1
View File
@@ -1,6 +1,6 @@
{
"name": "history",
"version": "0.15.0",
"version": "0.16.0",
"private": true,
"type": "module",
"description": "AI-driven alternate-history simulator — the LLM is the world engine (Pax Historia style).",
@@ -12,6 +12,7 @@
"generate:regions": "node scripts/generate-regions.mjs",
"coarsen:regions": "node scripts/coarsen-regions.mjs",
"cleanup:regions": "node scripts/cleanup-regions.mjs",
"smooth:regions": "node scripts/smooth-regions.mjs",
"typecheck": "tsc --noEmit",
"lint": "eslint .",
"e2e": "playwright test"
@@ -41,6 +42,7 @@
"@eslint/js": "^9.20.0",
"@playwright/test": "^1.62.1",
"@tailwindcss/vite": "^4.1.4",
"@turf/simplify": "^7.4.0",
"@turf/union": "^7.4.0",
"@types/better-sqlite3": "^7.6.13",
"@types/express": "^4.17.25",
File diff suppressed because one or more lines are too long
+36
View File
@@ -404,4 +404,40 @@ describe('regions.geo.json', () => {
` / max ${maxId} ${(100 * maxRatio).toFixed(1)}% (band 40–175%)`,
);
});
it('is at the Douglas–Peucker 50m fixed point: 1,796 regions, topology unchanged, 111,175 vertices (v0.16 smoothing)', () => {
// v0.16 (scripts/smooth-regions.mjs) thinned every ring with Douglas–Peucker at
// ~50 m. That removes redundant vertices only — it never moves a border, merges
// a part, or creates/removes a ring — so the topology signature is invariant
// (1,796 features, 5,026 parts, 5,027 rings, the single KZ-KZY hole) and the
// vertex total sits at the pass's fixed point (re-running changes nothing).
// Pinning it guards against an accidental re-coarsening or a different-tolerance
// re-smooth silently changing the shipped dataset.
const polysOf = (g: Geometry) => (g.type === 'Polygon' ? [g.coordinates] : g.coordinates);
let verts = 0,
rings = 0,
parts = 0,
holes = 0;
const holeIds: string[] = [];
for (const f of regions.features) {
const polys = polysOf(f.geometry);
parts += polys.length;
for (const poly of polys) {
rings += poly.length;
holes += Math.max(0, poly.length - 1);
if (poly.length > 1) holeIds.push(f.id);
for (const ring of poly) verts += ring.length;
}
}
expect(regions.features.length, 'region count').toBe(1796);
expect(verts, 'total vertices (incl. ring closure)').toBe(111_175); // was 111,891 pre-v0.16; 716 removed
expect(parts, 'parts').toBe(5026); // unchanged — no part merged/dropped
expect(rings, 'rings').toBe(5027); // unchanged — no ring added/dropped
expect(holes, 'holes').toBe(1); // the single KZ-KZY hole, unchanged
expect(holeIds, 'the one remaining hole is KZ-KZY').toEqual(['KZ-KZY']);
console.log(
`\n v0.16 smoothing: ${regions.features.length} regions, ${parts} parts, ${rings} rings, ` +
`${holes} hole (KZ-KZY), ${verts} vertices (50m DP fixed point; 716 removed from 111,891) — topology unchanged`,
);
});
});
+180
View File
@@ -0,0 +1,180 @@
#!/usr/bin/env node
/**
* scripts/smooth-regions.mjs — smooths jagged sub-national region boundaries.
*
* Run: npm run smooth:regions (in place on public/regions.geo.json)
* node scripts/smooth-regions.mjs [out] [src]
*
* The v0.14 coarsening (boolean dissolve) and Natural Earth's 1:10m digitization
* left some sub-national boundaries overly jagged — the worst being coastal /
* borderland regions that the dissolve split into many small, zig-zag fragments
* (e.g. SI-048 Kocevje, Slovenia, 57 parts; MK-36 Kavadartsi, N. Macedonia, 8
* parts). This cleans the jaggedness in place with **Douglas–Peucker
* vertex-thinning** (`@turf/simplify`), applied to every ring of every polygon.
*
* DOUGLAS–PEUCKER — `@turf/simplify` keeps a vertex only when it deviates from
* the chord joining its kept neighbours by more than the tolerance, removing
* the redundant "in-between" points of a shallow zig-zag. The tolerance is
* expressed in the GeoJSON degree space (its distance metric treats 1° lon
* == 1° lat); TOLERANCE_M (metres) is converted with the same ~111 km/° the
* area model uses, so TOLERANCE_M = 50 is a gentle ~50 m pass. Bump it to
* 100 / 200 for a stronger pass and re-run — the pass is idempotent either way.
*
* VERTEX-THINNING ONLY — DP only removes points. It never moves a border,
* never merges/dissolves regions or parts, and never creates or removes a
* ring, so topology is preserved exactly: the region count, every `id`,
* `properties.name`/`properties.parent`, the part count of every feature, the
* single real hole (KZ-KZY), and all 83 islands are all unchanged. Only the
* redundant vertices along each ring are dropped.
*
* Deterministic + idempotent: one @turf/simplify pass at a fixed tolerance, then
* 4-dp rounding (the same convention as the other scripts). Because the output
* is already at the tolerance's fixed point, re-running on the smoothed file
* yields a byte-identical file (sha256-verified). The 18 degenerate 3-point
* micro-rings (the ~0-area "dots" like Ragged Island / the Paracels) are kept
* verbatim — @turf/simplify requires ≥4 points and would otherwise throw — so
* no ring is ever dropped and every feature keeps its exact part/ring count.
*/
import { readFileSync, writeFileSync } from 'node:fs';
import path from 'node:path';
import { fileURLToPath } from 'node:url';
import { simplify } from '@turf/simplify';
const here = path.dirname(fileURLToPath(import.meta.url));
const root = path.resolve(here, '..');
const DEFAULT_FILE = path.join(root, 'public', 'regions.geo.json');
const args = process.argv.slice(2).filter((a) => !a.startsWith('-'));
const OUT = args[0] ? path.resolve(args[0]) : DEFAULT_FILE;
const SRC = args[1] ? path.resolve(args[1]) : DEFAULT_FILE;
// --- parameters ---------------------------------------------------------------
const TOLERANCE_M = 50; // DP tolerance in metres — ~50 m is gentle (bump to 100/200 for more)
const M_PER_DEG = 111_000; // ~111 km per degree (same scale as the area model's 111.32/110.574)
const TOL_DEG = TOLERANCE_M / M_PER_DEG; // @turf/simplify works in degree space → ≈ 0.00045°
const R4 = 4; // coordinate precision (decimal places) — same as the other scripts
// --- rounding / normalization (same conventions as the other scripts) ----------
const rN = (v, N) => Math.round(v * 10 ** N) / 10 ** N;
function roundRing(ring, N = R4) {
const out = [];
for (const [x, y] of ring) {
const p = [rN(x, N), rN(y, N)];
const last = out[out.length - 1];
if (!last || last[0] !== p[0] || last[1] !== p[1]) out.push(p);
}
if (out.length > 1 && out[0][0] === out[out.length - 1][0] && out[0][1] === out[out.length - 1][1]) out.pop();
if (out.length < 3) return ring; // degenerate — keep verbatim, never corrupt a ring
return [...out, [...out[0]]];
}
// --- tiny geometry helpers -----------------------------------------------------
const polysOf = (g) => (g.type === 'Polygon' ? [g.coordinates] : g.coordinates);
const ringsOf = (g) => polysOf(g).flatMap((p) => p);
const countVerts = (g) => ringsOf(g).reduce((s, r) => s + r.length, 0);
const countParts = (g) => (g.type === 'Polygon' ? 1 : g.coordinates.length);
const countRings = (g) => ringsOf(g).length;
const countHoles = (g) => polysOf(g).reduce((s, p) => s + Math.max(0, p.length - 1), 0);
/**
* Smooth one closed ring with a single @turf/simplify (Douglas–Peucker) pass.
* Only removes vertices — never adds, never drops the ring. Degenerate rings
* (<4 pts) are returned verbatim (simplify would throw); a ring DP would
* over-collapse below a valid 4-pt ring is kept as-is; any error keeps the
* original. Never throws.
*/
function smoothRing(ring) {
if (!Array.isArray(ring) || ring.length < 4) return ring; // keep degenerate verbatim
try {
const feat = { type: 'Feature', geometry: { type: 'Polygon', coordinates: [ring] } };
const next = simplify(feat, { tolerance: TOL_DEG }).geometry.coordinates[0];
if (!Array.isArray(next) || next.length < 4) return ring; // over-collapsed — keep original
const closed =
next[next.length - 1][0] === next[0][0] && next[next.length - 1][1] === next[0][1]
? next
: [...next, [next[0][0], next[0][1]]];
const rounded = roundRing(closed);
return rounded.length >= 4 ? rounded : ring;
} catch {
return ring;
}
}
/** Smooth every ring of a geometry. Never drops a ring or a part. */
function smoothGeom(geom) {
const newPolys = polysOf(geom).map((poly) => poly.map(smoothRing));
if (geom.type === 'Polygon' && newPolys.length === 1) return { type: 'Polygon', coordinates: newPolys[0] };
return { type: 'MultiPolygon', coordinates: newPolys };
}
// --- main ----------------------------------------------------------------------
function main() {
const t0 = Date.now();
const rawIn = readFileSync(SRC);
const fc = JSON.parse(rawIn);
const inBytes = Buffer.byteLength(rawIn);
console.log(`input: ${fc.features.length} regions (${(inBytes / 1024).toFixed(0)} KB) from ${SRC}`);
console.log(`mode: Douglas–Peucker via @turf/simplify, tolerance ${TOLERANCE_M} m (≈ ${TOL_DEG}°)`);
// before stats
let vb = 0, hb = 0, pb = 0, rb = 0, degenerate = 0;
for (const f of fc.features) {
vb += countVerts(f.geometry);
hb += countHoles(f.geometry);
pb += countParts(f.geometry);
rb += countRings(f.geometry);
for (const r of ringsOf(f.geometry)) if (r.length < 4) degenerate++;
}
const outFeatures = fc.features.map((f) => {
const g = smoothGeom(f.geometry);
return { type: 'Feature', id: f.id, properties: { name: f.properties.name, parent: f.properties.parent }, geometry: g };
});
const out = { type: 'FeatureCollection', features: outFeatures };
// after stats + change count (compare against the original, in order)
let va = 0, ha = 0, pa = 0, ra = 0, changed = 0;
for (let i = 0; i < out.features.length; i++) {
const f = out.features[i];
va += countVerts(f.geometry);
ha += countHoles(f.geometry);
pa += countParts(f.geometry);
ra += countRings(f.geometry);
if (JSON.stringify(f.geometry) !== JSON.stringify(fc.features[i].geometry)) changed++;
}
const afterVerts = new Map(out.features.map((f) => [f.id, countVerts(f.geometry)]));
const json = JSON.stringify(out);
writeFileSync(OUT, json + '\n');
const outBytes = Buffer.byteLength(json) + 1;
console.log(`\n--- RESULTS ---`);
console.log(`regions: ${fc.features.length} → ${out.features.length} (unchanged)`);
console.log(`vertices (incl. ring closure): ${vb} → ${va} removed ${vb - va} (${(100 * (vb - va) / vb).toFixed(2)}%)`);
console.log(`rings: ${rb} → ${ra} parts: ${pb} → ${pa} holes: ${hb} → ${ha} (all unchanged)`);
console.log(`regions whose geometry changed: ${changed} / ${fc.features.length}`);
console.log(`degenerate rings (<4 pts, kept verbatim): ${degenerate}`);
console.log(`file: ${(inBytes / 1024).toFixed(0)} KB → ${(outBytes / 1024).toFixed(0)} KB (${(outBytes / 1024 / 1024).toFixed(2)} MB)`);
// Diagnose the "messy" regions: the real problem is fragmentation — the
// dissolve left coherent regions as many disconnected parts. Rank by part
// count (then total vertices) so these surface; Kocelje/Kavadartsi are here.
const messy = fc.features
.map((f) => ({
id: f.id,
name: f.properties.name,
parent: f.properties.parent,
parts: countParts(f.geometry),
vb: countVerts(f.geometry),
}))
.sort((a, b) => b.parts - a.parts || b.vb - a.vb)
.slice(0, 15);
console.log(`\nMOST-MESSY (top ${messy.length} by fragmentation = parts, then vertices):`);
console.log(` id name country parts verts verts(after)`);
for (const m of messy)
console.log(` ${m.id.padEnd(10)} ${m.name.slice(0, 37).padEnd(38)} ${m.parent.padEnd(7)} ${String(m.parts).padStart(4)} ${String(m.vb).padStart(5)} ${String(afterVerts.get(m.id)).padStart(5)}`);
console.log(`\nwrote ${OUT}; elapsed ${((Date.now() - t0) / 1000).toFixed(1)}s`);
}
main();