Reduce native table startup with bounded subject subscriptions #63

Merged

michielbdejong merged 1 commit from codex/issue-52-native-startup into main

2026-10-20 09:39:56 +02:00

michielbdejong commented

2026-10-19 16:39:28 +02:00

Owner

Keep at most 100 subjects per runtime window, with pageSize: 1 and maxActivePages: 1 on every child subscription. Backend queries still determine complete-collection search, ordering and window membership.
Cancel pending startup and prevent later batches on a page/tab/schema change. Stop watchers and close every started subscription on cancellation or failure, including a stream failure during the final startup yield.
Notify window membership only when row presence or proxy identity changes. Ignore an ordered SDK move of the same row back to index zero; ordinary field edits continue through individual row observers.
Add native correctness/lifecycle tests and pnpm bench:windows, covering first/middle/last windows at 1k/10k/100k records per class. All 27 windows retain the correct 100 subjects. At each size, 100 sustained incoming updates produce exactly 100 row notifications, zero window notifications and a value matching a fresh read.

Measured behavior and tradeoffs

The follow-up end-to-end 100k run, with normal membership polling, records:

Operation	Before	After
Initial readiness	5.17 s	1.91 s
Search	2.03 s	2.13 s
Sort	8.38 s	2.09 s
Next page	4.17 s	1.019 s
Edit to paint	36 ms	32 ms
100 incoming updates, including 20 ms between writes	10.78 s	4.06 s

Each run still retains 100 ORM records and renders 13 grid rows. The verifier retains its complete graph; at 100k the final category tab measured 15.7 MB post-GC V8 heap and 1.64 GB renderer RSS.

More subscriptions add fixed overhead on small graphs: isolated 1k startup increases from 54 ms to about 312 ms, and end-to-end initial readiness from 480 ms to 946 ms. Recorded 1k/10k search, sort and paging remain below one second. These are development-server measurements, not production latency distributions or a supported row-count threshold.

Relates to #52; keep it open. Search and periodic membership queries still run on the main thread. The strict benchmark retains the original targets: 1k and 10k pass, while the three 100k search/sort/Next assertions still fail, including 1018.8 ms for Next. No target has been relaxed.

Behavior, reproduction and raw measurements.

Validation

362 unit tests pass, including pinned native local/incoming updates, bounded membership, deletion/reclassification and disposal.
All six typechecks, twelve production app builds and generated-artifact checks pass.
Full browser regression suite: 79 pass, with four existing skips.
All three window diagnostics pass; strict end-to-end benchmark has the documented 100k latency failures.
Broker-backed replication was not rerun; native tests use an isolated local verifier without a broker.

Opening a bounded 100-record table window at 100k records spent about 3.44 seconds in a single native multi-subject subscription. The runtime now opens one bounded SDK page per subject, starting four per task and yielding between batches. Isolated 100k window startup has a 316 ms median across nine samples, with a maximum 32 ms heartbeat delay while membership polling is paused. - Keep at most 100 subjects per runtime window, with `pageSize: 1` and `maxActivePages: 1` on every child subscription. Backend queries still determine complete-collection search, ordering and window membership. - Cancel pending startup and prevent later batches on a page/tab/schema change. Stop watchers and close every started subscription on cancellation or failure, including a stream failure during the final startup yield. - Notify window membership only when row presence or proxy identity changes. Ignore an ordered SDK move of the same row back to index zero; ordinary field edits continue through individual row observers. - Add native correctness/lifecycle tests and `pnpm bench:windows`, covering first/middle/last windows at 1k/10k/100k records per class. All 27 windows retain the correct 100 subjects. At each size, 100 sustained incoming updates produce exactly 100 row notifications, zero window notifications and a value matching a fresh read. ### Measured behavior and tradeoffs The follow-up end-to-end 100k run, with normal membership polling, records: | Operation | Before | After | | --- | ---: | ---: | | Initial readiness | 5.17 s | 1.91 s | | Search | 2.03 s | 2.13 s | | Sort | 8.38 s | 2.09 s | | Next page | 4.17 s | 1.019 s | | Edit to paint | 36 ms | 32 ms | | 100 incoming updates, including 20 ms between writes | 10.78 s | 4.06 s | Each run still retains 100 ORM records and renders 13 grid rows. The verifier retains its complete graph; at 100k the final category tab measured 15.7 MB post-GC V8 heap and 1.64 GB renderer RSS. More subscriptions add fixed overhead on small graphs: isolated 1k startup increases from 54 ms to about 312 ms, and end-to-end initial readiness from 480 ms to 946 ms. Recorded 1k/10k search, sort and paging remain below one second. These are development-server measurements, not production latency distributions or a supported row-count threshold. Relates to #52; **keep it open**. Search and periodic membership queries still run on the main thread. The strict benchmark retains the original targets: 1k and 10k pass, while the three 100k search/sort/Next assertions still fail, including 1018.8 ms for Next. No target has been relaxed. [Behavior, reproduction and raw measurements](https://git.elfaconsortium.eu/ELFA/database-apps/src/branch/codex/issue-52-native-startup/docs/table-scaling.md). ### Validation - 362 unit tests pass, including pinned native local/incoming updates, bounded membership, deletion/reclassification and disposal. - All six typechecks, twelve production app builds and generated-artifact checks pass. - Full browser regression suite: 79 pass, with four existing skips. - All three window diagnostics pass; strict end-to-end benchmark has the documented 100k latency failures. - Broker-backed replication was not rerun; native tests use an isolated local verifier without a broker.

michielbdejong added 1 commit

2026-10-19 16:39:28 +02:00

Reduce native window startup using bounded subject subscriptions 28a58d6391

michielbdejong referenced this pull request

2026-10-19 16:40:08 +02:00

Bound native/shared table loading and rendering; replace full-collection change serialization #52

michielbdejong merged commit 3017824ca0 into main

2026-10-20 09:39:56 +02:00

michielbdejong referenced this pull request from a commit

2026-10-20 09:39:58 +02:00

Merge native table startup improvements (#63)