ELFA/database-apps
15
0
Fork 0
Code Issues 5 Pull requests 1 Projects Releases Packages Wiki Activity

Bound runtime table projections and virtualize rows #61

Merged
michielbdejong merged 7 commits from codex/issue-52-bounded-tables into main 2026-10-19 16:11:53 +02:00
Conversation 0 Commits 7 Files changed 32 +2328 -181
michielbdejong commented 2026-10-19 15:21:21 +02:00
Owner
Copy link

Runtime table editors subscribed to every class and rendered every row. The active runtime table now retains one 100-record SDK page and the shared grid virtualizes its rows. At 1k, 10k and 100k records per class, the end-to-end benchmark retained 100 ORM records and mounted 13 grid rows.

  • Search and deterministic ordering run over the complete graph/class before loading a finite subject window. Membership refresh discovers incoming insertions, deletions and ordering changes; ordinary field changes notify their own rows.
  • Drafts, baselines, invalid text, errors and focus survive virtual rows, pages and tabs. The integration preserves #57 conflict reconciliation, #59 removed-record editors and #60 live schema refresh, including incompatible-field recovery.
  • Inactive tabs unmount, inspectors build content on demand, native mode avoids Atomic mirrors, and relationship choices/labels use bounded queries.
  • A finite native-page normalization fixes the pinned verifier's mismatch between equal-key initial ordering and later numeric patch indices.

Scope and remaining performance work

Relates to #52; leave the issue open. This PR accepts the bounded-loading improvements independently of the remaining 100k latency work. It does not claim the original sub-second query/navigation goal is met or establish a supported row-count threshold.

The end-to-end 100k run measured 36 ms edit-to-paint, 2.0 s search, 8.4 s sort and 4.2 s Next. The benchmark keeps asserting the original targets and fails those three 100k latency checks. Native queries can block editing despite the sampled edit latency.

The new pnpm bench:subscriptions diagnostic isolates 1/10/100-subject subscriptions at all three graph sizes, with three consecutive samples per case. At 100k, a 100-subject ordered subscription takes about 3.44 s, almost entirely before its initial message; JavaScript row setup takes about 1.4 ms. Its main-thread heartbeat is delayed about 3.43 s. An unordered window still takes about 1.77 s, while a one-subject subscription takes about 3 ms. These results prioritize native multi-subject evaluation/query planning and main-thread responsiveness. Final latency gates need repeated production-build measurements on representative hardware.

Behavior, reproduction, raw end-to-end measurements and startup samples are documented in docs/table-scaling.md and docs/benchmarks/. The verifier retains its full graph: at 100k the end-to-end run measured 15.7 MB post-GC V8 heap and 1.6 GB renderer RSS.

Validation

  • 358 unit tests pass, including actual pinned WASM updates and page-boundary insertion, deletion, reclassification, reordering and complete-collection search.
  • 18 focused browser scenarios pass, including membership, live schema refresh and conflict reconciliation.
  • All six typechecks, twelve production builds and generated-artifact checks pass.
  • Browser suite: 78 pass and four existing skips in the integrated run. Its remaining inspector scenario was interrupted by generator-triggered HMR, then passed three consecutive runs against frozen source. All 79 browser scenarios are validated.
  • Subscription profile: all 54 sampled subscriptions returned the correct subjects and retained their requested window. The end-to-end benchmark's known 100k latency failures remain documented.
  • Broker-backed replication was not rerun; the native harness uses an isolated local verifier without a broker.
Runtime table editors subscribed to every class and rendered every row. The active runtime table now retains one 100-record SDK page and the shared grid virtualizes its rows. At 1k, 10k and 100k records per class, the end-to-end benchmark retained 100 ORM records and mounted 13 grid rows. - Search and deterministic ordering run over the complete graph/class before loading a finite subject window. Membership refresh discovers incoming insertions, deletions and ordering changes; ordinary field changes notify their own rows. - Drafts, baselines, invalid text, errors and focus survive virtual rows, pages and tabs. The integration preserves #57 conflict reconciliation, #59 removed-record editors and #60 live schema refresh, including incompatible-field recovery. - Inactive tabs unmount, inspectors build content on demand, native mode avoids Atomic mirrors, and relationship choices/labels use bounded queries. - A finite native-page normalization fixes the pinned verifier's mismatch between equal-key initial ordering and later numeric patch indices. ### Scope and remaining performance work Relates to #52; **leave the issue open**. This PR accepts the bounded-loading improvements independently of the remaining 100k latency work. It does not claim the original sub-second query/navigation goal is met or establish a supported row-count threshold. The end-to-end 100k run measured 36 ms edit-to-paint, 2.0 s search, 8.4 s sort and 4.2 s Next. The benchmark keeps asserting the original targets and fails those three 100k latency checks. Native queries can block editing despite the sampled edit latency. The new `pnpm bench:subscriptions` diagnostic isolates 1/10/100-subject subscriptions at all three graph sizes, with three consecutive samples per case. At 100k, a 100-subject ordered subscription takes about 3.44 s, almost entirely before its initial message; JavaScript row setup takes about 1.4 ms. Its main-thread heartbeat is delayed about 3.43 s. An unordered window still takes about 1.77 s, while a one-subject subscription takes about 3 ms. These results prioritize native multi-subject evaluation/query planning and main-thread responsiveness. Final latency gates need repeated production-build measurements on representative hardware. Behavior, reproduction, raw end-to-end measurements and startup samples are documented in `docs/table-scaling.md` and `docs/benchmarks/`. The verifier retains its full graph: at 100k the end-to-end run measured 15.7 MB post-GC V8 heap and 1.6 GB renderer RSS. ### Validation - 358 unit tests pass, including actual pinned WASM updates and page-boundary insertion, deletion, reclassification, reordering and complete-collection search. - 18 focused browser scenarios pass, including membership, live schema refresh and conflict reconciliation. - All six typechecks, twelve production builds and generated-artifact checks pass. - Browser suite: 78 pass and four existing skips in the integrated run. Its remaining inspector scenario was interrupted by generator-triggered HMR, then passed three consecutive runs against frozen source. All 79 browser scenarios are validated. - Subscription profile: all 54 sampled subscriptions returned the correct subjects and retained their requested window. The end-to-end benchmark's known 100k latency failures remain documented. - Broker-backed replication was not rerun; the native harness uses an isolated local verifier without a broker.
michielbdejong changed title from WIP: Bound runtime table projections and virtualize rows to Bound runtime table projections and virtualize rows 2026-10-19 16:11:37 +02:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
ELFA/database-apps!61
No description provided.