Bound native/shared table loading and rendering; replace full-collection change serialization #52
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Problem and affected apps
The native table editor has unbounded client-side work as collections grow. This is in the shared application code, not evidence of a fundamental NextGraph engine limit. It also affects the main Atomic-labelled expense editor: both apps call
RuntimeExpenseEditor, which passesmode="native"toRuntimeSchemaTables.Verified against main source matching commit
be8938078d97a70519a991ade8b9207d07067c5f.Current behavior
RuntimeSchemaTables.tsx,Projectionserializes the complete subscribed row collection withJSON.stringifyduring render and uses that string as an effect dependency. A single-field update can therefore traverse and allocate a representation of the whole collection. Cost follows serialized data size, roughly rows × fields for uniformly sized rows.useRuntimeShape.tsopens a graph/shape subscription without ordering or paging limits.RuntimeSchemaTablesrepeatedly materializes row arrays, rebuilds record wrappers and relationship choices, and callsRuntimeAtomicStore.ingest()across the loaded groups. The Atomic mirror is instantiated and ingested even in native mode.TableWorkspace.tsxfilters and sorts loaded records in the browser and keeps all table panels mounted, including inactive tabs.SchemaTable.tsxrenders every matching row without virtualization. The RDF inspector also constructs all its rows even when its details element is collapsed.Consequently, work and memory follow loaded records and mounted tables rather than the viewport or changed rows. Removing only
JSON.stringifywill leave the other bottlenecks.Requested direction
pageSizeandmaxActivePages; its bounded-page behavior for incoming inserts requires explicit handling/testing. Use the appropriate SDK insertion/deletion APIs for ordered projections.Reproduction and acceptance evidence
Related: #51 addresses Atomic table membership and collection discovery. Keep that correctness fix distinct from this shared/native scalability work, while aligning the bounded collection interfaces.
Implementation and remaining latency work (19 October 2026)
PR #61 is merged. Runtime tables retain one 100-record page; the shared grid virtualizes rows, preserves drafts/focus and the #57/#59/#60 reconciliation behavior, and defers inactive tables, inspectors and relationship loading. Native boundary/update tests and 1k/10k/100k benchmark evidence are included.
Keep this issue open for the remaining performance work. The original sub-100ms editing and sub-second query/navigation timings remain measurement goals; they are not grounds to discard the bounded-loading improvements or to declare a supported capacity. Final latency gates need repeated production-build measurements on representative hardware.
The isolated startup profile sampled 54 correct bounded subscriptions. At 100k records per class, a 100-subject ordered subscription has median readiness of 3.44 seconds, about 1.4ms JavaScript row setup, and about 3.43 seconds of main-thread heartbeat delay. Unordered readiness is still 1.77 seconds; the first single-subject case is about 3ms. This prioritizes native multi-subject evaluation and query planning without establishing a particular scan algorithm or a graph-size limit.
Remaining work:
See reproduction and measurements, including
pnpm bench:tablesandpnpm bench:subscriptions. Broker-backed replication remains outside the local native benchmark's evidence.Follow-up native startup optimization
PR #63 was merged on 20 October 2026 after the measured performance and small-dataset tradeoff were accepted. It uses bounded single-subject SDK pages, with four startups per task, rather than one native 100-subject subscription. The application window still retains 100 records and renders 13 grid rows. Pending reads, later batches and watchers are canceled/closed together when a window changes or fails.
At 100k, isolated window startup has a 316ms median across nine samples at first/middle/last positions; maximum heartbeat delay is 32ms while membership polling is paused. All 27 window cases at 1k/10k/100k pass. At every size, 100 incoming updates generate exactly 100 row notifications and zero whole-window notifications, and agree with a fresh read.
The follow-up end-to-end 100k run improves initial readiness from 5.17s to 1.91s, sort from 8.38s to 2.09s, and Next from 4.17s to 1.019s. Search remains 2.13s. Small graphs pay additional subscription overhead: 1k initial readiness is 946ms versus the original 480ms, while measured 1k/10k search/sort/paging stay below one second.
This PR does not complete #52 or relax its original goals. The strict benchmark still fails the three 100k search/sort/Next assertions. Complete-collection query/polling costs and main-thread blocking remain follow-up work; final latency gates still need representative production measurements. The verifier continues to retain its full graph.
Validation: 362 unit tests, 79 browser tests with four existing skips, six typechecks, twelve production app builds and generated-artifact checks pass. Broker-backed replication was not rerun. Reproduction and raw measurements are included in the PR's
docs/table-scaling.mdanddocs/benchmarks/changes.Proposed first step in #58: use the published
useDeepSignalReact adapter inuseRuntimeShapeand remove the complete row-collection JSON snapshot fromProjection. React dependencies now follow changing proxy identities, while graph/schema subscription replacement and error handling are preserved.All 332 unit tests, six typechecks, 12 production builds and 28 browser tests passed locally. New browser coverage checks incoming scalar edits, nested set changes and row creation/deletion through distinct actual ORM subscriptions using the test engine.
This addresses the serialization/React bridge portion only. Bounded loading/rendering, relationship options, Atomic ingestion and performance measurements remain outstanding, so #52 stays open.