Record readers after a disk rewrite

Disk sort and trim publish a complete data.bin by atomic rename. Existing mmaps still refer to the previous inode. All cooperating clients must run the current ownership, recovery and reader-refresh protocol before publication is enabled. Disabling SHAREDDATA_LOCKFREE_READ does not provide compatibility with older clients: that flag controls index lookups only.

Historical upgrade context is in the changelog.

SharedNumpy slicing, records, count, recordssize, mtime, minchgid, next_bson_ptr and df now use a records-only read view. This view has its own header and record maps opened through one file descriptor. Readers publish only a single bundle reference; they never attach or remap a writer's Table. Two Table objects for one path maintain independent, EPOCH-validated bundles.

Stable non-owner reads check the always-mapped mutex EPOCH and the mapped capacity, with no path syscalls and no writer acquire. STRUCT/DIRTY can span the rename before the final EPOCH increment, including a writer death immediately after rename. Every read in these modes drops the superseded stable cache, reopens the current path and checks its device/inode. An overlapping read may retain its image, but the next read validates again. Four racing attempts raise RewriteError("record image changed repeatedly; retry read"); the reader never spins indefinitely or falls back to an old image. This bounds retries, not filesystem I/O latency. Plain reads do not repair indexes or take the exclusive write lock.

Old returned NumPy/DataFrame views remain valid while referenced. They are views of that image, not subscriptions. Call the SharedNumpy interface again for fresh data. Raw np.asarray(records), .view(np.ndarray), cached Table.hdr, cached pointers, and numeric locs held across a rewrite are outside this refresh API. HTTP GET, DataAPI table reads and .loc first wait out STRUCT/DIRTY as an index lookup does (capped waiters; a dead or released holder's DIRTY is repaired through acquire), so a sort or trim delays these reads rather than failing them. They retry the entire lookup and materialization up to four times when EPOCH changes, then raise with the last error chained. A lock holder inside its own STRUCT/DIRTY reads its writer image directly. This closes the new flag-off lookup-to-fresh-slice generation race on those paths. Numeric positions retained by external callers still require their own generation check. Row updates within one image are not atomic snapshots. Writable array views retain the existing access permissions but are not a supported substitute for locked upsert/extend/insert.

Held aliases refresh before writes, lookups, plain reads and decorated structural operations. An operation already in STRUCT skips recursive recovery. The per-thread ownership token still dies with Python thread state; dead-thread takeover is deliberately unsupported. A C caller must release in the same Python thread state in which it acquired.

Consumers and refresh paths

The repository's long-lived record pollers are IO/SyncTable.py (init_client, get_ids2send, socket/websocket publish loops), IO/ClientAPI.py (subscribe polling and HTTP publishing), and IO/ClientSocket.py (count/mtime handshake). IO/ServerSocket.py and IO/ServerWebSocket.py expose metadata and dispatch SyncTable publishers. These use the refreshed SharedNumpy properties/slicing without requiring changes at each call site. Socket and websocket payload selection shares the same tested get_ids2send path.

API/routes/tables.py GET without key filters (last, since, full scans) can keep a Table cached in SharedData.data and never acquire again. Its HEAD path previously accessed tbl.table.hdr directly; it now copies the current read header. Filtered GET and DataAPI/readers.py::read_table additionally use index lookups. Both HTTP GET and DataAPI check lookup and materialization together. SharedData.load_table metadata polling uses the refreshed properties; its hasindex field is invariant across sort/trim. SharedNumpy.df now invalidates its mmap-backed frame when the record address or count changes, including same-count sorts. Public callers retaining SharedNumpy references are covered by the same paths. Other raw record/index mmap accesses in this repository are writer/recovery internals under ownership. Cached raw pointers in other repositories/native code require a separate client inventory.

Platform limits

Windows disk-table use for these paths is claimed but unvalidated: the package declares OS independence, the quick start and development docs include Windows, and the repository ships Windows shared-mutex code, but the crash-safe record-replacement path has not been exercised on native Windows.

fsync_directory is a no-op on Windows. sync_file opens with write access there because Windows file flushing requires it. Neither behavior establishes Windows directory-entry power-loss durability or proves cold-open/rebuild compatibility: native Windows execution of these paths is unvalidated, including existing mapped index replacement sharing restrictions.

On non-Linux, ordinary SharedNumpy.write() skips automatic disk record sorting and saves the unsorted image. Explicit sort/trim and pending-rewrite recovery require Linux and the publisher's POSIX APIs. Refusal of an explicit structural operation can leave derived index flags invalid and mode DIRTY; it preserves record bytes. No crash-safe record publication guarantee is made off Linux.

Owner property reads call _sync_held_mapping, including a pending-descriptor existence check (a path stat) and possible remap, several times during upsert. Only stable non-owner reads avoid path syscalls. All reads currently consult the writer-lock registry, whose setdefault constructs a lock even for an existing entry. These costs and the exceptional DIRTY reopen cost require measurements.

A never-acquiring Table retains its original writer mmap: at most one deleted record generation can remain pinned until acquire/free, with disk usage not visible to du. Raw table.records.size and raw views retain old capacity. Disk read slices/records return np.ndarray, including with the index flag off; external dependencies on the previous SharedNumpy subclass are unverified.

Linux process-kill tests cannot prove host/power-loss durability. A disposable target-filesystem power-loss lab and coordinated old/native client upgrades remain release gates. No production lifecycle operation is part of this patch.