Table

A table is a file of row-oriented structured records, mapped into each process that opens it. Use it when an item has a key, fields have different types, and updates should replace an existing record or append a new one. Examples here contain invented identifiers and values, not data from a running application.

Why this class exists

Independent processes often need to reuse the same mutable records. Loading a separate DataFrame in every process duplicates memory and needs another mechanism to propagate updates. Fetching every local read through a server adds transport and serialization. A table instead maps its record file through the operating system's page cache: resident pages can be shared by local processes without a second serialized copy of the whole dataset.

Table supplies the schema, header and coordination; TableDisk owns the local file lifecycle; SharedNumpy exposes record operations; TableIndex maps keys to rows. This separation lets applications work with typed records while storage and index maintenance stay coordinated. It is not a distributed shared-memory system or a general SQL engine, and an in-place row update is not an atomic snapshot of the entire table.

The diagram is a conceptual indexed-write path, not a promise that every lookup is lock-free. Index lookup behavior depends on the reader protocol and configured mode; structural changes and recovery coordinate with readers. Read the reader guarantees before retaining views.

When to choose a table

• Mixed types: identifiers, timestamps, counters and numeric fields in one record.
• Primary-key lookup and insert-or-update semantics rather than a rectangular grid.
• Append-only records when an indexed update path is unnecessary.

For aligned numeric histories, use time series instead. For arbitrary SQL joins or multi-host writers, do not assume the local table protocol provides those capabilities. Latency measurements describe the actual API cost on synthetic, warm-cache data, separately from flush costs.

The classes:

Class	File	Role
Table	src/SharedData/Table.py	Schema, header, create/load, BSON side file, S3 head/tail, locks
TableDisk	src/SharedData/TableDisk.py	The disk file: malloc, reload, trim, delete
SharedNumpy	src/SharedData/SharedNumpy.py	The array SharedData.table() returns. upsert, extend, loc
TableIndex	src/SharedData/TableIndex.py	Hash of the primary key, plus date / symbol / portfolio indexes

SharedData.table() does not return the TableDisk. It returns table.records, a SharedNumpy. The disk object is records.table.

Address and schema

{user}/{database}/{period}/{source}/table/{tablename}

database selects the primary key (src/SharedData/Database.py):

Database	Primary key
TimeSeries	date
Symbols	symbol
Hashs	hash
Accounts	portfolio
MarketData	date, symbol
Text	date, hash
Portfolios	date, portfolio
Relationships	date, symbol, symbol1
Tags	date, tag, symbol
Signals, Risk, Positions	date, portfolio, symbol
Requests	date, portfolio, requestid
Orders	date, portfolio, clordid
Trades	date, portfolio, symbol, tradeid
Options	date, symbol, expiry, strike, callput

period is D1, M15, M1, or RT. An RT table is created with hasindex=False. A new indexed table whose period is shorter than a day must carry a partition in the name. Opening an existing table (no names, formats, value, size, or is_schemaless) does not create one, so an old unpartitioned file stays readable.

Name suffix	Partition
/2026 (4 digits)	yearly
/202601 (6 digits)	monthly
/20260115 (8 digits)	daily

The on-disk layout, from TableDisk.malloc():

{tablename}/data.bin
  [header]   count, recordssize, hasindex, …
  [records]  recdtype × recordssize

Header and records are separate np.memmap views of that file. A local writer is visible to other processes on the same machine through the page cache. flush / write is durability and the S3 upload, not the way another local process sees the row.

Open, create, write

from SharedData.SharedData import SharedData
import pandas as pd

shdata = SharedData(__file__, user='example')

# Open an existing indexed table. Fails with TableNotFound if it is absent.
records = shdata.table('MarketData', 'D1', 'DEMO', 'PRICES/2026')

# Create, or upsert into an existing table, from a frame indexed by the key.
idx = pd.MultiIndex.from_product(
    [pd.date_range('2026-01-01', periods=3, freq='D'), ['EXAMPLE']],
    names=['date', 'symbol'])
frame = pd.DataFrame({'close': [1.1, 1.2, 1.3]}, index=idx)
records = shdata.table(
    'MarketData', 'D1', 'DEMO', 'PRICES/2026',
    value=frame, overwrite=False)

Passing names, formats, and size creates an empty table with that dtype and capacity. value= on a later call to the same path upserts when the table has an index, and extends when it does not. overwrite=True replaces the existing image instead. To grow an existing table explicitly, pass size=bigger, grow=True; size alone remains a creation capacity so routines do not regrow trimmed tables on open. A smaller size never shrinks a table. POST /api/table requests growth when its payload supplies size.

upsert — indexed tables

records.upsert(rows) updates a row whose primary key is already present and inserts the rest. rows may be a DataFrame, a dict, a list of dicts, or a structured array. Rows with an invalid key or a date outside the partition are dropped. Near 99% full, the file grows (at least ~100 MB) and the indexes are remapped. Keep the object upsert returns: after a grow it can be a new SharedNumpy.

loc is read-only. records.loc[key] = ... raises. Writes go through upsert.

extend — tables with no index

records.extend(rows) appends. It raises if hasindex is set. The file grows by about 100 MB when the mapped capacity is short. Realtime (RT) tables use this path.

Schemaless columns

is_schemaless=True keeps the typed columns in the record array and stores extra fields in a BSON side file. Each row holds bson_ptr and bson_size. Table.compact_bson_if_worthwhile() rewrites that file only when the orphaned fraction is worth the lock. A multi-gigabyte file with a few megabytes orphaned is left alone.

Lock

Each table has one mutex, taken by acquire() and released by release(). upsert and extend take it for you (acquire=True). Pass acquire=False only when the caller already holds it. Do not call Logger while you hold it for a long scan: the log handler does file and HTTP I/O. The pattern is on the logging page.

Table.write() fills the header, partitions head and tail, and uploads. TableDisk.reload() fetches and applies a newer remote head or tail; with S3 off or no newer remote copy it leaves local rows alone. Acquiring the table refreshes mappings replaced by another local process. TableDisk.trim() returns unused capacity on supported POSIX platforms, keeping at least two record slots.

HTTP

The same container is GET / POST / PATCH / DELETE /api/table/{database}/{period}/{source}/{tablename}. Bulk bodies are BSON compressed with LZ4. The paths, query parameters, and error codes are in the Swagger spec.

Python callers on another host use SharedData.IO.ClientAPI (ClientAPI.get_table(...)), which speaks that API. A process on the machine that owns the file uses shdata.table() and skips the copy.