Orchestration: routines, schedules and the worker pool

SharedData pairs the mapped store with a control plane that runs the code producing and consuming the data. A routine is any program the Scheduler owns: a Python script, a module, or an external process that only reports in. A schedule is the list of routines that should be alive or should run during a day.

The routine model

A routine's identity is user@COMPUTER:Repo/folder/code, or user@COMPUTER:SharedData.Module for the modules shipped with the package. A schedule row adds what the Scheduler needs to run it:

Field	Meaning
name	Group shown on the SCHED page; routines with the same name roll up
computer	user@COMPUTER the routine belongs to
isrealtime	Long-running daemon (restarted when it dies) vs. a timed batch run
runtimes	HH:MM start times for batch routines, several separated by commas
dependencies	Routines in the same schedule, comma-separated; batch dependencies must complete, realtime dependencies must be running
isexternal	Runs where no worker listens; reports status over the API
args	Command-line arguments; secret-looking values are masked on pages
seq	Stored row order; daily run order is computed from run times and dependencies

Definitions and daily state are ordinary SQLite containers in the store — read where schedules live for the exact layout and editing rules.

Two kinds of routine

Realtime routines run continuously: feed collectors, stream consumers, API servers. They heartbeat through the logging plane; a routine that goes silent is detected and restarted. Batch routines start at their run times and run to completion: downloads, parsers, consolidations, analytics, backups.

Dependencies order programs within the same schedule, including programs in different source trees or on different machines. Batch routines wait for their batch dependencies to reach COMPLETED; realtime routines wait for their realtime dependencies to be RUNNING before starting or restarting. Dependencies between the two kinds are not enforced at runtime. That is how a download, its parser, the derived dataset and the report that consumes it stay ordered without any of them knowing about the others.

Batch routines marked ERROR or DELAYED are not automatically retried. Completing a scheduled batch instance does not subscribe it to later input changes. Realtime restarts depend on heartbeat age and startup grace periods (45 seconds to five minutes), rather than an immediate process-death signal.

The Scheduler's tick

The Scheduler is itself a routine. Every few seconds it walks each schedule and does two things: work out how every routine is doing (from what the routines logged), then send the commands that are due.

Commands travel through the worker pool. One worker per host listens for commands addressed to user@COMPUTER, validates the requested source revision — an incomplete or inconsistent installation is refused rather than launched — and starts the process with the routine's arguments. Scheduled routines emit lifecycle messages and heartbeats to the logging plane; workers also report launch failures and abnormal exits there. The Scheduler derives the day's state from those logs. Batch-job results are reported through post_workerpool. See worker repository synchronization for the validation rules and logging for what a routine may log.

A schedule can be switched to observe-only mode: the Scheduler keeps tracking status but sends no commands, which is how a new Scheduler can shadow a running one. See the control switch.

External routines

An external routine runs where no worker listens — for example a terminal application on a host outside the pool, or a process started by hand. It appears in the same schedule, reports status through the API, and is monitored like any other routine; it is just never started by the pool.

One-off work: batch jobs

Schedules cover work that recurs. For a one-off command — run this job once on a worker with an available batch slot — post_workerpool with job='batch' posts a batch job. Jobs have their own statuses (NEW/WAITING -> PENDING -> FETCHED -> RUNNING -> COMPLETED, with ERROR and CANCEL as other outcomes) and dependencies. Workers opt in with --batchjobs N; the SharedData.Routines.UpdateActiveJobs routine promotes due jobs whose dependencies completed. The monthly SQLite job store is opt-in; until migrated, jobs use the legacy table. See batch jobs.

The whole deployment watches itself

Schedule state and batch-job results are containers in the same store they operate on; heartbeats and lifecycle messages arrive through the logging plane. The SCHED page shows today's routines per schedule, grouped or filtered, with restart and stop controls that go through the same command path the Scheduler uses. Because the state is data, anything permitted to read the containers — an HTTP client, a script, an MCP agent — can answer "what ran, what failed, what is still waiting" without a separate monitoring system.

Limits

• The Scheduler runs on the API server's host and keeps its containers on local disk. Workers on remote hosts receive its commands through the API; running the Scheduler itself on a different host is not supported.
• Direct worker commands expire after 60 seconds and are removed from the queue when fetched, before execution. There is no exactly-once execution guarantee. Run one controlling Scheduler per schedule and design routines to be restartable: a routine killed mid-write leaves whatever it wrote.
• Batch dependencies order completion, not correctness — a dependency that completes with partial data still releases its dependents. Validate inputs inside the routine.
• Mapped-page sharing is per host; routines on different machines coordinate through the API, streams and the store's containers, not shared memory.