# UCE WASM Runtime Architecture

Status: current as of the W7e native-pipeline removal (June 2026). This document
describes the **runtime architecture as built** — the process topology, the
wasm membrane, the unified request dispatch, and the central WebSocket broker.
Native `.so` unit execution/dlopen fallback has been removed; the parser and
preprocessor remain only as the front-end that emits C++ for wasm side-module
compilation.

The guiding principle: **request code never shares an address space or an
allocator with the runtime.** Every unit runs as a WebAssembly module inside a
per-request workspace behind a narrow host membrane. Long-lived connection
state (WebSocket connections, listening HTTP sockets) lives in dedicated native
broker processes that own no unit code — they hold the connection and forward
the actual unit invocation to a clean-engine worker, exactly like a normal
page render.

---

## 1. Process topology

A single native binary (`uce_fastcgi`) forks into a small set of long-lived
roles. None of them is special-cased per request mode; the differences are
purely *which socket a process listens on* and *which function inside a unit
gets invoked*.

```
                          ┌────────────────────────────┐
        nginx  ──FastCGI──►  worker pool (N processes)  │  $FCGI_SOCKET_PATH (example `/run/uce/fastcgi.sock`)
   (port 80 etc.)          │  uniform unit renderers    │  (FastCGI + CLI)
                          └─────────────▲──────────────┘
                                        │ forward render (FastCGI, FCGI_SOCKET_PATH)
                                        │
   browser ──raw HTTP / WS──►  ┌────────┴─────────┐
        (HTTP_PORT 8080)       │   WS broker      │  owns HTTP_PORT + every
                               │  (1 process)     │  WS connection; renders
                               └────────▲─────────┘  nothing itself
                                        │ ws_* command flush
                                        │ (FastCGI, ws-broker.sock)
   on-demand serve_http   ──►  ┌────────┴─────────┐
        (per bind addr)        │ custom-server    │  owns one serve_http
                               │ dispatcher(s)    │  bind addr; forwards to pool
                               └──────────────────┘

   parent process: spawns/respawns all of the above + the proactive compiler.
```

| Process | Owns | Renders units? | Source |
|---|---|---|---|
| **Parent** | nothing; supervises children | no | `main()`, `init_base_process()` |
| **Worker** (×`WORKER_COUNT`) | `FCGI_SOCKET_PATH` (configured socket path; example `/run/uce/fastcgi.sock`) + `CLI_SOCKET_PATH` | **yes** — the only processes that run wasm | `listen_for_connections()` |
| **WS broker** (×1) | `HTTP_PORT` + every live WS connection + `WS_BROKER_SOCKET_PATH` | no — forwards to the pool | `run_ws_broker()` |
| **serve_http dispatcher** (×bind) | one custom-server bind address | no — forwards to the pool | `custom_server_http_dispatcher_loop()` |
| **Proactive compiler** | nothing; pre-compiles units | no | `run_proactive_compiler()` |

**only workers instantiate Wasmtime and run unit code.**
Every connection-owning process (broker, serve_http dispatcher) forwards the
request invocation back to a worker via `FCGI_SOCKET_PATH` using the minimal
FastCGI client in [`src/lib/fcgi_forward.h`](../src/lib/fcgi_forward.h). This is
forced by Wasmtime: an `Engine`/`Store` cannot be safely re-created across
`fork()`, and the brokers fork from the parent that already touched the
runtime. So the brokers hold the long-lived connection and units respond to
events the same way they respond to a page request — through a clean worker.

---

## 2. The membrane and the DValue ABI

Units are compiled to WebAssembly and linked against a host import surface (the
"membrane"). Host and guest exchange structured values as **UCEB** (a compact
binary `DValue` encoding — `ucb_encode`/`ucb_decode`); strings cross as
`std::string` on the native side throughout (no raw `char*` ownership across
the boundary).

- `DValue` is the universal value type: scalars plus an ordered child map
  (`_map`, a `std::map<String,DValue>`). `operator[]` is non-const (creates);
  `.key(k)` is the const probe returning `const DValue*`; `.each(fn)` iterates
  children yielding `const DValue&`.
- The request context (`params`/`get`/`post`/`cookies`/`session`, the raw body
  `in`, and—for WS—the connection context) is marshalled into a single `ctx`
  DValue, UCEB-encoded, and handed to the workspace. The response (body,
  headers, status, and any `meta` such as `ws_commands`) comes back the same
  way.

See [`docs/wasm-phase1-dvalue-abi.md`](wasm-phase1-dvalue-abi.md) for the wire
format details.

Decoder robustness note: UCEB input is untrusted across the wasm membrane. The
UCEB decoder must reject malformed magic/version/varint/length/trailing-data
inputs and excessive nesting with explicit errors, not native or wasm traps. Its
current hard nesting cap is intentionally low (64 levels) to stay well below the
guest stack limit. JSON decoding follows the same rule for malformed strings and
unicode escapes: validate bounds before every read and return an empty/partial
`DValue` rather than reading beyond the input.

---

## 3. Units, handlers, and export naming

A `.uce` unit compiles to a wasm module that exports one symbol per **handler**.
There is no per-mode machinery — a "CLI unit", "WebSocket unit", and "page" are
the same compiled artifact invoked at different exports. The dispatcher passes a
**handler string**; the host maps it to an export symbol:

```
__uce_<base>[_<sanitize(suffix)>]
```

`handler_export_symbol()` (`src/wasm/worker.cpp`) splits on the first `:`:

| Handler string | Export symbol |
|---|---|
| `render` | `__uce_render` |
| `cli` | `__uce_cli` |
| `websocket` | `__uce_websocket` |
| `once` | `__uce_once` (optional; absence is not an error) |
| `serve_http` | `__uce_serve_http` |
| `serve_http:named` | `__uce_serve_http_named` |
| `component:CARD` | `__uce_component_CARD` |
| `exists` | probe only — resolves the unit, loads nothing |

`sanitize_symbol_suffix()` keeps `[A-Za-z0-9_]` (mirrors `ascii_safe_name`).
`wasm_resolve_target(unit, handler)` (`src/wasm/core.cpp`) resolves the source
path and looks up the export's funcref slot; `exists` lets callers probe a unit
without instantiating it.

`wasm_backend_should_handle(request, entry_unit)` checks whether the wasm
backend is initialized and the requested artifact/handler is currently
available. If an artifact is cold or stale, dispatch compiles it on demand via
`get_shared_unit()` and rechecks. There is no native unit-execution fallback.
Workers identify cached unit artifacts by nanosecond mtime, ctime, and size.
Whole-second mtime alone is insufficient because a dependency-triggered rebuild
can replace a wasm artifact within the same second as its prior build.
Cold module compilation and deserialization are host work, so `load_unit()`
refreshes the epoch deadline before its first guest call. Otherwise a component
whose compilation outlasted the guest CPU budget would immediately trap in the
following allocator/relocation call even though no guest loop consumed it.

The proactive compiler and request workers coordinate through per-unit file
locks and a lock-protected demand-priority queue under `BIN_DIRECTORY`. Unit
compilation writes and validates a process-unique temporary wasm file, then
publishes it with an atomic rename. When proactive compilation is enabled,
read-only HTTP requests keep using the last complete artifact while requesting
that stale unit at the head of the compiler queue. Non-read requests never run
a stale entry artifact: they return `503 Service Unavailable` with
`Retry-After: 1`, allowing the client to retry after the priority rebuild.
CLI and explicit compile paths remain synchronous. A failed rebuild removes
availability and surfaces the compiler error rather than serving the old unit
indefinitely. The per-unit lock also keeps concurrent synchronous compilers
from waiting across a transitive graph when a last complete artifact is
available.

Before preprocessing, the compiler verifies that the worker can actually read
the source. An unreadable path is a compile failure with a persisted diagnostic;
it never becomes an apparently valid empty side module. Source signatures mark
unreadable inputs, so correcting access invalidates that failure and permits a
normal retry without changing signatures for ordinary readable files.

---

## 4. The workspace runtime

Each request gets a fresh **workspace** — a per-request wasm instance tree with
the membrane wired in. `wasm_worker_serve(worker, ctx, entry_unit, handler)`
(`src/wasm/worker.cpp`) is the single entry point for *every* mode:

1. Birth a request-scoped workspace with fresh per-request state. Current production workers instantiate a fresh workspace for each request; any future snapshot/CoW optimization must preserve that request-isolation contract.
2. Resolve `entry_unit` + `handler` to an export; components referenced at
   runtime are resolved on demand via the `uce_host_component_resolve` hostcall
   (`component_resolve()` → `__uce_<...>` slot), loading dependency modules
   lazily and recording resolve counts/timings.
3. Invoke the export. Unit code calls host functions (filesystem, sqlite, regex,
   markdown, time, tasks, `ws_*`, …) across the membrane.
4. Collect the response (`WasmResponse`: body, headers, status, `meta`).

Because the workspace owns no process-lifetime state, a unit crash or trap is
contained: it fails the one request and the worker stays healthy. Workers
suspend the native SIGSEGV/SIGILL recovery handler around the wasm call so that
Wasmtime's own trap signals are not escalated into a native fatal signal (see
`serve_via_wasm` in `handle_complete`).

Render workers are long-lived. The generic FastCGI transport retains a legacy
eight-connection recycle default, but `listen_for_connections()` disables it
for the UCE pool: recycling a healthy worker discards its Wasmtime engine and
module cache, adding a visible cold-start request. Request state remains bounded
by the fresh workspace and the normal per-request database/resource cleanup;
faulted workers still exit and are replaced by the parent.

### Task callbacks and workspace lifetime

`task()` and `task_repeat()` are fork-backed. The `uce_host_task_spawn` hostcall
captures the current `WasmWorkspace*`, but `src/lib/sys.cpp::task()` invokes the
captured callback only in the forked child, before the hostcall stack unwinds in
that child. The parent request may return and destroy its workspace; the child
is forked from the parent and keeps a private request-context copy of the per-request workspace. This means a delayed task callback can run after the spawning request
returns without dereferencing the parent's destroyed workspace. It is still a
callback into the inherited child workspace, not a fresh normal request
workspace; avoid adding host resources to `WasmWorkspace` that are invalid across
`fork()` unless task callback handling is changed to birth a fresh workspace.

---

## 5. Request dispatch (`handle_complete`)

`handle_complete()` (`src/linux_fastcgi.cpp`) is the worker's single dispatch
point. It selects a handler string from request params and calls the same
`serve_via_wasm(entry_unit, handler)` lambda for all of them:

```
UCE_WS == "1"            → serve_via_wasm(entry_unit, "websocket")
request.resources.is_cli → serve_via_wasm(entry_unit, "cli")
UCE_SERVE_HTTP == "1"    → serve_via_wasm(entry_unit, "serve_http"[:fn])
otherwise (page)         → serve_via_wasm(entry_unit, "render")
```

The `UCE_*` params are set by whichever broker forwarded the request:

- **Page render**: FastCGI nginx → `FCGI_SOCKET_PATH` directly; no `UCE_*` flags → `render`.
- **CLI**: the CLI socket sets `is_cli`.
- **serve_http**: the custom-server dispatcher sets `UCE_SERVE_HTTP=1` plus
  `UCE_SERVE_HTTP_FUNCTION` and rewrites `SCRIPT_FILENAME` to the configured
  unit (`custom_server_http_complete`).
- **WebSocket**: the WS broker sets `UCE_WS=1` and carries the connection
  identity as `UCE_WS_*` params (see §6). `handle_complete` rebuilds
  `request.resources.websocket_*` and `request.connection` from them before
  invoking `__uce_websocket`.

If a unit still cannot be served after the on-demand wasm compile, the worker
returns a clean 500 with the wasm/compile error; it does not execute native unit
code.

---

## 6. The central WebSocket broker

**One process owns the HTTP port
and every WebSocket connection**, so any unit's `ws_*` call can reach any or all
connections, and a unit-code crash (which happens in a worker) never drops live
connections.

### 6.1 Inbound: a WS frame → a worker render (non-blocking)

`ws_broker_ws_message(request, message, opcode)` fires when a complete
(reassembled) WS message arrives on a connection. It does **not** block the
broker loop:

1. Build FastCGI params: `SCRIPT_FILENAME`, `REQUEST_METHOD=GET`, a
   `REQUEST_URI` (required — `handle_request()` rejects requests without one
   *before* `on_complete` runs), `UCE_WS=1`, and the connection context as
   `UCE_WS_CONNECTION_ID / SCOPE / OPCODE / BINARY / CONNECTIONS / STATE`.
2. The message rides as `UCE_WS_MESSAGE` (base64) with an **empty STDIN body** —
   a non-empty STDIN makes the FastCGI transport flush a premature response
   before `on_complete` ever runs.
3. Connect to `FCGI_SOCKET_PATH` (non-blocking) and queue the encoded request in
   `ws_broker_outbound[fd]` with an enqueue timestamp.

`ws_broker_drain_outbound()` runs after every `process(50)` tick: it finishes
writing each queued request, then drains and discards the reply (the unit's
output comes back via the command socket, not this reply), closing the fd when
the worker closes its end. If a forward remains pending beyond
`WS_BROKER_OUTBOUND_TIMEOUT_SECONDS` (default `30`), the broker drops and
closes it so a wedged worker cannot pin broker fds/memory indefinitely.

### 6.2 Outbound: `ws_*` commands flushed back to the broker

Any unit code — not just WebSocket handlers — may call `ws_send` / `ws_send_to`
/ `ws_close`. In the workspace these **record dispatch commands** rather than
touching a socket (the workspace owns no connections); `wasm-core`'s `ws_*`
(`src/lib/sys.cpp`) append to `websocket_dispatch_commands`, and
`finish_response_meta` (`src/wasm/core.cpp`) emits them as `ws_commands`.
If the handler changed per-connection state, the core also emits
`ws_connection_state` even when no commands were emitted, and the native
backend flushes this state-only batch to the broker.

`wasm_backend_serve` (`src/wasm/backend.cpp`) flushes that batch at workspace
teardown — in **any** scenario, not just WS handlers — to the broker's command
socket (`WS_BROKER_SOCKET_PATH`, `/run/uce/ws-broker.sock`) via
`fcgi_forward_request` with `UCE_WS_DISPATCH=1`. This is the only path WS data
takes out of a workspace.

`ws_broker_apply_commands()` decodes the batch and applies each command against
the full registry it owns: `broadcast` (by scope), `send_to` (by connection id),
`close`. If the batch carries `connection_state`, it persists that onto the
matching live connection's `websocket_state`.

### 6.3 Un-upgraded HTTP on the WS port

The WS port can also receive ordinary (non-Upgrade) HTTP requests.
`ws_broker_complete()` routes by param: `UCE_WS_DISPATCH=1` → apply commands;
otherwise → `forward_request_to_worker()` — the *same* shared facility the
serve_http dispatcher uses, so there is no duplicated request-forwarding code.

### 6.4 The broker loop

`run_ws_broker()` drops the worker listeners it inherited
(`close_inherited_server_sockets`), installs permissive `on_request`/`on_data`
handlers (it renders nothing, so it accepts every request straight through to
`on_complete`), wires `on_complete=ws_broker_complete` and
`on_websocket_message=ws_broker_ws_message`, listens on `HTTP_PORT` and the
command socket, and loops `process(50)` + `drain_outbound(timeout)`. The
`timeout` comes from `WS_BROKER_OUTBOUND_TIMEOUT_SECONDS` (default `30`). The
design is **non-blocking outbound dispatch + async command-socket flush, all in
the broker's single epoll loop** — the broker never blocks on a worker.

The parent respawns the broker if it dies (`ws_broker_alive` / `ensure_ws_broker`
in `main()`). 

---

## 7. The serve_http facility

`serve_http` units are reachable on their own bind address via a custom-server
dispatcher (`custom_server_http_dispatcher_loop`). The dispatcher owns the bind
socket and, on each request, sets `UCE_SERVE_HTTP=1` + the configured unit/
function and calls `forward_request_to_worker()`. The worker then runs
`serve_via_wasm(entry_unit, "serve_http"[:fn])`. This is the same
hold-connection-forward-render model as the WS broker, sharing
`forward_request_to_worker` and `fcgi_forward.h`.

---

## 8. Build / object layout

The native side is split into separately-compiled objects so editing the wasm
runtime does not recompile the whole TU (`scripts/build_linux.sh`):

| Object | Contents |
|---|---|
| `bin/sqlite3.o` | sqlite amalgamation (cached) |
| `bin/wasm.o` | `src/wasm/wasm_module.cpp` (backend.cpp + worker.cpp + wasmtime) |
| `bin/main.o` | `src/linux_fastcgi.cpp` (includes `lib/uce_lib.cpp`) |

Linked into one `-rdynamic` binary. ODR hazards across the objects are handled
deliberately: `context` and `my_pid`/`parent_pid` are `extern` (guarded for the
wasm core vs. unit builds), `operator new`/`delete` live in `types.cpp`, and
header free-functions are `inline`. The wasm backend exposes only declarations
(`src/wasm/backend.h`) to `main.o`.

---

## 9. Configuration keys

| Key | Default | Meaning |
|---|---|---|
| `WASM_BACKEND_VERBOSE` | `0` | Emit `X-UCE-Wasm-*` workspace timing headers (benchmark only). |
| `FCGI_SOCKET_PATH` | runtime-configured (`/run/uce/fastcgi.sock` in this doc) | Worker pool FastCGI socket (brokers forward here). |
| `CLI_SOCKET_PATH` | `/run/uce/cli.sock` | Worker CLI/admin socket. Keep private; reference `CLI_SOCKET_MODE` is `0600`. |
| `FCGI_SOCKET_MODE` | `0666` | Permission mode applied to `FCGI_SOCKET_PATH` after bind; set tighter if nginx/Apache can use a trusted group. |
| `CLI_SOCKET_MODE` | `0600` | Permission mode applied to `CLI_SOCKET_PATH`; set `0660` only for a trusted admin group. |
| `HTTP_PORT` | `8080` | Raw HTTP + WebSocket port — owned by the WS broker. |
| `WS_BROKER_SOCKET_PATH` | `/run/uce/ws-broker.sock` | Broker command socket for `ws_*` flushes. |
| `WS_BROKER_OUTBOUND_TIMEOUT_SECONDS` | `30` | Max lifetime in seconds for queued WS broker forwards before drop. |
| `WORKER_COUNT` | `4` | Number of uniform worker processes. |

---

## 10. Testing

- **Regression gate**: `scripts/run_cli_tests.sh --include-wasm-kill` runs the
  in-runtime CLI suite (`site/tests/cli_runner.uce`) plus the site test pages and
  `scripts/test_dependency_invalidation.sh`. The latter changes a transitive
  `#load`, then replaces a warmed worker artifact while preserving its
  whole-second mtime to prove both compiler and worker caches invalidate it. It
  also rejects an unreadable unit without publishing a wasm artifact, restores
  its permissions and proves the next CLI request compiles it, then sends 48
  requests and asserts the observed worker PID set does not exceed
  `WORKER_COUNT`, guarding against accidental reintroduction of request-count
  recycling. It also races a source edit against an active unit compile after
  the generated C++ snapshot has been written. The compiler must either retry
  and serve the post-edit source or fail closed; it must never stamp current
  source metadata onto wasm produced from an older snapshot.
  `scripts/test_cold_component_deadline.sh` separately compiles a deliberately
  cold component that exceeds the development epoch window and proves the
  parent request still renders it. The dependency-invalidation gate also holds
  parent and child compile locks across a transitive source edit, proves a
  warmed HTTP request returns the last atomically published result without
  waiting, proves a POST returns a prompt retryable 503 without executing its
  old mutation handler, and verifies demand-priority convergence. A CLI request
  still waits and returns only the current dependency result after rebuild
  completion.
  The component page also calls a parent once without its optional relative
  child and again with that child activated. This guards the component-slot
  cache invariant that a cache hit must restore both the function-table slot
  and the resolved unit path; otherwise nested relative targets resolve from
  the caller after a repeated parent invocation.

- **WebSocket end-to-end**: a headless client performs a raw WS handshake to
  `:HTTP_PORT` with path `/site/tests/websockets.ws.uce` (self-resolving
  `SCRIPT_FILENAME`) and asserts the `hello-ack` frame — exercising the full
  broker → worker → broker → client chain across process boundaries.
