# UCE WASM Runtime Architecture Status: current as of the W7e native-pipeline removal (June 2026). This document describes the **runtime architecture as built** — the process topology, the wasm membrane, the unified request dispatch, and the central WebSocket broker. Native `.so` unit execution/dlopen fallback has been removed; the parser and preprocessor remain only as the front-end that emits C++ for wasm side-module compilation. The guiding principle: **request code never shares an address space or an allocator with the runtime.** Every unit runs as a WebAssembly module inside a per-request workspace behind a narrow host membrane. Long-lived connection state (WebSocket connections, listening HTTP sockets) lives in dedicated native broker processes that own no unit code — they hold the connection and forward the actual unit invocation to a clean-engine worker, exactly like a normal page render. --- ## 1. Process topology A single native binary (`uce_fastcgi`) forks into a small set of long-lived roles. None of them is special-cased per request mode; the differences are purely *which socket a process listens on* and *which function inside a unit gets invoked*. ``` ┌────────────────────────────┐ nginx ──FastCGI──► worker pool (N processes) │ $FCGI_SOCKET_PATH (example `/run/uce/fastcgi.sock`) (port 80 etc.) │ uniform unit renderers │ (FastCGI + CLI) └─────────────▲──────────────┘ │ forward render (FastCGI, FCGI_SOCKET_PATH) │ browser ──raw HTTP / WS──► ┌────────┴─────────┐ (HTTP_PORT 8080) │ WS broker │ owns HTTP_PORT + every │ (1 process) │ WS connection; renders └────────▲─────────┘ nothing itself │ ws_* command flush │ (FastCGI, ws-broker.sock) on-demand serve_http ──► ┌────────┴─────────┐ (per bind addr) │ custom-server │ owns one serve_http │ dispatcher(s) │ bind addr; forwards to pool └──────────────────┘ parent process: spawns/respawns all of the above + the proactive compiler. ``` | Process | Owns | Renders units? | Source | |---|---|---|---| | **Parent** | nothing; supervises children | no | `main()`, `init_base_process()` | | **Worker** (×`WORKER_COUNT`) | `FCGI_SOCKET_PATH` (configured socket path; example `/run/uce/fastcgi.sock`) + `CLI_SOCKET_PATH` | **yes** — the only processes that run wasm | `listen_for_connections()` | | **WS broker** (×1) | `HTTP_PORT` + every live WS connection + `WS_BROKER_SOCKET_PATH` | no — forwards to the pool | `run_ws_broker()` | | **serve_http dispatcher** (×bind) | one custom-server bind address | no — forwards to the pool | `custom_server_http_dispatcher_loop()` | | **Proactive compiler** | nothing; pre-compiles units | no | `run_proactive_compiler()` | **only workers instantiate Wasmtime and run unit code.** Every connection-owning process (broker, serve_http dispatcher) forwards the request invocation back to a worker via `FCGI_SOCKET_PATH` using the minimal FastCGI client in [`src/lib/fcgi_forward.h`](../src/lib/fcgi_forward.h). This is forced by Wasmtime: an `Engine`/`Store` cannot be safely re-created across `fork()`, and the brokers fork from the parent that already touched the runtime. So the brokers hold the long-lived connection and units respond to events the same way they respond to a page request — through a clean worker. --- ## 2. The membrane and the DValue ABI Units are compiled to WebAssembly and linked against a host import surface (the "membrane"). Host and guest exchange structured values as **UCEB** (a compact binary `DValue` encoding — `ucb_encode`/`ucb_decode`); strings cross as `std::string` on the native side throughout (no raw `char*` ownership across the boundary). - `DValue` is the universal value type: scalars plus an ordered child map (`_map`, a `std::map`). `operator[]` is non-const (creates); `.key(k)` is the const probe returning `const DValue*`; `.each(fn)` iterates children yielding `const DValue&`. - The request context (`params`/`get`/`post`/`cookies`/`session`, the raw body `in`, and—for WS—the connection context) is marshalled into a single `ctx` DValue, UCEB-encoded, and handed to the workspace. The response (body, headers, status, and any `meta` such as `ws_commands`) comes back the same way. See [`docs/wasm-phase1-dvalue-abi.md`](wasm-phase1-dvalue-abi.md) for the wire format details. Decoder robustness note: UCEB input is untrusted across the wasm membrane. The UCEB decoder must reject malformed magic/version/varint/length/trailing-data inputs and excessive nesting with explicit errors, not native or wasm traps. Its current hard nesting cap is intentionally low (64 levels) to stay well below the guest stack limit. JSON decoding follows the same rule for malformed strings and unicode escapes: validate bounds before every read and return an empty/partial `DValue` rather than reading beyond the input. --- ## 3. Units, handlers, and export naming A `.uce` unit compiles to a wasm module that exports one symbol per **handler**. There is no per-mode machinery — a "CLI unit", "WebSocket unit", and "page" are the same compiled artifact invoked at different exports. The dispatcher passes a **handler string**; the host maps it to an export symbol: ``` __uce_[_] ``` `handler_export_symbol()` (`src/wasm/worker.cpp`) splits on the first `:`: | Handler string | Export symbol | |---|---| | `render` | `__uce_render` | | `cli` | `__uce_cli` | | `websocket` | `__uce_websocket` | | `once` | `__uce_once` (optional; absence is not an error) | | `serve_http` | `__uce_serve_http` | | `serve_http:named` | `__uce_serve_http_named` | | `component:CARD` | `__uce_component_CARD` | | `exists` | probe only — resolves the unit, loads nothing | `sanitize_symbol_suffix()` keeps `[A-Za-z0-9_]` (mirrors `ascii_safe_name`). `wasm_resolve_target(unit, handler)` (`src/wasm/core.cpp`) resolves the source path and looks up the export's funcref slot; `exists` lets callers probe a unit without instantiating it. `wasm_backend_should_handle(request, entry_unit)` checks whether the wasm backend is initialized and the requested artifact/handler is currently available. If an artifact is cold or stale, dispatch compiles it on demand via `get_shared_unit()` and rechecks. There is no native unit-execution fallback. --- ## 4. The workspace runtime Each request gets a fresh **workspace** — a per-request wasm instance tree with the membrane wired in. `wasm_worker_serve(worker, ctx, entry_unit, handler)` (`src/wasm/worker.cpp`) is the single entry point for *every* mode: 1. Birth a request-scoped workspace with fresh per-request state. Current production workers instantiate a fresh workspace for each request; any future snapshot/CoW optimization must preserve that request-isolation contract. 2. Resolve `entry_unit` + `handler` to an export; components referenced at runtime are resolved on demand via the `uce_host_component_resolve` hostcall (`component_resolve()` → `__uce_<...>` slot), loading dependency modules lazily and recording resolve counts/timings. 3. Invoke the export. Unit code calls host functions (filesystem, sqlite, regex, markdown, time, tasks, `ws_*`, …) across the membrane. 4. Collect the response (`WasmResponse`: body, headers, status, `meta`). Because the workspace owns no process-lifetime state, a unit crash or trap is contained: it fails the one request and the worker stays healthy. Workers suspend the native SIGSEGV/SIGILL recovery handler around the wasm call so that Wasmtime's own trap signals are not escalated into a native fatal signal (see `serve_via_wasm` in `handle_complete`). ### Task callbacks and workspace lifetime `task()` and `task_repeat()` are fork-backed. The `uce_host_task_spawn` hostcall captures the current `WasmWorkspace*`, but `src/lib/sys.cpp::task()` invokes the captured callback only in the forked child, before the hostcall stack unwinds in that child. The parent request may return and destroy its workspace; the child is forked from the parent and keeps a private request-context copy of the per-request workspace. This means a delayed task callback can run after the spawning request returns without dereferencing the parent's destroyed workspace. It is still a callback into the inherited child workspace, not a fresh normal request workspace; avoid adding host resources to `WasmWorkspace` that are invalid across `fork()` unless task callback handling is changed to birth a fresh workspace. --- ## 5. Request dispatch (`handle_complete`) `handle_complete()` (`src/linux_fastcgi.cpp`) is the worker's single dispatch point. It selects a handler string from request params and calls the same `serve_via_wasm(entry_unit, handler)` lambda for all of them: ``` UCE_WS == "1" → serve_via_wasm(entry_unit, "websocket") request.resources.is_cli → serve_via_wasm(entry_unit, "cli") UCE_SERVE_HTTP == "1" → serve_via_wasm(entry_unit, "serve_http"[:fn]) otherwise (page) → serve_via_wasm(entry_unit, "render") ``` The `UCE_*` params are set by whichever broker forwarded the request: - **Page render**: FastCGI nginx → `FCGI_SOCKET_PATH` directly; no `UCE_*` flags → `render`. - **CLI**: the CLI socket sets `is_cli`. - **serve_http**: the custom-server dispatcher sets `UCE_SERVE_HTTP=1` plus `UCE_SERVE_HTTP_FUNCTION` and rewrites `SCRIPT_FILENAME` to the configured unit (`custom_server_http_complete`). - **WebSocket**: the WS broker sets `UCE_WS=1` and carries the connection identity as `UCE_WS_*` params (see §6). `handle_complete` rebuilds `request.resources.websocket_*` and `request.connection` from them before invoking `__uce_websocket`. If a unit still cannot be served after the on-demand wasm compile, the worker returns a clean 500 with the wasm/compile error; it does not execute native unit code. --- ## 6. The central WebSocket broker **One process owns the HTTP port and every WebSocket connection**, so any unit's `ws_*` call can reach any or all connections, and a unit-code crash (which happens in a worker) never drops live connections. ### 6.1 Inbound: a WS frame → a worker render (non-blocking) `ws_broker_ws_message(request, message, opcode)` fires when a complete (reassembled) WS message arrives on a connection. It does **not** block the broker loop: 1. Build FastCGI params: `SCRIPT_FILENAME`, `REQUEST_METHOD=GET`, a `REQUEST_URI` (required — `handle_request()` rejects requests without one *before* `on_complete` runs), `UCE_WS=1`, and the connection context as `UCE_WS_CONNECTION_ID / SCOPE / OPCODE / BINARY / CONNECTIONS / STATE`. 2. The message rides as `UCE_WS_MESSAGE` (base64) with an **empty STDIN body** — a non-empty STDIN makes the FastCGI transport flush a premature response before `on_complete` ever runs. 3. Connect to `FCGI_SOCKET_PATH` (non-blocking) and queue the encoded request in `ws_broker_outbound[fd]` with an enqueue timestamp. `ws_broker_drain_outbound()` runs after every `process(50)` tick: it finishes writing each queued request, then drains and discards the reply (the unit's output comes back via the command socket, not this reply), closing the fd when the worker closes its end. If a forward remains pending beyond `WS_BROKER_OUTBOUND_TIMEOUT_SECONDS` (default `30`), the broker drops and closes it so a wedged worker cannot pin broker fds/memory indefinitely. ### 6.2 Outbound: `ws_*` commands flushed back to the broker Any unit code — not just WebSocket handlers — may call `ws_send` / `ws_send_to` / `ws_close`. In the workspace these **record dispatch commands** rather than touching a socket (the workspace owns no connections); `wasm-core`'s `ws_*` (`src/lib/sys.cpp`) append to `websocket_dispatch_commands`, and `finish_response_meta` (`src/wasm/core.cpp`) emits them as `ws_commands`. If the handler changed per-connection state, the core also emits `ws_connection_state` even when no commands were emitted, and the native backend flushes this state-only batch to the broker. `wasm_backend_serve` (`src/wasm/backend.cpp`) flushes that batch at workspace teardown — in **any** scenario, not just WS handlers — to the broker's command socket (`WS_BROKER_SOCKET_PATH`, `/run/uce/ws-broker.sock`) via `fcgi_forward_request` with `UCE_WS_DISPATCH=1`. This is the only path WS data takes out of a workspace. `ws_broker_apply_commands()` decodes the batch and applies each command against the full registry it owns: `broadcast` (by scope), `send_to` (by connection id), `close`. If the batch carries `connection_state`, it persists that onto the matching live connection's `websocket_state`. ### 6.3 Un-upgraded HTTP on the WS port The WS port can also receive ordinary (non-Upgrade) HTTP requests. `ws_broker_complete()` routes by param: `UCE_WS_DISPATCH=1` → apply commands; otherwise → `forward_request_to_worker()` — the *same* shared facility the serve_http dispatcher uses, so there is no duplicated request-forwarding code. ### 6.4 The broker loop `run_ws_broker()` drops the worker listeners it inherited (`close_inherited_server_sockets`), installs permissive `on_request`/`on_data` handlers (it renders nothing, so it accepts every request straight through to `on_complete`), wires `on_complete=ws_broker_complete` and `on_websocket_message=ws_broker_ws_message`, listens on `HTTP_PORT` and the command socket, and loops `process(50)` + `drain_outbound(timeout)`. The `timeout` comes from `WS_BROKER_OUTBOUND_TIMEOUT_SECONDS` (default `30`). The design is **non-blocking outbound dispatch + async command-socket flush, all in the broker's single epoll loop** — the broker never blocks on a worker. The parent respawns the broker if it dies (`ws_broker_alive` / `ensure_ws_broker` in `main()`). --- ## 7. The serve_http facility `serve_http` units are reachable on their own bind address via a custom-server dispatcher (`custom_server_http_dispatcher_loop`). The dispatcher owns the bind socket and, on each request, sets `UCE_SERVE_HTTP=1` + the configured unit/ function and calls `forward_request_to_worker()`. The worker then runs `serve_via_wasm(entry_unit, "serve_http"[:fn])`. This is the same hold-connection-forward-render model as the WS broker, sharing `forward_request_to_worker` and `fcgi_forward.h`. --- ## 8. Build / object layout The native side is split into separately-compiled objects so editing the wasm runtime does not recompile the whole TU (`scripts/build_linux.sh`): | Object | Contents | |---|---| | `bin/sqlite3.o` | sqlite amalgamation (cached) | | `bin/wasm.o` | `src/wasm/wasm_module.cpp` (backend.cpp + worker.cpp + wasmtime) | | `bin/main.o` | `src/linux_fastcgi.cpp` (includes `lib/uce_lib.cpp`) | Linked into one `-rdynamic` binary. ODR hazards across the objects are handled deliberately: `context` and `my_pid`/`parent_pid` are `extern` (guarded for the wasm core vs. unit builds), `operator new`/`delete` live in `types.cpp`, and header free-functions are `inline`. The wasm backend exposes only declarations (`src/wasm/backend.h`) to `main.o`. --- ## 9. Configuration keys | Key | Default | Meaning | |---|---|---| | `WASM_BACKEND_VERBOSE` | `0` | Emit `X-UCE-Wasm-*` workspace timing headers (benchmark only). | | `FCGI_SOCKET_PATH` | runtime-configured (`/run/uce/fastcgi.sock` in this doc) | Worker pool FastCGI socket (brokers forward here). | | `CLI_SOCKET_PATH` | `/run/uce/cli.sock` | Worker CLI/admin socket. Keep private; reference `CLI_SOCKET_MODE` is `0600`. | | `FCGI_SOCKET_MODE` | `0666` | Permission mode applied to `FCGI_SOCKET_PATH` after bind; set tighter if nginx/Apache can use a trusted group. | | `CLI_SOCKET_MODE` | `0600` | Permission mode applied to `CLI_SOCKET_PATH`; set `0660` only for a trusted admin group. | | `HTTP_PORT` | `8080` | Raw HTTP + WebSocket port — owned by the WS broker. | | `WS_BROKER_SOCKET_PATH` | `/run/uce/ws-broker.sock` | Broker command socket for `ws_*` flushes. | | `WS_BROKER_OUTBOUND_TIMEOUT_SECONDS` | `30` | Max lifetime in seconds for queued WS broker forwards before drop. | | `WORKER_COUNT` | `4` | Number of uniform worker processes. | --- ## 10. Testing - **Regression gate**: `scripts/run_cli_tests.sh --include-wasm-kill` runs the in-runtime CLI suite (`site/tests/cli_runner.uce`) plus the site test pages. - **WebSocket end-to-end**: a headless client performs a raw WS handshake to `:HTTP_PORT` with path `/site/tests/websockets.ws.uce` (self-resolving `SCRIPT_FILENAME`) and asserts the `hello-ack` frame — exercising the full broker → worker → broker → client chain across process boundaries.