403 lines
22 KiB
Markdown
403 lines
22 KiB
Markdown
# UCE WASM Runtime Architecture
|
||
|
||
Status: current as of the W7e native-pipeline removal (June 2026). This document
|
||
describes the **runtime architecture as built** — the process topology, the
|
||
wasm membrane, the unified request dispatch, and the central WebSocket broker.
|
||
Native `.so` unit execution/dlopen fallback has been removed; the parser and
|
||
preprocessor remain only as the front-end that emits C++ for wasm side-module
|
||
compilation.
|
||
|
||
The guiding principle: **request code never shares an address space or an
|
||
allocator with the runtime.** Every unit runs as a WebAssembly module inside a
|
||
per-request workspace behind a narrow host membrane. Long-lived connection
|
||
state (WebSocket connections, listening HTTP sockets) lives in dedicated native
|
||
broker processes that own no unit code — they hold the connection and forward
|
||
the actual unit invocation to a clean-engine worker, exactly like a normal
|
||
page render.
|
||
|
||
---
|
||
|
||
## 1. Process topology
|
||
|
||
A single native binary (`uce_fastcgi`) forks into a small set of long-lived
|
||
roles. None of them is special-cased per request mode; the differences are
|
||
purely *which socket a process listens on* and *which function inside a unit
|
||
gets invoked*.
|
||
|
||
```
|
||
┌────────────────────────────┐
|
||
nginx ──FastCGI──► worker pool (N processes) │ $FCGI_SOCKET_PATH (example `/run/uce/fastcgi.sock`)
|
||
(port 80 etc.) │ uniform unit renderers │ (FastCGI + CLI)
|
||
└─────────────▲──────────────┘
|
||
│ forward render (FastCGI, FCGI_SOCKET_PATH)
|
||
│
|
||
browser ──raw HTTP / WS──► ┌────────┴─────────┐
|
||
(HTTP_PORT 8080) │ WS broker │ owns HTTP_PORT + every
|
||
│ (1 process) │ WS connection; renders
|
||
└────────▲─────────┘ nothing itself
|
||
│ ws_* command flush
|
||
│ (FastCGI, ws-broker.sock)
|
||
on-demand serve_http ──► ┌────────┴─────────┐
|
||
(per bind addr) │ custom-server │ owns one serve_http
|
||
│ dispatcher(s) │ bind addr; forwards to pool
|
||
└──────────────────┘
|
||
|
||
parent process: spawns/respawns all of the above + the proactive compiler.
|
||
```
|
||
|
||
| Process | Owns | Renders units? | Source |
|
||
|---|---|---|---|
|
||
| **Parent** | nothing; supervises children | no | `main()`, `init_base_process()` |
|
||
| **Worker** (×`WORKER_COUNT`) | `FCGI_SOCKET_PATH` (configured socket path; example `/run/uce/fastcgi.sock`) + `CLI_SOCKET_PATH` | **yes** — the only processes that run wasm | `listen_for_connections()` |
|
||
| **WS broker** (×1) | `HTTP_PORT` + every live WS connection + `WS_BROKER_SOCKET_PATH` | no — forwards to the pool | `run_ws_broker()` |
|
||
| **serve_http dispatcher** (×bind) | one custom-server bind address | no — forwards to the pool | `custom_server_http_dispatcher_loop()` |
|
||
| **Proactive compiler** | nothing; pre-compiles units | no | `run_proactive_compiler()` |
|
||
|
||
**only workers instantiate Wasmtime and run unit code.**
|
||
Every connection-owning process (broker, serve_http dispatcher) forwards the
|
||
request invocation back to a worker via `FCGI_SOCKET_PATH` using the minimal
|
||
FastCGI client in [`src/lib/fcgi_forward.h`](../src/lib/fcgi_forward.h). This is
|
||
forced by Wasmtime: an `Engine`/`Store` cannot be safely re-created across
|
||
`fork()`, and the brokers fork from the parent that already touched the
|
||
runtime. So the brokers hold the long-lived connection and units respond to
|
||
events the same way they respond to a page request — through a clean worker.
|
||
|
||
---
|
||
|
||
## 2. The membrane and the DValue ABI
|
||
|
||
Units are compiled to WebAssembly and linked against a host import surface (the
|
||
"membrane"). Host and guest exchange structured values as **UCEB** (a compact
|
||
binary `DValue` encoding — `ucb_encode`/`ucb_decode`); strings cross as
|
||
`std::string` on the native side throughout (no raw `char*` ownership across
|
||
the boundary).
|
||
|
||
- `DValue` is the universal value type: scalars plus an ordered child map
|
||
(`_map`, a `std::map<String,DValue>`). `operator[]` is non-const (creates);
|
||
`.key(k)` is the const probe returning `const DValue*`; `.each(fn)` iterates
|
||
children yielding `const DValue&`.
|
||
- The request context (`params`/`get`/`post`/`cookies`/`session`, the raw body
|
||
`in`, and—for WS—the connection context) is marshalled into a single `ctx`
|
||
DValue, UCEB-encoded, and handed to the workspace. The response (body,
|
||
headers, status, and any `meta` such as `ws_commands`) comes back the same
|
||
way.
|
||
|
||
See [`docs/wasm-phase1-dvalue-abi.md`](wasm-phase1-dvalue-abi.md) for the wire
|
||
format details.
|
||
|
||
Decoder robustness note: UCEB input is untrusted across the wasm membrane. The
|
||
UCEB decoder must reject malformed magic/version/varint/length/trailing-data
|
||
inputs and excessive nesting with explicit errors, not native or wasm traps. Its
|
||
current hard nesting cap is intentionally low (64 levels) to stay well below the
|
||
guest stack limit. JSON decoding follows the same rule for malformed strings and
|
||
unicode escapes: validate bounds before every read and return an empty/partial
|
||
`DValue` rather than reading beyond the input.
|
||
|
||
---
|
||
|
||
## 3. Units, handlers, and export naming
|
||
|
||
A `.uce` unit compiles to a wasm module that exports one symbol per **handler**.
|
||
There is no per-mode machinery — a "CLI unit", "WebSocket unit", and "page" are
|
||
the same compiled artifact invoked at different exports. The dispatcher passes a
|
||
**handler string**; the host maps it to an export symbol:
|
||
|
||
```
|
||
__uce_<base>[_<sanitize(suffix)>]
|
||
```
|
||
|
||
`handler_export_symbol()` (`src/wasm/worker.cpp`) splits on the first `:`:
|
||
|
||
| Handler string | Export symbol |
|
||
|---|---|
|
||
| `render` | `__uce_render` |
|
||
| `cli` | `__uce_cli` |
|
||
| `websocket` | `__uce_websocket` |
|
||
| `once` | `__uce_once` (optional; absence is not an error) |
|
||
| `serve_http` | `__uce_serve_http` |
|
||
| `serve_http:named` | `__uce_serve_http_named` |
|
||
| `component:CARD` | `__uce_component_CARD` |
|
||
| `exists` | probe only — resolves the unit, loads nothing |
|
||
|
||
`sanitize_symbol_suffix()` keeps `[A-Za-z0-9_]` (mirrors `ascii_safe_name`).
|
||
`wasm_resolve_target(unit, handler)` (`src/wasm/core.cpp`) resolves the source
|
||
path and looks up the export's funcref slot; `exists` lets callers probe a unit
|
||
without instantiating it.
|
||
|
||
`wasm_backend_should_handle(request, entry_unit)` checks whether the wasm
|
||
backend is initialized and the requested artifact/handler is currently
|
||
available. If an artifact is cold or stale, dispatch compiles it on demand via
|
||
`get_shared_unit()` and rechecks. There is no native unit-execution fallback.
|
||
Workers identify cached unit artifacts by nanosecond mtime, ctime, and size.
|
||
Whole-second mtime alone is insufficient because a dependency-triggered rebuild
|
||
can replace a wasm artifact within the same second as its prior build.
|
||
Cold module compilation and deserialization are host work, so `load_unit()`
|
||
refreshes the epoch deadline before its first guest call. Otherwise a component
|
||
whose compilation outlasted the guest CPU budget would immediately trap in the
|
||
following allocator/relocation call even though no guest loop consumed it.
|
||
|
||
The proactive compiler and request workers coordinate through per-unit file
|
||
locks and a lock-protected demand-priority queue under `BIN_DIRECTORY`. Unit
|
||
compilation writes and validates a process-unique temporary wasm file, then
|
||
publishes it with an atomic rename. When proactive compilation is enabled,
|
||
read-only HTTP requests keep using the last complete artifact while requesting
|
||
that stale unit at the head of the compiler queue. Non-read requests never run
|
||
a stale entry artifact: they return `503 Service Unavailable` with
|
||
`Retry-After: 1`, allowing the client to retry after the priority rebuild.
|
||
CLI and explicit compile paths remain synchronous. A failed rebuild removes
|
||
availability and surfaces the compiler error rather than serving the old unit
|
||
indefinitely. The per-unit lock also keeps concurrent synchronous compilers
|
||
from waiting across a transitive graph when a last complete artifact is
|
||
available.
|
||
|
||
Before preprocessing, the compiler verifies that the worker can actually read
|
||
the source. An unreadable path is reported as a source-read failure with a
|
||
persisted diagnostic; it never becomes an apparently valid empty side module or
|
||
a generic compiler error. Source signatures mark unreadable inputs, so correcting
|
||
access invalidates that failure and permits a normal retry without changing
|
||
signatures for ordinary readable files. The private CLI harness may label its
|
||
intentional unreadable-source regression request as an expected source-read
|
||
failure, keeping production log scans focused on real runtime failures while
|
||
preserving the failing compile result and diagnostic artifacts.
|
||
|
||
---
|
||
|
||
## 4. The workspace runtime
|
||
|
||
Each request gets a fresh **workspace** — a per-request wasm instance tree with
|
||
the membrane wired in. `wasm_worker_serve(worker, ctx, entry_unit, handler)`
|
||
(`src/wasm/worker.cpp`) is the single entry point for *every* mode:
|
||
|
||
1. Birth a request-scoped workspace with fresh per-request state. Current production workers instantiate a fresh workspace for each request; any future snapshot/CoW optimization must preserve that request-isolation contract.
|
||
2. Resolve `entry_unit` + `handler` to an export; components referenced at
|
||
runtime are resolved on demand via the `uce_host_component_resolve` hostcall
|
||
(`component_resolve()` → `__uce_<...>` slot), loading dependency modules
|
||
lazily and recording resolve counts/timings.
|
||
3. Invoke the export. Unit code calls host functions (filesystem, sqlite, regex,
|
||
markdown, time, tasks, `ws_*`, …) across the membrane.
|
||
4. Collect the response (`WasmResponse`: body, headers, status, `meta`).
|
||
|
||
Because the workspace owns no process-lifetime state, a unit crash or trap is
|
||
contained: it fails the one request and the worker stays healthy. Workers
|
||
suspend the native SIGSEGV/SIGILL recovery handler around the wasm call so that
|
||
Wasmtime's own trap signals are not escalated into a native fatal signal (see
|
||
`serve_via_wasm` in `handle_complete`).
|
||
|
||
Render workers are long-lived. The generic FastCGI transport retains a legacy
|
||
eight-connection recycle default, but `listen_for_connections()` disables it
|
||
for the UCE pool: recycling a healthy worker discards its Wasmtime engine and
|
||
module cache, adding a visible cold-start request. Request state remains bounded
|
||
by the fresh workspace and the normal per-request database/resource cleanup;
|
||
faulted workers still exit and are replaced by the parent.
|
||
|
||
### Task callbacks and workspace lifetime
|
||
|
||
`task()` and `task_repeat()` are fork-backed. The `uce_host_task_spawn` hostcall
|
||
captures the current `WasmWorkspace*`, but `src/lib/sys.cpp::task()` invokes the
|
||
captured callback only in the forked child, before the hostcall stack unwinds in
|
||
that child. The parent request may return and destroy its workspace; the child
|
||
is forked from the parent and keeps a private request-context copy of the per-request workspace. This means a delayed task callback can run after the spawning request
|
||
returns without dereferencing the parent's destroyed workspace. It is still a
|
||
callback into the inherited child workspace, not a fresh normal request
|
||
workspace; avoid adding host resources to `WasmWorkspace` that are invalid across
|
||
`fork()` unless task callback handling is changed to birth a fresh workspace.
|
||
|
||
---
|
||
|
||
## 5. Request dispatch (`handle_complete`)
|
||
|
||
`handle_complete()` (`src/linux_fastcgi.cpp`) is the worker's single dispatch
|
||
point. It selects a handler string from request params and calls the same
|
||
`serve_via_wasm(entry_unit, handler)` lambda for all of them:
|
||
|
||
```
|
||
UCE_WS == "1" → serve_via_wasm(entry_unit, "websocket")
|
||
request.resources.is_cli → serve_via_wasm(entry_unit, "cli")
|
||
UCE_SERVE_HTTP == "1" → serve_via_wasm(entry_unit, "serve_http"[:fn])
|
||
otherwise (page) → serve_via_wasm(entry_unit, "render")
|
||
```
|
||
|
||
The `UCE_*` params are set by whichever broker forwarded the request:
|
||
|
||
- **Page render**: FastCGI nginx → `FCGI_SOCKET_PATH` directly; no `UCE_*` flags → `render`.
|
||
- **CLI**: the CLI socket sets `is_cli`.
|
||
- **serve_http**: the custom-server dispatcher sets `UCE_SERVE_HTTP=1` plus
|
||
`UCE_SERVE_HTTP_FUNCTION` and rewrites `SCRIPT_FILENAME` to the configured
|
||
unit (`custom_server_http_complete`).
|
||
- **WebSocket**: the WS broker sets `UCE_WS=1` and carries the connection
|
||
identity as `UCE_WS_*` params (see §6). `handle_complete` rebuilds
|
||
`request.resources.websocket_*` and `request.connection` from them before
|
||
invoking `__uce_websocket`.
|
||
|
||
If a unit still cannot be served after the on-demand wasm compile, the worker
|
||
returns a clean 500 with the wasm/compile error; it does not execute native unit
|
||
code.
|
||
|
||
---
|
||
|
||
## 6. The central WebSocket broker
|
||
|
||
**One process owns the HTTP port
|
||
and every WebSocket connection**, so any unit's `ws_*` call can reach any or all
|
||
connections, and a unit-code crash (which happens in a worker) never drops live
|
||
connections.
|
||
|
||
### 6.1 Inbound: a WS frame → a worker render (non-blocking)
|
||
|
||
`ws_broker_ws_message(request, message, opcode)` fires when a complete
|
||
(reassembled) WS message arrives on a connection. It does **not** block the
|
||
broker loop:
|
||
|
||
1. Build FastCGI params: `SCRIPT_FILENAME`, `REQUEST_METHOD=GET`, a
|
||
`REQUEST_URI` (required — `handle_request()` rejects requests without one
|
||
*before* `on_complete` runs), `UCE_WS=1`, and the connection context as
|
||
`UCE_WS_CONNECTION_ID / SCOPE / OPCODE / BINARY / CONNECTIONS / STATE`.
|
||
2. The message rides as `UCE_WS_MESSAGE` (base64) with an **empty STDIN body** —
|
||
a non-empty STDIN makes the FastCGI transport flush a premature response
|
||
before `on_complete` ever runs.
|
||
3. Connect to `FCGI_SOCKET_PATH` (non-blocking) and queue the encoded request in
|
||
`ws_broker_outbound[fd]` with an enqueue timestamp.
|
||
|
||
`ws_broker_drain_outbound()` runs after every `process(50)` tick: it finishes
|
||
writing each queued request, then drains and discards the reply (the unit's
|
||
output comes back via the command socket, not this reply), closing the fd when
|
||
the worker closes its end. If a forward remains pending beyond
|
||
`WS_BROKER_OUTBOUND_TIMEOUT_SECONDS` (default `30`), the broker drops and
|
||
closes it so a wedged worker cannot pin broker fds/memory indefinitely.
|
||
|
||
### 6.2 Outbound: `ws_*` commands flushed back to the broker
|
||
|
||
Any unit code — not just WebSocket handlers — may call `ws_send` / `ws_send_to`
|
||
/ `ws_close`. In the workspace these **record dispatch commands** rather than
|
||
touching a socket (the workspace owns no connections); `wasm-core`'s `ws_*`
|
||
(`src/lib/sys.cpp`) append to `websocket_dispatch_commands`, and
|
||
`finish_response_meta` (`src/wasm/core.cpp`) emits them as `ws_commands`.
|
||
If the handler changed per-connection state, the core also emits
|
||
`ws_connection_state` even when no commands were emitted, and the native
|
||
backend flushes this state-only batch to the broker.
|
||
|
||
`wasm_backend_serve` (`src/wasm/backend.cpp`) flushes that batch at workspace
|
||
teardown — in **any** scenario, not just WS handlers — to the broker's command
|
||
socket (`WS_BROKER_SOCKET_PATH`, `/run/uce/ws-broker.sock`) via
|
||
`fcgi_forward_request` with `UCE_WS_DISPATCH=1`. This is the only path WS data
|
||
takes out of a workspace.
|
||
|
||
`ws_broker_apply_commands()` decodes the batch and applies each command against
|
||
the full registry it owns: `broadcast` (by scope), `send_to` (by connection id),
|
||
`close`. If the batch carries `connection_state`, it persists that onto the
|
||
matching live connection's `websocket_state`.
|
||
|
||
### 6.3 Un-upgraded HTTP on the WS port
|
||
|
||
The WS port can also receive ordinary (non-Upgrade) HTTP requests.
|
||
`ws_broker_complete()` routes by param: `UCE_WS_DISPATCH=1` → apply commands;
|
||
otherwise → `forward_request_to_worker()` — the *same* shared facility the
|
||
serve_http dispatcher uses, so there is no duplicated request-forwarding code.
|
||
|
||
### 6.4 The broker loop
|
||
|
||
`run_ws_broker()` drops the worker listeners it inherited
|
||
(`close_inherited_server_sockets`), installs permissive `on_request`/`on_data`
|
||
handlers (it renders nothing, so it accepts every request straight through to
|
||
`on_complete`), wires `on_complete=ws_broker_complete` and
|
||
`on_websocket_message=ws_broker_ws_message`, listens on `HTTP_PORT` and the
|
||
command socket, and loops `process(50)` + `drain_outbound(timeout)`. The
|
||
`timeout` comes from `WS_BROKER_OUTBOUND_TIMEOUT_SECONDS` (default `30`). The
|
||
design is **non-blocking outbound dispatch + async command-socket flush, all in
|
||
the broker's single epoll loop** — the broker never blocks on a worker.
|
||
|
||
The parent respawns the broker if it dies (`ws_broker_alive` / `ensure_ws_broker`
|
||
in `main()`).
|
||
|
||
---
|
||
|
||
## 7. The serve_http facility
|
||
|
||
`serve_http` units are reachable on their own bind address via a custom-server
|
||
dispatcher (`custom_server_http_dispatcher_loop`). The dispatcher owns the bind
|
||
socket and, on each request, sets `UCE_SERVE_HTTP=1` + the configured unit/
|
||
function and calls `forward_request_to_worker()`. The worker then runs
|
||
`serve_via_wasm(entry_unit, "serve_http"[:fn])`. This is the same
|
||
hold-connection-forward-render model as the WS broker, sharing
|
||
`forward_request_to_worker` and `fcgi_forward.h`.
|
||
|
||
---
|
||
|
||
## 8. Build / object layout
|
||
|
||
The native side is split into separately-compiled objects so editing the wasm
|
||
runtime does not recompile the whole TU (`scripts/build_linux.sh`):
|
||
|
||
| Object | Contents |
|
||
|---|---|
|
||
| `bin/sqlite3.o` | sqlite amalgamation (cached) |
|
||
| `bin/wasm.o` | `src/wasm/wasm_module.cpp` (backend.cpp + worker.cpp + wasmtime) |
|
||
| `bin/main.o` | `src/linux_fastcgi.cpp` (includes `lib/uce_lib.cpp`) |
|
||
|
||
Linked into one `-rdynamic` binary. ODR hazards across the objects are handled
|
||
deliberately: `context` and `my_pid`/`parent_pid` are `extern` (guarded for the
|
||
wasm core vs. unit builds), `operator new`/`delete` live in `types.cpp`, and
|
||
header free-functions are `inline`. The wasm backend exposes only declarations
|
||
(`src/wasm/backend.h`) to `main.o`.
|
||
|
||
---
|
||
|
||
## 9. Configuration keys
|
||
|
||
| Key | Default | Meaning |
|
||
|---|---|---|
|
||
| `WASM_BACKEND_VERBOSE` | `0` | Emit `X-UCE-Wasm-*` workspace timing headers (benchmark only). |
|
||
| `FCGI_SOCKET_PATH` | runtime-configured (`/run/uce/fastcgi.sock` in this doc) | Worker pool FastCGI socket (brokers forward here). |
|
||
| `CLI_SOCKET_PATH` | `/run/uce/cli.sock` | Worker CLI/admin socket. Keep private; reference `CLI_SOCKET_MODE` is `0600`. |
|
||
| `FCGI_SOCKET_MODE` | `0666` | Permission mode applied to `FCGI_SOCKET_PATH` after bind; set tighter if nginx/Apache can use a trusted group. |
|
||
| `CLI_SOCKET_MODE` | `0600` | Permission mode applied to `CLI_SOCKET_PATH`; set `0660` only for a trusted admin group. |
|
||
| `HTTP_PORT` | `8080` | Raw HTTP + WebSocket port — owned by the WS broker. |
|
||
| `WS_BROKER_SOCKET_PATH` | `/run/uce/ws-broker.sock` | Broker command socket for `ws_*` flushes. |
|
||
| `WS_BROKER_OUTBOUND_TIMEOUT_SECONDS` | `30` | Max lifetime in seconds for queued WS broker forwards before drop. |
|
||
| `WORKER_COUNT` | `4` | Number of uniform worker processes. |
|
||
|
||
---
|
||
|
||
## 10. Testing
|
||
|
||
- **Regression gate**: `scripts/run_cli_tests.sh --include-wasm-kill` runs the
|
||
in-runtime CLI suite (`site/tests/cli_runner.uce`) plus the site test pages and
|
||
`scripts/test_dependency_invalidation.sh`. The shell helper runs the CLI suite
|
||
as named groups (`demo`, `http`, `site`, `doc-gate`, `security`,
|
||
`task-lifetime`, `starter`, `tcp`, and optional `wasm-kill`) so a slow or
|
||
failing group is visible before the rest of the gate runs. `doc-gate` can take
|
||
several minutes on a cold runtime because it compiles generated documentation
|
||
examples; `UCE_CLI_TEST_TIMEOUT` controls the per-group curl timeout and
|
||
defaults to 900 seconds. Individual doc-page checks retry transient empty
|
||
frontend reads, but still fail persistent status, marker, compile, runtime, or
|
||
placeholder-example errors. The dependency-invalidation gate changes a transitive
|
||
`#load`, then replaces a warmed worker artifact while preserving its
|
||
whole-second mtime to prove both compiler and worker caches invalidate it. It
|
||
also rejects an unreadable unit without publishing a wasm artifact, restores
|
||
its permissions and proves the next CLI request compiles it, then sends 48
|
||
requests and asserts the observed worker PID set does not exceed
|
||
`WORKER_COUNT`, guarding against accidental reintroduction of request-count
|
||
recycling. It also races a source edit against an active unit compile after
|
||
the generated C++ snapshot has been written. The compiler must either retry
|
||
and serve the post-edit source or fail closed; it must never stamp current
|
||
source metadata onto wasm produced from an older snapshot.
|
||
`scripts/test_cold_component_deadline.sh` separately compiles a deliberately
|
||
cold component that exceeds the development epoch window and proves the
|
||
parent request still renders it. The dependency-invalidation gate also holds
|
||
parent and child compile locks across a transitive source edit, proves a
|
||
warmed HTTP request returns the last atomically published result without
|
||
waiting, proves a POST returns a prompt retryable 503 without executing its
|
||
old mutation handler, and verifies demand-priority convergence. A CLI request
|
||
still waits and returns only the current dependency result after rebuild
|
||
completion.
|
||
The component page also calls a parent once without its optional relative
|
||
child and again with that child activated. This guards the component-slot
|
||
cache invariant that a cache hit must restore both the function-table slot
|
||
and the resolved unit path; otherwise nested relative targets resolve from
|
||
the caller after a repeated parent invocation.
|
||
|
||
- **WebSocket end-to-end**: a headless client performs a raw WS handshake to
|
||
`:HTTP_PORT` with path `/site/tests/websockets.ws.uce` (self-resolving
|
||
`SCRIPT_FILENAME`) and asserts the `hello-ack` frame — exercising the full
|
||
broker → worker → broker → client chain across process boundaries.
|