334 lines
17 KiB
Markdown
334 lines
17 KiB
Markdown
# UCE WASM Runtime Architecture
|
||
|
||
Status: current as of the W7e native-pipeline removal (June 2026). This document
|
||
describes the **runtime architecture as built** — the process topology, the
|
||
wasm membrane, the unified request dispatch, and the central WebSocket broker.
|
||
Native `.so` unit execution/dlopen fallback has been removed; the parser and
|
||
preprocessor remain only as the front-end that emits C++ for wasm side-module
|
||
compilation.
|
||
|
||
The guiding principle: **request code never shares an address space or an
|
||
allocator with the runtime.** Every unit runs as a WebAssembly module inside a
|
||
per-request workspace behind a narrow host membrane. Long-lived connection
|
||
state (WebSocket connections, listening HTTP sockets) lives in dedicated native
|
||
broker processes that own no unit code — they hold the connection and forward
|
||
the actual unit invocation to a clean-engine worker, exactly like a normal
|
||
page render.
|
||
|
||
---
|
||
|
||
## 1. Process topology
|
||
|
||
A single native binary (`uce_fastcgi`) forks into a small set of long-lived
|
||
roles. None of them is special-cased per request mode; the differences are
|
||
purely *which socket a process listens on* and *which function inside a unit
|
||
gets invoked*.
|
||
|
||
```
|
||
┌────────────────────────────┐
|
||
nginx ──FastCGI──► worker pool (N processes) │ $FCGI_SOCKET_PATH (example `/run/uce/fastcgi.sock`)
|
||
(port 80 etc.) │ uniform unit renderers │ (FastCGI + CLI)
|
||
└─────────────▲──────────────┘
|
||
│ forward render (FastCGI, FCGI_SOCKET_PATH)
|
||
│
|
||
browser ──raw HTTP / WS──► ┌────────┴─────────┐
|
||
(HTTP_PORT 8080) │ WS broker │ owns HTTP_PORT + every
|
||
│ (1 process) │ WS connection; renders
|
||
└────────▲─────────┘ nothing itself
|
||
│ ws_* command flush
|
||
│ (FastCGI, ws-broker.sock)
|
||
on-demand serve_http ──► ┌────────┴─────────┐
|
||
(per bind addr) │ custom-server │ owns one serve_http
|
||
│ dispatcher(s) │ bind addr; forwards to pool
|
||
└──────────────────┘
|
||
|
||
parent process: spawns/respawns all of the above + the proactive compiler.
|
||
```
|
||
|
||
| Process | Owns | Renders units? | Source |
|
||
|---|---|---|---|
|
||
| **Parent** | nothing; supervises children | no | `main()`, `init_base_process()` |
|
||
| **Worker** (×`WORKER_COUNT`) | `FCGI_SOCKET_PATH` (configured socket path; example `/run/uce/fastcgi.sock`) + `CLI_SOCKET_PATH` | **yes** — the only processes that run wasm | `listen_for_connections()` |
|
||
| **WS broker** (×1) | `HTTP_PORT` + every live WS connection + `WS_BROKER_SOCKET_PATH` | no — forwards to the pool | `run_ws_broker()` |
|
||
| **serve_http dispatcher** (×bind) | one custom-server bind address | no — forwards to the pool | `custom_server_http_dispatcher_loop()` |
|
||
| **Proactive compiler** | nothing; pre-compiles units | no | `run_proactive_compiler()` |
|
||
|
||
**only workers instantiate Wasmtime and run unit code.**
|
||
Every connection-owning process (broker, serve_http dispatcher) forwards the
|
||
request invocation back to a worker via `FCGI_SOCKET_PATH` using the minimal
|
||
FastCGI client in [`src/lib/fcgi_forward.h`](../src/lib/fcgi_forward.h). This is
|
||
forced by Wasmtime: an `Engine`/`Store` cannot be safely re-created across
|
||
`fork()`, and the brokers fork from the parent that already touched the
|
||
runtime. So the brokers hold the long-lived connection and units respond to
|
||
events the same way they respond to a page request — through a clean worker.
|
||
|
||
---
|
||
|
||
## 2. The membrane and the DValue ABI
|
||
|
||
Units are compiled to WebAssembly and linked against a host import surface (the
|
||
"membrane"). Host and guest exchange structured values as **UCEB** (a compact
|
||
binary `DValue` encoding — `ucb_encode`/`ucb_decode`); strings cross as
|
||
`std::string` on the native side throughout (no raw `char*` ownership across
|
||
the boundary).
|
||
|
||
- `DValue` is the universal value type: scalars plus an ordered child map
|
||
(`_map`, a `std::map<String,DValue>`). `operator[]` is non-const (creates);
|
||
`.key(k)` is the const probe returning `const DValue*`; `.each(fn)` iterates
|
||
children yielding `const DValue&`.
|
||
- The request context (`params`/`get`/`post`/`cookies`/`session`, the raw body
|
||
`in`, and—for WS—the connection context) is marshalled into a single `ctx`
|
||
DValue, UCEB-encoded, and handed to the workspace. The response (body,
|
||
headers, status, and any `meta` such as `ws_commands`) comes back the same
|
||
way.
|
||
|
||
See [`docs/wasm-phase1-dvalue-abi.md`](wasm-phase1-dvalue-abi.md) for the wire
|
||
format details.
|
||
|
||
Decoder robustness note: UCEB input is untrusted across the wasm membrane. The
|
||
UCEB decoder must reject malformed magic/version/varint/length/trailing-data
|
||
inputs and excessive nesting with explicit errors, not native or wasm traps. Its
|
||
current hard nesting cap is intentionally low (64 levels) to stay well below the
|
||
guest stack limit. JSON decoding follows the same rule for malformed strings and
|
||
unicode escapes: validate bounds before every read and return an empty/partial
|
||
`DValue` rather than reading beyond the input.
|
||
|
||
---
|
||
|
||
## 3. Units, handlers, and export naming
|
||
|
||
A `.uce` unit compiles to a wasm module that exports one symbol per **handler**.
|
||
There is no per-mode machinery — a "CLI unit", "WebSocket unit", and "page" are
|
||
the same compiled artifact invoked at different exports. The dispatcher passes a
|
||
**handler string**; the host maps it to an export symbol:
|
||
|
||
```
|
||
__uce_<base>[_<sanitize(suffix)>]
|
||
```
|
||
|
||
`handler_export_symbol()` (`src/wasm/worker.cpp`) splits on the first `:`:
|
||
|
||
| Handler string | Export symbol |
|
||
|---|---|
|
||
| `render` | `__uce_render` |
|
||
| `cli` | `__uce_cli` |
|
||
| `websocket` | `__uce_websocket` |
|
||
| `once` | `__uce_once` (optional; absence is not an error) |
|
||
| `serve_http` | `__uce_serve_http` |
|
||
| `serve_http:named` | `__uce_serve_http_named` |
|
||
| `component:CARD` | `__uce_component_CARD` |
|
||
| `exists` | probe only — resolves the unit, loads nothing |
|
||
|
||
`sanitize_symbol_suffix()` keeps `[A-Za-z0-9_]` (mirrors `ascii_safe_name`).
|
||
`wasm_resolve_target(unit, handler)` (`src/wasm/core.cpp`) resolves the source
|
||
path and looks up the export's funcref slot; `exists` lets callers probe a unit
|
||
without instantiating it.
|
||
|
||
`wasm_backend_should_handle(request, entry_unit)` checks whether the wasm
|
||
backend is initialized and the requested artifact/handler is currently
|
||
available. If an artifact is cold or stale, dispatch compiles it on demand via
|
||
`get_shared_unit()` and rechecks. There is no native unit-execution fallback.
|
||
|
||
---
|
||
|
||
## 4. The workspace runtime
|
||
|
||
Each request gets a fresh **workspace** — a per-request wasm instance tree with
|
||
the membrane wired in. `wasm_worker_serve(worker, ctx, entry_unit, handler)`
|
||
(`src/wasm/worker.cpp`) is the single entry point for *every* mode:
|
||
|
||
1. Birth a request-scoped workspace with fresh per-request state. Current production workers instantiate a fresh workspace for each request; any future snapshot/CoW optimization must preserve that request-isolation contract.
|
||
2. Resolve `entry_unit` + `handler` to an export; components referenced at
|
||
runtime are resolved on demand via the `uce_host_component_resolve` hostcall
|
||
(`component_resolve()` → `__uce_<...>` slot), loading dependency modules
|
||
lazily and recording resolve counts/timings.
|
||
3. Invoke the export. Unit code calls host functions (filesystem, sqlite, regex,
|
||
markdown, time, tasks, `ws_*`, …) across the membrane.
|
||
4. Collect the response (`WasmResponse`: body, headers, status, `meta`).
|
||
|
||
Because the workspace owns no process-lifetime state, a unit crash or trap is
|
||
contained: it fails the one request and the worker stays healthy. Workers
|
||
suspend the native SIGSEGV/SIGILL recovery handler around the wasm call so that
|
||
Wasmtime's own trap signals are not escalated into a native fatal signal (see
|
||
`serve_via_wasm` in `handle_complete`).
|
||
|
||
### Task callbacks and workspace lifetime
|
||
|
||
`task()` and `task_repeat()` are fork-backed. The `uce_host_task_spawn` hostcall
|
||
captures the current `WasmWorkspace*`, but `src/lib/sys.cpp::task()` invokes the
|
||
captured callback only in the forked child, before the hostcall stack unwinds in
|
||
that child. The parent request may return and destroy its workspace; the child
|
||
is forked from the parent and keeps a private request-context copy of the per-request workspace. This means a delayed task callback can run after the spawning request
|
||
returns without dereferencing the parent's destroyed workspace. It is still a
|
||
callback into the inherited child workspace, not a fresh normal request
|
||
workspace; avoid adding host resources to `WasmWorkspace` that are invalid across
|
||
`fork()` unless task callback handling is changed to birth a fresh workspace.
|
||
|
||
---
|
||
|
||
## 5. Request dispatch (`handle_complete`)
|
||
|
||
`handle_complete()` (`src/linux_fastcgi.cpp`) is the worker's single dispatch
|
||
point. It selects a handler string from request params and calls the same
|
||
`serve_via_wasm(entry_unit, handler)` lambda for all of them:
|
||
|
||
```
|
||
UCE_WS == "1" → serve_via_wasm(entry_unit, "websocket")
|
||
request.resources.is_cli → serve_via_wasm(entry_unit, "cli")
|
||
UCE_SERVE_HTTP == "1" → serve_via_wasm(entry_unit, "serve_http"[:fn])
|
||
otherwise (page) → serve_via_wasm(entry_unit, "render")
|
||
```
|
||
|
||
The `UCE_*` params are set by whichever broker forwarded the request:
|
||
|
||
- **Page render**: FastCGI nginx → `FCGI_SOCKET_PATH` directly; no `UCE_*` flags → `render`.
|
||
- **CLI**: the CLI socket sets `is_cli`.
|
||
- **serve_http**: the custom-server dispatcher sets `UCE_SERVE_HTTP=1` plus
|
||
`UCE_SERVE_HTTP_FUNCTION` and rewrites `SCRIPT_FILENAME` to the configured
|
||
unit (`custom_server_http_complete`).
|
||
- **WebSocket**: the WS broker sets `UCE_WS=1` and carries the connection
|
||
identity as `UCE_WS_*` params (see §6). `handle_complete` rebuilds
|
||
`request.resources.websocket_*` and `request.connection` from them before
|
||
invoking `__uce_websocket`.
|
||
|
||
If a unit still cannot be served after the on-demand wasm compile, the worker
|
||
returns a clean 500 with the wasm/compile error; it does not execute native unit
|
||
code.
|
||
|
||
---
|
||
|
||
## 6. The central WebSocket broker
|
||
|
||
**One process owns the HTTP port
|
||
and every WebSocket connection**, so any unit's `ws_*` call can reach any or all
|
||
connections, and a unit-code crash (which happens in a worker) never drops live
|
||
connections.
|
||
|
||
### 6.1 Inbound: a WS frame → a worker render (non-blocking)
|
||
|
||
`ws_broker_ws_message(request, message, opcode)` fires when a complete
|
||
(reassembled) WS message arrives on a connection. It does **not** block the
|
||
broker loop:
|
||
|
||
1. Build FastCGI params: `SCRIPT_FILENAME`, `REQUEST_METHOD=GET`, a
|
||
`REQUEST_URI` (required — `handle_request()` rejects requests without one
|
||
*before* `on_complete` runs), `UCE_WS=1`, and the connection context as
|
||
`UCE_WS_CONNECTION_ID / SCOPE / OPCODE / BINARY / CONNECTIONS / STATE`.
|
||
2. The message rides as `UCE_WS_MESSAGE` (base64) with an **empty STDIN body** —
|
||
a non-empty STDIN makes the FastCGI transport flush a premature response
|
||
before `on_complete` ever runs.
|
||
3. Connect to `FCGI_SOCKET_PATH` (non-blocking) and queue the encoded request in
|
||
`ws_broker_outbound[fd]` with an enqueue timestamp.
|
||
|
||
`ws_broker_drain_outbound()` runs after every `process(50)` tick: it finishes
|
||
writing each queued request, then drains and discards the reply (the unit's
|
||
output comes back via the command socket, not this reply), closing the fd when
|
||
the worker closes its end. If a forward remains pending beyond
|
||
`WS_BROKER_OUTBOUND_TIMEOUT_SECONDS` (default `30`), the broker drops and
|
||
closes it so a wedged worker cannot pin broker fds/memory indefinitely.
|
||
|
||
### 6.2 Outbound: `ws_*` commands flushed back to the broker
|
||
|
||
Any unit code — not just WebSocket handlers — may call `ws_send` / `ws_send_to`
|
||
/ `ws_close`. In the workspace these **record dispatch commands** rather than
|
||
touching a socket (the workspace owns no connections); `wasm-core`'s `ws_*`
|
||
(`src/lib/sys.cpp`) append to `websocket_dispatch_commands`, and
|
||
`finish_response_meta` (`src/wasm/core.cpp`) emits them as `ws_commands`.
|
||
If the handler changed per-connection state, the core also emits
|
||
`ws_connection_state` even when no commands were emitted, and the native
|
||
backend flushes this state-only batch to the broker.
|
||
|
||
`wasm_backend_serve` (`src/wasm/backend.cpp`) flushes that batch at workspace
|
||
teardown — in **any** scenario, not just WS handlers — to the broker's command
|
||
socket (`WS_BROKER_SOCKET_PATH`, `/run/uce/ws-broker.sock`) via
|
||
`fcgi_forward_request` with `UCE_WS_DISPATCH=1`. This is the only path WS data
|
||
takes out of a workspace.
|
||
|
||
`ws_broker_apply_commands()` decodes the batch and applies each command against
|
||
the full registry it owns: `broadcast` (by scope), `send_to` (by connection id),
|
||
`close`. If the batch carries `connection_state`, it persists that onto the
|
||
matching live connection's `websocket_state`.
|
||
|
||
### 6.3 Un-upgraded HTTP on the WS port
|
||
|
||
The WS port can also receive ordinary (non-Upgrade) HTTP requests.
|
||
`ws_broker_complete()` routes by param: `UCE_WS_DISPATCH=1` → apply commands;
|
||
otherwise → `forward_request_to_worker()` — the *same* shared facility the
|
||
serve_http dispatcher uses, so there is no duplicated request-forwarding code.
|
||
|
||
### 6.4 The broker loop
|
||
|
||
`run_ws_broker()` drops the worker listeners it inherited
|
||
(`close_inherited_server_sockets`), installs permissive `on_request`/`on_data`
|
||
handlers (it renders nothing, so it accepts every request straight through to
|
||
`on_complete`), wires `on_complete=ws_broker_complete` and
|
||
`on_websocket_message=ws_broker_ws_message`, listens on `HTTP_PORT` and the
|
||
command socket, and loops `process(50)` + `drain_outbound(timeout)`. The
|
||
`timeout` comes from `WS_BROKER_OUTBOUND_TIMEOUT_SECONDS` (default `30`). The
|
||
design is **non-blocking outbound dispatch + async command-socket flush, all in
|
||
the broker's single epoll loop** — the broker never blocks on a worker.
|
||
|
||
The parent respawns the broker if it dies (`ws_broker_alive` / `ensure_ws_broker`
|
||
in `main()`).
|
||
|
||
---
|
||
|
||
## 7. The serve_http facility
|
||
|
||
`serve_http` units are reachable on their own bind address via a custom-server
|
||
dispatcher (`custom_server_http_dispatcher_loop`). The dispatcher owns the bind
|
||
socket and, on each request, sets `UCE_SERVE_HTTP=1` + the configured unit/
|
||
function and calls `forward_request_to_worker()`. The worker then runs
|
||
`serve_via_wasm(entry_unit, "serve_http"[:fn])`. This is the same
|
||
hold-connection-forward-render model as the WS broker, sharing
|
||
`forward_request_to_worker` and `fcgi_forward.h`.
|
||
|
||
---
|
||
|
||
## 8. Build / object layout
|
||
|
||
The native side is split into separately-compiled objects so editing the wasm
|
||
runtime does not recompile the whole TU (`scripts/build_linux.sh`):
|
||
|
||
| Object | Contents |
|
||
|---|---|
|
||
| `bin/sqlite3.o` | sqlite amalgamation (cached) |
|
||
| `bin/wasm.o` | `src/wasm/wasm_module.cpp` (backend.cpp + worker.cpp + wasmtime) |
|
||
| `bin/main.o` | `src/linux_fastcgi.cpp` (includes `lib/uce_lib.cpp`) |
|
||
|
||
Linked into one `-rdynamic` binary. ODR hazards across the objects are handled
|
||
deliberately: `context` and `my_pid`/`parent_pid` are `extern` (guarded for the
|
||
wasm core vs. unit builds), `operator new`/`delete` live in `types.cpp`, and
|
||
header free-functions are `inline`. The wasm backend exposes only declarations
|
||
(`src/wasm/backend.h`) to `main.o`.
|
||
|
||
---
|
||
|
||
## 9. Configuration keys
|
||
|
||
| Key | Default | Meaning |
|
||
|---|---|---|
|
||
| `WASM_BACKEND_VERBOSE` | `0` | Emit `X-UCE-Wasm-*` workspace timing headers (benchmark only). |
|
||
| `FCGI_SOCKET_PATH` | runtime-configured (`/run/uce/fastcgi.sock` in this doc) | Worker pool FastCGI socket (brokers forward here). |
|
||
| `CLI_SOCKET_PATH` | `/run/uce/cli.sock` | Worker CLI/admin socket. Keep private; reference `CLI_SOCKET_MODE` is `0600`. |
|
||
| `FCGI_SOCKET_MODE` | `0666` | Permission mode applied to `FCGI_SOCKET_PATH` after bind; set tighter if nginx/Apache can use a trusted group. |
|
||
| `CLI_SOCKET_MODE` | `0600` | Permission mode applied to `CLI_SOCKET_PATH`; set `0660` only for a trusted admin group. |
|
||
| `HTTP_PORT` | `8080` | Raw HTTP + WebSocket port — owned by the WS broker. |
|
||
| `WS_BROKER_SOCKET_PATH` | `/run/uce/ws-broker.sock` | Broker command socket for `ws_*` flushes. |
|
||
| `WS_BROKER_OUTBOUND_TIMEOUT_SECONDS` | `30` | Max lifetime in seconds for queued WS broker forwards before drop. |
|
||
| `WORKER_COUNT` | `4` | Number of uniform worker processes. |
|
||
|
||
---
|
||
|
||
## 10. Testing
|
||
|
||
- **Regression gate**: `scripts/run_cli_tests.sh --include-wasm-kill` runs the
|
||
in-runtime CLI suite (`site/tests/cli_runner.uce`) plus the site test pages.
|
||
|
||
- **WebSocket end-to-end**: a headless client performs a raw WS handshake to
|
||
`:HTTP_PORT` with path `/site/tests/websockets.ws.uce` (self-resolving
|
||
`SCRIPT_FILENAME`) and asserts the `hello-ack` frame — exercising the full
|
||
broker → worker → broker → client chain across process boundaries.
|
||
|
||
|