8587fbc5aa
- WS: a dedicated broker process owns HTTP_PORT + every connection; it forwards renders to the worker pool over uce.sock (non-blocking) and applies ws_* command batches flushed back at workspace teardown. Removes the now-dead per-worker websocket executor (-509 lines). - Dispatch: unify CLI / WebSocket / serve_http / page render through one serve_via_wasm(entry_unit, handler) path; handler string -> __uce_<handler> export symbol. - W7d: rewrite zip.uce to the membrane return-value error contract (no C++ try/catch), error-reporting.uce to genuine wasm traps instead of throw, and sharedunit.uce to unit_info(); empty the native-only token gate. - Membrane: wire ls / mkdir / file_mtime through new uce_host_file_list / uce_host_file_mkdir / uce_host_file_mtime hostcalls (resolve_guest_file gains directory support). Fixes /doc/index.uce listing nothing; adds a regression assertion that the index enumerates items. - Docs: add docs/wasm-runtime-architecture.md; record the W7e staged native- deletion plan in WASM-PROPOSAL.md. Verified: scripts/run_cli_tests.sh --include-wasm-kill -> 87 passed, 0 failed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
319 lines
15 KiB
Markdown
319 lines
15 KiB
Markdown
# UCE WASM Runtime Architecture
|
||
|
||
Status: current as of the W7 cutover (June 2026). This document describes the
|
||
**runtime architecture as built** — the process topology, the wasm membrane,
|
||
the unified request dispatch, and the central WebSocket broker. For the
|
||
historical motivation and the phased migration plan, see
|
||
[`WASM-PROPOSAL.md`](../WASM-PROPOSAL.md); this file is the steady-state
|
||
reference that proposal points at.
|
||
|
||
The guiding principle: **request code never shares an address space or an
|
||
allocator with the runtime.** Every unit runs as a WebAssembly module inside a
|
||
per-request workspace behind a narrow host membrane. Long-lived connection
|
||
state (WebSocket connections, listening HTTP sockets) lives in dedicated native
|
||
broker processes that own no unit code — they hold the connection and forward
|
||
the actual unit invocation to a clean-engine worker, exactly like a normal
|
||
page render.
|
||
|
||
---
|
||
|
||
## 1. Process topology
|
||
|
||
A single native binary (`uce_fastcgi`) forks into a small set of long-lived
|
||
roles. None of them is special-cased per request mode; the differences are
|
||
purely *which socket a process listens on* and *which function inside a unit
|
||
gets invoked*.
|
||
|
||
```
|
||
┌────────────────────────────┐
|
||
nginx ──FastCGI──► worker pool (N processes) │ /run/uce.sock
|
||
(port 80 etc.) │ uniform unit renderers │ (FastCGI + CLI)
|
||
└─────────────▲──────────────┘
|
||
│ forward render (FastCGI, uce.sock)
|
||
│
|
||
browser ──raw HTTP / WS──► ┌────────┴─────────┐
|
||
(HTTP_PORT 8080) │ WS broker │ owns HTTP_PORT + every
|
||
│ (1 process) │ WS connection; renders
|
||
└────────▲─────────┘ nothing itself
|
||
│ ws_* command flush
|
||
│ (FastCGI, ws-broker.sock)
|
||
on-demand serve_http ──► ┌────────┴─────────┐
|
||
(per bind addr) │ custom-server │ owns one serve_http
|
||
│ dispatcher(s) │ bind addr; forwards to pool
|
||
└──────────────────┘
|
||
|
||
parent process: spawns/respawns all of the above + the proactive compiler.
|
||
```
|
||
|
||
| Process | Owns | Renders units? | Source |
|
||
|---|---|---|---|
|
||
| **Parent** | nothing; supervises children | no | `main()`, `init_base_process()` |
|
||
| **Worker** (×`WORKER_COUNT`) | `FCGI_SOCKET_PATH` (`/run/uce.sock`) + `CLI_SOCKET_PATH` | **yes** — the only processes that run wasm | `listen_for_connections()` |
|
||
| **WS broker** (×1) | `HTTP_PORT` + every live WS connection + `WS_BROKER_SOCKET_PATH` | no — forwards to the pool | `run_ws_broker()` |
|
||
| **serve_http dispatcher** (×bind) | one custom-server bind address | no — forwards to the pool | `custom_server_http_dispatcher_loop()` |
|
||
| **Proactive compiler** | nothing; pre-compiles units | no | `run_proactive_compiler()` |
|
||
|
||
The key invariant: **only workers instantiate Wasmtime and run unit code.**
|
||
Every connection-owning process (broker, serve_http dispatcher) forwards the
|
||
real invocation back to a worker over `/run/uce.sock` using the minimal
|
||
FastCGI client in [`src/lib/fcgi_forward.h`](../src/lib/fcgi_forward.h). This is
|
||
forced by Wasmtime: an `Engine`/`Store` cannot be safely re-created across
|
||
`fork()`, and the brokers fork from the parent that already touched the
|
||
runtime. So the brokers hold the long-lived connection and units respond to
|
||
events the same way they respond to a page request — through a clean worker.
|
||
|
||
---
|
||
|
||
## 2. The membrane and the DValue ABI
|
||
|
||
Units are compiled to WebAssembly and linked against a host import surface (the
|
||
"membrane"). Host and guest exchange structured values as **UCEB** (a compact
|
||
binary `DValue` encoding — `ucb_encode`/`ucb_decode`); strings cross as
|
||
`std::string` on the native side throughout (no raw `char*` ownership across
|
||
the boundary).
|
||
|
||
- `DValue` is the universal value type: scalars plus an ordered child map
|
||
(`_map`, a `std::map<String,DValue>`). `operator[]` is non-const (creates);
|
||
`.key(k)` is the const probe returning `const DValue*`; `.each(fn)` iterates
|
||
children yielding `const DValue&`.
|
||
- The request context (`params`/`get`/`post`/`cookies`/`session`, the raw body
|
||
`in`, and—for WS—the connection context) is marshalled into a single `ctx`
|
||
DValue, UCEB-encoded, and handed to the workspace. The response (body,
|
||
headers, status, and any `meta` such as `ws_commands`) comes back the same
|
||
way.
|
||
|
||
See [`docs/wasm-phase1-dvalue-abi.md`](wasm-phase1-dvalue-abi.md) for the wire
|
||
format details.
|
||
|
||
---
|
||
|
||
## 3. Units, handlers, and export naming
|
||
|
||
A `.uce` unit compiles to a wasm module that exports one symbol per **handler**.
|
||
There is no per-mode machinery — a "CLI unit", "WebSocket unit", and "page" are
|
||
the same compiled artifact invoked at different exports. The dispatcher passes a
|
||
**handler string**; the host maps it to an export symbol:
|
||
|
||
```
|
||
__uce_<base>[_<sanitize(suffix)>]
|
||
```
|
||
|
||
`handler_export_symbol()` (`src/wasm/worker.cpp`) splits on the first `:`:
|
||
|
||
| Handler string | Export symbol |
|
||
|---|---|
|
||
| `render` | `__uce_render` |
|
||
| `cli` | `__uce_cli` |
|
||
| `websocket` | `__uce_websocket` |
|
||
| `once` | `__uce_once` (optional; absence is not an error) |
|
||
| `serve_http` | `__uce_serve_http` |
|
||
| `serve_http:named` | `__uce_serve_http_named` |
|
||
| `component:CARD` | `__uce_component_CARD` |
|
||
| `exists` | probe only — resolves the unit, loads nothing |
|
||
|
||
`sanitize_symbol_suffix()` keeps `[A-Za-z0-9_]` (mirrors `ascii_safe_name`).
|
||
`wasm_resolve_target(unit, handler)` (`src/wasm/core.cpp`) resolves the source
|
||
path and looks up the export's funcref slot; `exists` lets callers probe a unit
|
||
without instantiating it.
|
||
|
||
`wasm_backend_should_handle(request, entry_unit)` gates whether the wasm path
|
||
takes the request (vs. the legacy native compiler path, still selectable while
|
||
the native pipeline is retired). With `WASM_BACKEND_ENABLED=1` this is the
|
||
default for all unit execution.
|
||
|
||
---
|
||
|
||
## 4. The workspace runtime
|
||
|
||
Each request gets a fresh **workspace** — a per-request wasm instance tree with
|
||
the membrane wired in. `wasm_worker_serve(worker, ctx, entry_unit, handler)`
|
||
(`src/wasm/worker.cpp`) is the single entry point for *every* mode:
|
||
|
||
1. Birth a workspace (CoW-snapshot-based where available).
|
||
2. Resolve `entry_unit` + `handler` to an export; components referenced at
|
||
runtime are resolved on demand via the `uce_host_component_resolve` hostcall
|
||
(`component_resolve()` → `__uce_<...>` slot), loading dependency modules
|
||
lazily and recording resolve counts/timings.
|
||
3. Invoke the export. Unit code calls host functions (filesystem, sqlite, regex,
|
||
markdown, time, tasks, `ws_*`, …) across the membrane.
|
||
4. Collect the response (`WasmResponse`: body, headers, status, `meta`).
|
||
|
||
Because the workspace owns no process-lifetime state, a unit crash or trap is
|
||
contained: it fails the one request and the worker stays healthy. Workers
|
||
suspend the native SIGSEGV/SIGILL recovery handler around the wasm call so that
|
||
Wasmtime's own trap signals are not escalated into a native fatal signal (see
|
||
`serve_via_wasm` in `handle_complete`).
|
||
|
||
---
|
||
|
||
## 5. Request dispatch (`handle_complete`)
|
||
|
||
`handle_complete()` (`src/linux_fastcgi.cpp`) is the worker's single dispatch
|
||
point. It selects a handler string from request params and calls the same
|
||
`serve_via_wasm(entry_unit, handler)` lambda for all of them:
|
||
|
||
```
|
||
UCE_WS == "1" → serve_via_wasm(entry_unit, "websocket")
|
||
request.resources.is_cli → serve_via_wasm(entry_unit, "cli")
|
||
UCE_SERVE_HTTP == "1" → serve_via_wasm(entry_unit, "serve_http"[:fn])
|
||
otherwise (page) → serve_via_wasm(entry_unit, "render")
|
||
```
|
||
|
||
The `UCE_*` params are set by whichever broker forwarded the request:
|
||
|
||
- **Page render**: nginx → `/run/uce.sock` directly; no `UCE_*` flags → `render`.
|
||
- **CLI**: the CLI socket sets `is_cli`.
|
||
- **serve_http**: the custom-server dispatcher sets `UCE_SERVE_HTTP=1` plus
|
||
`UCE_SERVE_HTTP_FUNCTION` and rewrites `SCRIPT_FILENAME` to the configured
|
||
unit (`custom_server_http_complete`).
|
||
- **WebSocket**: the WS broker sets `UCE_WS=1` and carries the connection
|
||
identity as `UCE_WS_*` params (see §6). `handle_complete` rebuilds
|
||
`request.resources.websocket_*` and `request.connection` from them before
|
||
invoking `__uce_websocket`.
|
||
|
||
Each branch has a native-compiler fallback (`compiler_invoke*`) used only when
|
||
`wasm_backend_should_handle` declines — the holdout path during native
|
||
retirement.
|
||
|
||
---
|
||
|
||
## 6. The central WebSocket broker
|
||
|
||
The broker is the architectural centerpiece. **One process owns the HTTP port
|
||
and every WebSocket connection**, so any unit's `ws_*` call can reach any or all
|
||
connections, and a unit-code crash (which happens in a worker) never drops live
|
||
connections.
|
||
|
||
### 6.1 Why central, and why it forwards
|
||
|
||
Earlier designs gave each worker its connections and captured `ws_*` output to
|
||
return inline. That was wrong: connections couldn't outlive a worker, and one
|
||
unit could only talk to connections it happened to own. The broker fixes both —
|
||
it is the sole connection registry and data broker between connected clients and
|
||
the units that handle them. It renders nothing itself; like every other
|
||
connection owner it forwards unit invocation to the worker pool.
|
||
|
||
### 6.2 Inbound: a WS frame → a worker render (non-blocking)
|
||
|
||
`ws_broker_ws_message(request, message, opcode)` fires when a complete
|
||
(reassembled) WS message arrives on a connection. It does **not** block the
|
||
broker loop:
|
||
|
||
1. Build FastCGI params: `SCRIPT_FILENAME`, `REQUEST_METHOD=GET`, a
|
||
`REQUEST_URI` (required — `handle_request()` rejects requests without one
|
||
*before* `on_complete` runs), `UCE_WS=1`, and the connection context as
|
||
`UCE_WS_CONNECTION_ID / SCOPE / OPCODE / BINARY / CONNECTIONS / STATE`.
|
||
2. The message rides as `UCE_WS_MESSAGE` (base64) with an **empty STDIN body** —
|
||
a non-empty STDIN makes the FastCGI transport flush a premature response
|
||
before `on_complete` ever runs.
|
||
3. Connect to `/run/uce.sock` (non-blocking) and queue the encoded request in
|
||
`ws_broker_outbound[fd]`.
|
||
|
||
`ws_broker_drain_outbound()` runs after every `process(50)` tick: it finishes
|
||
writing each queued request, then drains and discards the reply (the unit's
|
||
output comes back via the command socket, not this reply), closing the fd when
|
||
the worker closes its end.
|
||
|
||
### 6.3 Outbound: `ws_*` commands flushed back to the broker
|
||
|
||
Any unit code — not just WebSocket handlers — may call `ws_send` / `ws_send_to`
|
||
/ `ws_close`. In the workspace these **record dispatch commands** rather than
|
||
touching a socket (the workspace owns no connections); `wasm-core`'s `ws_*`
|
||
(`src/lib/sys.cpp`) append to `websocket_dispatch_commands`, and
|
||
`finish_response_meta` (`src/wasm/core.cpp`) emits them as `ws_commands` (plus
|
||
`ws_connection_state` if the handler mutated per-connection state).
|
||
|
||
`wasm_backend_serve` (`src/wasm/backend.cpp`) flushes that batch at workspace
|
||
teardown — in **any** scenario, not just WS handlers — to the broker's command
|
||
socket (`WS_BROKER_SOCKET_PATH`, `/run/uce/ws-broker.sock`) via
|
||
`fcgi_forward_request` with `UCE_WS_DISPATCH=1`. This is the only path WS data
|
||
takes out of a workspace.
|
||
|
||
`ws_broker_apply_commands()` decodes the batch and applies each command against
|
||
the full registry it owns: `broadcast` (by scope), `send_to` (by connection id),
|
||
`close`. If the batch carries `connection_state`, it persists that onto the
|
||
matching live connection's `websocket_state`.
|
||
|
||
### 6.4 Un-upgraded HTTP on the WS port
|
||
|
||
The WS port can also receive ordinary (non-Upgrade) HTTP requests.
|
||
`ws_broker_complete()` routes by param: `UCE_WS_DISPATCH=1` → apply commands;
|
||
otherwise → `forward_request_to_worker()` — the *same* shared facility the
|
||
serve_http dispatcher uses, so there is no duplicated request-forwarding code.
|
||
|
||
### 6.5 The broker loop
|
||
|
||
`run_ws_broker()` drops the worker listeners it inherited
|
||
(`close_inherited_server_sockets`), installs permissive `on_request`/`on_data`
|
||
handlers (it renders nothing, so it accepts every request straight through to
|
||
`on_complete`), wires `on_complete=ws_broker_complete` and
|
||
`on_websocket_message=ws_broker_ws_message`, listens on `HTTP_PORT` and the
|
||
command socket, and loops `process(50)` + `drain_outbound()`. The design is
|
||
**non-blocking outbound dispatch + async command-socket flush, all in the
|
||
broker's single epoll loop** — the broker never blocks on a worker.
|
||
|
||
The parent respawns the broker if it dies (`ws_broker_alive` / `ensure_ws_broker`
|
||
in `main()`). A broker restart loses live connections (acceptable: crashes
|
||
happen in workers, so the broker stays up in practice), but no unit state is at
|
||
risk because the broker holds none.
|
||
|
||
---
|
||
|
||
## 7. The serve_http facility
|
||
|
||
`serve_http` units are reachable on their own bind address via a custom-server
|
||
dispatcher (`custom_server_http_dispatcher_loop`). The dispatcher owns the bind
|
||
socket and, on each request, sets `UCE_SERVE_HTTP=1` + the configured unit/
|
||
function and calls `forward_request_to_worker()`. The worker then runs
|
||
`serve_via_wasm(entry_unit, "serve_http"[:fn])`. This is the same
|
||
hold-connection-forward-render model as the WS broker, sharing
|
||
`forward_request_to_worker` and `fcgi_forward.h`.
|
||
|
||
---
|
||
|
||
## 8. Build / object layout
|
||
|
||
The native side is split into separately-compiled objects so editing the wasm
|
||
runtime does not recompile the whole TU (`scripts/build_linux.sh`):
|
||
|
||
| Object | Contents |
|
||
|---|---|
|
||
| `bin/sqlite3.o` | sqlite amalgamation (cached) |
|
||
| `bin/wasm.o` | `src/wasm/wasm_module.cpp` (backend.cpp + worker.cpp + wasmtime) |
|
||
| `bin/main.o` | `src/linux_fastcgi.cpp` (includes `lib/uce_lib.cpp`) |
|
||
|
||
Linked into one `-rdynamic` binary. ODR hazards across the objects are handled
|
||
deliberately: `context` and `my_pid`/`parent_pid` are `extern` (guarded for the
|
||
wasm core vs. unit builds), `operator new`/`delete` live in `types.cpp`, and
|
||
header free-functions are `inline`. The wasm backend exposes only declarations
|
||
(`src/wasm/backend.h`) to `main.o`.
|
||
|
||
---
|
||
|
||
## 9. Configuration keys
|
||
|
||
| Key | Default | Meaning |
|
||
|---|---|---|
|
||
| `WASM_BACKEND_ENABLED` | `1` | Route unit execution through the wasm path. |
|
||
| `WASM_BACKEND_VERBOSE` | `0` | Emit `X-UCE-Wasm-*` workspace timing headers (benchmark only). |
|
||
| `FCGI_SOCKET_PATH` | `/run/uce.sock` | Worker pool FastCGI socket (brokers forward here). |
|
||
| `CLI_SOCKET_PATH` | `/run/uce/cli.sock` | Worker CLI socket. |
|
||
| `HTTP_PORT` | `8080` | Raw HTTP + WebSocket port — owned by the WS broker. |
|
||
| `WS_BROKER_SOCKET_PATH` | `/run/uce/ws-broker.sock` | Broker command socket for `ws_*` flushes. |
|
||
| `WORKER_COUNT` | `4` | Number of uniform worker processes. |
|
||
|
||
---
|
||
|
||
## 10. Testing
|
||
|
||
- **Regression gate**: `scripts/run_cli_tests.sh --include-wasm-kill` runs the
|
||
in-runtime CLI suite (`site/tests/cli_runner.uce`) plus the site test pages.
|
||
Current baseline: **86 passed, 0 failed**.
|
||
- **WebSocket end-to-end**: a headless client performs a raw WS handshake to
|
||
`:HTTP_PORT` with path `/site/tests/websockets.ws.uce` (self-resolving
|
||
`SCRIPT_FILENAME`) and asserts the `hello-ack` frame — exercising the full
|
||
broker → worker → broker → client chain across process boundaries.
|
||
|
||
All builds, runs, and installs happen on the dev host (`k-uce` / uce-dev) over
|
||
SSH; the local checkout is edit-only.
|