wasm runtime: central WS broker, unified handlers, W7d holdouts, membrane completeness
- WS: a dedicated broker process owns HTTP_PORT + every connection; it forwards renders to the worker pool over uce.sock (non-blocking) and applies ws_* command batches flushed back at workspace teardown. Removes the now-dead per-worker websocket executor (-509 lines). - Dispatch: unify CLI / WebSocket / serve_http / page render through one serve_via_wasm(entry_unit, handler) path; handler string -> __uce_<handler> export symbol. - W7d: rewrite zip.uce to the membrane return-value error contract (no C++ try/catch), error-reporting.uce to genuine wasm traps instead of throw, and sharedunit.uce to unit_info(); empty the native-only token gate. - Membrane: wire ls / mkdir / file_mtime through new uce_host_file_list / uce_host_file_mkdir / uce_host_file_mtime hostcalls (resolve_guest_file gains directory support). Fixes /doc/index.uce listing nothing; adds a regression assertion that the index enumerates items. - Docs: add docs/wasm-runtime-architecture.md; record the W7e staged native- deletion plan in WASM-PROPOSAL.md. Verified: scripts/run_cli_tests.sh --include-wasm-kill -> 87 passed, 0 failed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
root
@@ -0,0 +1,318 @@
|
||||
# UCE WASM Runtime Architecture
|
||||
|
||||
Status: current as of the W7 cutover (June 2026). This document describes the
|
||||
**runtime architecture as built** — the process topology, the wasm membrane,
|
||||
the unified request dispatch, and the central WebSocket broker. For the
|
||||
historical motivation and the phased migration plan, see
|
||||
[`WASM-PROPOSAL.md`](../WASM-PROPOSAL.md); this file is the steady-state
|
||||
reference that proposal points at.
|
||||
|
||||
The guiding principle: **request code never shares an address space or an
|
||||
allocator with the runtime.** Every unit runs as a WebAssembly module inside a
|
||||
per-request workspace behind a narrow host membrane. Long-lived connection
|
||||
state (WebSocket connections, listening HTTP sockets) lives in dedicated native
|
||||
broker processes that own no unit code — they hold the connection and forward
|
||||
the actual unit invocation to a clean-engine worker, exactly like a normal
|
||||
page render.
|
||||
|
||||
---
|
||||
|
||||
## 1. Process topology
|
||||
|
||||
A single native binary (`uce_fastcgi`) forks into a small set of long-lived
|
||||
roles. None of them is special-cased per request mode; the differences are
|
||||
purely *which socket a process listens on* and *which function inside a unit
|
||||
gets invoked*.
|
||||
|
||||
```
|
||||
┌────────────────────────────┐
|
||||
nginx ──FastCGI──► worker pool (N processes) │ /run/uce.sock
|
||||
(port 80 etc.) │ uniform unit renderers │ (FastCGI + CLI)
|
||||
└─────────────▲──────────────┘
|
||||
│ forward render (FastCGI, uce.sock)
|
||||
│
|
||||
browser ──raw HTTP / WS──► ┌────────┴─────────┐
|
||||
(HTTP_PORT 8080) │ WS broker │ owns HTTP_PORT + every
|
||||
│ (1 process) │ WS connection; renders
|
||||
└────────▲─────────┘ nothing itself
|
||||
│ ws_* command flush
|
||||
│ (FastCGI, ws-broker.sock)
|
||||
on-demand serve_http ──► ┌────────┴─────────┐
|
||||
(per bind addr) │ custom-server │ owns one serve_http
|
||||
│ dispatcher(s) │ bind addr; forwards to pool
|
||||
└──────────────────┘
|
||||
|
||||
parent process: spawns/respawns all of the above + the proactive compiler.
|
||||
```
|
||||
|
||||
| Process | Owns | Renders units? | Source |
|
||||
|---|---|---|---|
|
||||
| **Parent** | nothing; supervises children | no | `main()`, `init_base_process()` |
|
||||
| **Worker** (×`WORKER_COUNT`) | `FCGI_SOCKET_PATH` (`/run/uce.sock`) + `CLI_SOCKET_PATH` | **yes** — the only processes that run wasm | `listen_for_connections()` |
|
||||
| **WS broker** (×1) | `HTTP_PORT` + every live WS connection + `WS_BROKER_SOCKET_PATH` | no — forwards to the pool | `run_ws_broker()` |
|
||||
| **serve_http dispatcher** (×bind) | one custom-server bind address | no — forwards to the pool | `custom_server_http_dispatcher_loop()` |
|
||||
| **Proactive compiler** | nothing; pre-compiles units | no | `run_proactive_compiler()` |
|
||||
|
||||
The key invariant: **only workers instantiate Wasmtime and run unit code.**
|
||||
Every connection-owning process (broker, serve_http dispatcher) forwards the
|
||||
real invocation back to a worker over `/run/uce.sock` using the minimal
|
||||
FastCGI client in [`src/lib/fcgi_forward.h`](../src/lib/fcgi_forward.h). This is
|
||||
forced by Wasmtime: an `Engine`/`Store` cannot be safely re-created across
|
||||
`fork()`, and the brokers fork from the parent that already touched the
|
||||
runtime. So the brokers hold the long-lived connection and units respond to
|
||||
events the same way they respond to a page request — through a clean worker.
|
||||
|
||||
---
|
||||
|
||||
## 2. The membrane and the DValue ABI
|
||||
|
||||
Units are compiled to WebAssembly and linked against a host import surface (the
|
||||
"membrane"). Host and guest exchange structured values as **UCEB** (a compact
|
||||
binary `DValue` encoding — `ucb_encode`/`ucb_decode`); strings cross as
|
||||
`std::string` on the native side throughout (no raw `char*` ownership across
|
||||
the boundary).
|
||||
|
||||
- `DValue` is the universal value type: scalars plus an ordered child map
|
||||
(`_map`, a `std::map<String,DValue>`). `operator[]` is non-const (creates);
|
||||
`.key(k)` is the const probe returning `const DValue*`; `.each(fn)` iterates
|
||||
children yielding `const DValue&`.
|
||||
- The request context (`params`/`get`/`post`/`cookies`/`session`, the raw body
|
||||
`in`, and—for WS—the connection context) is marshalled into a single `ctx`
|
||||
DValue, UCEB-encoded, and handed to the workspace. The response (body,
|
||||
headers, status, and any `meta` such as `ws_commands`) comes back the same
|
||||
way.
|
||||
|
||||
See [`docs/wasm-phase1-dvalue-abi.md`](wasm-phase1-dvalue-abi.md) for the wire
|
||||
format details.
|
||||
|
||||
---
|
||||
|
||||
## 3. Units, handlers, and export naming
|
||||
|
||||
A `.uce` unit compiles to a wasm module that exports one symbol per **handler**.
|
||||
There is no per-mode machinery — a "CLI unit", "WebSocket unit", and "page" are
|
||||
the same compiled artifact invoked at different exports. The dispatcher passes a
|
||||
**handler string**; the host maps it to an export symbol:
|
||||
|
||||
```
|
||||
__uce_<base>[_<sanitize(suffix)>]
|
||||
```
|
||||
|
||||
`handler_export_symbol()` (`src/wasm/worker.cpp`) splits on the first `:`:
|
||||
|
||||
| Handler string | Export symbol |
|
||||
|---|---|
|
||||
| `render` | `__uce_render` |
|
||||
| `cli` | `__uce_cli` |
|
||||
| `websocket` | `__uce_websocket` |
|
||||
| `once` | `__uce_once` (optional; absence is not an error) |
|
||||
| `serve_http` | `__uce_serve_http` |
|
||||
| `serve_http:named` | `__uce_serve_http_named` |
|
||||
| `component:CARD` | `__uce_component_CARD` |
|
||||
| `exists` | probe only — resolves the unit, loads nothing |
|
||||
|
||||
`sanitize_symbol_suffix()` keeps `[A-Za-z0-9_]` (mirrors `ascii_safe_name`).
|
||||
`wasm_resolve_target(unit, handler)` (`src/wasm/core.cpp`) resolves the source
|
||||
path and looks up the export's funcref slot; `exists` lets callers probe a unit
|
||||
without instantiating it.
|
||||
|
||||
`wasm_backend_should_handle(request, entry_unit)` gates whether the wasm path
|
||||
takes the request (vs. the legacy native compiler path, still selectable while
|
||||
the native pipeline is retired). With `WASM_BACKEND_ENABLED=1` this is the
|
||||
default for all unit execution.
|
||||
|
||||
---
|
||||
|
||||
## 4. The workspace runtime
|
||||
|
||||
Each request gets a fresh **workspace** — a per-request wasm instance tree with
|
||||
the membrane wired in. `wasm_worker_serve(worker, ctx, entry_unit, handler)`
|
||||
(`src/wasm/worker.cpp`) is the single entry point for *every* mode:
|
||||
|
||||
1. Birth a workspace (CoW-snapshot-based where available).
|
||||
2. Resolve `entry_unit` + `handler` to an export; components referenced at
|
||||
runtime are resolved on demand via the `uce_host_component_resolve` hostcall
|
||||
(`component_resolve()` → `__uce_<...>` slot), loading dependency modules
|
||||
lazily and recording resolve counts/timings.
|
||||
3. Invoke the export. Unit code calls host functions (filesystem, sqlite, regex,
|
||||
markdown, time, tasks, `ws_*`, …) across the membrane.
|
||||
4. Collect the response (`WasmResponse`: body, headers, status, `meta`).
|
||||
|
||||
Because the workspace owns no process-lifetime state, a unit crash or trap is
|
||||
contained: it fails the one request and the worker stays healthy. Workers
|
||||
suspend the native SIGSEGV/SIGILL recovery handler around the wasm call so that
|
||||
Wasmtime's own trap signals are not escalated into a native fatal signal (see
|
||||
`serve_via_wasm` in `handle_complete`).
|
||||
|
||||
---
|
||||
|
||||
## 5. Request dispatch (`handle_complete`)
|
||||
|
||||
`handle_complete()` (`src/linux_fastcgi.cpp`) is the worker's single dispatch
|
||||
point. It selects a handler string from request params and calls the same
|
||||
`serve_via_wasm(entry_unit, handler)` lambda for all of them:
|
||||
|
||||
```
|
||||
UCE_WS == "1" → serve_via_wasm(entry_unit, "websocket")
|
||||
request.resources.is_cli → serve_via_wasm(entry_unit, "cli")
|
||||
UCE_SERVE_HTTP == "1" → serve_via_wasm(entry_unit, "serve_http"[:fn])
|
||||
otherwise (page) → serve_via_wasm(entry_unit, "render")
|
||||
```
|
||||
|
||||
The `UCE_*` params are set by whichever broker forwarded the request:
|
||||
|
||||
- **Page render**: nginx → `/run/uce.sock` directly; no `UCE_*` flags → `render`.
|
||||
- **CLI**: the CLI socket sets `is_cli`.
|
||||
- **serve_http**: the custom-server dispatcher sets `UCE_SERVE_HTTP=1` plus
|
||||
`UCE_SERVE_HTTP_FUNCTION` and rewrites `SCRIPT_FILENAME` to the configured
|
||||
unit (`custom_server_http_complete`).
|
||||
- **WebSocket**: the WS broker sets `UCE_WS=1` and carries the connection
|
||||
identity as `UCE_WS_*` params (see §6). `handle_complete` rebuilds
|
||||
`request.resources.websocket_*` and `request.connection` from them before
|
||||
invoking `__uce_websocket`.
|
||||
|
||||
Each branch has a native-compiler fallback (`compiler_invoke*`) used only when
|
||||
`wasm_backend_should_handle` declines — the holdout path during native
|
||||
retirement.
|
||||
|
||||
---
|
||||
|
||||
## 6. The central WebSocket broker
|
||||
|
||||
The broker is the architectural centerpiece. **One process owns the HTTP port
|
||||
and every WebSocket connection**, so any unit's `ws_*` call can reach any or all
|
||||
connections, and a unit-code crash (which happens in a worker) never drops live
|
||||
connections.
|
||||
|
||||
### 6.1 Why central, and why it forwards
|
||||
|
||||
Earlier designs gave each worker its connections and captured `ws_*` output to
|
||||
return inline. That was wrong: connections couldn't outlive a worker, and one
|
||||
unit could only talk to connections it happened to own. The broker fixes both —
|
||||
it is the sole connection registry and data broker between connected clients and
|
||||
the units that handle them. It renders nothing itself; like every other
|
||||
connection owner it forwards unit invocation to the worker pool.
|
||||
|
||||
### 6.2 Inbound: a WS frame → a worker render (non-blocking)
|
||||
|
||||
`ws_broker_ws_message(request, message, opcode)` fires when a complete
|
||||
(reassembled) WS message arrives on a connection. It does **not** block the
|
||||
broker loop:
|
||||
|
||||
1. Build FastCGI params: `SCRIPT_FILENAME`, `REQUEST_METHOD=GET`, a
|
||||
`REQUEST_URI` (required — `handle_request()` rejects requests without one
|
||||
*before* `on_complete` runs), `UCE_WS=1`, and the connection context as
|
||||
`UCE_WS_CONNECTION_ID / SCOPE / OPCODE / BINARY / CONNECTIONS / STATE`.
|
||||
2. The message rides as `UCE_WS_MESSAGE` (base64) with an **empty STDIN body** —
|
||||
a non-empty STDIN makes the FastCGI transport flush a premature response
|
||||
before `on_complete` ever runs.
|
||||
3. Connect to `/run/uce.sock` (non-blocking) and queue the encoded request in
|
||||
`ws_broker_outbound[fd]`.
|
||||
|
||||
`ws_broker_drain_outbound()` runs after every `process(50)` tick: it finishes
|
||||
writing each queued request, then drains and discards the reply (the unit's
|
||||
output comes back via the command socket, not this reply), closing the fd when
|
||||
the worker closes its end.
|
||||
|
||||
### 6.3 Outbound: `ws_*` commands flushed back to the broker
|
||||
|
||||
Any unit code — not just WebSocket handlers — may call `ws_send` / `ws_send_to`
|
||||
/ `ws_close`. In the workspace these **record dispatch commands** rather than
|
||||
touching a socket (the workspace owns no connections); `wasm-core`'s `ws_*`
|
||||
(`src/lib/sys.cpp`) append to `websocket_dispatch_commands`, and
|
||||
`finish_response_meta` (`src/wasm/core.cpp`) emits them as `ws_commands` (plus
|
||||
`ws_connection_state` if the handler mutated per-connection state).
|
||||
|
||||
`wasm_backend_serve` (`src/wasm/backend.cpp`) flushes that batch at workspace
|
||||
teardown — in **any** scenario, not just WS handlers — to the broker's command
|
||||
socket (`WS_BROKER_SOCKET_PATH`, `/run/uce/ws-broker.sock`) via
|
||||
`fcgi_forward_request` with `UCE_WS_DISPATCH=1`. This is the only path WS data
|
||||
takes out of a workspace.
|
||||
|
||||
`ws_broker_apply_commands()` decodes the batch and applies each command against
|
||||
the full registry it owns: `broadcast` (by scope), `send_to` (by connection id),
|
||||
`close`. If the batch carries `connection_state`, it persists that onto the
|
||||
matching live connection's `websocket_state`.
|
||||
|
||||
### 6.4 Un-upgraded HTTP on the WS port
|
||||
|
||||
The WS port can also receive ordinary (non-Upgrade) HTTP requests.
|
||||
`ws_broker_complete()` routes by param: `UCE_WS_DISPATCH=1` → apply commands;
|
||||
otherwise → `forward_request_to_worker()` — the *same* shared facility the
|
||||
serve_http dispatcher uses, so there is no duplicated request-forwarding code.
|
||||
|
||||
### 6.5 The broker loop
|
||||
|
||||
`run_ws_broker()` drops the worker listeners it inherited
|
||||
(`close_inherited_server_sockets`), installs permissive `on_request`/`on_data`
|
||||
handlers (it renders nothing, so it accepts every request straight through to
|
||||
`on_complete`), wires `on_complete=ws_broker_complete` and
|
||||
`on_websocket_message=ws_broker_ws_message`, listens on `HTTP_PORT` and the
|
||||
command socket, and loops `process(50)` + `drain_outbound()`. The design is
|
||||
**non-blocking outbound dispatch + async command-socket flush, all in the
|
||||
broker's single epoll loop** — the broker never blocks on a worker.
|
||||
|
||||
The parent respawns the broker if it dies (`ws_broker_alive` / `ensure_ws_broker`
|
||||
in `main()`). A broker restart loses live connections (acceptable: crashes
|
||||
happen in workers, so the broker stays up in practice), but no unit state is at
|
||||
risk because the broker holds none.
|
||||
|
||||
---
|
||||
|
||||
## 7. The serve_http facility
|
||||
|
||||
`serve_http` units are reachable on their own bind address via a custom-server
|
||||
dispatcher (`custom_server_http_dispatcher_loop`). The dispatcher owns the bind
|
||||
socket and, on each request, sets `UCE_SERVE_HTTP=1` + the configured unit/
|
||||
function and calls `forward_request_to_worker()`. The worker then runs
|
||||
`serve_via_wasm(entry_unit, "serve_http"[:fn])`. This is the same
|
||||
hold-connection-forward-render model as the WS broker, sharing
|
||||
`forward_request_to_worker` and `fcgi_forward.h`.
|
||||
|
||||
---
|
||||
|
||||
## 8. Build / object layout
|
||||
|
||||
The native side is split into separately-compiled objects so editing the wasm
|
||||
runtime does not recompile the whole TU (`scripts/build_linux.sh`):
|
||||
|
||||
| Object | Contents |
|
||||
|---|---|
|
||||
| `bin/sqlite3.o` | sqlite amalgamation (cached) |
|
||||
| `bin/wasm.o` | `src/wasm/wasm_module.cpp` (backend.cpp + worker.cpp + wasmtime) |
|
||||
| `bin/main.o` | `src/linux_fastcgi.cpp` (includes `lib/uce_lib.cpp`) |
|
||||
|
||||
Linked into one `-rdynamic` binary. ODR hazards across the objects are handled
|
||||
deliberately: `context` and `my_pid`/`parent_pid` are `extern` (guarded for the
|
||||
wasm core vs. unit builds), `operator new`/`delete` live in `types.cpp`, and
|
||||
header free-functions are `inline`. The wasm backend exposes only declarations
|
||||
(`src/wasm/backend.h`) to `main.o`.
|
||||
|
||||
---
|
||||
|
||||
## 9. Configuration keys
|
||||
|
||||
| Key | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `WASM_BACKEND_ENABLED` | `1` | Route unit execution through the wasm path. |
|
||||
| `WASM_BACKEND_VERBOSE` | `0` | Emit `X-UCE-Wasm-*` workspace timing headers (benchmark only). |
|
||||
| `FCGI_SOCKET_PATH` | `/run/uce.sock` | Worker pool FastCGI socket (brokers forward here). |
|
||||
| `CLI_SOCKET_PATH` | `/run/uce/cli.sock` | Worker CLI socket. |
|
||||
| `HTTP_PORT` | `8080` | Raw HTTP + WebSocket port — owned by the WS broker. |
|
||||
| `WS_BROKER_SOCKET_PATH` | `/run/uce/ws-broker.sock` | Broker command socket for `ws_*` flushes. |
|
||||
| `WORKER_COUNT` | `4` | Number of uniform worker processes. |
|
||||
|
||||
---
|
||||
|
||||
## 10. Testing
|
||||
|
||||
- **Regression gate**: `scripts/run_cli_tests.sh --include-wasm-kill` runs the
|
||||
in-runtime CLI suite (`site/tests/cli_runner.uce`) plus the site test pages.
|
||||
Current baseline: **86 passed, 0 failed**.
|
||||
- **WebSocket end-to-end**: a headless client performs a raw WS handshake to
|
||||
`:HTTP_PORT` with path `/site/tests/websockets.ws.uce` (self-resolving
|
||||
`SCRIPT_FILENAME`) and asserts the `hello-ack` frame — exercising the full
|
||||
broker → worker → broker → client chain across process boundaries.
|
||||
|
||||
All builds, runs, and installs happen on the dev host (`k-uce` / uce-dev) over
|
||||
SSH; the local checkout is edit-only.
|
||||
Reference in New Issue
Block a user