uce/docs/wasm-runtime-architecture.md

UCE WASM Runtime Architecture

Status: current as of the W7e native-pipeline removal (June 2026). This document describes the runtime architecture as built — the process topology, the wasm membrane, the unified request dispatch, and the central WebSocket broker. Native .so unit execution/dlopen fallback has been removed; the parser and preprocessor remain only as the front-end that emits C++ for wasm side-module compilation. For the historical motivation and the phased migration plan, see WASM-PROPOSAL.md; this file is the steady-state reference that proposal points at.

The guiding principle: request code never shares an address space or an allocator with the runtime. Every unit runs as a WebAssembly module inside a per-request workspace behind a narrow host membrane. Long-lived connection state (WebSocket connections, listening HTTP sockets) lives in dedicated native broker processes that own no unit code — they hold the connection and forward the actual unit invocation to a clean-engine worker, exactly like a normal page render.

---

1. Process topology

A single native binary (uce_fastcgi) forks into a small set of long-lived roles. None of them is special-cased per request mode; the differences are purely which socket a process listens on and which function inside a unit gets invoked.

                          ┌────────────────────────────┐
        nginx  ──FastCGI──►  worker pool (N processes)  │  /run/uce.sock
   (port 80 etc.)          │  uniform unit renderers    │  (FastCGI + CLI)
                          └─────────────▲──────────────┘
                                        │ forward render (FastCGI, uce.sock)
                                        │
   browser ──raw HTTP / WS──►  ┌────────┴─────────┐
        (HTTP_PORT 8080)       │   WS broker      │  owns HTTP_PORT + every
                               │  (1 process)     │  WS connection; renders
                               └────────▲─────────┘  nothing itself
                                        │ ws_* command flush
                                        │ (FastCGI, ws-broker.sock)
   on-demand serve_http   ──►  ┌────────┴─────────┐
        (per bind addr)        │ custom-server    │  owns one serve_http
                               │ dispatcher(s)    │  bind addr; forwards to pool
                               └──────────────────┘

   parent process: spawns/respawns all of the above + the proactive compiler.

Process	Owns	Renders units?	Source
Parent	nothing; supervises children	no	main(), init_base_process()
Worker (×WORKER_COUNT)	FCGI_SOCKET_PATH (/run/uce.sock) + CLI_SOCKET_PATH	yes — the only processes that run wasm	listen_for_connections()
WS broker (×1)	HTTP_PORT + every live WS connection + WS_BROKER_SOCKET_PATH	no — forwards to the pool	run_ws_broker()
serve_http dispatcher (×bind)	one custom-server bind address	no — forwards to the pool	custom_server_http_dispatcher_loop()
Proactive compiler	nothing; pre-compiles units	no	run_proactive_compiler()

The key invariant: only workers instantiate Wasmtime and run unit code. Every connection-owning process (broker, serve_http dispatcher) forwards the real invocation back to a worker over /run/uce.sock using the minimal FastCGI client in src/lib/fcgi_forward.h. This is forced by Wasmtime: an Engine/Store cannot be safely re-created across fork(), and the brokers fork from the parent that already touched the runtime. So the brokers hold the long-lived connection and units respond to events the same way they respond to a page request — through a clean worker.

---

2. The membrane and the DValue ABI

Units are compiled to WebAssembly and linked against a host import surface (the "membrane"). Host and guest exchange structured values as UCEB (a compact binary DValue encoding — ucb_encode/ucb_decode); strings cross as std::string on the native side throughout (no raw char* ownership across the boundary).

• DValue is the universal value type: scalars plus an ordered child map (_map, a std::map<String,DValue>). operator[] is non-const (creates); .key(k) is the const probe returning const DValue*; .each(fn) iterates children yielding const DValue&.
• The request context (params/get/post/cookies/session, the raw body in, and—for WS—the connection context) is marshalled into a single ctx DValue, UCEB-encoded, and handed to the workspace. The response (body, headers, status, and any meta such as ws_commands) comes back the same way.

See docs/wasm-phase1-dvalue-abi.md for the wire format details.

---

3. Units, handlers, and export naming

A .uce unit compiles to a wasm module that exports one symbol per handler. There is no per-mode machinery — a "CLI unit", "WebSocket unit", and "page" are the same compiled artifact invoked at different exports. The dispatcher passes a handler string; the host maps it to an export symbol:

__uce_<base>[_<sanitize(suffix)>]

handler_export_symbol() (src/wasm/worker.cpp) splits on the first ::

Handler string	Export symbol
render	__uce_render
cli	__uce_cli
websocket	__uce_websocket
once	__uce_once (optional; absence is not an error)
serve_http	__uce_serve_http
serve_http:named	__uce_serve_http_named
component:CARD	__uce_component_CARD
exists	probe only — resolves the unit, loads nothing

sanitize_symbol_suffix() keeps [A-Za-z0-9_] (mirrors ascii_safe_name). wasm_resolve_target(unit, handler) (src/wasm/core.cpp) resolves the source path and looks up the export's funcref slot; exists lets callers probe a unit without instantiating it.

wasm_backend_should_handle(request, entry_unit) checks whether the wasm backend is initialized and the requested artifact/handler is currently available. If an artifact is cold or stale, dispatch compiles it on demand via get_shared_unit() and rechecks. There is no native unit-execution fallback.

---

4. The workspace runtime

Each request gets a fresh workspace — a per-request wasm instance tree with the membrane wired in. wasm_worker_serve(worker, ctx, entry_unit, handler) (src/wasm/worker.cpp) is the single entry point for every mode:

1. Birth a workspace (CoW-snapshot-based where available).
2. Resolve entry_unit + handler to an export; components referenced at runtime are resolved on demand via the uce_host_component_resolve hostcall (component_resolve() → __uce_<...> slot), loading dependency modules lazily and recording resolve counts/timings.
3. Invoke the export. Unit code calls host functions (filesystem, sqlite, regex, markdown, time, tasks, ws_*, …) across the membrane.
4. Collect the response (WasmResponse: body, headers, status, meta).

Because the workspace owns no process-lifetime state, a unit crash or trap is contained: it fails the one request and the worker stays healthy. Workers suspend the native SIGSEGV/SIGILL recovery handler around the wasm call so that Wasmtime's own trap signals are not escalated into a native fatal signal (see serve_via_wasm in handle_complete).

---

5. Request dispatch (handle_complete)

handle_complete() (src/linux_fastcgi.cpp) is the worker's single dispatch point. It selects a handler string from request params and calls the same serve_via_wasm(entry_unit, handler) lambda for all of them:

UCE_WS == "1"            → serve_via_wasm(entry_unit, "websocket")
request.resources.is_cli → serve_via_wasm(entry_unit, "cli")
UCE_SERVE_HTTP == "1"    → serve_via_wasm(entry_unit, "serve_http"[:fn])
otherwise (page)         → serve_via_wasm(entry_unit, "render")

The UCE_* params are set by whichever broker forwarded the request:

• Page render: nginx → /run/uce.sock directly; no UCE_* flags → render.
• CLI: the CLI socket sets is_cli.
• serve_http: the custom-server dispatcher sets UCE_SERVE_HTTP=1 plus UCE_SERVE_HTTP_FUNCTION and rewrites SCRIPT_FILENAME to the configured unit (custom_server_http_complete).
• WebSocket: the WS broker sets UCE_WS=1 and carries the connection identity as UCE_WS_* params (see §6). handle_complete rebuilds request.resources.websocket_* and request.connection from them before invoking __uce_websocket.

If a unit still cannot be served after the on-demand wasm compile, the worker returns a clean 500 with the wasm/compile error; it does not execute native unit code.

---

6. The central WebSocket broker

The broker is the architectural centerpiece. One process owns the HTTP port and every WebSocket connection, so any unit's ws_* call can reach any or all connections, and a unit-code crash (which happens in a worker) never drops live connections.

6.1 Why central, and why it forwards

Earlier designs gave each worker its connections and captured ws_* output to return inline. That was wrong: connections couldn't outlive a worker, and one unit could only talk to connections it happened to own. The broker fixes both — it is the sole connection registry and data broker between connected clients and the units that handle them. It renders nothing itself; like every other connection owner it forwards unit invocation to the worker pool.

6.2 Inbound: a WS frame → a worker render (non-blocking)

ws_broker_ws_message(request, message, opcode) fires when a complete (reassembled) WS message arrives on a connection. It does not block the broker loop:

1. Build FastCGI params: SCRIPT_FILENAME, REQUEST_METHOD=GET, a REQUEST_URI (required — handle_request() rejects requests without one before on_complete runs), UCE_WS=1, and the connection context as UCE_WS_CONNECTION_ID / SCOPE / OPCODE / BINARY / CONNECTIONS / STATE.
2. The message rides as UCE_WS_MESSAGE (base64) with an empty STDIN body — a non-empty STDIN makes the FastCGI transport flush a premature response before on_complete ever runs.
3. Connect to /run/uce.sock (non-blocking) and queue the encoded request in ws_broker_outbound[fd].

ws_broker_drain_outbound() runs after every process(50) tick: it finishes writing each queued request, then drains and discards the reply (the unit's output comes back via the command socket, not this reply), closing the fd when the worker closes its end.

6.3 Outbound: ws_* commands flushed back to the broker

Any unit code — not just WebSocket handlers — may call ws_send / ws_send_to / ws_close. In the workspace these record dispatch commands rather than touching a socket (the workspace owns no connections); wasm-core's ws_* (src/lib/sys.cpp) append to websocket_dispatch_commands, and finish_response_meta (src/wasm/core.cpp) emits them as ws_commands (plus ws_connection_state if the handler mutated per-connection state).

wasm_backend_serve (src/wasm/backend.cpp) flushes that batch at workspace teardown — in any scenario, not just WS handlers — to the broker's command socket (WS_BROKER_SOCKET_PATH, /run/uce/ws-broker.sock) via fcgi_forward_request with UCE_WS_DISPATCH=1. This is the only path WS data takes out of a workspace.

ws_broker_apply_commands() decodes the batch and applies each command against the full registry it owns: broadcast (by scope), send_to (by connection id), close. If the batch carries connection_state, it persists that onto the matching live connection's websocket_state.

6.4 Un-upgraded HTTP on the WS port

The WS port can also receive ordinary (non-Upgrade) HTTP requests. ws_broker_complete() routes by param: UCE_WS_DISPATCH=1 → apply commands; otherwise → forward_request_to_worker() — the same shared facility the serve_http dispatcher uses, so there is no duplicated request-forwarding code.

6.5 The broker loop

run_ws_broker() drops the worker listeners it inherited (close_inherited_server_sockets), installs permissive on_request/on_data handlers (it renders nothing, so it accepts every request straight through to on_complete), wires on_complete=ws_broker_complete and on_websocket_message=ws_broker_ws_message, listens on HTTP_PORT and the command socket, and loops process(50) + drain_outbound(). The design is non-blocking outbound dispatch + async command-socket flush, all in the broker's single epoll loop — the broker never blocks on a worker.

The parent respawns the broker if it dies (ws_broker_alive / ensure_ws_broker in main()). A broker restart loses live connections (acceptable: crashes happen in workers, so the broker stays up in practice), but no unit state is at risk because the broker holds none.

---

7. The serve_http facility

serve_http units are reachable on their own bind address via a custom-server dispatcher (custom_server_http_dispatcher_loop). The dispatcher owns the bind socket and, on each request, sets UCE_SERVE_HTTP=1 + the configured unit/ function and calls forward_request_to_worker(). The worker then runs serve_via_wasm(entry_unit, "serve_http"[:fn]). This is the same hold-connection-forward-render model as the WS broker, sharing forward_request_to_worker and fcgi_forward.h.

---

8. Build / object layout

The native side is split into separately-compiled objects so editing the wasm runtime does not recompile the whole TU (scripts/build_linux.sh):

Object	Contents
bin/sqlite3.o	sqlite amalgamation (cached)
bin/wasm.o	src/wasm/wasm_module.cpp (backend.cpp + worker.cpp + wasmtime)
bin/main.o	src/linux_fastcgi.cpp (includes lib/uce_lib.cpp)

Linked into one -rdynamic binary. ODR hazards across the objects are handled deliberately: context and my_pid/parent_pid are extern (guarded for the wasm core vs. unit builds), operator new/delete live in types.cpp, and header free-functions are inline. The wasm backend exposes only declarations (src/wasm/backend.h) to main.o.

---

9. Configuration keys

Key	Default	Meaning
WASM_BACKEND_VERBOSE	0	Emit X-UCE-Wasm-* workspace timing headers (benchmark only).
FCGI_SOCKET_PATH	/run/uce.sock	Worker pool FastCGI socket (brokers forward here).
CLI_SOCKET_PATH	/run/uce/cli.sock	Worker CLI socket.
HTTP_PORT	8080	Raw HTTP + WebSocket port — owned by the WS broker.
WS_BROKER_SOCKET_PATH	/run/uce/ws-broker.sock	Broker command socket for ws_* flushes.
WORKER_COUNT	4	Number of uniform worker processes.

---

10. Testing

• Regression gate: scripts/run_cli_tests.sh --include-wasm-kill runs the in-runtime CLI suite (site/tests/cli_runner.uce) plus the site test pages. Current baseline: 87 passed, 0 failed.
• WebSocket end-to-end: a headless client performs a raw WS handshake to :HTTP_PORT with path /site/tests/websockets.ws.uce (self-resolving SCRIPT_FILENAME) and asserts the hello-ack frame — exercising the full broker → worker → broker → client chain across process boundaries.

All builds, runs, and installs happen on the dev host (k-uce / uce-dev) over SSH; the local checkout is edit-only.