udo/uce
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
uce/WASM-PROPOSAL.md

26 KiB
Raw Blame History

WASM-PROPOSAL: WebAssembly Unit Runtime for UCE

  • Status: updated design guide (2026-06-12)
  • Scope: replace the native unit pipeline (generated C++ → clang → .so → dlopen) with per-unit WebAssembly modules executed in a per-request, runtime-linked workspace, exposing the same API surface to page code, enabling memory safety, better execution control, and paving the way for supporting more source languages in the future.

1. Motivation

Two long-standing structural problems and one strategic opportunity share a single root cause: request code shares an address space and an allocator with the runtime.

  1. The arena attempt failed for a structural reason. The GLOBAL_ARENA_ALLOCATOR design in _scratchpad.cpp swapped the global operator new/delete against a current_memory_arena. A single global allocator cannot distinguish request-lifetime allocations from process-lifetime ones: during a request, server-lifetime structures (compile registry, sessions, config trees, unit statics) also allocate. Under the arena those dangle on reset; with delete as a no-op, system-allocated objects released mid-request leak. The lifetime distinction lives in the type system and call graph, not at the allocator boundary — fixing it in-process means threading PMR allocators through String, DValue, and every container.

  2. Fault recovery is best-effort, not sound. The current SIGSEGV → sigsetjmp/siglongjmp recovery performs no unwinding, skips destructors, can resume over corrupted worker state, and (see RECOMMENDATIONS.md 1.5) cannot reliably produce a useful backtrace. A faulting unit can have scribbled on runtime memory before the signal.

  3. UCE can only ever host fully-trusted code. A .uce page is arbitrary native code with full process privileges. Multi-tenant or user-supplied page hosting is structurally impossible in the native model.

A WASM execution model deletes the shared-fate fact itself rather than patching around it:

  • Arena by construction: each request runs in its own linear memory, dropped wholesale at request end. Request-lifetime memory physically cannot outlive the request; server-lifetime state physically cannot live inside it. The _scratchpad.cpp design becomes correct because the boundary is structural, not typological.
  • Sound recovery: a trap (null deref, OOB, stack exhaustion) is a defined host-side error that unwinds cleanly, with a precise guest stack trace, and the unit cannot have touched host memory. Render error page, drop workspace, keep serving — actually correct, not hopeful.
  • Capability security: page code gets exactly the imported API surface and nothing else. Multi-tenant hosting becomes possible.
  • Secondary wins: architecture-independent unit artifacts (no clang required on prod), safe module unload/replace (vs. never-safe dlclose), first-class limits (linear-memory cap = RAM limit, epoch/fuel = CPU timeout), per-request memory stats for free.

Expected outcome: ~1.2–2× compute slowdown vs. native (still far ahead of interpreted runtimes); a real ABI/membrane design; ownership of a custom loader; toolchain rough edges (§10). More isolated, shared-nothing PHP architecture but with a good story about websockets, generic sockets, background tasks, and other long running processes.

2. Rejected alternatives (recorded so they stay rejected)

  • In-process PMR arena. Requires re-typedefing String/DValue and threading allocators through the entire codebase; the failed global-swap shortcut is the only cheap version and it is unsound (§1.1).
  • One instance per component. UCE components are function calls, not RPCs: callees receive the context by reference, mutate context.call, share the ob_* capture stack and ONCE dedup state. Per-component instances force serialize/copy/deserialize of the context on every component() call (a dozen+ per page in the starter), require host-mediation of the ob stack, and silently change reference semantics to copy semantics. Rejected.
  • One linked module per app ("the blob"). Introduces an "app" concept UCE does not have, makes every edit a global relink, turns the artifact cache into a build graph — webpack's worst traits without its benefits. Rejected. The file stays the unit.
  • Eager pre-loading of known units at worker warm-up. Rejected; loading is strictly lazy, on first explicit call, preserving current semantics.

3. Architecture overview

3.1 Execution model

host process (linux_fastcgi, per worker)
│
├─ vendored wasm runtime (§10.1)
├─ unit artifact cache: one PIC .wasm per unit (replaces per-unit .so)
├─ loader (§6): dylink parsing, base allocation, GOT resolution,
│   symbol registry, ABI stamp check
├─ core snapshot: "core module, initialized" memory+table image
│
└─ per request: WORKSPACE
   ├─ linear memory (CoW-born from core snapshot)
   ├─ shared funcref table
   ├─ core module instance  (uce_lib + libc compiled to wasm)
   ├─ unit module instances (loaded lazily, on first call, incl. mid-request)
   └─ host handle table     (sqlite/mysql/file/socket handles + closers)
  • Workspace = request. Born from the core snapshot, dropped at request end. Memory drop is the arena; handle-table drop is resource cleanup (this generalizes and replaces the per-connector cleanup_*_connections() pattern — RECOMMENDATIONS.md 1.7 / 5.1 become structurally unrepresentable).
  • One unit = one PIC wasm module. Compile, cache, and invalidation granularity stay per-file. The .uce → C++ translation pipeline is unchanged; only the compile target changes (clang --target=wasm32-wasi -fPIC + wasm-ld -shared).
  • Strictly lazy loading. A unit's module is instantiated into a workspace the first time that workspace calls it — including mid-request. This is the wasm equivalent of today's compile-and-dlopen-on-first-hit and requires no restart, no fallback path. Placement memoization (deterministic bases so repeat instantiation is cheaper) is a permitted optimization; it must not change the loading policy.
  • In-flight isolation. Module versions are immutable; a recompiled unit becomes a new module. Running workspaces keep what they loaded; new workspaces get the new version. Old modules are dropped when unreferenced (safe unload — impossible with dlclose).

3.2 Memory model

  • One heap, one allocator, one DValue implementation — all owned by the core module. Unit modules import malloc/free/runtime symbols via GOT; the loader rejects any unit module that defines rather than imports them (two allocators on one heap is the one fatal misconfiguration).
  • Arena workspace allocator. Because the workspace heap is dropped wholesale, the core's allocator may be a bump allocator with no-op free — the _scratchpad.cpp design, now correct by construction. Per-deployment flag; fallback is wasi-libc dlmalloc. Memory stats = heap pointer − base (replaces the tracking operator new in types.h).
  • Unit statics reset per request (workspace is born from the core snapshot, which does not include unit data; unit data segments initialize at unit load within the workspace). This is more shared-nothing than today, where .so statics persist across requests within a worker. Cross-request state must use explicit host facilities (sessions, caches). Breaking change — must be called out in docs and checked against the site/ tree during Phase 5.

3.3 The host membrane

Exactly three currencies cross between host and workspace:

  1. scalars (i32/i64/f64),
  2. byte buffers (ptr+len into linear memory; inbound buffers are placed via the core's exported allocator), on the C++ side we strictly prefer the binary-safe std::string as a container for buffers
  3. handles (opaque u32 indices into the per-workspace host handle table; each entry carries a closer callback).

Everything pointer-shaped stays on its own side. Hostcall surface budget: 30–60 functions (§5.1). Host errors return as error values; traps are reserved for unit faults. Nothing throws across the membrane.

DValues cross the membrane only as the versioned wire encoding (§5.3), at the following sites: request context in (once), response out (once), some hostcalls.

It is expected that many if not most of the toolset API functions we expose to the unit developer will be judiciously split across the membrane: have a relatively minimal wasm part that calls into the runtime host, and then the runtime implementation which does most of the work.

3.4 DValue inside the workspace: no serialization, ever

Within a workspace, all modules share one address space, one toolchain, one set of headers — the C/C++ ABI is intact across module boundaries. Therefore:

  • A DValue is a pointer (an i32 offset into linear memory).
  • component(path, context) resolves path → table index (host registry or guest-resident map) and call_indirects, passing the context pointer. Reference semantics, mutation visibility, shared ob stack, working ONCE — identical to today.
  • Function pointers are shared-table indices, valid across modules: virtual calls, std::function callbacks (dv_map lambdas) work across units.
  • Cost: one call_indirect (single-digit ns) + GOT loads for cross-module symbols — the same shape of overhead native PIC pays through the PLT/GOT, i.e. what dlopened .so units pay today.

Encode/decode is not part of internal component calls. It exists only at the membrane (§3.3) and on the cross-instance plane (§4).

3.5 The DValue C ABI (load-bearing, build it first)

The stable contract of the workspace is a C ABI, not the C++ class: the core exports extern "C" accessors over an opaque uce_dvalue* (§5.2), plus the string/ob/print helpers. C++ units may bypass it and use the class directly (same headers, zero cost — a private fast path). Every other workspace language uses the C surface.

This ABI is versioned: every unit artifact carries a custom section (uce.abi: core ABI version + toolchain fingerprint); the loader refuses stale units and triggers lazy recompilation (units are lazily compiled anyway, so this costs nothing structurally).

Phase 1 of the implementation plan is to introduce this C ABI in the current native runtime — it is useful immediately (plugin surface, testability) and de-risks the rest.

4. Two component-call planes / multi-language support

The design stratifies languages by one question: can the toolchain produce a PIC linear-memory module that adopts a foreign allocator?

Plane A — workspace peers (C++, C, Rust, Zig, …):

  • Join the workspace as PIC modules importing core symbols.
  • Must adopt the core allocator (Rust: #[global_allocator]; Zig: allocator parameter) and must not unwind across boundaries (panic=abort / catch-at-edge).
  • Access DValues through the C ABI: pointer semantics, no copies, ns-scale calls. Idiomatic wrappers per language (e.g. Rust DValue<'request> — the borrow checker enforces the arena invariant).

Plane B — runtime-carrying languages (JS, Python, Go, C#, …):

  • No support currently planned.

The semantic rule (enforced by the loader, not by convention): cross-plane components do not receive the mutable context. They get an explicit interface — props in (copied by definition), rendered output and declared results back. Plane A keeps the full "here's the world, mutate it" contract. A component() call must never silently change mutation semantics based on the callee's implementation language.

The cross-instance call mechanism is shared by: Plane B units, and future cross-trust-boundary components (multi-tenant). Serialization boundaries and isolation boundaries are the same lines, by design.

5. ABI sketches (to be finalized in Phase 0/1)

5.1 Hostcall surface (grouped; target ≤ 60 functions)

request:   uce_host_ctx_read(buf) → len          // wire-encoded context, once
response:  uce_host_respond(status, hdrs_buf, body_buf)
           uce_host_stream_write(buf)            // chunked/streaming path
log:       uce_host_log(level, buf)
sqlite:    uce_host_sqlite_connect(path_buf) → handle | err
           uce_host_sqlite_query(handle, sql_buf, params_buf) → result_buf | err
           uce_host_sqlite_cursor_*(...)         // optional row-cursor variant
           uce_host_sqlite_insert_id/affected/error/disconnect(handle)
mysql:     (same shape; existing connector APIs are already handle-shaped)
files:     uce_host_file_read/write/stat/list(path_buf, ...)   // policy-gated
session:   uce_host_session_get/set(key_buf, val_buf)
http:      uce_host_http_request(req_buf) → handle/result_buf  // outbound
misc:      uce_host_time(), uce_host_random(buf), uce_host_env(key_buf)
loader:    uce_host_component_resolve(path_buf) → table_index  // may load (§6)
ws:        uce_host_ws_send(buf), event delivery via render entry re-invocation

Conventions: all errors as result codes + uce_host_last_error(buf); inbound buffers placed via the core's exported uce_alloc; no hostcall traps on bad input (clamp/error instead).

5.2 DValue C ABI (core exports; sketch)

typedef struct uce_dvalue uce_dvalue;                 // opaque; workspace-owned

uce_dvalue*  uce_dv_root(void);                   // request context
uce_dvalue*  uce_dv_get(uce_dvalue*, const char* key, size_t klen); // create-on-write
uce_dvalue*  uce_dv_find(uce_dvalue*, const char* key, size_t klen); // NULL if absent
const char* uce_dv_value(uce_dvalue*, size_t* len);
void        uce_dv_set_value(uce_dvalue*, const char* v, size_t vlen);
size_t      uce_dv_count(uce_dvalue*);
int         uce_dv_is_list(uce_dvalue*);
/* iteration */
uce_dv_iter uce_dv_iter_begin(uce_dvalue*);
int         uce_dv_iter_next(uce_dvalue*, uce_dv_iter*,
                                const char** key, size_t* klen, uce_dvalue** child);
/* encode/decode at the membrane */
size_t      uce_dv_encode(uce_dvalue*, char* buf, size_t cap);   // → UCEB1
uce_dvalue*  uce_dv_decode(const char* buf, size_t len);
/* ob / print / helpers: uce_print, uce_ob_start, uce_ob_get_close,
   uce_html_escape, uce_json_encode, ... (mirror uce_lib surface) */

No unwinding across this surface; C++ exceptions are caught at the edge and surfaced as error returns where fallible.

5.3 Wire encoding "UCEB1" (membrane + cross-instance only)

Length-prefixed binary tree; not JSON. Sketch (finalize against DValue's actual fields — value + ordered children):

node  := value children
value := varint len, bytes (utf-8)
children := varint count, count × ( key: varint len + bytes, node )
flags := one leading byte per node reserved (bit0: is_list hint)
header := "UCEB" u8 version

This encoding is a versioned protocol from day one (header byte). It is the Plane B contract and the membrane format; internal calls never see it.

UCEB1 encoding/decoding should also be exposed to the unit developer so they can make use of fast serialization/deserialization: matching our existing API conventions these should be ucb_encode(DValue val) and ucb_decode(String val). This may also be a worthwhile target for session variables storage (either change session to DValue or add StringMap support to UCEB1 ser/de).

5.4 Unit module contract

custom sections:  dylink.0 (standard), uce.abi { abi_version, toolchain_id }
imports:          env.memory, env.__indirect_function_table,
                  env.__memory_base, env.__table_base,
                  GOT.mem.* / GOT.func.* (resolved by loader),
                  core symbols (malloc, uce_dv_*, uce_print, ...)
exports:          uce_unit_setup, uce_unit_render,
                  uce_unit_component, uce_unit_websocket
                  (same roles as today's UCE_SETUP/RENDER/COMPONENT/WEBSOCKET
                   dlsym symbols in compiler.cpp)
forbidden:        defining malloc/free/operator new, own memory, start fn
                  with side effects beyond data init

6. The loader (host-side, custom, load-bearing)

Owned code, ~1–2k lines, vendored-runtime-adjacent. Reference logic: Emscripten's dylink loader (the ABI is the stable, battle-tested part; the server-side loader is what doesn't exist off the shelf).

Per load(unit) into a workspace:

  1. Fetch compiled module from artifact cache (compile on miss — today's lazy-compile path, retargeted).
  2. Verify uce.abi stamp against the core; on mismatch, recompile unit.
  3. Verify import discipline (no allocator/runtime definitions; §3.2).
  4. Parse dylink.0: data size/alignment, table slots needed.
  5. Allocate __memory_base (bump within workspace data region) and __table_base (append to shared table).
  6. Instantiate with bases; resolve GOT.* imports against the workspace symbol registry (core symbols + previously loaded units); register the unit's exports.
  7. Register entry points in the path → table-index dispatch map.

uce_host_component_resolve(path) consults the dispatch map and calls load() on miss — this is how lazy, programmatic, mid-request loading works with no special cases.

Placement memoization (optional, later): record each unit's first-assigned bases; reuse across workspaces so instantiation is cheaper and snapshot growth (below) stays consistent. Does not change the lazy policy.

Core snapshot: the only pre-built state is "core module, initialized" — memory bytes + table state captured once per core build. Workspaces are born from it via CoW (mmap(MAP_PRIVATE) of the snapshot image; the host owns the Memory object, so OS-level CoW is available). No units are pre-fed.

7. Request lifecycle (replaces the native flow in linux_fastcgi.cpp)

1. accept request (fastcgi, websockets message, socket event, CLI request)
2. workspace = birth_from_core_snapshot()          (CoW, ~µs)
3. write wire-encoded context into workspace; core decodes → context DValue
4. resolve entry unit (load on first call); call uce_unit_render(ctx_ptr)
5. component(path) inside guest → resolve hostcall → (lazy load) →
   call_indirect — reference semantics throughout
6. I/O via hostcalls; resources land in the workspace handle table
7. on return: encode response/headers out; write FastCGI response
   on trap: defined error → render error page with guest stack trace;
   workspace state is irrelevant because…
8. drop workspace: linear memory gone (arena), handle table closed
   (generalized resource cleanup), instances released

CPU limit: epoch/fuel interruption → same path as trap. Memory limit: linear memory max → allocation failure / trap → same path.

8. What carries over unchanged

  • The .uce → C++ translation, parser, and page semantics.
  • The lazy compile-on-first-request model and per-unit artifact caching (different artifact format).
  • The host-side connectors (sqlite/mysql) — already handle-shaped APIs; they move behind hostcalls with the same .uce-visible signatures.
  • The site tree, docs, demo, and the network test suite (tests/run_network_tests.py) — which becomes the parity harness (§9).
  • nginx/FastCGI front-end integration, worker model, websocket event flow (events re-enter via the websocket entry point).

9. Implementation plan

Phases are sequential; each has an exit criterion. No phase except 0 and 2's scaffolding produces throwaway work. All dependencies must be vendored.

Phase 0 — toolchain & runtime. Validate: wasi-sdk -fPIC + wasm-ld -shared on a representative generated unit; cross-module C++ calls with shared memory/table; exceptions decision (wasm EH vs. error-code discipline at unit boundaries — pick one, record it); vendored runtime selection. Candidates: WAMR (C, small, designed for embedding, easiest to vendor and patch — fits the project's vendoring practice and would be preferred) vs. Wasmtime (fastest, best AOT/CoW machinery, Rust — heavier to vendor/patch, use only if blocked on WAMR). Selection criteria: imported-memory + shared-table support, AOT artifact quality, patchability. Exit: a two-module (core stub + unit stub) hello-world linked at runtime by a minimal loader, in the chosen vendored runtime.

Phase 1 — DValue C ABI in the native runtime. Introduce uce_dv_* (§5.2) and the UCEB1 codec in src/lib/, used natively. Zero wasm dependency; immediately testable; freezes the contract everything else builds on. Exit: codec round-trip + accessor tests in the existing suite; ABI doc checked in.

Phase 2 — core module + membrane. Compile uce_lib (+ wasi-libc) to wasm as the core module; implement the hostcall surface (§5.1) in the host; temporary scaffolding allowed: one statically-linked unit + core to validate codegen and membrane without the loader. Exit: one real .uce page (e.g. site/tests/core.uce) renders correctly through the membrane. Scaffolding is marked throwaway.

Phase 3 — the loader + workspace. Implement §6 in full: dylink parsing, base allocation, GOT resolution, ABI/import verification, lazy mid-request loading, path dispatch. Per-request workspace birth/drop (plain memcpy birth is fine here; CoW is Phase 4). Exit: the uce-starter renders end-to-end with components loading lazily; tests/run_network_tests.py --match starter passes against the wasm worker.

Phase 4 — production mechanics. Core snapshot + CoW birth; bump-allocator flag; epoch/memory limits; trap → error-page path with guest stack traces (this supersedes the signal/longjmp machinery and closes RECOMMENDATIONS.md 1.5 structurally); handle-table cleanup (closes 1.7/5.1); artifact/ABI versioning end-to-end. Exit: kill-tests (deliberate null-deref page, infinite-loop page, OOM page) produce clean error pages and an unharmed worker.

Phase 5 — parity & performance. Full network suite green on the wasm worker; differential native-vs-wasm runs on the site tree; audit site/ for cross-request-static reliance (§3.2 breaking change); benchmark suite (template-heavy page, sqlite page, component-heavy starter page) with budgets: ≤2× native page latency, workspace birth ≤100µs, internal component call overhead within 10× native call cost. Exit: numbers published in this document, all tests and reviews pass.

Phase 6 — second plane (deferred until wanted). Cross-instance call mechanism (props-in/output-out, UCEB1), first Plane B language binding, loader enforcement of the cross-plane context rule (§4).

The native .so backend remains in-tree (as a reference) and selectable by config but we switch over to the wasm backend as soon as it's available and test only on that; both backends share the Phase 1 C ABI.

10. Risks & mitigations

Risk Mitigation
wasi-sdk PIC / shared-library maturity (least-trodden toolchain path) Phase 0 spike before any commitment; pin toolchain versions; statically link libc into the core and export from there (avoid shared wasi-libc entirely)
C++ exceptions × PIC × wasm EH Phase 0 decision point; fallback is error-code discipline at unit entry points (units already have a uniform entry shape)
Custom loader correctness (GOT, bases, relocation) Small, contained (~1–2k lines); crib logic from Emscripten's reference loader; fuzz with adversarial modules; loader rejects > loader guesses
Vendored runtime patches drift from upstream Same practice as vendored SQLite: provenance + patch files under docs/patches/; pin upstream tag; tests gate upgrades
Performance regression beyond budget Phase 5 gates; bump allocator and placement memoization in reserve; native backend retained
Multi-module DWARF / debugging story Trap stack traces cover the production case (better than today); accept weaker interactive debugging; keep native backend for local deep-debugging
Unit-statics semantic change breaks existing pages Phase 5 audit of site/; documented migration note; host-side cache facility if a real need surfaces

11. Decisions

  1. Exceptions: wasm EH or error codes at unit boundaries? Error codes.
  2. WebSocket granularity: workspace per event (pure arena, statics reset per event) or per connection (state across events, bounded lifetime)? Per-event. Should be compatible with WS since the WS contract is: runtime holds and brokers connections, makes request to unit's WS() {...} directive, unit may decide to send data back over WS to any or even all connected clients.
  3. UCEB1 final layout vs. DValue's actual field set (value/children/list flag) — somewhat open, finalize in Phase 1.
  4. Cursor vs. bulk as the default for sqlite_query results at the membrane (offer both; pick the default after Phase 5 benchmarks, tending towards cursor right now).
  5. Streaming output: today's ob model buffers; does the membrane expose uce_host_stream_write from day one or post-MVP? Undecided. ob is a critical abstraction to our whole execution model and whether we expose direct stream write or not, the existing PHP-like ob contract/paradigm must stay in place.
  6. Core snapshot rebuild cadence once placement memoization lands (dead-base reclamation policy).

12. Summary

The module is the unit (file-grained, lazily compiled, lazily loaded — including mid-request). The instance set is the workspace (one per request; shared memory + table; born from a core-only CoW snapshot; dropped wholesale — the arena, done right). The contract is the DValue C ABI inside the workspace (pointer semantics, no serialization) and the UCEB1 wire encoding + handles at every true address-space boundary (host membrane, Plane B languages, future trust boundaries). Serialization boundaries and isolation boundaries are the same lines; component calls stay function calls; and the file stays the unit.