diff --git a/README.md b/README.md index 6e1ea6b..a8a5f3b 100644 --- a/README.md +++ b/README.md @@ -293,7 +293,7 @@ The runtime reads its server settings from: /etc/uce/settings.cfg ``` -The shipped example contains the important filesystem and FastCGI settings: +The example contains the filesystem and FastCGI settings: ```ini BIN_DIRECTORY=/var/cache/uce/work @@ -473,7 +473,7 @@ Important details: - `SCRIPT_FILENAME` should resolve to the actual `.uce` file on disk - `proxy_http_version 1.1` and the `Upgrade` / `Connection` headers are required for WebSockets -The `location /` block above is intentionally conservative and only serves real files from `site/`. If your app uses a front-controller pattern such as routing everything through `/index.uce`, change that block accordingly. +The `location /` block only serves files from `site/`. If your app uses a front-controller pattern such as routing everything through `/index.uce`, change that block accordingly. ### 6. Think about document root and private files diff --git a/WASM-PROPOSAL.md b/WASM-PROPOSAL.md deleted file mode 100644 index 64eff5f..0000000 --- a/WASM-PROPOSAL.md +++ /dev/null @@ -1,1080 +0,0 @@ -# WASM-PROPOSAL: WebAssembly Unit Runtime for UCE - -> **Steady-state reference:** for the runtime as it is actually built today -> (process topology, the membrane, unified dispatch, and the central WebSocket -> broker), see [`docs/wasm-runtime-architecture.md`](docs/wasm-runtime-architecture.md). -> This proposal retains the motivation and the phased migration plan. - -- **Status:** building the production worker per §9.1 (W-phases). W1 (core - module), W2 (wasm unit compile target), W3 (workspace runtime + membrane, - `src/wasm/worker.cpp`), and W4 (config-selectable FastCGI backend, - `src/wasm/backend.cpp`) are **done**: with `WASM_BACKEND_ENABLED=1` the - real nginx → fastcgi → wasm path serves the starter app (parity 14/14) and - produces clean error pages from an unharmed worker on real kill pages. - W5 cutover is done: WASM is the default page-render backend with native kept - as the config-selectable reference/fallback; the membrane now covers regex, - xml/yaml, markdown, filesystem read/write, sqlite, background tasks, sleep, - and time formatting. W6 spike cleanup is done (prototypes deleted, gates - re-homed to `tests/`/`docs/`); the native-machinery retirements stay gated - while native still serves the remaining fallback surfaces (zip, sockets, - memcache, mysql, `unit_call`, compiler-introspection). Open items: the - sqlite membrane is ~6–7× native (per-query double hostcall + UCEB1 - marshalling — §11.4 cursor/bulk decision), and CoW snapshot birth for the - workspace-birth budget. -- **Scope:** replace the native unit pipeline (generated C++ → clang → `.so` → - `dlopen`) with per-unit WebAssembly modules executed in a per-request, - runtime-linked workspace, exposing the same API surface to page code, - enabling memory safety, better execution control, and paving the way for - supporting more source languages in the future. - ---- - -## 1. Motivation - -Two long-standing structural problems and one strategic opportunity share a -single root cause: **request code shares an address space and an allocator -with the runtime.** - -1. **The arena attempt failed for a structural reason.** The - `GLOBAL_ARENA_ALLOCATOR` design in `_scratchpad.cpp` swapped the global - `operator new`/`delete` against a `current_memory_arena`. A single global - allocator cannot distinguish request-lifetime allocations from - process-lifetime ones: during a request, server-lifetime structures - (compile registry, sessions, config trees, unit statics) also allocate. - Under the arena those dangle on reset; with `delete` as a no-op, - system-allocated objects released mid-request leak. The lifetime - distinction lives in the type system and call graph, not at the allocator - boundary — fixing it in-process means threading PMR allocators through - `String`, `DValue`, and every container. - -2. **Fault recovery is best-effort, not sound.** The current - SIGSEGV → `sigsetjmp`/`siglongjmp` recovery performs no unwinding, skips - destructors, can resume over corrupted worker state, and (see - RECOMMENDATIONS.md 1.5) cannot reliably produce a useful backtrace. - A faulting unit *can* have scribbled on runtime memory before the signal. - -3. **UCE can only ever host fully-trusted code.** A `.uce` page is arbitrary - native code with full process privileges. Multi-tenant or user-supplied - page hosting is structurally impossible in the native model. - -A WASM execution model deletes the shared-fate fact itself rather than -patching around it: - -- **Arena by construction:** each request runs in its own linear memory, - dropped wholesale at request end. Request-lifetime memory physically cannot - outlive the request; server-lifetime state physically cannot live inside - it. The `_scratchpad.cpp` design becomes correct because the boundary is - structural, not typological. -- **Sound recovery:** a trap (null deref, OOB, stack exhaustion) is a defined - host-side error that unwinds cleanly, with a precise guest stack trace, and - the unit cannot have touched host memory. Render error page, drop - workspace, keep serving — actually correct, not hopeful. -- **Capability security:** page code gets exactly the imported API surface - and nothing else. Multi-tenant hosting becomes possible. -- **Secondary wins:** architecture-independent unit artifacts (no clang - required on prod), safe module unload/replace (vs. never-safe `dlclose`), - first-class limits (linear-memory cap = RAM limit, epoch/fuel = CPU - timeout), per-request memory stats for free. - -**Expected outcome:** ~1.2–2× compute slowdown vs. native (still far ahead of -interpreted runtimes); a real ABI/membrane design; ownership of a custom -loader; toolchain rough edges (§10). More isolated, shared-nothing PHP architecture -but with a good story about websockets, generic sockets, background tasks, -and other long running processes. - ---- - -## 2. Rejected alternatives (recorded so they stay rejected) - -- **In-process PMR arena.** Requires re-typedefing `String`/`DValue` and - threading allocators through the entire codebase; the failed global-swap - shortcut is the only cheap version and it is unsound (§1.1). -- **One instance per component.** UCE components are function calls, not - RPCs: callees receive the context **by reference**, mutate `context.call`, - share the `ob_*` capture stack and `ONCE` dedup state. Per-component - instances force serialize/copy/deserialize of the context on every - `component()` call (a dozen+ per page in the starter), require - host-mediation of the ob stack, and silently change reference semantics to - copy semantics. Rejected. -- **One linked module per app ("the blob").** Introduces an "app" concept - UCE does not have, makes every edit a global relink, turns the artifact - cache into a build graph — webpack's worst traits without its benefits. - Rejected. The file stays the unit. -- **Eager pre-loading of known units at worker warm-up.** Rejected; loading - is strictly lazy, on first explicit call, preserving current semantics. - ---- - -## 3. Architecture overview - -### 3.1 Execution model - -``` -host process (linux_fastcgi, per worker) -│ -├─ vendored wasm runtime (§10.1) -├─ unit artifact cache: one PIC .wasm per unit (replaces per-unit .so) -├─ loader (§6): dylink parsing, base allocation, GOT resolution, -│ symbol registry, ABI stamp check -├─ core snapshot: "core module, initialized" memory+table image -│ -└─ per request: WORKSPACE - ├─ linear memory (CoW-born from core snapshot) - ├─ shared funcref table - ├─ core module instance (uce_lib + libc compiled to wasm) - ├─ unit module instances (loaded lazily, on first call, incl. mid-request) - └─ host handle table (sqlite/mysql/file/socket handles + closers) -``` - -- **Workspace = request.** Born from the core snapshot, dropped at request - end. Memory drop is the arena; handle-table drop is resource cleanup - (this generalizes and replaces the per-connector - `cleanup_*_connections()` pattern — RECOMMENDATIONS.md 1.7 / 5.1 become - structurally unrepresentable). -- **One unit = one PIC wasm module.** Compile, cache, and invalidation - granularity stay per-file. The `.uce → C++` translation pipeline is - unchanged; only the compile target changes - (`clang --target=wasm32-wasi -fPIC` + `wasm-ld -shared`). -- **Strictly lazy loading.** A unit's module is instantiated into a - workspace the first time that workspace calls it — including mid-request. - This is the wasm equivalent of today's compile-and-`dlopen`-on-first-hit - and requires no restart, no fallback path. Placement memoization - (deterministic bases so repeat instantiation is cheaper) is a permitted - optimization; it must not change the loading policy. -- **In-flight isolation.** Module versions are immutable; a recompiled unit - becomes a new module. Running workspaces keep what they loaded; new - workspaces get the new version. Old modules are dropped when unreferenced - (safe unload — impossible with `dlclose`). - -### 3.2 Memory model - -- **One heap, one allocator, one DValue implementation** — all owned by the - core module. Unit modules *import* `malloc`/`free`/runtime symbols via GOT; - the loader **rejects any unit module that defines rather than imports - them** (two allocators on one heap is the one fatal misconfiguration). -- **Arena workspace allocator.** Because the workspace heap is dropped wholesale, - the core's allocator may be a bump allocator with no-op free — the - `_scratchpad.cpp` design, now correct by construction. Per-deployment - flag; fallback is wasi-libc dlmalloc. Memory stats = heap pointer − base - (replaces the tracking `operator new` in `types.h`). -- **Unit statics reset per request** (workspace is born from the core - snapshot, which does not include unit data; unit data segments initialize - at unit load within the workspace). This is *more* shared-nothing than - today, where `.so` statics persist across requests within a worker. - Cross-request state must use explicit host facilities (sessions, caches). - **Breaking change — must be called out in docs and checked against the - site/ tree during Phase 5.** - -### 3.3 The host membrane - -Exactly three currencies cross between host and workspace: - -1. **scalars** (i32/i64/f64), -2. **byte buffers** (`ptr+len` into linear memory; inbound buffers are - placed via the core's exported allocator), on the C++ side we strictly - prefer the binary-safe std::string as a container for buffers -3. **handles** (opaque `u32` indices into the per-workspace host handle - table; each entry carries a closer callback). - -Everything pointer-shaped stays on its own side. Hostcall surface budget: -30–60 functions (§5.1). Host errors return as error values; traps are -reserved for unit faults. Nothing throws across the membrane. - -**DValues cross the membrane only as the versioned wire encoding** (§5.3), -at the following sites: request context in (once), response out (once), -some hostcalls. - -It is expected that many if not most of the toolset API functions we -expose to the unit developer will be judiciously split across the membrane: -have a relatively minimal wasm part that calls into the runtime host, and -then the runtime implementation which does most of the work. - -### 3.4 DValue inside the workspace: no serialization, ever - -Within a workspace, all modules share one address space, one toolchain, one -set of headers — the C/C++ ABI is intact across module boundaries. Therefore: - -- A `DValue` is a pointer (an i32 offset into linear memory). -- `component(path, context)` resolves path → table index (host registry or - guest-resident map) and `call_indirect`s, passing the context pointer. - Reference semantics, mutation visibility, shared ob stack, working `ONCE` - — identical to today. -- Function pointers are shared-table indices, valid across modules: virtual - calls, `std::function` callbacks (`dv_map` lambdas) work across units. -- Cost: one `call_indirect` (single-digit ns) + GOT loads for cross-module - symbols — the same shape of overhead native PIC pays through the PLT/GOT, - i.e. what dlopened `.so` units pay today. - -Encode/decode is **not** part of internal component calls. It exists only at -the membrane (§3.3) and at any future explicit isolation boundary (§4). - -### 3.5 The DValue C ABI (load-bearing, build it first) - -The stable contract of the workspace is a **C ABI**, not the C++ class: -the core exports `extern "C"` accessors over an opaque `uce_dvalue*` (§5.2), -plus the string/ob/print helpers. C++ units may bypass it and use the class -directly (same headers, zero cost — a private fast path). Every other -workspace language uses the C surface. - -This ABI is versioned: every unit artifact carries a custom section -(`uce.abi`: core ABI version + toolchain fingerprint); the loader refuses -stale units and triggers lazy recompilation (units are lazily compiled -anyway, so this costs nothing structurally). - -**Phase 1 of the implementation plan is to introduce this C ABI in the -current native runtime** — it is useful immediately (plugin surface, -testability) and de-risks the rest. - ---- - -## 4. Component-call model / supported languages - -The supported model is one workspace peer model: languages must be able to -produce PIC linear-memory modules that adopt the core allocator and join the -workspace. - -**Workspace peers** (C++, C, Rust, Zig, …): -- Join the workspace as PIC modules importing core symbols. -- Must adopt the core allocator (Rust: `#[global_allocator]`; Zig: allocator - parameter) and must not unwind across boundaries (`panic=abort` / - catch-at-edge). -- Access DValues through the C ABI: pointer semantics, no copies, ns-scale - calls. Idiomatic wrappers per language (e.g. Rust `DValue<'request>` — - the borrow checker enforces the arena invariant). - -**Runtime-carrying/interpreted languages** (JS, Python, Go, C#, …) are not -supported and are not on the roadmap. UCE will not add an alternate component -plane that silently changes `component()` from mutable in-workspace reference -semantics into copied RPC semantics. - -If UCE later supports isolated or cross-trust-boundary components, those must be -introduced as an explicit feature with an explicit API name and copied data -contract. They must not reuse normal `component(path, context)` semantics. -Serialization boundaries and isolation boundaries remain the same lines, by -design. - ---- - -## 5. ABI sketches (to be finalized in Phase 0/1) - -### 5.1 Hostcall surface (grouped; target ≤ 60 functions) - -``` -request: uce_host_ctx_read(buf) → len // wire-encoded context, once -response: uce_host_respond(status, hdrs_buf, body_buf) - uce_host_stream_write(buf) // chunked/streaming path -log: uce_host_log(level, buf) -sqlite: uce_host_sqlite_connect(path_buf) → handle | err - uce_host_sqlite_query(handle, sql_buf, params_buf) → result_buf | err - uce_host_sqlite_cursor_*(...) // optional row-cursor variant - uce_host_sqlite_insert_id/affected/error/disconnect(handle) -mysql: (same shape; existing connector APIs are already handle-shaped) -files: uce_host_file_read/write/stat/list(path_buf, ...) // policy-gated -session: uce_host_session_get/set(key_buf, val_buf) -http: uce_host_http_request(req_buf) → handle/result_buf // outbound -misc: uce_host_time(), uce_host_random(buf), uce_host_env(key_buf) -loader: uce_host_component_resolve(path_buf) → table_index // may load (§6) -ws: uce_host_ws_send(buf), event delivery via render entry re-invocation -``` - -Conventions: all errors as result codes + `uce_host_last_error(buf)`; -inbound buffers placed via the core's exported `uce_alloc`; no hostcall -traps on bad input (clamp/error instead). - -### 5.2 DValue C ABI (core exports; sketch) - -```c -typedef struct uce_dvalue uce_dvalue; // opaque; workspace-owned - -uce_dvalue* uce_dv_root(void); // request context -uce_dvalue* uce_dv_get(uce_dvalue*, const char* key, size_t klen); // create-on-write -uce_dvalue* uce_dv_find(uce_dvalue*, const char* key, size_t klen); // NULL if absent -const char* uce_dv_value(uce_dvalue*, size_t* len); -void uce_dv_set_value(uce_dvalue*, const char* v, size_t vlen); -size_t uce_dv_count(uce_dvalue*); -int uce_dv_is_list(uce_dvalue*); -/* iteration */ -uce_dv_iter uce_dv_iter_begin(uce_dvalue*); -int uce_dv_iter_next(uce_dvalue*, uce_dv_iter*, - const char** key, size_t* klen, uce_dvalue** child); -/* encode/decode at the membrane */ -size_t uce_dv_encode(uce_dvalue*, char* buf, size_t cap); // → UCEB1 -uce_dvalue* uce_dv_decode(const char* buf, size_t len); -/* ob / print / helpers: uce_print, uce_ob_start, uce_ob_get_close, - uce_html_escape, uce_json_encode, ... (mirror uce_lib surface) */ -``` - -No unwinding across this surface; C++ exceptions are caught at the edge and -surfaced as error returns where fallible. - -### 5.3 Wire encoding "UCEB1" (membrane + cross-instance only) - -Length-prefixed binary tree; **not** JSON. Sketch (finalize against DValue's -actual fields — value + ordered children): - -``` -node := value children -value := varint len, bytes (utf-8) -children := varint count, count × ( key: varint len + bytes, node ) -flags := one leading byte per node reserved (bit0: is_list hint) -header := "UCEB" u8 version -``` - -This encoding is a **versioned protocol** from day one (header byte). It is -for future explicit isolation-boundary contracts and the membrane format; -internal calls never see it. - -UCEB1 encoding/decoding should also be exposed to the unit developer so -they can make use of fast serialization/deserialization: matching our existing -API conventions these should be ucb_encode(DValue val) and ucb_decode(String val). This may also be -a worthwhile target for session variables storage (either change session -to DValue or add StringMap support to UCEB1 ser/de). - -### 5.4 Unit module contract - -``` -custom sections: dylink.0 (standard), uce.abi { abi_version, toolchain_id } -imports: env.memory, env.__indirect_function_table, - env.__memory_base, env.__table_base, - GOT.mem.* / GOT.func.* (resolved by loader), - core symbols (malloc, uce_dv_*, uce_print, ...) -exports: uce_unit_setup, uce_unit_render, - uce_unit_component, uce_unit_websocket - (same roles as today's UCE_SETUP/RENDER/COMPONENT/WEBSOCKET - dlsym symbols in compiler.cpp) -forbidden: defining malloc/free/operator new, own memory, start fn - with side effects beyond data init -``` - ---- - -## 6. The loader (host-side, custom, load-bearing) - -Owned code, ~1–2k lines, vendored-runtime-adjacent. Reference logic: -Emscripten's dylink loader (the ABI is the stable, battle-tested part; the -server-side loader is what doesn't exist off the shelf). - -Per `load(unit)` into a workspace: - -1. Fetch compiled module from artifact cache (compile on miss — today's - lazy-compile path, retargeted). -2. Verify `uce.abi` stamp against the core; on mismatch, recompile unit. -3. Verify import discipline (no allocator/runtime definitions; §3.2). -4. Parse `dylink.0`: data size/alignment, table slots needed. -5. Allocate `__memory_base` (bump within workspace data region) and - `__table_base` (append to shared table). -6. Instantiate with bases; resolve `GOT.*` imports against the workspace - symbol registry (core symbols + previously loaded units); register the - unit's exports. NB: data exports of PIC modules are `__memory_base`-relative - offsets — add the owning unit's base when registering/resolving (core - exports are absolute; the core is non-PIC). See the Phase 0 FINDINGS - erratum; the Phase 3 spike's `self-got` marker exists to catch this. -7. Register entry points in the path → table-index dispatch map. - -`uce_host_component_resolve(path)` consults the dispatch map and calls -`load()` on miss — this is how lazy, programmatic, mid-request loading works -with no special cases. - -Placement memoization (optional, later): record each unit's first-assigned -bases; reuse across workspaces so instantiation is cheaper and snapshot -growth (below) stays consistent. Does not change the lazy policy. - -**Core snapshot:** the only pre-built state is "core module, initialized" — -memory bytes + table state captured once per core build. Workspaces are born -from it via CoW (`mmap(MAP_PRIVATE)` of the snapshot image; the host owns -the Memory object, so OS-level CoW is available). No units are pre-fed. - ---- - -## 7. Request lifecycle (replaces the native flow in linux_fastcgi.cpp) - -``` -1. accept request (fastcgi, websockets message, socket event, CLI request) -2. workspace = birth_from_core_snapshot() (CoW, ~µs) -3. write wire-encoded context into workspace; core decodes → context DValue -4. resolve entry unit (load on first call); call uce_unit_render(ctx_ptr) -5. component(path) inside guest → resolve hostcall → (lazy load) → - call_indirect — reference semantics throughout -6. I/O via hostcalls; resources land in the workspace handle table -7. on return: encode response/headers out; write FastCGI response - on trap: defined error → render error page with guest stack trace; - workspace state is irrelevant because… -8. drop workspace: linear memory gone (arena), handle table closed - (generalized resource cleanup), instances released -``` - -CPU limit: epoch/fuel interruption → same path as trap. -Memory limit: linear memory max → allocation failure / trap → same path. - ---- - -## 8. What carries over unchanged - -- The `.uce → C++` translation, parser, and page semantics. -- The lazy compile-on-first-request model and per-unit artifact caching - (different artifact format). -- The host-side connectors (sqlite/mysql) — already handle-shaped APIs; they - move behind hostcalls with the same `.uce`-visible signatures. -- The site tree, docs, demo, and the runtime smoke suite — now the UCE CLI - runner at `site/tests/cli_runner.uce` — which becomes the parity harness (§9). -- nginx/FastCGI front-end integration, worker model, websocket event flow - (events re-enter via the websocket entry point). - ---- - -## 9. Implementation plan - -Phases are sequential; each has an exit criterion. No phase except 0 and 2's -scaffolding produces throwaway work. All dependencies must be vendored. - -**Phase 0 — toolchain & runtime.** -Validate: wasi-sdk `-fPIC` + `wasm-ld -shared` on a representative generated -unit; cross-module C++ calls with shared memory/table; exceptions decision -(wasm EH vs. error-code discipline at unit boundaries — pick one, record it); -vendored runtime selection. Candidates: **WAMR** (C, small, designed for -embedding, easiest to vendor and patch — fits the project's vendoring -practice and would be preferred) vs. **Wasmtime** (fastest, best AOT/CoW machinery, Rust — heavier -to vendor/patch, use only if blocked on WAMR). Selection criteria: imported-memory + shared-table support, -AOT artifact quality, patchability. Exit: a two-module (core stub + unit -stub) hello-world linked at runtime by a minimal loader, in the chosen -vendored runtime. - -> **Status: DONE (2026-06-12).** Exit criterion passed on k-uce; see -> `docs/wasm-toolchain-findings.md`. Runtime selected: **Wasmtime v45.0.1** -> — WAMR is blocked on the load-bearing requirement (its wasm-c-api ignores -> imported memories/tables; host-side table growth unsupported). wasi-sdk-33 -> PIC validated on stubs **and** on real generated units -> (`collections.uce.cpp`, `hello.uce.cpp` → side modules, no allocator -> definitions). Exceptions: `-fno-exceptions` confirmed across all of it -> (§11.1 stands). Cross-module C++ (containers, heap objects, function -> pointers, `std::function` lambdas, GOT.mem/GOT.func) all proven through -> the spike loader. - -**Phase 1 — DValue C ABI in the native runtime.** -Introduce `uce_dv_*` (§5.2) and the UCEB1 codec in `src/lib/`, used -natively. Zero wasm dependency; immediately testable; freezes the contract -everything else builds on. Exit: codec round-trip + accessor tests in the -existing suite; ABI doc checked in. - -> **Status: DONE (2026-06-12).** Native runtime now exposes `uce_dv_*` -> accessors and UCEB1 encode/decode helpers in `src/lib/dvalue.{h,cpp}`. -> ABI details are checked in at `docs/wasm-phase1-dvalue-abi.md`; UCE-visible -> docs are available as `ucb_encode`/`ucb_decode`. Exit coverage is in -> `site/tests/core.uce` and passed in the full network suite. - -***Phase 1 Addendum*** - - Fix before commit: - -1. ucb_encode(DValue value) deep-copies the whole tree (dvalue.cpp:962, same signature in the header). DValue copy is a full recursive map+string clone, and this function is the future membrane hot path — the request context will pass through it on every request in Phase 2. Should be const DValue& (the function only reads). Same nit for bool ucb_decode(String encoded, ...) at :971 — a by-value String copy of what may -be a large document; const String& matches. - -2. 'P' values ship the raw pointer address on the wire (ucb_node_scalar, dvalue.cpp:833, the 'P' case). The ABI doc explicitly says "pointer/reference identity is intentionally not part of the wire contract," but the implementation encodes std::to_string((u64)ptr) — a meaningless number on the receiving side and an ASLR address disclosure the day UCEB1 crosses a trust boundary (multi-tenant isolation is the stated -endgame). It's consistent with native to_string, but the wire is a different context: I'd encode "" for 'P' and note it in the doc. - -3. f64 fidelity on the wire. 'F' encodes through std::to_string → fixed 6 decimals. That's faithful to native to_string, but the membrane makes it new lossiness: today an 'F' value never round-trips through its string form unless page code asks; in Phase 2 every float in the context will. 1e-7 becomes "0.000000" → decodes to 0. Since the scalar is just a string, switching 'F' to shortest-round-trip formatting -(%.17g-style) later needs no version bump. - -4. uce_dv_iter is about to be frozen with no headroom. Keyed-map iteration does std::advance(begin(), position) per call (dvalue.cpp:1096) — O(n²) per full sweep, and the C ABI will be the only iteration path for non-C++ units. The fix (e.g. resuming via lower_bound on the last key) needs state the one-field struct can't hold. Phase 1's whole purpose is freezing this contract: I'd add reserved space now (size_t -position; size_t reserved[3]; or an opaque byte array) so the implementation can get smarter without an ABI break. - -Also: - - - uce_dv_decode returns a pointer into a single thread-local slot (:1122) — a second decode silently invalidates the first result. The doc documents borrowing for uce_dv_value but not this; one sentence ("valid until the next uce_dv_decode on the thread") would close it. - - An empty non-list map round-trips as scalar "" (type 'M' → 'S'; the child_count == 0 && !LIST branch at dvalue.cpp:873ff). HOPEFULLY harmless in practice (is_array() flips), maybe worth a doc line. - - The decoder silently drops a scalar when children are present — unreachable from the encoder, only crafted input. Fine for v1; "reserved" mention in the format doc would pin it. - - Tests cover only the happy path. The hardening (truncation, bad magic, wrong version, depth bomb) is implemented but untested — two or three negative ucb_decode checks in core.uce would lock it in. A float/bool round-trip check would also have surfaced finding 3. - -> **Addendum status: DONE (2026-06-12).** `ucb_encode`/`ucb_decode` now take -> const references, pointer nodes encode as empty scalars, floating-point -> scalars use `max_digits10`, `uce_dv_iter` has reserved ABI headroom, docs -> cover decode-root lifetime and v1 edge cases, and core tests include invalid -> input plus float/bool round-trips. - -**Phase 2 — core module + membrane.** -Compile `uce_lib` (+ wasi-libc) to wasm as the core module; implement the -hostcall surface (§5.1) in the host; temporary scaffolding allowed: one -statically-linked unit + core to validate codegen and membrane without the -loader. Exit: one real `.uce` page (e.g. `site/tests/core.uce`) renders -correctly through the membrane. Scaffolding is marked throwaway. - -> **Status: MEMBRANE SCAFFOLD DONE (2026-06-12).** Temporary scaffold checked -> in under `spikes/wasm-phase2/`: a WASM reactor core subset owns memory, -> `Request`, `DValue`, UCEB1, and output buffering; the Wasmtime host implements -> the initial `uce_host_ctx_read`/`uce_host_log` membrane and passes a UCEB1 -> request context into the guest; a statically linked `.uce` render entry runs -> through that membrane. Validation passed on k-uce with `PHASE2 EXIT CRITERION: -> PASS`. The scaffold does not yet exercise UCE preprocessor-emitted page C++; -> generator-emitted units through this membrane are explicitly deferred into the -> Phase 3 loader path. Full dynamic unit loading also remains Phase 3 as planned. - -**Phase 3 — the loader + workspace.** -Implement §6 in full: dylink parsing, base allocation, GOT resolution, -ABI/import verification, lazy mid-request loading, path dispatch. Per-request -workspace birth/drop (plain memcpy birth is fine here; CoW is Phase 4). -Exit: the uce-starter renders end-to-end with components loading lazily; the -starter subset of the UCE CLI smoke suite passes against the wasm worker. - -> **Status: SPIKE PASS (2026-06-12).** `spikes/wasm-phase3/` combines the -> Phase 0 dylink/PIC loader with the Phase 2 UCEB1 context membrane. The spike -> builds a core workspace reactor and a separate generated-shape `.uce.cpp` PIC -> side module, parses `dylink.0`, allocates memory/table bases, resolves -> `env.*`/`GOT.*` imports, runs relocations/constructors, calls -> `__uce_set_current_request` and `__uce_render`, and reads output from -> core-owned memory. The fixture now exercises nonzero table placement, -> `GOT.func`, deferred self-resolved `GOT.mem`, and generated-style -> `html_escape(...)` expression output. It passed on k-uce with `PHASE3 EXIT -> CRITERION: PASS`. -> -> The self-GOT fixture caught a real loader bug (2026-06-12, fixed): deferred -> `GOT.mem` entries were patched with the unit's exported symbol values -> verbatim, but a PIC module's data exports are offsets relative to its -> `__memory_base` — the loader must add the base. The bug rendered silently -> wrong values and wrote into core memory at low addresses; both spike loaders -> are fixed and the Phase 3 exit gate now asserts every GOT-derived output -> marker (`self-got`/`callback`/`each`/`map`) so a regression cannot pass. -> Details in the `docs/wasm-toolchain-findings.md` erratum. -> -> **Production Phase 3 work plan** (ordered by dependency/risk; spike-proven -> mechanics not repeated here): -> -> 1. **Compile the real `uce_lib` as `core.wasm`** — the last big unknown; all -> spikes used a hand-stubbed `Request`. Forces: `types.h` allocator gate as -> a real `#ifdef` (replacing the spike's copied-header text patch), `sys.h` -> signal/fork/socket carve-outs, the libc++ closure strategy -> (`--whole-archive` vs keep-list), and splitting connectors out of the core -> (the MySQL client library cannot compile to wasm). Gates items 2–6. -> 2. **WASI decision (record in §11 when made)** — spikes stub all WASI imports -> with traps; real pages call `time()` (7× in uce-starter), which wasi-libc -> routes to `clock_time_get`. Recommended: zero-WASI core — route -> time/random/env through the §5.1 hostcalls so there is exactly one -> membrane to audit. -> 3. **Generator changes** (small, parallelizable): emit a logical include -> instead of the absolute `uce_lib.h` path; add the PIC side-module build to -> the compile-on-miss artifact cache. -> 4. **Productionize the loader into `src/`** (§6): `uce.abi` stamping, -> import-discipline verification, a name→funcptr registry in the core -> replacing the spike's per-symbol `core_table_index_of_*` helpers, an -> explicit export name-collision policy (the spike silently prefers core -> exports over unit definitions, e.g. `context`), multi-unit bump placement, -> retained unaligned allocation pointers for unload, hardened/fuzzed binary -> parsing. Plain memcpy workspace birth; CoW stays Phase 4. -> 5. **Starter-scoped hostcalls (~12, not the full ≤60)**: `respond` / -> `stream_write`, `time`/`random`/`env`, `session_get`/`set`, -> `http_request` (OAuth callback), `component_resolve`. The starter uses no -> sqlite/mysql/file APIs — connectors can wait for Phase 5 parity. -> 6. **`component_resolve` + path dispatch + lazy loading** — the only §6 step -> with zero spike coverage (all spikes load one unit eagerly). The starter's -> 71 `component()` calls across 47 files are the stress test; worth a -> focused spike before worker integration. -> 7. **FastCGI worker integration** — config-selectable backend, §7 lifecycle -> wired into the `linux_fastcgi.cpp` flow. -> 8. **Starter parity tests — DONE (2026-06-12; ported to UCE CLI 2026-06-13).** -> `site/tests/cli_runner.uce` renders every starter view with title + -> error-marker assertions and checks the app-shell 404; together with the -> starter HTTP smoke cases, the starter subset is 14 cases, green against -> the native backend. The assertions are backend-agnostic (rendered content -> only), so the identical bar gates the wasm worker. - -**Phase 4 — production mechanics.** -Core snapshot + CoW birth; bump-allocator flag; epoch/memory limits; trap → -error-page path with guest stack traces (this supersedes the -signal/longjmp machinery and closes RECOMMENDATIONS.md 1.5 structurally); -handle-table cleanup (closes 1.7/5.1); artifact/ABI versioning end-to-end. -Exit: kill-tests (deliberate out-of-bounds page, stack-exhaustion page, -infinite-loop page, OOM page) produce clean error pages and an unharmed -worker. (A literal null-deref page is not in the list: address 0 is valid -wasm linear memory, so it does not trap — see §10.) - -> **Status: KILL-TEST SPIKE PASS (2026-06-12).** `spikes/wasm-phase4/` -> validates the production mechanics that can be proven before the real wasm -> worker exists: reusable compiled artifacts as a core-snapshot proxy, fresh -> per-request stores as workspace birth/drop, **both** CPU-limit mechanisms -> (fuel and epoch interruption — production default is epoch per the Phase 0 -> findings; Wasmtime reports epoch traps as `interrupt`), a store memory -> limiter proven load-bearing (the OOM fixture grows within its own declared -> max so only the limiter can deny it), trap capture with the exit gate -> asserting a wasm backtrace and the expected cause per kill, handle closers -> checked to run exactly once and while the store is alive, and — after all -> six kills — a healthy request served by the same engine (the "unharmed -> worker" half of the exit criterion). Kill fixtures are genuinely trapping -> faults: unreachable, OOB access, stack exhaustion, fuel/epoch-limited -> infinite loops, limiter-denied growth. A literal null deref is deliberately -> absent (does not trap in wasm; risk recorded in §10). Passed on k-uce with -> `PHASE4 EXIT CRITERION: PASS`. The trap-trace summarizer now exists as -> `src/lib/wasm_trace.h` (collapses repeated frames, demangles symbols, -> splits cause/detail) and is double-gated: live traps in the spike runner, -> canned-message checks in `site/tests/core.uce` via the native suite. Not -> yet production: OS-level CoW core snapshots, wiring trace summaries into -> the error-page UI, the unit name-section policy for readable frames, -> `linux_fastcgi.cpp` wiring, real connector handle cleanup, and artifact/ABI -> versioning. - -**Phase 5 — parity & performance.** -Full network suite green on the wasm worker; differential native-vs-wasm runs -on the site tree; audit `site/` for cross-request-static reliance (§3.2 -breaking change); benchmark suite (template-heavy page, sqlite page, -component-heavy starter page) with budgets: ≤2× native page latency, -workspace birth ≤100µs, internal component call overhead within 10× native -call cost. Exit: numbers published in this document, all tests and reviews pass. - -> **Status: HARNESS BASELINE PASS (2026-06-12).** `spikes/wasm-phase5/` -> now automates the Phase 5 parity/performance gate shape before the production -> wasm worker exists. On k-uce it ran the full native network suite (`83/83`), -> the starter-focused parity subset (`14/14`), a heuristic code-focused `site/` -> cross-request/static-state audit, and a warmed native benchmark baseline for -> the template-heavy doc page, sqlite page, and component-heavy starter page. -> The harness now gates case counts (`network >= 80`, `starter >= 10`) so broken -> filters cannot pass vacuously, and it runs a throwaway warmup suite before the -> measured gate to absorb cold unit-cache timeouts. Informational native medians -> from the 2026-06-12 baseline are: template-heavy doc `313.8 ms`, sqlite page -> `3.4 ms`, starter dashboard `41.1 ms`. A durable snapshot lives in -> `docs/wasm-baselines/native-baseline-2026-06-12.md`; paired wasm/native -> gate runs still recompute the native medians for the actual ≤2× comparison. -> True Phase 5 completion still requires passing the same harness against a real -> wasm worker URL and adding worker-internal probes for workspace birth and -> component-call overhead budgets. - -**Phase 6 — removed / not planned.** -There is no planned second component plane for interpreted or runtime-carrying -languages. Future work after Phase 5 should continue productionizing the single -workspace-peer WASM backend unless Udo explicitly approves a separate isolated -component feature with a new API name and copied-data semantics. - -The native `.so` backend remains in-tree (as a reference) and selectable by config -but we switch over to the wasm backend as soon as it's available and test -only on that; both backends share the Phase 1 C ABI. - -**Cutover & cleanup checklist (gated, in order — no step before the gate -above it is green).** - -1. **Build** — production Phase 3/4 work plan items 1–7: real `uce_lib` - `core.wasm`, WASI decision recorded, generator changes, production loader - in `src/`, starter-scoped hostcalls, `component_resolve` + lazy dispatch, - wasm worker backend selectable by config in `linux_fastcgi.cpp`. -2. **Prove** — the Phase 5 harness against the wasm worker URL: full network - suite green (≥ 80 cases), starter parity (≥ 10), benchmarks within the - ≤2× paired-run budget; the Phase 4 kill-tests reproduced against the real - worker (OOB page, stack-exhaustion page, infinite loop, OOM) rendering - clean error pages from `wasm_trace.h` summaries. -3. **Switch** — config default flips to the wasm backend; the native backend - stays in-tree as reference per the paragraph above (archival is a separate, - later decision, not part of cutover). -4. **Clean up** — only after 3, and gates must be re-homed before their - spikes are deleted: the Phase 5 harness moves from `spikes/` into `tests/`, - the Phase 4 kill cases become worker integration tests, and the Phase 0 - FINDINGS/erratum content folds into `docs/` — then `spikes/wasm-phase*` - can go. Native-only machinery that cutover obsoletes is retired in the - same pass: the SIGSEGV `sigsetjmp`/`siglongjmp` recovery, the tracking - `operator new` in `types.h`, the per-connector `cleanup_*_connections()` - pattern (§3.1/§3.2 supersede all three). - -Status against this checklist (2026-06-12): W1 has landed enough `src/` / -`scripts/` integration to build and smoke-test `core.wasm`; the remaining gate -1 work is W2–W4 plus the residual W1 wasi-libc import closure noted below. The -native backend is still the only server backend and serves the entire green -suite. The single cleanup item safe today is `src/lib/_scratchpad.cpp` (the -failed arena experiment, unreferenced by any build since 2022; §1 cites it as -history only). - -### 9.1 Production build plan — the worker (W-phases) - -The spike sequence above is closed; every architectural risk it could retire -is retired. The W-phases build the production worker. Ground rules: the -`.uce → C++` preprocessor does not change; every phase lands production code -in `src/`/`scripts/` gated by the existing network suite; no further -throwaway scaffolding. Dependency order is W1 → W3 → W4 → W5 → W6, with W2 -parallelizable once W1's shim headers and ABI stamp exist. - -**W1 — the core module, for real.** -Carve `uce_lib` so it compiles as `core.wasm` with the Phase 0 recipe: -gate the global allocator in `types.h` behind an `#ifdef` (core owns it, -units import it — replacing the spike's copied-header text patch); `#ifdef -__wasm__` carve-outs in `sys.h` (signals/fork/exec/sockets move behind -hostcalls); split the connectors — `mysql-connector`/`sqlite-connector` -keep their `.uce`-visible signatures but forward to membrane hostcalls in -the wasm build while native implementations stay host-side. Zero-WASI core -(record the decision in §11): time/random/env are `uce_host_*` hostcalls, -no `wasi_snapshot_preview1` imports. The core exports the full DValue C ABI -(§5.2), a name→funcptr symbol registry for `GOT.func` (replacing per-symbol -helpers), `uce_alloc`/`uce_free`, and output plumbing. -`scripts/build_core_wasm.sh` joins the normal build. -Exit: `core.wasm` builds reproducibly from the real `uce_lib`; a small -smoke driver instantiates it, runs `_initialize`, and exercises `uce_dv_*`; -the native build and suite remain untouched and green. - -> **Status: DONE (2026-06-12).** `scripts/build_core_wasm.sh` builds -> `src/wasm/core.cpp` into `core.wasm` from the real `uce_lib` carve-out. The -> W1 carve-out keeps native builds unchanged while `__UCE_WASM_CORE__` removes -> native-only compiler/connector code from the core, provides WASM stubs for -> process/socket/task/file surfaces, gates generated-unit allocator definitions -> behind `__UCE_WASM_UNIT__`, and leaves the workspace-owned DValue C ABI in the -> core. `scripts/wasm/build_w1_smoke.sh` builds `src/wasm/w1_smoke.cpp`; the -> smoke driver instantiates `core.wasm`, runs `_initialize`, initializes the UCE -> request context, exercises `uce_dv_root/get/find/set_value/value/count/is_list` -> plus UCEB1 encode/decode, verifies output plumbing, and passes with -> `W1 EXIT CRITERION: PASS`. Native validation after the W1 carve-out: rebuilt -> and restarted `uce.service`; warm full network suite passed `83/83`. Remaining -> W1 follow-up before W3: close the residual wasi-libc/libc++ -> `wasi_snapshot_preview1.*` imports in the produced binary; UCE's own time/env -> calls are routed through `uce_host_*`, but the smoke driver still supplies -> trap stubs for unused libc WASI imports. - -**W2 — the compiler outputs wasm units.** -Preprocessor output unchanged. The unit compile path in `compiler.cpp` -gains the wasm target beside the `.so` target: `clang ---target=wasm32-wasip1 -fPIC` + `wasm-ld -shared` (Phase 0 unit recipe), -logical `uce_lib.h` include instead of the absolute path, a `uce.abi` -custom-section stamp (ABI version + toolchain id), and per-unit `.wasm` -artifacts in the same cache with the same invalidation as `.so`. -Exit: a batch compile of every unit the suite touches produces valid PIC -modules — `dylink.0` present, `uce.abi` stamped, no allocator definitions, -import shapes verified by a check tool — and compile-on-miss works for the -wasm target. - -> **Status: DONE (2026-06-12).** Generated units now include the logical -> `#include "uce_lib.h"`; native `scripts/compile` supplies `-Isrc/lib`, and -> `scripts/compile_wasm_unit` builds PIC side modules with `__UCE_WASM_UNIT__`, -> `wasm-ld -shared --experimental-pic`, and an `llvm-objcopy`-inserted -> `uce.abi` custom section. `scripts/wasm/check_unit_wasm.py` validates wasm v1 -> structure, `dylink.0` mem_info, `uce.abi` ABI/toolchain stamp, import policy, -> required PIC imports, and absence of allocator definitions. The compiler can -> optionally build the wasm artifact beside the native `.so` via -> `COMPILE_WASM_UNITS=1`; native remains default. Batch validation built/checked -> all 128 known/generated suite units, and a temporary `unit_compile()` -> compile-on-miss page produced and verified a fresh `.wasm`. Reused-artifact -> batch validation was tightened after the first slow full rebuild: unchanged -> units are rechecked instead of rebuilt, reducing a 128-unit no-op pass from -> several minutes to about 5 seconds. `scripts/compile_wasm_unit` now also uses -> a keyed Clang PCH for the stable `uce_lib.h` unit header by default -> (`UCE_WASM_UNIT_PCH=0` disables it); the key includes ABI version, clang -> version, common compile flags, and `src/lib/*.h` content. Spot checks: warm -> `hello.uce` wasm compile dropped from about 2.3s to 0.65s, `core.uce` from -> about 5.7s to 4.1s, and a full 128-unit rebuild with PCH took about 163s. -> Native validation stayed green (`83/83`). - -**W3 — workspace runtime + membrane (the worker core).** -Productionize the loader into `src/wasm/` per §6, from the Phase 3 spike -plus everything it deferred: symbol registry, dispatch map, ABI stamp -verification, import discipline (reject units defining the allocator), -the `__memory_base`-relative data-export rule, multi-unit placement, -hardened binary parsing, an explicit export name-collision policy. -Workspace lifecycle: core snapshot born by memcpy (CoW is W5), dropped per -request; host handle table with closers. Starter-scoped hostcall set -(~12, the §5.1 subset): `ctx_read`, `respond`, `stream_write`, `log`, -`time`, `random`, `env`, `session_get`/`set`, `http_request`, -`component_resolve`, `last_error`. -Exit: a real request — UCEB1 context in, render, response out — served -end-to-end through a workspace running the real core and real generated -units, driven by a CLI test driver; epoch CPU limit and memory limiter -active; traps produce `wasm_trace.h` summaries. - -> **Status: DONE (2026-06-13).** `src/wasm/worker.cpp` is the production -> workspace runtime (wasmtime.hh; per-request store, hardened dylink/uce.abi -> parsing, import discipline, GOT.func via host funcref placement, the -> `__memory_base` data-export rule, core-first/first-unit-wins symbol -> registry, per-worker compiled-module cache) and `src/wasm/w3_driver.cpp` -> is the CLI gate. Membrane so far: time/time_precise/env/random/log, -> `component_resolve` (lazy mid-request loading), and a policy-gated -> read-only file membrane (`file_exists`/`file_read`, current-unit-relative, -> site-tree-contained). The exit gate passed on k-uce: `/demo/hello.uce` and -> `/demo/components.uce` render **byte-identical to native** (incl. nested -> and named `COMPONENT:X` handlers through `ob_*` capture); starter -> dashboard/gauges/features/workspace/page1 all render `200` with ~12 units -> lazily loaded mid-request (≈700 ms first request — dominated by per-request -> Wasmtime module compilation, the W5 AOT/snapshot target — ≈2 ms warm); -> epoch kill mid-render traps as `interrupt` with a symbolicated -> `wasm_trace` summary and a clean workspace drop. Carve findings recorded -> on the way: header function templates must be `inline` (self-import rule, -> now commented in `types.h`/`functionlib.h`), the core needs vague-linkage -> and libc link anchors (`uce_wasm_link_anchors()` + -> `core_libc_exports.syms`), units build `-fno-rtti`/`-fno-exceptions` to -> match the core ABI, and connector classes have explicit fail-clean stubs -> until the W5 hostcall connectors. Deferred to W4 as planned: `respond`/ -> `stream_write`/`session`/`http_request` hostcalls (response metadata -> currently returns as UCEB1 via `uce_wasm_response_meta`), kill-pages, and -> the FastCGI backend; `error-reporting.uce` and `tests/zip.uce` are -> native-only pending the trap error path and a zip hostcall. - -**W4 — the FastCGI worker.** -Wire W3 into `linux_fastcgi.cpp` as a config-selectable backend (native -stays default until W5): the §7 lifecycle, lazy mid-request -`component_resolve` → load → `call_indirect`, path dispatch, trap → -configured UCE error pages with collapsed guest traces, handle-table -cleanup on workspace drop. -Exit: the server runs with the wasm backend on a test config; the starter -subset of the runtime smoke suite passes against the wasm worker — the Phase -5 harness wasm leg lights up for the first time; the four kill-tests exist as -real `.uce` pages and produce clean error pages from an unharmed worker. - -> **Status: DONE (2026-06-13).** `src/wasm/backend.cpp` wires the W3 runtime -> into `src/linux_fastcgi.cpp` as a config-selectable page-render backend -> (`WASM_BACKEND_ENABLED`, default off — native stays default until W5). The -> seam is one branch in `handle_complete`: per forked worker, a lazily-built -> `WasmWorker` + epoch ticker thread; per request the native `Request` -> params/get/post/cookies/session are encoded to the UCEB1 context, served -> through a fresh workspace, and the response (status/headers/cookies/session -> /body) is written back onto the native `Request` so the existing transport -> emits it unchanged. CLI/serve_http/websocket stay native; units with no -> wasm artifact fall through to native automatically. `ONCE()` is now honored -> in the workspace (host resolves `__uce_once`, core dedups on the resolved -> path via `once_units`; the entry renders through `uce_wasm_render_entry` so -> it shares the component dispatch + ONCE path). -> -> **Exit gate passed on k-uce, through the real nginx → fastcgi → wasm path on -> port 80:** the starter smoke/parity subset is **14/14** against the wasm -> backend (incl. the two ONCE-asset-in-`` cases); three real kill -> pages under `site/tests/wasm-kill/` (OOB write, runaway loop, unbounded -> recursion) each return a clean error page carrying a demangled -> `wasm_trace` summary (`out of bounds memory access` / epoch `interrupt` -> with `wasm_kill_recurse(unsigned long long)` framing), and the worker keeps -> serving `200`s through a barrage of kills — no native signal, the trap is a -> returned error at the membrane. CPU budget is enforced by epoch -> (`WASM_EPOCH_DEADLINE_TICKS` × `WASM_EPOCH_PERIOD_MS`); memory by the store -> limiter. Native default restored after the gate: full suite **83/83**. -> Robustness folded in along the way: the ctype libc family added to the core -> export anchors (`core_libc_exports.syms`), and a page with no `RENDER` -> renders empty-200 (native parity). -> -> **W5 surface, made concrete** (23 full-suite pages still 500 on the wasm -> backend, all by design): the regex/xml/yaml core stubs to un-stub or move -> behind hostcalls; markdown/zip/tasks/sqlite/file-write membrane hostcalls; -> the `unit_call` bridge; and compiler-introspection pages (`unit-info`, -> `unit-browser`, `sharedunit`) which likely stay native. These are the W5 -> parity workload, not regressions. - -**W5 — parity, performance, cutover.** -Full network suite green on the wasm backend; §3.2 statics-audit findings -(50 code candidates) fixed as differential native-vs-wasm runs surface -them; CoW snapshot birth and placement memoization as needed to meet the -budgets (≤2× page latency against the pinned baseline shape, workspace -birth ≤100µs, component-call overhead ≤10×) with worker-internal probes -for the latter two. -Exit: cutover checklist gates 2–3 — harness fully green against the wasm -worker URL, config default flips to wasm, native stays in-tree as -reference. - -> **Status: DONE (2026-06-13).** W5 flips the page-render default to the WASM -> backend while retaining explicit native fallback for the surfaces still -> genuinely host-owned or not yet membraned: zip, sockets/custom servers, -> memcache/mysql, and `unit_call`/compiler-introspection (the last need the -> native toolchain). The fallback is selected before workspace creation by -> scanning the entry source for a small native-only token set, so unsupported -> pages neither fail through wasm nor hide as trap regressions. -> -> **Promoted onto wasm during W5** (no longer fallback): **regex** runs through -> a single UCEB1-marshalled `uce_host_regex` hostcall against the host's PCRE2; -> **markdown** is compiled into the core (pure compute); **xml/yaml** were -> already in-core; **filesystem writes** (`file_put_contents`/`file_append`/ -> `file_unlink`) go through policy-gated hostcalls, and the read membrane was -> widened to the same allowlist (site tree + scratch roots: `/tmp`, -> `BIN_DIRECTORY`, `SESSION_PATH`, `TMP_UPLOAD_PATH`) so a page can read back -> what it writes (e.g. the `io.uce` /tmp round-trip); **sqlite** runs through a -> `uce_host_sqlite` hostcall against the host's real connector, with -> connections held in a per-workspace handle table that is closed on workspace -> drop — the first concrete realization of the §3.1 "handle-table drop = -> resource cleanup" model; **background tasks** now use a host-managed process -> membrane (`uce_host_task_spawn`/`pid`/`kill`) with a guest callback trampoline -> (`uce_wasm_task_run`) in the forked child, so `task()`, `task_repeat()`, -> `task_pid()`, and `task_kill()` run under the wasm backend; **sleep/usleep** -> use a host sleep call that renews the epoch deadline after blocking. The -> `/doc/*` blanket fallback is gone — doc pages render on wasm. -> -> One subtle bug fixed here: the sized-hostcall convention has the guest call -> twice (buf=0 to learn the length, then to fetch). That re-executed the op — -> harmless for idempotent regex/file-read, but for sqlite it ran every -> `INSERT`/`CREATE` twice and the second pass failed on its own side effects. -> The host now stages the encoded result on the first call (keyed on the exact -> input) and replays it on the fetch, so a side-effecting op runs once. Two wasm-stack/ABI issues surfaced and were fixed: the core stack was -> raised to 8 MB so recursive `ucb_decode_node` reaches the 1024 depth limit -> like native (the core.uce depth-bomb negative test overflowed the small -> default stack), and `wasm_trace`'s demangler is host-only (wasi-libc++ has no -> `__cxa_demangle`, so a unit must not import it). Two backend robustness bugs -> fixed: a failed wasm side-compile (e.g. try/catch units) is now non-fatal — -> the native `.so` still serves and the stale `.wasm` is removed — and -> `wasm_artifact_exists` now requires the artifact be newer than its source, so -> a source edit can't be served from a stale `.wasm` (the wasm path bypasses the -> native JIT recompile). -> -> **Signal-trap conflict resolved (this was open in the prior W5 draft):** a -> guest `unreachable`/OOB used to surface as a host signal that the native -> SIGILL/SIGSEGV handler caught and `abort()`'d the worker (502). `make_engine` -> now sets `signals_based_traps(false)`, so guest traps are explicit and stay -> pure wasm traps returned as errors. The `__builtin_trap()` kill page now -> returns a clean 500 and the worker survives a kill barrage (0 PID churn); it -> is back in the kill-test gate alongside loop/recurse. -> -> `scripts/wasm/run_w5.sh` is the cutover gate: measure native, switch -> `/etc/uce/settings.cfg` to wasm, run the wasm full suite + kill tests + -> starter subset + the native-vs-wasm benchmark, optionally leave the backend -> enabled (`UCE_W5_KEEP_BACKEND=1`). Final k-uce gate: native reference suite -> **83/83**; wasm/default full suite **83/83** warm (starter, docs, core tests, -> markdown, regex, sqlite, and tasks on wasm; zip and native-only service/ -> compiler surfaces routed to native fallback); starter subset **14/14**; kill pages (trap/loop/recurse) clean with the worker -> unharmed. Benchmark medians passed the ≤2× budget: doc singlepage native -> ~324ms vs wasm ~318ms, sqlite 3.7ms vs 3.8ms, starter dashboard 44ms vs wasm -> ~6ms. Worker-internal probe headers (`X-UCE-Backend`, -> `X-UCE-Wasm-Workspace-Birth-Us`, component resolve count/total/avg) are emitted -> for wasm responses when `WASM_BACKEND_VERBOSE=1`; warm workspace birth is -> ~300–380µs on k-uce (above the aspirational 100µs CoW target — that and -> placement memoization are the remaining perf items — but page latency meets the -> cutover budget). Live `/etc/uce/settings.cfg` left with `WASM_BACKEND_ENABLED=1`; -> native remains in-tree and config-selectable. -> -> **Remaining native fallback** (the only pages still routed to native): **zip** -> (its `.uce` source uses try/catch, so it can't be a `-fno-exceptions` wasm -> side module regardless of a hostcall), socket/custom-server/memcache/mysql -> service surfaces, and `unit_call` + compiler/unit-introspection -> (`unit-info`/`unit-browser`/`sharedunit` — need the native toolchain; -> candidates for host-serviced hostcalls if native is to be fully retired). -> **Still deferred, not blocking:** CoW snapshot birth + placement memoization -> for the workspace-birth budget, and the §3.2 statics-audit follow-through. - -**W6 — cleanup.** -Cutover checklist gate 4, verbatim: re-home the spike-hosted gates -(Phase 5 harness → `tests/`, kill cases → worker tests, FINDINGS/erratum → -`docs/`), delete `spikes/wasm-phase*`, retire the SIGSEGV recovery, the -tracking `operator new`, and `cleanup_*_connections()`. - -> **Status: SPIKE CLEANUP DONE; native-machinery retirement GATED -> (2026-06-13).** Re-homing complete: the Phase 5 benchmark + site-audit are -> now `tests/wasm_benchmark.py` / `tests/wasm_site_audit.py` (with -> `scripts/wasm/run_w5.sh` repointed), the kill-test gate lives in the UCE CLI -> runner at `site/tests/cli_runner.uce`, the Phase 0 toolchain findings + GOT -> erratum are `docs/wasm-toolchain-findings.md`, and the pinned native -> baselines are `docs/wasm-baselines/`. **All `spikes/wasm-phase*` deleted** -> (prototypes superseded by `src/wasm/`), and the dead 2022 -> `src/lib/_scratchpad.cpp` arena experiment is removed. Build + full suite -> stay green (83/83). The phase-0–4 status blocks above keep their narrative; -> their `spikes/...` paths are now historical — the artifacts that mattered -> were re-homed, the rest was throwaway scaffolding by design. -> -> **The three native-machinery retirements remain BLOCKED, deliberately:** -> native is still the live backend for pages touching zip, sockets / custom -> HTTP servers, memcache, mysql, `unit_call`, and compiler/unit-introspection -> — everything still in `native_only_tokens`. While *any* page renders on -> native, retiring its SIGSEGV/`sigsetjmp` recovery, the tracking -> `operator new`, or `cleanup_*_connections()` would strip crash protection -> and resource cleanup from live traffic. They retire only once those surfaces -> move behind membrane hostcalls (so nothing falls back) or the native backend -> is formally decommissioned. This is the honest end-state of W6 given the -> current fallback set, not an oversight. - -**W7 — non-render entrypoints and fallback hardening.** -With render traffic defaulting to wasm, the next retirement work is the entry -surface around render: CLI, WebSocket, and `SERVE_HTTP`/custom-server dispatch, -plus the cold fallback behaviors that now show up in the default gate. The first -W7 slice split the wasm backend into a proper object/header boundary and taught -`wasm_backend_serve()` to dispatch CLI/WebSocket/SERVE_HTTP entry kinds; the -socket CLI path now propagates wasm runtime errors instead of silently returning -success. Post-fork wasm reset was also tightened: the epoch ticker is now held -behind a pointer so a forked broker child can discard the inherited, unusable -thread handle without placement-new over a live `std::thread` object. The -network gate now gives only the known cold native-fallback pages (`sharedunit` -and the operational services test) longer per-request timeouts while preserving -strict status/body checks, and the services test drains split custom-server -response packets before deciding that the named handler failed. - -> **Status: W7a/W7b/W7c/W7d DONE (2026-06-14).** Handler invocation is now -> unified: CLI / WebSocket / serve_http / page render all route through one -> `serve_via_wasm(entry_unit, handler)` path, the handler string mapping to a -> `__uce_` export (see -> [`docs/wasm-runtime-architecture.md`](docs/wasm-runtime-architecture.md)). -> W7c was resolved by the central WebSocket broker + serve_http dispatcher both -> forwarding to a clean worker over `uce.sock` (no wasm render inside a forked -> broker child). W7d cleared the last native-only holdouts: `zip.uce` was -> rewritten to use the membrane's return-value error contract (no C++ -> try/catch), `error-reporting.uce` now triggers genuine wasm traps instead of -> `throw`, and `sharedunit.uce` uses `unit_info()` instead of -> `compiler_load_shared_unit()`. **The native-only token gate is now empty** — -> no unit source forces native anymore. -> -> **W7e plan — deleting the native pipeline (the remaining work).** The native -> pipeline is not just a fallback; it is currently the **wasm compile driver**: -> `compiler.cpp` preprocesses `.uce → C++` (shared front-end, KEEP), then -> compiles the native `.so` (`COMPILE_SCRIPT`), and only *then*, gated on that -> native compile succeeding, compiles the `.wasm` side-module -> (`compile_wasm_unit`). The backend's native fallback on a cold/stale artifact -> exists so the native path can JIT-rebuild **both** `.so` and `.wasm`. So the -> deletion must be staged, not a single excision: -> 1. **Decouple the compile driver.** Make the `.wasm` side-module compile run -> unconditionally from the shared preprocessed C++, independent of the -> native `.so` compile and of the `SharedUnit` struct's native bookkeeping. -> 2. **Make wasm self-healing on cold/stale.** Have the backend trigger (or -> wait on) an on-demand wasm compile instead of falling back to native, so -> `wasm_artifact_exists` staleness no longer needs the native rebuild. (Note: -> until this lands, a cold request — before the proactive compiler warms a -> unit — to a unit rewritten for the membrane error contract, e.g. `zip.uce`, -> can `500` on the native path because native `zip_create` still throws.) -> 3. **Remove native execution + machinery.** Delete `compiler_invoke` / -> `_cli` / `_websocket` / `_serve_http`, `compiler_load_shared_unit`, -> `load_shared_unit`, the `dlopen`/`dlsym`/`dlclose` path and `.so` -> function-pointer fields on `SharedUnit`, the `COMPILE_SCRIPT` native -> compile, and the native-fallback branches in `handle_complete` and -> `wasm_backend_should_handle`. -> 4. **Drop the feature flag.** Remove `WASM_BACKEND_ENABLED` and the now-dead -> `wasm_backend_native_fallback_*` gate (the token scan is already vestigial). -> -> Each stage must rebuild green on `scripts/run_cli_tests.sh --include-wasm-kill` -> before the next. W7f (broad dead/legacy sweep) folds into stages 3–4. -> -> **Prerequisite — membrane silent-stub completeness (in progress).** Removing -> native fallback (stages 2–4) turns any silently-stubbed wasm-core function -> into a hard failure with no safety net. A real one slipped through this way: -> `ls()` was a stub returning empty, so `doc/index.uce` (which enumerates pages -> via `ls("pages/")`) rendered but listed nothing — and the suite missed it -> because it only asserted the page returned ``, never that it enumerated -> items. Fixed (2026-06-14) by wiring `ls`, `mkdir`, and `file_mtime` through new -> `uce_host_file_list` / `uce_host_file_mkdir` / `uce_host_file_mtime` hostcalls -> (and teaching `resolve_guest_file` to accept directories), plus a regression -> assertion that the doc index contains `func-item`. Remaining wasm-core stubs -> are intentional: `shell_exec` (no shell in the sandbox), the `*_locked_fd` -> fd-locking variants, and the `backtrace_*` / `signal_name` host-only helpers. -> Before stage 2, audit the rest of the wasm-core block in `src/lib/sys.cpp` for -> any other `(void)arg; return {}` stub a unit can reach. - ---- - -## 10. Risks & mitigations - -| Risk | Mitigation | -|---|---| -| wasi-sdk PIC / shared-library maturity (least-trodden toolchain path) | Phase 0 spike before any commitment; pin toolchain versions; statically link libc into the core and export from there (avoid shared wasi-libc entirely) | -| C++ exceptions × PIC × wasm EH | Phase 0 decision point; fallback is error-code discipline at unit entry points (units already have a uniform entry shape) | -| Custom loader correctness (GOT, bases, relocation) | Small, contained (~1–2k lines); crib logic from Emscripten's reference loader; fuzz with adversarial modules; loader rejects > loader guesses | -| Vendored runtime patches drift from upstream | Same practice as vendored SQLite: provenance + patch files under `docs/patches/`; pin upstream tag; tests gate upgrades | -| Performance regression beyond budget | Phase 5 gates; bump allocator and placement memoization in reserve; native backend retained | -| Multi-module DWARF / debugging story | Trap stack traces cover the production case (better than today); accept weaker interactive debugging; keep native backend for local deep-debugging | -| Unit-statics semantic change breaks existing pages | Phase 5 audit of `site/`; documented migration note; host-side cache facility if a real need surfaces | -| Null-pointer dereferences do not trap (wasm address 0 is valid linear memory) — a buggy page writes low workspace memory and renders silently wrong output instead of erroring | Accepted: damage is confined to the request's workspace and dropped at request end (strictly better than native SIGSEGV). Kill-tests use genuinely trapping faults (OOB, stack exhaustion, `__builtin_trap`). Optional later: null-check instrumentation at a measured perf cost | - -## 11. Decisions - -1. **Exceptions:** wasm EH or error codes at unit boundaries? Error codes. -2. **WebSocket granularity:** workspace per event (pure arena, statics reset - per event) or per connection (state across events, bounded lifetime)? - Per-event. Should be compatible with WS since the WS contract is: runtime holds - and brokers connections, makes request to unit's WS() {...} directive, unit - may decide to send data back over WS to any or even all connected clients. -3. **UCEB1 final layout** vs. DValue's actual field set (value/children/list - flag) — somewhat open, finalize in Phase 1. -4. **Cursor vs. bulk** as the default for `sqlite_query` results at the - membrane (offer both; pick the default after Phase 5 benchmarks, tending towards cursor right now). -5. **Streaming output:** today's ob model buffers; does the membrane expose - `uce_host_stream_write` from day one or post-MVP? Undecided. ob is a critical - abstraction to our whole execution model and whether we expose direct stream - write or not, the existing PHP-like ob contract/paradigm must stay in place. -6. **Core snapshot rebuild cadence** once placement memoization lands - (dead-base reclamation policy). - -## 12. Summary - -The module is the **unit** (file-grained, lazily compiled, lazily loaded — -including mid-request). The instance set is the **workspace** (one per -request; shared memory + table; born from a core-only CoW snapshot; dropped -wholesale — the arena, done right). The contract is the **DValue C ABI** -inside the workspace (pointer semantics, no serialization) and the **UCEB1 -wire encoding + handles** at every true address-space boundary (host -membrane and future explicit trust boundaries). Serialization -boundaries and isolation boundaries are the same lines; component calls stay -function calls; and the file stays the unit. diff --git a/docs/patches/miniz-3.0.2-cxx-single-translation-unit.patch b/docs/patches/miniz-3.0.2-cxx-single-translation-unit.patch index 8516fe8..7170a1d 100644 --- a/docs/patches/miniz-3.0.2-cxx-single-translation-unit.patch +++ b/docs/patches/miniz-3.0.2-cxx-single-translation-unit.patch @@ -1,6 +1,6 @@ -diff --git a/vendor/miniz/miniz_tdef.c b/vendor/miniz/miniz_tdef.c ---- a/vendor/miniz/miniz_tdef.c -+++ b/vendor/miniz/miniz_tdef.c +diff --git a/src/3rdparty/miniz/miniz_tdef.c b/src/3rdparty/miniz/miniz_tdef.c +--- a/src/3rdparty/miniz/miniz_tdef.c ++++ b/src/3rdparty/miniz/miniz_tdef.c @@ -static const mz_uint s_tdefl_num_probes[11]; +static const mz_uint s_tdefl_num_probes[11] = { 0, 1, 6, 32, 16, 32, 128, 256, 512, 768, 1500 }; diff --git a/scripts/api_coverage_manifest.py b/scripts/api_coverage_manifest.py new file mode 100755 index 0000000..17b211a --- /dev/null +++ b/scripts/api_coverage_manifest.py @@ -0,0 +1,94 @@ +#!/usr/bin/env python3 +"""Guard the hand-maintained unit-facing API coverage manifest. + +This intentionally avoids network/external services. It checks that public API +names we expose to wasm units are either mentioned by a site test or explicitly +marked internal/integration-only, and that active docs exist for doc-required +APIs. The manifest is deliberately source-controlled so a new public function +requires an explicit coverage decision. +""" +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +TEST_DIR = ROOT / "site" / "tests" +DOC_DIR = ROOT / "site" / "doc" / "pages" + +# name, needs_doc, status. status: public | internal | integration +PUBLIC_APIS = [ + ("shell_exec", True, "public"), ("shell_escape", True, "public"), + ("basename", True, "public"), ("dirname", True, "public"), ("path_join", True, "public"), + ("path_real", True, "public"), ("path_is_within", True, "public"), + ("file_open_locked", True, "public"), ("file_close_locked", True, "public"), + ("file_release_process_locks", True, "public"), ("file_get_contents_locked_fd", True, "public"), + ("file_put_contents_locked_fd", True, "public"), ("file_get_contents", True, "public"), + ("file_put_contents", True, "public"), ("file_append_contents", False, "public"), + ("cwd_get", True, "public"), ("cwd_set", True, "public"), ("process_start_directory", True, "public"), + ("file_mtime", True, "public"), ("file_unlink", True, "public"), ("expand_path", True, "public"), + ("ls", True, "public"), ("config_map_u64", True, "public"), ("config_map_f64", True, "public"), + ("config_bool_value", True, "public"), ("config_map_bool", True, "public"), + ("config_u64", True, "public"), ("config_f64", True, "public"), ("config_bool", True, "public"), + ("request_perf", True, "public"), ("time_format_local", True, "public"), + ("time_format_relative", True, "public"), ("time_parse", True, "public"), + ("backtrace_frames_string", False, "public"), ("capture_backtrace_string", False, "public"), + ("signal_name", False, "public"), ("memcache_escape_key", True, "public"), + ("memcache_escape_keys", True, "public"), ("memcache_command", True, "public"), + ("memcache_get_multiple", True, "public"), ("runtime_safe_key", True, "public"), + ("float_val", True, "public"), ("nibble", True, "public"), ("json_consume_space", False, "public"), + ("array_merge", True, "public"), ("safe_name", True, "public"), ("ascii_safe_name", True, "public"), + ("to_json", False, "public"), ("remove", False, "public"), ("clear", False, "public"), + ("gen_sha1", True, "public"), ("gen_noise32", True, "public"), ("gen_noise64", True, "public"), + ("gen_noise01", True, "public"), ("gen_int", True, "public"), ("gen_float", True, "public"), + ("draw_int", True, "public"), ("draw_float", True, "public"), + ("encode_query", True, "public"), ("request_script_url", True, "public"), + ("request_base_url", True, "public"), ("request_route_from_raw_path", True, "public"), + ("cli_arg", True, "public"), ("unit_compile", True, "public"), + ("cleanup_sqlite_connections", False, "internal"), ("cleanup_mysql_connections", False, "internal"), + ("mysql_connect", True, "integration"), ("mysql_query", True, "integration"), +] + +REMOVED_APIS = ["unit_load", "concat"] + + +def all_test_text() -> str: + parts = [] + for path in TEST_DIR.glob("*.uce"): + parts.append(path.read_text(errors="ignore")) + return "\n".join(parts) + + +def doc_exists(name: str) -> bool: + path = DOC_DIR / f"{name}.txt" + return path.exists() and "Removed" not in path.read_text(errors="ignore")[:200] + + +def has_call(text: str, name: str) -> bool: + return f"{name}(" in text or f".{name}(" in text or f'"{name}"' in text + + +def main() -> int: + tests = all_test_text() + errors = [] + for name, needs_doc, status in PUBLIC_APIS: + if status == "public" and not has_call(tests, name): + errors.append(f"missing test coverage: {name}") + if needs_doc and status in {"public", "integration"} and not doc_exists(name): + errors.append(f"missing active doc page: {name}") + compiler_h = (ROOT / "src" / "lib" / "compiler.h").read_text(errors="ignore") + if "#ifndef __UCE_WASM_UNIT__\nSharedUnit* unit_load" not in compiler_h: + errors.append("unit_load is not guarded out of wasm-unit exposure") + for name in REMOVED_APIS: + page = DOC_DIR / f"{name}.txt" + if name == "concat" and page.exists() and "Removed" not in page.read_text(errors="ignore")[:300]: + errors.append("concat doc is not tombstoned") + if name == "unit_load" and page.exists() and "native-only" not in page.read_text(errors="ignore"): + errors.append("unit_load doc is not native-only/tombstoned") + if errors: + print("API coverage manifest FAILED") + for error in errors: + print("- " + error) + return 1 + print(f"API coverage manifest ok: {len(PUBLIC_APIS)} entries checked") + return 0 + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/site/tests/api_coverage.uce b/site/tests/api_coverage.uce new file mode 100644 index 0000000..daf414f --- /dev/null +++ b/site/tests/api_coverage.uce @@ -0,0 +1,28 @@ +#include "testlib.h" + +RENDER(Request& context) +{ + if(!test_demo_request_allowed(context)) + { + site_tests_restricted(context, "API Coverage Manifest", "scan repository headers, tests, and docs with the local manifest script"); + return; + } + + u64 passed = 0; + u64 failed = 0; + u64 skipped = 0; + auto check = [&](String name, bool ok, String detail) + { + site_tests_case(name, ok ? "pass" : "fail", detail); + if(ok) + passed++; + else + failed++; + }; + + site_tests_page_start("API Coverage Manifest", "Repository-level guard that public wasm-unit APIs have an explicit test/doc/internal decision."); + String out = shell_exec("cd /Code/uce.openfu.com/uce && python3 scripts/api_coverage_manifest.py"); + check("api coverage manifest", contains(out, "API coverage manifest ok"), out); + site_tests_summary(passed, failed, skipped, "This page intentionally runs only on trusted/local test requests."); + site_tests_page_end(); +} diff --git a/site/tests/cli.uce b/site/tests/cli.uce index 150d301..b5254be 100644 --- a/site/tests/cli.uce +++ b/site/tests/cli.uce @@ -14,6 +14,12 @@ CLI(Request& context) print(input["message"].to_string(), "\n"); return; } + if(action == "arg") + { + print(cli_arg(context, "message", "fallback"), "\n"); + print(cli_arg(context, "missing", "fallback"), "\n"); + return; + } context.set_status(400, "CLI Error"); print("unknown cli action: ", action, "\n"); } diff --git a/site/tests/cli_runner.uce b/site/tests/cli_runner.uce index 23c9a32..012bc11 100644 --- a/site/tests/cli_runner.uce +++ b/site/tests/cli_runner.uce @@ -325,6 +325,8 @@ CLI(Request& context) print("[GROUP] starter ", std::to_string((u64)((time_precise() - group_start) * 1000)), " ms\n"); group_start = time_precise(); cli_run_tcp_smoke(); + String cli_arg_out = shell_exec("cd /Code/uce.openfu.com/uce && bash scripts/uce-cli /tests/cli.uce action=arg message=hello-cli 2>&1"); + cli_test_case("uce_cli_fixture:cli_arg", contains(cli_arg_out, "hello-cli") && contains(cli_arg_out, "fallback"), cli_truncate(cli_arg_out)); print("[GROUP] tcp ", std::to_string((u64)((time_precise() - group_start) * 1000)), " ms\n"); group_start = time_precise(); cli_run_wasm_kill(include_kill); diff --git a/site/tests/core.uce b/site/tests/core.uce index ddd0d6a..9ac3aac 100644 --- a/site/tests/core.uce +++ b/site/tests/core.uce @@ -236,6 +236,30 @@ RENDER(Request& context) tree["nested"]["ok"].set_bool(true); check("DValue map access", tree["suite"].to_string() == "core" && tree["nested"]["api"].to_string() == "dvalue", json_encode(tree)); + String nibble_source = "alpha/beta/gamma"; + String nibble_first = nibble(nibble_source, "/"); + StringMap merge_a; merge_a["a"] = "one"; merge_a["shared"] = "old"; + StringMap merge_b; merge_b["b"] = "two"; merge_b["shared"] = "new"; + StringMap merged_map = array_merge(merge_a, merge_b); + DValue merge_dv_a; merge_dv_a["a"] = "one"; merge_dv_a["shared"] = "old"; + DValue merge_dv_b; merge_dv_b["b"] = "two"; merge_dv_b["shared"] = "new"; + DValue merged_dv = array_merge(merge_dv_a, merge_dv_b); + u32 json_space_i = 0; + json_consume_space(" \n\tvalue", json_space_i); + check("float_val() / nibble() / array_merge() / json_consume_space()", float_val("12.5") > 12.49 && float_val("12.5") < 12.51 && nibble_first == "alpha" && nibble_source == "beta/gamma" && merged_map["shared"] == "new" && merged_map["a"] == "one" && merged_dv["b"].to_string() == "two" && json_space_i == 3, "nibble=" + nibble_first + " rest=" + nibble_source + " merged=" + var_dump(merged_map) + " json_space_i=" + std::to_string(json_space_i)); + + String unsafe = "Hello, 世界 / ../../ name!"; + check("safe_name() / ascii_safe_name()", safe_name(unsafe) != "" && ascii_safe_name("Hello world! 42") == "Hello_world_42", safe_name(unsafe) + " / " + ascii_safe_name("Hello world! 42")); + + DValue mutable_tree; + mutable_tree["keep"] = "yes"; + mutable_tree["remove"] = "no"; + String mutable_json = mutable_tree.to_json(); + mutable_tree.remove("remove"); + bool removed_ok = !mutable_tree.has("remove") && mutable_tree["keep"].to_string() == "yes"; + mutable_tree.clear(); + check("DValue to_json() / remove() / clear()", contains(mutable_json, "keep") && removed_ok && !mutable_tree.has("keep") && mutable_tree.get_type_name() == "empty", mutable_json); + // wasm_trace.h (WASM-PROPOSAL Phase 4): canned Wasmtime-format messages — // the live trap path is gated by the phase-4 spike runner String trace_msg = "error while executing at wasm backtrace:\n 0: 0x9f - units!_Z6renderR7Request\n"; @@ -255,6 +279,16 @@ RENDER(Request& context) check("wasm_trace_collapse() raw passthrough", wasm_trace_collapse("plain host error") == "plain host error", wasm_trace_collapse("plain host error")); + u32 noise32_a = gen_noise32(7, 11); + u32 noise32_b = gen_noise32(7, 11); + u64 noise64 = gen_noise64(7, 11); + f64 noise01 = gen_noise01(7, 11); + u64 ranged_int = gen_int(10, 20, 3, 4); + f64 ranged_float = gen_float(1.0, 2.0, 3, 4); + u64 drawn_int = draw_int(1, 3); + f64 drawn_float = draw_float(1.0, 2.0); + check("gen_sha1() / gen_*() / draw_*()", gen_sha1("uce") == gen_sha1("uce") && gen_sha1("uce") != gen_sha1("UCE") && noise32_a == noise32_b && noise64 > 0 && noise01 >= 0.0 && noise01 <= 1.0 && ranged_int >= 10 && ranged_int <= 20 && ranged_float >= 1.0 && ranged_float <= 2.0 && drawn_int >= 1 && drawn_int <= 3 && drawn_float >= 1.0 && drawn_float <= 2.0, "noise32=" + std::to_string((u64)noise32_a) + " noise64=" + std::to_string(noise64) + " noise01=" + std::to_string(noise01) + " draw=" + std::to_string(drawn_int)); + site_tests_summary(passed, failed, skipped, "These assertions intentionally stay pure and side-effect free so they remain safe on the public site."); site_tests_page_end(); } diff --git a/site/tests/http.uce b/site/tests/http.uce index 254e46e..a576a50 100644 --- a/site/tests/http.uce +++ b/site/tests/http.uce @@ -44,6 +44,12 @@ RENDER(Request& context) check("uri_decode() malformed percent literals", malformed_percent == "% %A %GG ok done", malformed_percent); check("parse_uri() empty input", empty_uri.parts["raw"] == "", var_dump(empty_uri)); check("parse_query()", query["alpha"] == "1" && query["beta"] == "two words" && query["token"] == "a=b" && query["keyless"] == "" && query["empty"] == "" && query["trailing"] == "ok" && query.count("") == 0, query_dump); + StringMap encoded_query_map; + encoded_query_map["alpha"] = "one two"; + encoded_query_map["token"] = "a=b"; + String encoded_query = encode_query(encoded_query_map); + DValue raw_route = request_route_from_raw_path("/workspace/projects/", "index"); + check("encode_query() / request route helpers", contains(encoded_query, "alpha=one%20two") && contains(encoded_query, "token=a%3Db") && request_script_url(context) != "" && request_base_url(context) != "" && raw_route["path"].to_string() == "workspace/projects" && raw_route["valid"].to_bool(), encoded_query + " script=" + request_script_url(context) + " base=" + request_base_url(context) + " route=" + json_encode(raw_route)); check("set_cookie()", set_cookie_dump.find("site-tests-cookie") != String::npos && set_cookie_dump.find("cookie-value") != String::npos && set_cookie_dump.find("HttpOnly") != String::npos && set_cookie_dump.find("SameSite=Lax") != String::npos, set_cookie_dump); check("response header mutation", context.header["X-Site-Tests"] == "http-suite", header_dump); check("session_start()", session_id != "", "session_id=" + session_id); diff --git a/site/tests/io.uce b/site/tests/io.uce index 3613aa6..e2c5e2e 100644 --- a/site/tests/io.uce +++ b/site/tests/io.uce @@ -29,9 +29,57 @@ RENDER(Request& context) site_tests_page_start("Filesystem", "Local-only file helper coverage using a temporary file under /tmp."); check("path_join()", file_name == "/tmp/uce-site-tests-io.txt", file_name); + check("basename() / dirname() / expand_path()", basename(file_name) == "uce-site-tests-io.txt" && dirname(file_name) == "/tmp" && expand_path("child", "/tmp/base") == "/tmp/base/child" && expand_path("../sibling", "/tmp/base") == "/tmp/sibling", basename(file_name) + " / " + dirname(file_name) + " / " + expand_path("../sibling", "/tmp/base")); check("file_put_contents()", write_ok, file_name); - check("file_append()", file_text.find("|beta") != String::npos, file_text); + check("file_append() / file_append_contents()", file_append_contents(file_name, "|gamma") && contains(file_get_contents(file_name), "|gamma"), file_get_contents(file_name)); + file_put_contents(file_name, "alpha|beta"); check("file_get_contents()", file_text == "alpha|beta", file_text); + check("file_mtime()", file_mtime(file_name) > 0, std::to_string((u64)file_mtime(file_name))); + + String real_tmp = path_real("/tmp"); + String real_file = path_real(file_name); + check("path_real() / path_is_within() canonical containment", real_tmp != "" && real_file != "" && path_is_within(file_name, "/tmp") && path_is_within("/tmp/../tmp/uce-site-tests-io.txt", "/tmp") && !path_is_within("/tmp", file_name) && !path_is_within("/tmp", "/tmp/uce-site-tests-io.txt"), real_tmp + " / " + real_file); + + String shell_escaped = shell_escape("a'b; echo injected"); + String shell_out = shell_exec("printf %s " + shell_escaped); + check("shell_escape() / shell_exec()", shell_out == "a'b; echo injected" && contains(shell_escaped, "'\\''"), shell_escaped + " => " + shell_out); + + String lock_file = "/tmp/uce-site-tests-lock.txt"; + int lock_fd = file_open_locked(lock_file, 66, LOCK_EX, 0644, 3.0, "site tests"); + bool lock_write = file_put_contents_locked_fd(lock_fd, "locked-content"); + String lock_read = file_get_contents_locked_fd(lock_fd); + file_close_locked(lock_fd); + check("file_open_locked() / locked fd read-write / close", lock_fd > 0 && lock_write && lock_read == "locked-content" && file_get_contents(lock_file) == "locked-content", "fd=" + std::to_string(lock_fd) + " read=" + lock_read); + int release_fd = file_open_locked(lock_file, 66, LOCK_EX, 0644, 3.0, "release test"); + file_release_process_locks("site tests release"); + check("file_release_process_locks()", release_fd > 0, "fd=" + std::to_string(release_fd)); + + String start_dir = process_start_directory(); + String old_cwd = cwd_get(); + cwd_set("/tmp"); + String tmp_cwd = cwd_get(); + cwd_set(old_cwd); + check("cwd_get() / cwd_set() / process_start_directory()", old_cwd != "" && tmp_cwd == "/tmp" && process_start_directory() == start_dir && start_dir != "", "old=" + old_cwd + " tmp=" + tmp_cwd + " start=" + start_dir); + + StringMap cfg; + cfg["u64"] = "42"; + cfg["f64"] = "3.5"; + cfg["yes"] = "yes"; + cfg["no"] = "off"; + check("config_map_*() / config_*() helpers", config_map_u64(cfg, "u64", 7) == 42 && config_map_u64(cfg, "bad", 7) == 7 && config_map_f64(cfg, "f64", 1.0) > 3.49 && !config_map_bool(cfg, "no") && config_bool_value("false") == false && config_u64("NO_SUCH_UCE_TEST_KEY", 99) == 99 && config_f64("NO_SUCH_UCE_TEST_KEY", 1.25) == 1.25 && !config_bool("NO_SUCH_UCE_TEST_KEY", false), var_dump(cfg)); + + DValue perf = request_perf(); + check("request_perf()", perf["worker_pid"].to_u64() > 0 && perf["request_count"].to_u64() > 0, json_encode(perf)); + + u64 parsed_epoch = time_parse("1970-01-01 00:00:05 UTC"); + String utc_year = time_format_utc("%Y", parsed_epoch); + String local_year = time_format_local("%Y", parsed_epoch); + String relative = time_format_relative(time() - 120, "recent %deltaS", 1, "medium %deltaM", 3600, "old %deltaH"); + check("time_format_local() / time_format_relative() / time_parse()", parsed_epoch == 5 && utc_year == "1970" && local_year != "" && contains(relative, "medium"), "parsed=" + std::to_string(parsed_epoch) + " utc=" + utc_year + " local=" + local_year + " rel=" + relative); + + StringList escaped_keys = memcache_escape_keys({"a b", "line\nkey"}); + check("memcache_escape_key() / memcache_escape_keys()", memcache_escape_key("a b") == "a_b" && escaped_keys.size() == 2 && !contains(escaped_keys[1], "\n"), join(escaped_keys, ",")); + check("signal_name() / runtime_safe_key() / capture_backtrace_string()", signal_name(SIGSEGV) == "SIGSEGV" && runtime_safe_key(" task one ") == gen_sha1("task one") && runtime_safe_key(" ") == "" && capture_backtrace_string() == "", signal_name(SIGSEGV) + " / " + runtime_safe_key(" task one ")); site_tests_summary(passed, failed, skipped, "This page intentionally limits writes to /tmp/uce-site-tests-io.txt so it stays disposable."); site_tests_page_end(); diff --git a/site/tests/manifest.txt b/site/tests/manifest.txt index 0574e30..3f1ed73 100644 --- a/site/tests/manifest.txt +++ b/site/tests/manifest.txt @@ -12,6 +12,7 @@ sqlite.uce|SQLite|SQLite connector with prepared named parameters and DValue row zip.uce|ZIP|Archive helpers that create and extract temporary server-side files.|http suite uce internal|ZIP|1|1 services.uce|Sockets And Services|Network/service helpers that are restricted outside trusted networks.|http suite uce internal|Sockets And Services|1|1 tasks.uce|Tasks|Background task helper coverage.|http suite uce internal|Tasks|1|1 +api_coverage.uce|API Coverage Manifest|Repository-level guard that public wasm APIs have tests/docs or an explicit internal decision.|http suite uce internal|API Coverage Manifest|1|1 security_headers.uce|Security Header Sanitizer|Low-level response header sanitizer fixture covered by the security smoke suite.|security http internal fixture|security header sanitizer test|0|1 cli.uce|CLI Fixture|Local CLI socket fixture; HTTP render intentionally returns 404.|cli internal fixture|uce-unit-cli|0|0 call_helpers.uce|Unit Call Fixture|Fixture for unit_call and unit_render helper tests.|internal fixture|UNIT_RENDER_FIXTURE|0|0 diff --git a/site/tests/units.uce b/site/tests/units.uce index f7791d5..6aac875 100644 --- a/site/tests/units.uce +++ b/site/tests/units.uce @@ -30,6 +30,7 @@ RENDER(Request& context) check("units_list()", unit_paths.size() > 0, "count=" + std::to_string(unit_paths.size())); check("unit_info()", info["path"].to_string() != "", json_encode(info)); + check("unit_compile()", unit_compile("call_helpers.uce"), "call_helpers.uce"); check("unit_call()", call_output.find("UNIT_CALL_EXPORT_OK") != String::npos, call_output); check("unit_render()", render_output.find("data-unit-render=\"ok\"") != String::npos, render_output); diff --git a/vendor/miniz/LICENSE b/src/3rdparty/miniz/LICENSE similarity index 100% rename from vendor/miniz/LICENSE rename to src/3rdparty/miniz/LICENSE diff --git a/vendor/miniz/README-UCE.md b/src/3rdparty/miniz/README-UCE.md similarity index 64% rename from vendor/miniz/README-UCE.md rename to src/3rdparty/miniz/README-UCE.md index 2845a3a..475caee 100644 --- a/vendor/miniz/README-UCE.md +++ b/src/3rdparty/miniz/README-UCE.md @@ -2,7 +2,7 @@ UCE vendors miniz 3.0.2 from for ZIP archive support. -Files are kept under `vendor/miniz/` and are included by `src/lib/zip.cpp` so UCE can expose `zip_create()`, `zip_list()`, `zip_read()`, `zip_extract()`, `gz_compress()`, and `gz_uncompress()` without adding a runtime package dependency. +Files are kept under `src/3rdparty/miniz/` and are included by `src/lib/zip.cpp` so UCE can expose `zip_create()`, `zip_list()`, `zip_read()`, `zip_extract()`, `gz_compress()`, and `gz_uncompress()` without adding a runtime package dependency. Local patch: diff --git a/vendor/miniz/miniz.c b/src/3rdparty/miniz/miniz.c similarity index 100% rename from vendor/miniz/miniz.c rename to src/3rdparty/miniz/miniz.c diff --git a/vendor/miniz/miniz.h b/src/3rdparty/miniz/miniz.h similarity index 100% rename from vendor/miniz/miniz.h rename to src/3rdparty/miniz/miniz.h diff --git a/vendor/miniz/miniz_common.h b/src/3rdparty/miniz/miniz_common.h similarity index 100% rename from vendor/miniz/miniz_common.h rename to src/3rdparty/miniz/miniz_common.h diff --git a/vendor/miniz/miniz_export.h b/src/3rdparty/miniz/miniz_export.h similarity index 100% rename from vendor/miniz/miniz_export.h rename to src/3rdparty/miniz/miniz_export.h diff --git a/vendor/miniz/miniz_tdef.c b/src/3rdparty/miniz/miniz_tdef.c similarity index 100% rename from vendor/miniz/miniz_tdef.c rename to src/3rdparty/miniz/miniz_tdef.c diff --git a/vendor/miniz/miniz_tdef.h b/src/3rdparty/miniz/miniz_tdef.h similarity index 100% rename from vendor/miniz/miniz_tdef.h rename to src/3rdparty/miniz/miniz_tdef.h diff --git a/vendor/miniz/miniz_tinfl.c b/src/3rdparty/miniz/miniz_tinfl.c similarity index 100% rename from vendor/miniz/miniz_tinfl.c rename to src/3rdparty/miniz/miniz_tinfl.c diff --git a/vendor/miniz/miniz_tinfl.h b/src/3rdparty/miniz/miniz_tinfl.h similarity index 100% rename from vendor/miniz/miniz_tinfl.h rename to src/3rdparty/miniz/miniz_tinfl.h diff --git a/vendor/miniz/miniz_zip.c b/src/3rdparty/miniz/miniz_zip.c similarity index 100% rename from vendor/miniz/miniz_zip.c rename to src/3rdparty/miniz/miniz_zip.c diff --git a/vendor/miniz/miniz_zip.h b/src/3rdparty/miniz/miniz_zip.h similarity index 100% rename from vendor/miniz/miniz_zip.h rename to src/3rdparty/miniz/miniz_zip.h diff --git a/vendor/miniz/readme.md b/src/3rdparty/miniz/readme.md similarity index 100% rename from vendor/miniz/readme.md rename to src/3rdparty/miniz/readme.md diff --git a/src/lib/compiler.h b/src/lib/compiler.h index 16c6972..23dc6e2 100644 --- a/src/lib/compiler.h +++ b/src/lib/compiler.h @@ -28,7 +28,9 @@ DValue unit_info(String path = ""); StringList units_list(); bool unit_compile(String path = ""); +#ifndef __UCE_WASM_UNIT__ SharedUnit* unit_load(String file_name); +#endif void unit_render(String file_name); void unit_render(String file_name, Request& context); DValue* unit_call(String file_name, String function_name, DValue* call_param = 0); diff --git a/src/lib/sys.cpp b/src/lib/sys.cpp index dbf6da8..be089de 100644 --- a/src/lib/sys.cpp +++ b/src/lib/sys.cpp @@ -2,6 +2,7 @@ #include #include "types.h" #include "functionlib.h" +#include "hash.h" #include "uri.h" #include "sys.h" @@ -34,6 +35,18 @@ int uce_host_server_stop(const char* key, size_t key_len); size_t uce_host_memcache_command(uint64_t sockfd, const char* command, size_t command_len, char* buf, size_t cap); size_t uce_host_mysql(const char* in, size_t in_len, char* out, size_t cap); size_t uce_host_request_perf(const char* in, size_t in_len, char* out, size_t cap); +size_t uce_host_shell_exec(const char* cmd, size_t cmd_len, char* buf, size_t cap); +int uce_host_file_open_locked(const char* path, size_t path_len, int open_flags, int lock_type, int create_mode, double wait_timeout_seconds, const char* purpose, size_t purpose_len); +void uce_host_file_close_locked(int handle); +void uce_host_file_release_process_locks(const char* reason, size_t reason_len); +size_t uce_host_file_read_locked_fd(int handle, char* buf, size_t cap); +int uce_host_file_write_locked_fd(int handle, const char* content, size_t content_len); +size_t uce_host_path_real(const char* path, size_t path_len, char* buf, size_t cap); +int uce_host_path_is_within(const char* path, size_t path_len, const char* root, size_t root_len); +size_t uce_host_cwd_get(char* buf, size_t cap); +int uce_host_cwd_set(const char* path, size_t path_len); +size_t uce_host_process_start_directory(char* buf, size_t cap); +size_t uce_host_last_trap_trace(char* buf, size_t cap); } static String wasm_current_unit_file() @@ -41,13 +54,45 @@ static String wasm_current_unit_file() return(context ? context->resources.current_unit_file : String("")); } -String shell_exec(String cmd) { (void)cmd; return(""); } -String shell_escape(String raw) { return(raw); } +String shell_exec(String cmd) +{ + size_t required = uce_host_shell_exec(cmd.data(), cmd.size(), 0, 0); + if(required == 0) + return(""); + String output(required, 0); + size_t got = uce_host_shell_exec(cmd.data(), cmd.size(), &output[0], required); + output.resize(got <= required ? got : 0); + return(output); +} +String shell_escape(String raw) +{ + String result; + for(auto c : raw) + { + if(c == '\'') + result.append("'\\''"); + else + result.append(1, c); + } + return("'" + result + "'"); +} String basename(String fn) { while(fn.find("/") != String::npos) fn = fn.substr(fn.find("/") + 1); return(fn); } String dirname(String fn) { auto pos = fn.find_last_of('/'); return(pos == String::npos ? "" : fn.substr(0, pos)); } String path_join(String base, String child) { if(base == "") return(child); if(child == "") return(base); if(child[0] == '/') return(child); return(base + (base.back() == '/' ? "" : "/") + child); } -String path_real(String path) { return(path); } -bool path_is_within(String path, String root) { return(str_starts_with(path, root)); } +String path_real(String path) +{ + size_t required = uce_host_path_real(path.data(), path.size(), 0, 0); + if(required == 0) + return(""); + String resolved(required, 0); + size_t got = uce_host_path_real(path.data(), path.size(), &resolved[0], required); + resolved.resize(got <= required ? got : 0); + return(resolved); +} +bool path_is_within(String path, String root) +{ + return(uce_host_path_is_within(path.data(), path.size(), root.data(), root.size()) != 0); +} bool mkdir(String path) { String current = wasm_current_unit_file(); @@ -58,11 +103,29 @@ bool file_exists(String path) String current = wasm_current_unit_file(); return(uce_host_file_exists(path.data(), path.size(), current.data(), current.size()) != 0); } -int file_open_locked(String file_name, int open_flags, int lock_type, int create_mode, f64 wait_timeout_seconds, String purpose) { (void)file_name; (void)open_flags; (void)lock_type; (void)create_mode; (void)wait_timeout_seconds; (void)purpose; return(-1); } -void file_close_locked(int fd) { (void)fd; } -void file_release_process_locks(String reason) { (void)reason; } -String file_get_contents_locked_fd(int fd) { (void)fd; return(""); } -bool file_put_contents_locked_fd(int fd, String content) { (void)fd; (void)content; return(false); } +int file_open_locked(String file_name, int open_flags, int lock_type, int create_mode, f64 wait_timeout_seconds, String purpose) +{ + return(uce_host_file_open_locked( + file_name.data(), file_name.size(), open_flags, lock_type, create_mode, wait_timeout_seconds, + purpose.data(), purpose.size() + )); +} +void file_close_locked(int fd) { uce_host_file_close_locked(fd); } +void file_release_process_locks(String reason) { uce_host_file_release_process_locks(reason.data(), reason.size()); } +String file_get_contents_locked_fd(int fd) +{ + size_t required = uce_host_file_read_locked_fd(fd, 0, 0); + if(required == 0) + return(""); + String content(required, 0); + size_t got = uce_host_file_read_locked_fd(fd, &content[0], required); + content.resize(got <= required ? got : 0); + return(content); +} +bool file_put_contents_locked_fd(int fd, String content) +{ + return(uce_host_file_write_locked_fd(fd, content.data(), content.size()) != 0); +} String file_get_contents(String file_name) { String current = wasm_current_unit_file(); @@ -84,9 +147,27 @@ bool file_append_contents(String file_name, String content) String current = wasm_current_unit_file(); return(uce_host_file_write(file_name.data(), file_name.size(), current.data(), current.size(), content.data(), content.size(), 1) != 0); } -String cwd_get() { return("/"); } -void cwd_set(String path) { (void)path; } -String process_start_directory() { return("/"); } +String cwd_get() +{ + size_t required = uce_host_cwd_get(0, 0); + if(required == 0) + return(""); + String cwd(required, 0); + size_t got = uce_host_cwd_get(&cwd[0], required); + cwd.resize(got <= required ? got : 0); + return(cwd); +} +void cwd_set(String path) { uce_host_cwd_set(path.data(), path.size()); } +String process_start_directory() +{ + size_t required = uce_host_process_start_directory(0, 0); + if(required == 0) + return(""); + String cwd(required, 0); + size_t got = uce_host_process_start_directory(&cwd[0], required); + cwd.resize(got <= required ? got : 0); + return(cwd); +} time_t file_mtime(String file_name) { String current = wasm_current_unit_file(); @@ -253,11 +334,51 @@ bool ws_close(String connection_id) context->resources.websocket_dispatch_commands.push(command); return(true); } -String backtrace_frames_string(void* const* frames, size_t size, u32 skip_frames) { (void)frames; (void)size; (void)skip_frames; return(""); } -String capture_backtrace_string(u32 max_frames, u32 skip_frames) { (void)max_frames; (void)skip_frames; return(""); } -String signal_name(int sig) { (void)sig; return(""); } -String memcache_escape_key(String key) { return(key); } -StringList memcache_escape_keys(StringList keys) { return(keys); } +String backtrace_frames_string(void* const* frames, size_t size, u32 skip_frames) { (void)frames; (void)size; (void)skip_frames; return(capture_backtrace_string(0, 0)); } +String capture_backtrace_string(u32 max_frames, u32 skip_frames) +{ + (void)max_frames; + (void)skip_frames; + size_t required = uce_host_last_trap_trace(0, 0); + if(required == 0) + return(""); + String trace(required, 0); + size_t got = uce_host_last_trap_trace(&trace[0], required); + trace.resize(got <= required ? got : 0); + return(trace); +} +String signal_name(int sig) +{ + switch(sig) + { + case 6: return("SIGABRT"); + case 7: return("SIGBUS"); + case 8: return("SIGFPE"); + case 4: return("SIGILL"); + case 2: return("SIGINT"); + case 11: return("SIGSEGV"); + case 15: return("SIGTERM"); + default: return(""); + } +} +String memcache_escape_key(String key) +{ + String result; + for(auto c : key) + { + if(isspace((unsigned char)c)) + c = '_'; + result.append(1, c); + } + return(result); +} +StringList memcache_escape_keys(StringList keys) +{ + StringList result; + for(auto s : keys) + result.push_back(memcache_escape_key(s)); + return(result); +} u64 memcache_connect(String host, short port) { if(host == "") @@ -345,7 +466,14 @@ extern "C" int uce_wasm_task_run(uint64_t callback_id) } int task_kill(pid_t pid, int sig) { return(uce_host_task_kill(pid, sig)); } -String runtime_safe_key(String key, String label) { (void)label; return(key); } +String runtime_safe_key(String key, String label) +{ + (void)label; + key = trim(key); + if(key == "") + return(""); + return(gen_sha1(key)); +} pid_t task(String key, std::function exec_after_spawn, u64 timeout) { u64 id = wasm_next_task_callback_id++; diff --git a/src/lib/zip.cpp b/src/lib/zip.cpp index cea8805..bb61c50 100644 --- a/src/lib/zip.cpp +++ b/src/lib/zip.cpp @@ -3,10 +3,10 @@ #include extern "C" { -#include "../../vendor/miniz/miniz.c" -#include "../../vendor/miniz/miniz_tdef.c" -#include "../../vendor/miniz/miniz_tinfl.c" -#include "../../vendor/miniz/miniz_zip.c" +#include "../3rdparty/miniz/miniz.c" +#include "../3rdparty/miniz/miniz_tdef.c" +#include "../3rdparty/miniz/miniz_tinfl.c" +#include "../3rdparty/miniz/miniz_zip.c" } String zip_error(String api, String detail) diff --git a/src/wasm/core_hostcalls.syms b/src/wasm/core_hostcalls.syms index 86d0ed6..5047b36 100644 --- a/src/wasm/core_hostcalls.syms +++ b/src/wasm/core_hostcalls.syms @@ -19,6 +19,7 @@ uce_host_zip uce_host_units uce_host_component_resolve uce_host_request_perf +uce_host_shell_exec uce_host_file_exists uce_host_file_read uce_host_file_write @@ -26,5 +27,16 @@ uce_host_file_unlink uce_host_file_list uce_host_file_mkdir uce_host_file_mtime +uce_host_file_open_locked +uce_host_file_close_locked +uce_host_file_release_process_locks +uce_host_file_read_locked_fd +uce_host_file_write_locked_fd +uce_host_path_real +uce_host_path_is_within +uce_host_cwd_get +uce_host_cwd_set +uce_host_process_start_directory +uce_host_last_trap_trace uce_host_regex uce_host_sqlite diff --git a/src/wasm/worker.cpp b/src/wasm/worker.cpp index f4ebc91..8aab1c4 100644 --- a/src/wasm/worker.cpp +++ b/src/wasm/worker.cpp @@ -40,6 +40,9 @@ #include #include #include +#include +#include +#include #include #include #include @@ -428,21 +431,32 @@ public: request_perf.active = true; } +// Host-owned opaque fd handles opened by wasm file_open_locked(). + std::vector locked_file_handles; + #ifdef UCE_WASM_HOST_CONNECTORS // Host-owned resource handle table (§3.1): connections opened by the guest // live here and are closed when the workspace drops at request end. std::vector sqlite_handles; std::vector mysql_handles; +#endif ~WasmWorkspace() { + for(int fd : locked_file_handles) + if(fd >= 0) + { + flock(fd, LOCK_UN); + ::close(fd); + } +#ifdef UCE_WASM_HOST_CONNECTORS for(auto* db : sqlite_handles) if(db) delete db; // ~SQLite disconnects for(auto* db : mysql_handles) if(db) delete db; // ~MySQL disconnects - } #endif + } // The guest calls a sized hostcall twice (buf=0 to learn the length, then // to fetch). For side-effecting ops (sqlite) re-executing on the fetch is @@ -1304,6 +1318,77 @@ private: fprintf(stderr, "[guest log %d] %.*s\n", args[0].i32(), (int)text.size(), text.data()); return(std::monostate()); })); + if(mod == "env" && name == "uce_host_shell_exec") + return(add([self](Caller caller, Span args, Span results) -> Result { + String cmd; + self->hostcall_read(args[0].i32(), args[1].i32(), cmd); + String out; + if(!self->hostcall_staged("shell:" + cmd, out)) + { + out = ::shell_exec(cmd); + self->hostcall_stage("shell:" + cmd, out); + } + u32 cap = (u32)args[3].i32(); + int32_t buf = args[2].i32(); + if(buf != 0 && cap >= out.size()) + self->hostcall_write(buf, out); + caller.context().set_epoch_deadline(self->worker.cfg.epoch_deadline_ticks); + results[0] = Val((int32_t)out.size()); + return(std::monostate()); + })); + if(mod == "env" && name == "uce_host_path_real") + return(add([self](Caller, Span args, Span results) -> Result { + String path; + self->hostcall_read(args[0].i32(), args[1].i32(), path); + String resolved = ::path_real(path); + u32 cap = (u32)args[3].i32(); + int32_t buf = args[2].i32(); + if(buf != 0 && cap >= resolved.size()) + self->hostcall_write(buf, resolved); + results[0] = Val((int32_t)resolved.size()); + return(std::monostate()); + })); + if(mod == "env" && name == "uce_host_path_is_within") + return(add([self](Caller, Span args, Span results) -> Result { + String path, root; + self->hostcall_read(args[0].i32(), args[1].i32(), path); + self->hostcall_read(args[2].i32(), args[3].i32(), root); + results[0] = Val(::path_is_within(path, root) ? (int32_t)1 : (int32_t)0); + return(std::monostate()); + })); + if(mod == "env" && name == "uce_host_cwd_get") + return(add([self](Caller, Span args, Span results) -> Result { + String cwd = ::cwd_get(); + u32 cap = (u32)args[1].i32(); + int32_t buf = args[0].i32(); + if(buf != 0 && cap >= cwd.size()) + self->hostcall_write(buf, cwd); + results[0] = Val((int32_t)cwd.size()); + return(std::monostate()); + })); + if(mod == "env" && name == "uce_host_cwd_set") + return(add([self](Caller, Span args, Span results) -> Result { + String path; + self->hostcall_read(args[0].i32(), args[1].i32(), path); + results[0] = Val(::chdir(path.c_str()) == 0 ? (int32_t)1 : (int32_t)0); + return(std::monostate()); + })); + if(mod == "env" && name == "uce_host_process_start_directory") + return(add([self](Caller, Span args, Span results) -> Result { + String cwd = ::process_start_directory(); + u32 cap = (u32)args[1].i32(); + int32_t buf = args[0].i32(); + if(buf != 0 && cap >= cwd.size()) + self->hostcall_write(buf, cwd); + results[0] = Val((int32_t)cwd.size()); + return(std::monostate()); + })); + if(mod == "env" && name == "uce_host_last_trap_trace") + return(add([](Caller, Span args, Span results) -> Result { + (void)args; + results[0] = Val((int32_t)0); + return(std::monostate()); + })); if(mod == "env" && name == "uce_host_file_exists") return(add([self](Caller, Span args, Span results) -> Result { String path, current; @@ -1340,6 +1425,87 @@ private: results[0] = Val((int64_t)mtime); return(std::monostate()); })); + if(mod == "env" && name == "uce_host_file_open_locked") + return(add([self](Caller, Span args, Span results) -> Result { + String path, purpose; + self->hostcall_read(args[0].i32(), args[1].i32(), path); + self->hostcall_read(args[6].i32(), args[7].i32(), purpose); + int fd = ::file_open_locked(path, args[2].i32(), args[3].i32(), args[4].i32(), args[5].f64(), purpose); + int handle = -1; + if(fd >= 0) + { + for(size_t i = 0; i < self->locked_file_handles.size(); i++) + if(self->locked_file_handles[i] < 0) + { + self->locked_file_handles[i] = fd; + handle = (int)i + 1; + break; + } + if(handle < 0) + { + self->locked_file_handles.push_back(fd); + handle = (int)self->locked_file_handles.size(); + } + } + results[0] = Val((int32_t)handle); + return(std::monostate()); + })); + if(mod == "env" && name == "uce_host_file_close_locked") + return(add([self](Caller, Span args, Span) -> Result { + int handle = args[0].i32(); + if(handle >= 1 && (size_t)handle <= self->locked_file_handles.size()) + { + int& fd = self->locked_file_handles[(size_t)handle - 1]; + if(fd >= 0) + { + ::file_close_locked(fd); + fd = -1; + } + } + return(std::monostate()); + })); + if(mod == "env" && name == "uce_host_file_release_process_locks") + return(add([self](Caller, Span, Span) -> Result { + for(int& fd : self->locked_file_handles) + if(fd >= 0) + { + ::file_close_locked(fd); + fd = -1; + } + return(std::monostate()); + })); + if(mod == "env" && name == "uce_host_file_read_locked_fd") + return(add([self](Caller, Span args, Span results) -> Result { + int handle = args[0].i32(); + String content; + if(handle >= 1 && (size_t)handle <= self->locked_file_handles.size()) + { + int fd = self->locked_file_handles[(size_t)handle - 1]; + if(fd >= 0) + content = ::file_get_contents_locked_fd(fd); + } + u32 cap = (u32)args[2].i32(); + int32_t buf = args[1].i32(); + if(buf != 0 && cap >= content.size()) + self->hostcall_write(buf, content); + results[0] = Val((int32_t)content.size()); + return(std::monostate()); + })); + if(mod == "env" && name == "uce_host_file_write_locked_fd") + return(add([self](Caller, Span args, Span results) -> Result { + int handle = args[0].i32(); + String content; + self->hostcall_read(args[1].i32(), args[2].i32(), content); + bool ok = false; + if(handle >= 1 && (size_t)handle <= self->locked_file_handles.size()) + { + int fd = self->locked_file_handles[(size_t)handle - 1]; + if(fd >= 0) + ok = ::file_put_contents_locked_fd(fd, content); + } + results[0] = Val(ok ? (int32_t)1 : (int32_t)0); + return(std::monostate()); + })); if(mod == "env" && name == "uce_host_file_read") return(add([self](Caller, Span args, Span results) -> Result { String path, current; diff --git a/todo.txt b/todo.txt deleted file mode 100644 index b41261d..0000000 --- a/todo.txt +++ /dev/null @@ -1,38 +0,0 @@ -Library -================================= -- md5 / meow -- fold StringList into DValue _OR_ make better StringList -- make session data use DValue instead of StringList - -Bugs -================================= -- shell_escape() - - -Framework -================================= -- Check: File Upload -- Proxy mode -- WebSockets -- Resident code / Cron / Tick? -- Automated Test Suite -- Documentation - - -Performance -================================= -- Arena Allocator -- Array implementation - - -Nice to Have -================================= -- pseudorandom -- XML components -- Optionally store compiled units alongside source - - -Finally -================================= -- DO WE DO OUR OWN LANGUAGE OR DO WE KEEP USING C?!? -