uce/docs/wasm-toolchain-findings.md

Phase 0 findings — toolchain & runtime spike

• Status: EXIT CRITERION PASSED (2026-06-12, on k-uce)
• Runtime selected: Wasmtime (v45.0.1, C API). WAMR rejected — evidence below.
• Exceptions decision (§11.1, error codes / -fno-exceptions) confirmed viable: both the stubs and two real generated units compile with -fno-exceptions, no try/catch blocker anywhere.

The exit criterion ran end-to-end: a core stub (libc/libc++ statically linked, owns memory/allocator) and a unit stub (PIC, dylink.0) were linked at runtime by loader.cpp and produced:

hello from unit; unit-data-segment-ok; counter=7; mapsum=3; core-string+unit[cb:42][got-func-ok][fn:42]
core_counter (in linear memory): before=7 after=8
PHASE0 EXIT CRITERION: PASS

which validates, in one render call: unit data-segment relocation (__memory_base), GOT.mem read and write of a core global, std::string /std::map in unit code on the core's heap, a heap C++ object created in core and mutated/read by the unit, function pointers crossing unit→core→unit through the shared table, GOT.func resolution, and a std::function lambda allocated in the unit and invoked by core. That is the §3.4 contract ("DValue inside the workspace: no serialization, ever") demonstrated at the ABI level.

Toolchain pins

What	Version	Where on k-uce
wasi-sdk	33 (clang 22.1.0-wasi-sdk)	/opt/wasi-sdk
target triple	wasm32-wasip1 (wasm32-wasi is deprecated)	—
Wasmtime C API	v45.0.1 (prebuilt x86_64-linux release)	/opt/wasmtime
WAMR (rejected)	WAMR-2.4.4, built from source	/opt/wamr
cmake / ninja	3.31.6 / 1.12.1 (apt)	—

Runtime selection: why not WAMR

WAMR was tried first per §9 ("preferred ... use Wasmtime only if blocked"). We are blocked, on the load-bearing requirement itself:

1. WAMR's wasm-c-api ignores imported memories and tables. At unit instantiation it logs "doesn't support import memories and tables for now, ignore them" (wasm_c_api.c) and gives the instance its own memory/table — which silently destroys the shared-workspace model.
2. Host-side wasm_table_grow / wasm_memory_grow are explicitly unsupported ("Only allow growing a table via the opcode table.grow").
3. Its build banner lists Import/Export of Mutable Globals as unsupported — the dylink ABI imports __stack_pointer and every GOT.* entry as a mutable global.

Its multi-module feature is name-based auto-resolution, not host-orchestrated dylink (no host-computed __memory_base/__table_base, no GOT). Making WAMR fit means implementing import binding through the c-api layer and runtime internals — a runtime-development project, not a patch.

Wasmtime v45.0.1 passed everything on the first run through the standard wasm.h C API: host-created funcref table shared by both instances, exported memory imported by the unit, host-created (mutable) globals, cross-instance export→import wiring. The remaining §9 criteria also favor it: AOT artifacts (.cwasm precompilation) for the unit cache, epoch interruption for CPU limits, and built-in copy-on-write memory-image instantiation for the §6 core snapshot (machinery we'd have had to build ourselves on WAMR).

Trade-off accepted (was already in §10): Rust codebase, heavier to vendor/patch. Pin the release artifact (lib + headers, checksummed) the way sqlite is vendored; building from source stays possible but is not the default path. The C API .so is ~27 MB.

Module build recipe (what build_modules.sh settled on)

Core (non-PIC reactor, owns libc/libc++/allocator):

clang++ --target=wasm32-wasip1 -mexec-model=reactor -O1 -fno-exceptions \
    core.cpp -o core.wasm \
    -Wl,--export-all -Wl,--import-table \
    -Wl,--export=__stack_pointer -Wl,--export=__heap_base \
    -Wl,--undefined=_ZTVN10__cxxabiv117__class_type_infoE

Unit (PIC side module):

clang++ --target=wasm32-wasip1 -fPIC -fvisibility=default \
    -fvisibility-inlines-hidden -O1 -fno-exceptions -c unit.cpp
wasm-ld -shared --experimental-pic --unresolved-symbols=import-dynamic \
    --Bsymbolic unit.o -o unit.wasm --export=<entry>

Hard-won flag findings:

1. --unresolved-symbols=import-dynamic is required for the side-module link; undefined symbols then become env.* function imports and GOT.* globals exactly per the Emscripten dylink ABI.
2. -fvisibility-inlines-hidden is mandatory. Without it one libc++ vague-linkage lambda (std::map tree-emplace internals, missing libc++'s usual hide-from-ABI attribute) is emitted as both an export and an import of the unit — a self-import the loader cannot satisfy at instantiation time without lazy-binding trampolines. --Bsymbolic alone did not bind it.
3. Core symbol closure: --export-all only exports what got linked. The unit needed __cxxabiv1::__class_type_info's vtable (RTTI machinery behind std::function), which the core never references — forced in with --undefined=. The production core needs a closure strategy: --whole-archive for libc/libc++/libc++abi, or a curated keep-list. The loader also implements the complementary fallback (resolve GOT.mem of weak data from the unit's own exports post-instantiation, patching the provisional GOT global).
4. --import-table on the core + a host-created table is the right shape (see loader notes); --export-table/--growable-table was the first attempt and died on WAMR's host-grow limitation, but host-created stays the better design under Wasmtime too: the loader picks table size (core's declared minimum + headroom) before any instantiation.
5. wasm32-wasi triple is deprecated in wasi-sdk 33 → use wasm32-wasip1.

Loader notes (loader.cpp, ~450 lines, standard wasm-c-api)

Sequence proven: instantiate core (45 WASI imports satisfied with named trap stubs — none was ever called) → parse dylink.0 (mem_info: memsize/align, tablesize) → __memory_base = call core's exported malloc → __table_base = bump pointer starting at core's table-import minimum → build the unit's import vector (memory/table/__stack_pointer shared from core; env.* functions from core exports; GOT.mem.* as host mutable i32 globals holding addresses read from core's exported data-symbol globals) → instantiate → patch deferred GOT entries → __wasm_apply_data_relocs → __wasm_call_ctors → call the entry export.

• Erratum (found in Phase 3): self-resolved GOT.mem values must add __memory_base. A PIC module's exported data symbols are i32 globals holding offsets relative to its __memory_base, not absolute addresses — the linker adds the base when patching deferred GOT entries (there is no __wasm_apply_global_relocs export to do it). Copying the export verbatim reads/writes core memory at low addresses and renders silently wrong values; the Phase 3 fixture's self-got/callback markers exist to catch exactly this. GOT.mem entries resolved from the core's exports are absolute already (the core is non-PIC) and need no adjustment.
• GOT.func is resolved guest-side: the core exports a helper returning (intptr_t)&func — taking the address forces a link-time elem entry and a wasm function pointer is its table index. No host-side funcref injection is needed at all (it was WAMR-unsupported; under Wasmtime it would work but the guest-side registry is simpler and runtime-agnostic). The production core should carry a name→funcptr registry (dlsym-shaped) for its API surface.
• wasi-libc gotcha: _initialize has a double-init guard ending in __builtin_trap(). WAMR runs _initialize automatically at instantiation (so calling it again traps "unreachable"); Wasmtime does not (so you must call it). Cost one debugging round; recorded here so it never costs another.
• Export-name wasm_name_t may include the trailing NUL in size (WAMR did); trim when indexing exports by name.

Real generated units (delegated grind — full log in realunit-report.md)

site/demo/collections.uce.cpp and hello.uce.cpp (taken verbatim from the live unit cache at /tmp/uce/work/...) both compile and link as PIC side modules with dylink.0, no allocator definitions, with only shim-level intervention. collections.wasm: 42 KB, 52 imports — including exactly the predicted GOT.mem.context for the global Request*. Friction points that become Phase 2 work items:

1. types.h defines global operator new/delete in every unit — must be gated (#ifdef) out of side-module builds; allocator ownership belongs to the core (§3.2: "the one fatal misconfiguration").
2. sys.h includes <signal.h> → wasi needs -D_WASI_EMULATED_SIGNAL (+ -lwasi-emulated-signal in the core) or an #ifdef __wasm__ carve-out; signals/fork/exec/sockets in sys.h have no wasi equivalent and move behind hostcalls anyway (§5.1).
3. Generated units include uce_lib.h by absolute path — the preprocessor should emit a logical include so the wasm build can supply its own include order.
4. Header-inline connector wrappers (MySQL etc.) get pulled into every unit regardless of use; the §3.3 membrane split (thin wasm-side shim, host-side implementation) resolves this and shrinks unit import lists.
5. The real-unit compile used the pre--fvisibility-inlines-hidden flag set (parallel work); the final unit flag set above should be used from Phase 2 on.

Implications for the next phases

• Phase 1 (DValue C ABI, native): unaffected by any of this; proceed as written.
• Phase 2 (core module + membrane): add the closure strategy (whole-archive), the GOT.func name→funcptr registry, the types.h allocator gate, the signal emulation define, and the preprocessor include change. Compile uce_lib with the core recipe above.
• Phase 3 (loader): loader.cpp here is the skeleton — dylink parsing, base allocation, GOT resolution, and init sequencing are all proven; what remains is the registry/dispatch layer, ABI stamping, and multi-unit placement.
• Phase 4: use Wasmtime's epoch interruption for CPU limits and its memory-image/CoW instantiation for the core snapshot rather than building either by hand.

Artifacts (on k-uce, not in git)

• /tmp/uce/wasm-phase0/{core.wasm,unit.wasm,loader} — exit-criterion run
• /tmp/uce/wasm-phase0/realunit/ — real-unit compiles + shim tree + inspector dumps
• /opt/wasi-sdk, /opt/wasmtime, /opt/wamr — toolchains/runtimes