uce/API_TODO.md

API_TODO — trans-membrane OS surface expansion

Status: briefing / design spec for the next round of uce_host_* membrane work. Implementation delegated to the pi agent + sub-agents.

Why

UCE units run as wasm in a per-request workspace; the only way they touch the OS is via uce_host_* hostcalls (see docs/wasm-runtime-architecture.md). That constrained, auditable surface is a feature — it's the security boundary sysadmins can reason about. The current file API (file_get_contents / file_put_contents / file_append / file_exists / file_unlink / ls / file_mkdir / file_mtime / path_* / cwd_*) is simple and good but is whole-file only and lacks several OS capabilities real apps need. This doc widens what units can do within the existing path roots — not where.

Cross-cutting design principles (apply to EVERY new call)

1. Requests must complete fast — never block indefinitely. Every potentially blocking call takes an explicit timeout/deadline and fails fast on expiry with a clear result, rather than spinning or blocking forever. In particular:
   • File locks expire. Auto-locking (writes take exclusive, reads wait for writers) must use a bounded wait (deadline, e.g. config UCE_FILE_LOCK_TIMEOUT_MS, sane default ~2000ms). On timeout the op returns a clear "lock timeout" error — it does NOT hang the request. This applies to file_put_contents/file_append/file_get_contents and the new streaming handles below.
   • http_request, blocking shell_exec, job_await, etc. all take timeouts.
2. Fire-and-forget + check back later. Long work goes to the background and the request returns immediately with a job id; a later request (any worker) checks the outcome. A single host-side job registry backs this (see "Async jobs"). This is the model for background shell, async http, and any long op — not per-request blocking.
3. Simple AND versatile. Prefer one small, composable primitive over many special cases. Structured inputs/outputs are DValue (UCEB across the membrane). Streaming uses opaque host handles (u64). Errors are returned values (no exceptions in the guest).
4. One capability per named hostcall (blocklist granularity — see Future). Don't bundle unrelated capabilities into one mega-hostcall.
5. Always policy-gated. Every path goes through resolve_guest_file / resolve_guest_write (roots); handles/stat/rename/etc. gated the same way.
6. No exceptions/RTTI in unit-facing code; return values + status DValues.

API groups to implement

1. Streaming / handle-based file I/O (P0 — the keystone)

Whole-file String I/O can't handle large or partial files. Add a handle API. Locking stays automatic and lifetime-scoped (open-for-write = exclusive, open-for-read = shared, released on file_close), with the bounded-wait rule from principle 1 — this is NOT the old manual lock API, the user never calls a lock function.

u64    file_open(String path, String mode)        // "r" "w" "a" "r+"; 0 = error (incl. lock timeout)
String file_read(u64 h, u64 len)                  // sequential
String file_pread(u64 h, u64 offset, u64 len)     // positional
u64    file_write(u64 h, String data)             // sequential; returns bytes written
u64    file_pwrite(u64 h, u64 offset, String data)
s64    file_seek(u64 h, s64 offset, int whence)   // + file_tell(h)
void   file_close(u64 h)                          // releases lock + fd

Keep the existing whole-file file_get_contents/put_contents/append (now auto-locking) as the easy path.

2. Metadata (P0/P1)

DValue file_stat(String path)   // { exists, size, mtime, ctime, mode, is_dir, is_file, is_symlink }
DValue dir_list(String path)    // [ { name, type, size, mtime } ... ]; optional glob/recursive flags

(file_mtime stays; dir_list is the richer ls so callers don't N+1 file_stat.)

3. Structural ops (P1)

bool file_rename(String from, String to)   // atomic replace; enables write-temp-then-rename
bool file_copy(String from, String to)     // host-side, preserves mode
bool file_truncate(String path, u64 size)  // (or on a handle)
bool dir_remove(String path, bool recursive)

4. Niceties (P2)

String file_temp(String prefix)            // race-free temp file under a write root (mkstemp)
bool   file_chmod(String path, u32 mode)
bool   file_symlink(String target, String linkpath); String file_readlink(String path)
// durability: file_put_contents/file_write gain an optional fsync flag/variant

5. Outbound HTTP (with TLS) (high value)

socket_* is raw TCP only — no secure outbound. Add a real client:

DValue http_request(DValue req)   // req: { method, url, headers{}, body, timeout_ms, follow_redirects }
                                  // -> { status, headers{}, body, error }  (TLS/https supported)
u64    http_request_async(DValue req)  // -> job_id (see Async jobs)

6. Crypto (extended) (high value)

hash.h has sha1 + rng only.

String sha256(String data); String sha256_hex(String data)
String hmac_sha256(String key, String data)         // + _hex
String base64_encode(String data); String base64_decode(String b64)
String random_bytes(u64 n)                            // CSPRNG
bool   crypto_equal(String a, String b)               // constant-time compare

7. Shell (extended + background) (high value)

Current shell_exec is blocking + output-only.

DValue shell_exec(DValue spec)   // spec: { cmd, stdin, env{}, timeout_ms } -> { exit_code, stdout, stderr, timed_out }
u64    shell_spawn(DValue spec)  // background -> job_id; non-blocking; request returns immediately

(Keep a simple shell_exec(String) convenience overload returning stdout.)

8. Async jobs (the unifying "check back later" primitive)

Host-side job registry, survives the spawning request, reachable from any later request/worker by id.

DValue job_status(u64 job_id)            // { state: pending|running|done|failed, ... }
DValue job_result(u64 job_id)            // blocks up to a small default; { done, ... , result }
DValue job_await(u64 job_id, u64 timeout_ms)  // bounded wait; returns even if not done (state=running)
bool   job_cancel(u64 job_id)

shell_spawn and http_request_async produce job ids consumed here. Jobs have a TTL and are reaped; results are retrievable until collected or TTL expires.

Explicitly OUT of scope (declined)

• DNS resolution API, UDP sockets, setenv / environment mutation.

Future (design influences now, NOT implemented here)

• Configurable trans-membrane blocklist. A server config will let sysadmins disable specific hostcalls by name (e.g. turn off shell_exec, http_request) to harden a deployment. This is WHY principle 4 (one capability per named hostcall) matters — keep the granularity clean so each is independently gateable. A blocked call should return a clear "disabled by policy" error.
• Native app extensions. Longer term: let app developers ship their own native compiled modules (.so living in the site tree) that units can call across the membrane — a per-app plugin mechanism. Out of scope now, but the hostcall/membrane shape should not preclude a future generic "call host extension" dispatch.

Implementation notes

• Each call = a discrete uce_host_* hostcall: add to src/wasm/core_hostcalls.syms, register in src/wasm/worker.cpp, declare/implement the wasm-core wrapper in src/lib/sys.cpp (+ header) mirroring existing hostcalls (sized-buffer for bytes, sized-DValue/UCEB for structured returns).
• Reuse native impls where they exist; policy-gate every path.
• Each group lands suite-gated (run_cli_tests --include-wasm-kill, 0 failed) on the build host; add tests + doc pages per new function (function pages, and method/2_ pages where a class gains methods).