uce/API_TODO.md

# API_TODO — trans-membrane OS surface expansion

Status: briefing / design spec for the next round of `uce_host_*` membrane work.
Implementation delegated to the pi agent + sub-agents.

## Why

UCE units run as wasm in a per-request workspace; the **only** way they touch the
OS is via `uce_host_*` hostcalls (see `docs/wasm-runtime-architecture.md`). That
constrained, auditable surface is a feature — it's the security boundary
sysadmins can reason about. The current file API (`file_get_contents` /
`file_put_contents` / `file_append` / `file_exists` / `file_unlink` / `ls` /
`file_mkdir` / `file_mtime` / `path_*` / `cwd_*`) is simple and good but is
**whole-file only** and lacks several OS capabilities real apps need. This doc
widens *what* units can do **within the existing path roots** — not *where*.

## Cross-cutting design principles (apply to EVERY new call)

1. **Requests must complete fast — never block indefinitely.** Every potentially
   blocking call takes an explicit timeout/deadline and **fails fast** on expiry
   with a clear result, rather than spinning or blocking forever. In particular:
   - **File locks expire.** Auto-locking (writes take exclusive, reads wait for
     writers) must use a **bounded** wait (deadline, e.g. config
     `UCE_FILE_LOCK_TIMEOUT_MS`, sane default ~2000ms). On timeout the op returns
     a clear "lock timeout" error — it does NOT hang the request. This applies to
     `file_put_contents`/`file_append`/`file_get_contents` and the new streaming
     handles below.
   - `http_request`, blocking `shell_exec`, `job_await`, etc. all take timeouts.
2. **Fire-and-forget + check back later.** Long work goes to the background and
   the request returns immediately with a **job id**; a *later* request (any
   worker) checks the outcome. A single host-side **job registry** backs this
   (see "Async jobs"). This is the model for background shell, async http, and
   any long op — not per-request blocking.
3. **Simple AND versatile.** Prefer one small, composable primitive over many
   special cases. Structured inputs/outputs are `DValue` (UCEB across the
   membrane). Streaming uses opaque host **handles** (u64). Errors are returned
   values (no exceptions in the guest).
4. **One capability per named hostcall** (blocklist granularity — see Future).
   Don't bundle unrelated capabilities into one mega-hostcall.
5. **Always policy-gated.** Every path goes through `resolve_guest_file` /
   `resolve_guest_write` (roots); handles/stat/rename/etc. gated the same way.
6. **No exceptions/RTTI** in unit-facing code; return values + status DValues.

## API groups to implement

### 1. Streaming / handle-based file I/O  (P0 — the keystone)
Whole-file String I/O can't handle large or partial files. Add a handle API.
Locking stays automatic and **lifetime-scoped** (open-for-write = exclusive,
open-for-read = shared, released on `file_close`), with the bounded-wait rule
from principle 1 — this is NOT the old manual lock API, the user never calls a
lock function.
```
u64    file_open(String path, String mode)        // "r" "w" "a" "r+"; 0 = error (incl. lock timeout)
String file_read(u64 h, u64 len)                  // sequential
String file_pread(u64 h, u64 offset, u64 len)     // positional
u64    file_write(u64 h, String data)             // sequential; returns bytes written
u64    file_pwrite(u64 h, u64 offset, String data)
s64    file_seek(u64 h, s64 offset, int whence)   // + file_tell(h)
void   file_close(u64 h)                          // releases lock + fd
```
Keep the existing whole-file `file_get_contents`/`put_contents`/`append` (now
auto-locking) as the easy path.

### 2. Metadata  (P0/P1)
```
DValue file_stat(String path)   // { exists, size, mtime, ctime, mode, is_dir, is_file, is_symlink }
DValue dir_list(String path)    // [ { name, type, size, mtime } ... ]; optional glob/recursive flags
```
(`file_mtime` stays; `dir_list` is the richer `ls` so callers don't N+1 `file_stat`.)

### 3. Structural ops  (P1)
```
bool file_rename(String from, String to)   // atomic replace; enables write-temp-then-rename
bool file_copy(String from, String to)     // host-side, preserves mode
bool file_truncate(String path, u64 size)  // (or on a handle)
bool dir_remove(String path, bool recursive)
```

### 4. Niceties  (P2)
```
String file_temp(String prefix)            // race-free temp file under a write root (mkstemp)
bool   file_chmod(String path, u32 mode)
bool   file_symlink(String target, String linkpath); String file_readlink(String path)
// durability: file_put_contents/file_write gain an optional fsync flag/variant
```

### 5. Outbound HTTP (with TLS)  (high value)
`socket_*` is raw TCP only — no secure outbound. Add a real client:
```
DValue http_request(DValue req)   // req: { method, url, headers{}, body, timeout_ms, follow_redirects }
                                  // -> { status, headers{}, body, error }  (TLS/https supported)
u64    http_request_async(DValue req)  // -> job_id (see Async jobs)
```

### 6. Crypto (extended)  (high value)
`hash.h` has sha1 + rng only.
```
String sha256(String data); String sha256_hex(String data)
String hmac_sha256(String key, String data)         // + _hex
String base64_encode(String data); String base64_decode(String b64)
String random_bytes(u64 n)                            // CSPRNG
bool   crypto_equal(String a, String b)               // constant-time compare
```

### 7. Shell (extended + background)  (high value)
Current `shell_exec` is blocking + output-only.
```
DValue shell_exec(DValue spec)   // spec: { cmd, stdin, env{}, timeout_ms } -> { exit_code, stdout, stderr, timed_out }
u64    shell_spawn(DValue spec)  // background -> job_id; non-blocking; request returns immediately
```
(Keep a simple `shell_exec(String)` convenience overload returning stdout.)

### 8. Async jobs  (the unifying "check back later" primitive)
Host-side job registry, survives the spawning request, reachable from any later
request/worker by id.
```
DValue job_status(u64 job_id)            // { state: pending|running|done|failed, ... }
DValue job_result(u64 job_id)            // blocks up to a small default; { done, ... , result }
DValue job_await(u64 job_id, u64 timeout_ms)  // bounded wait; returns even if not done (state=running)
bool   job_cancel(u64 job_id)
```
`shell_spawn` and `http_request_async` produce job ids consumed here. Jobs have a
TTL and are reaped; results are retrievable until collected or TTL expires.

## Explicitly OUT of scope (declined)
- DNS resolution API, UDP sockets, `setenv` / environment mutation.

## Future (design influences now, NOT implemented here)
- **Configurable trans-membrane blocklist.** A server config will let sysadmins
  disable specific hostcalls by name (e.g. turn off `shell_exec`, `http_request`)
  to harden a deployment. This is WHY principle 4 (one capability per named
  hostcall) matters — keep the granularity clean so each is independently
  gateable. A blocked call should return a clear "disabled by policy" error.
- **Native app extensions.** Longer term: let app developers ship their own
  native compiled modules (`.so` living in the site tree) that units can call
  across the membrane — a per-app plugin mechanism. Out of scope now, but the
  hostcall/membrane shape should not preclude a future generic
  "call host extension" dispatch.

## Implementation notes
- Each call = a discrete `uce_host_*` hostcall: add to `src/wasm/core_hostcalls.syms`,
  register in `src/wasm/worker.cpp`, declare/implement the wasm-core wrapper in
  `src/lib/sys.cpp` (+ header) mirroring existing hostcalls (sized-buffer for
  bytes, sized-DValue/UCEB for structured returns).
- Reuse native impls where they exist; policy-gate every path.
- Each group lands suite-gated (`run_cli_tests --include-wasm-kill`, 0 failed) on
  the build host; add tests + doc pages per new function (function pages, and
  method/`2_` pages where a class gains methods).