From 1a5c6547b9eef25e61cf7c15ac4bbdb49ac6a361 Mon Sep 17 00:00:00 2001 From: root Date: Mon, 15 Jun 2026 12:00:46 +0000 Subject: [PATCH] post-W7 cleanup --- README.md | 75 +-- docs/cli-test-port-plan.md | 90 --- docs/setup.md | 565 ++++++++++++++++++ docs/wasm-phase1-dvalue-abi.md | 110 ---- docs/wasm-runtime-architecture.md | 38 +- etc/uce/settings.cfg | 4 + scripts/build_core_wasm.sh | 2 +- scripts/check_unit_wasm.py | 235 ++++++++ scripts/compile_wasm_unit | 2 +- site/demo/index.uce | 2 +- site/demo/markdown-example.md | 2 +- site/doc/pages/1_WS.txt | 4 +- site/doc/pages/cwd_set.txt | 2 +- site/doc/pages/ws_scope.txt | 2 +- site/errors/compiling.uce | 2 +- .../uce-starter/js/u-howler-demo.html | 2 +- .../uce-starter/views/auth/callback.uce | 2 +- .../uce-starter/views/workspace/index.uce | 2 +- site/info/index.uce | 32 +- site/info/js/u-howler-demo.html | 2 +- site/tests/sqlite.uce | 2 +- src/fastcgi/src/fcgicc.cc | 2 + src/lib/functionlib.cpp | 2 +- src/lib/sys.cpp | 2 +- src/wasm/backend.cpp | 2 +- src/wasm/core.cpp | 2 +- src/wasm/w3_driver.cpp | 4 +- src/wasm/worker.cpp | 6 +- 28 files changed, 896 insertions(+), 301 deletions(-) delete mode 100644 docs/cli-test-port-plan.md create mode 100644 docs/setup.md delete mode 100644 docs/wasm-phase1-dvalue-abi.md create mode 100755 scripts/check_unit_wasm.py diff --git a/README.md b/README.md index cb94042..4d14efc 100644 --- a/README.md +++ b/README.md @@ -13,8 +13,8 @@ UCE is a PHP-inspired server-side runtime that lets you build web pages and hand - WebSocket pages can additionally expose `WS(Request& context)` - local CLI/admin/test entrypoints can expose `CLI(Request& context)` and are invoked through the Unix CLI socket - sub-rendering and components pass structured data through `context.props` -- nginx can forward normal `.uce` requests and ordinary `.ws.uce` page loads to the FastCGI socket, while real WebSocket upgrade requests for `.ws.uce` endpoints go to the built-in HTTP/WebSocket listener -- the nginx-published application tree lives under `site/` +- nginx can forward normal `.uce` requests to the FastCGI socket, while WebSocket upgrade requests for `.uce` endpoints go to the built-in HTTP/WebSocket listener +- the example application tree lives under `site/`; deployments should publish app files to a normal web root such as `/var/www/html` - you can include C++ code as much as you want, but only .uce files called via API functions and entry points will be pre-processed - the preprocessor has two jobs: - allow for inline HTML within C++ and the use of templating tags inside of that HTML @@ -45,8 +45,10 @@ The current build expects: - `mysql_config` - PCRE2 development headers and library (`libpcre2-dev` on Debian / Ubuntu) - standard Linux development headers for `dl`, `pthread`, sockets, and backtrace support +- Wasmtime C API / C++ headers, defaulting to `/opt/wasmtime` or `WASMTIME_HOME` +- WASI SDK tools, defaulting to `/opt/wasi-sdk` or `WASI_SDK`, for `scripts/build_core_wasm.sh` and unit compilation -SQLite is vendored under `src/3rdparty/sqlite/` and compiled by `scripts/build_linux.sh`; no system SQLite package is required. +SQLite and miniz are vendored under `src/3rdparty/`; no system SQLite or zlib package is required for those helpers. The binary is written to: @@ -184,7 +186,7 @@ The runtime keeps the socket lifecycle in-process and exposes a low-boilerplate - `ws_send_to(connection_id, message[, binary])` - `ws_close([connection_id])` -By default, the WebSocket scope is the current page file, so `ws_send()` queues a message for clients connected to that same `.ws.uce` endpoint. +By default, the WebSocket scope is the current page file, so `ws_send()` queues a message for clients connected to that same `.uce` endpoint. Each live WebSocket connection owns a broker-side `DValue` exposed to page code as `context.connection`. Mutations to that tree persist for the life of the socket and are visible on later `WS(Request& context)` calls for the same client. @@ -248,8 +250,8 @@ Representative test pages: The intended production shape is: - nginx serves static files directly -- nginx forwards `.uce` requests and ordinary `.ws.uce` page loads to the UCE FastCGI Unix socket -- nginx proxies actual WebSocket upgrade requests for `.ws.uce` endpoints to the runtime's built-in HTTP/WebSocket listener +- nginx forwards ordinary `.uce` page loads to the UCE FastCGI Unix socket +- nginx proxies WebSocket upgrade requests for `.uce` endpoints to the runtime's built-in HTTP/WebSocket listener - systemd keeps the runtime built, started, and restarted on failure The repository ships the pieces used for this: @@ -264,7 +266,7 @@ On a Debian or Ubuntu host, start with the packages needed to build and run UCE ```bash apt update -apt install -y nginx clang mariadb-client libmariadb-dev libpcre2-dev build-essential +apt install -y nginx clang mariadb-client libmariadb-dev libpcre2-dev build-essential curl rsync ca-certificates ``` The exact package names may vary by distro. The important requirements are: @@ -274,16 +276,26 @@ The exact package names may vary by distro. The important requirements are: - `mysql_config` - PCRE2 development headers and library (`libpcre2-dev` on Debian / Ubuntu) - normal Linux development headers for threads, sockets, `dl`, and backtrace support +- Wasmtime C API / C++ headers installed at `/opt/wasmtime` or configured with `WASMTIME_HOME` +- WASI SDK installed at `/opt/wasi-sdk` or configured with `WASI_SDK` ### 2. Put the repo on the server This README assumes the repository lives at: ```bash -/Code/uce.openfu.com/uce +/opt/uce ``` -That is what the shipped `scripts/systemd/uce.service` file currently uses as its `WorkingDirectory` and build path. If you deploy somewhere else, update that unit file before enabling the service. +The examples below use that path for the runtime. Publish public application files under the normal web root, for example: + +```bash +cd /opt/uce +mkdir -p /var/www/html +rsync -a site/ /var/www/html/ +``` + +If you deploy somewhere else, update the systemd unit's `WorkingDirectory`, build path, and `ExecStart` path before enabling the service. ### 3. Configure `/etc/uce/settings.cfg` @@ -304,7 +316,7 @@ FCGI_SOCKET_PATH=/run/uce/fastcgi.sock FCGI_PORT=9993 PRECOMPILE_FILES_IN= -SITE_DIRECTORY=site +SITE_DIRECTORY=/var/www/html PROACTIVE_COMPILE_CHECK_INTERVAL=60 WORKER_COUNT=4 @@ -328,7 +340,7 @@ HTTP_PORT=8080 Proactive compilation settings: -- `SITE_DIRECTORY=site` tells the runtime which tree to scan on startup for `.uce` files when `PRECOMPILE_FILES_IN` is left empty. +- `SITE_DIRECTORY=/var/www/html` tells the runtime which public web tree to scan on startup for `.uce` files when `PRECOMPILE_FILES_IN` is left empty. - `PRECOMPILE_FILES_IN=` can override that startup scan root with a different absolute or runtime-relative directory. - `PROACTIVE_COMPILE_CHECK_INTERVAL=60` controls how often the low-priority background compiler rechecks known `.uce` files for stale or missing wasm modules. @@ -395,12 +407,14 @@ That script: - writes Debian maintainer scripts for systemd reload/enable handling - follows a more PHP-like/FHS deployment shape with immutable runtime files under `/usr/lib`, config under `/etc`, cache/state under `/var`, and the FastCGI socket under `/run/uce/` -### 5. Configure nginx for `.uce` and `.ws.uce` +### 5. Configure nginx for `.uce` and WebSocket upgrades -You need two nginx paths for `.ws.uce` endpoints: +Any `.uce` unit can expose `WS(Request& context)`. WebSocket upgrade requests for `.uce` paths should be routed to the runtime's HTTP/WebSocket listener. -- FastCGI for ordinary `.uce` requests and plain `.ws.uce` page renders -- HTTP proxying only for actual WebSocket upgrade traffic on `.ws.uce` endpoints +You need two transport paths for `.uce` endpoints: + +- FastCGI for ordinary `.uce` page renders +- HTTP proxying only for WebSocket upgrade traffic on `.uce` endpoints If you use WebSockets, add this `map` in the nginx `http` block: @@ -417,7 +431,7 @@ Then use a server block along these lines: server { listen 80; server_name example.com; - root /Code/uce.openfu.com/uce/site; + root /var/www/html; index index.uce index.html; @@ -426,16 +440,6 @@ server { } location ~ \.uce$ { - include fastcgi_params; - fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; - fastcgi_param DOCUMENT_ROOT $document_root; - fastcgi_param SCRIPT_NAME $fastcgi_script_name; - fastcgi_param DOCUMENT_URI $uri; - fastcgi_param REQUEST_URI $request_uri; - fastcgi_pass unix:/run/uce/fastcgi.sock; - } - - location ~ \.ws\.uce$ { error_page 418 = @uce_websocket; if ($http_upgrade = "websocket") { return 418; @@ -453,7 +457,6 @@ server { location @uce_websocket { proxy_http_version 1.1; proxy_set_header Host $host; - proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header Upgrade $http_upgrade; @@ -465,19 +468,19 @@ server { Important details: -- `.ws.uce` must be matched before the more general `.uce` rule - `fastcgi_pass` should point at the same socket path as `FCGI_SOCKET_PATH` - `proxy_pass` should point at the runtime's `HTTP_PORT` -- ordinary `GET /page.ws.uce` page renders should stay on FastCGI -- only upgrade requests for `/page.ws.uce` should go through the HTTP/WebSocket listener -- `SCRIPT_FILENAME` should resolve to the actual `.uce` file on disk +- ordinary `GET /page.uce` page renders should stay on FastCGI +- only upgrade requests for `/page.uce` should go through the HTTP/WebSocket listener +- `SCRIPT_FILENAME` should resolve to the requested `.uce` file on disk - `proxy_http_version 1.1` and the `Upgrade` / `Connection` headers are required for WebSockets +- socket-capable pages are ordinary `.uce` units; route client WebSocket upgrade requests to the HTTP/WebSocket listener -The `location /` block only serves files from `site/`. If your app uses a front-controller pattern such as routing everything through `/index.uce`, change that block accordingly. +The `location /` block only serves files from `/var/www/html`. If your app uses a front-controller pattern such as routing everything through `/index.uce`, change that block accordingly. ### 6. Think about document root and private files -Point nginx at `site/`, not the repository root. The repo still contains source, scripts, packaging files, and operational assets that are not meant to be public. +Point nginx at `/var/www/html`, not the runtime repository root. The repo still contains source, scripts, packaging files, and operational assets that are not meant to be public. At minimum, explicitly block internal directories that should never be served directly. For example: @@ -487,7 +490,7 @@ location ~ ^/(src|scripts|etc|bin|work|dist|pkg)/ { } ``` -If nginx is rooted at `site/`, most of those paths will not be reachable anyway, which is the preferred setup. +If nginx is rooted at `/var/www/html`, most of those paths will not be reachable anyway, which is the preferred setup. ### 7. Reload nginx and verify the deployment @@ -506,7 +509,7 @@ curl -i http://127.0.0.1/test/index.uce curl -i http://127.0.0.1/doc/index.uce ``` -If WebSockets are enabled, also verify a `.ws.uce` endpoint through nginx rather than talking to the runtime directly. +If WebSockets are enabled, also verify a `.uce` endpoint that defines `WS(Request& context)` through nginx rather than talking to the runtime directly. ### 8. Troubleshooting @@ -515,7 +518,7 @@ Common failure modes: - `502 Bad Gateway` Usually means `uce.service` is down, the Unix socket path does not match, or the request crashed before sending a valid response. - WebSocket upgrade fails - Check that nginx is routing `.ws.uce` to `proxy_pass`, not `fastcgi_pass`, and that `HTTP_PORT` is reachable on localhost. + Check that nginx is routing WebSocket upgrade requests to `proxy_pass`, not `fastcgi_pass`, and that `HTTP_PORT` is reachable on localhost. - Requests compile but immediately crash Check `journalctl -u uce.service`. Generated units carry an ABI metadata sidecar and should be recompiled automatically after runtime ABI changes, but clearing stale artifacts under `BIN_DIRECTORY` is still a useful last-resort recovery step if the cache has been damaged manually. - nginx serves raw source or internal files diff --git a/docs/cli-test-port-plan.md b/docs/cli-test-port-plan.md deleted file mode 100644 index 0128d05..0000000 --- a/docs/cli-test-port-plan.md +++ /dev/null @@ -1,90 +0,0 @@ -# CLI Unit Test Port Plan - -## Objective - -Replace the Python-based network test runner and plugins with UCE unit tests invoked through the runtime's CLI socket path, keeping equivalent coverage for HTTP smoke, site suites, security checks, starter parity, TCP/WebSocket listener probes, and wasm kill checks. The final invocation should be a bash script that calls UCE CLI units; Python test files should be removed only after UCE coverage is in place and validated. - -## Success Criteria - -- [x] A bash command runs the full test suite through the CLI socket and exits nonzero on failure. -- [x] UCE CLI tests cover every current Python plugin behavior or explicitly document a deliberate replacement. -- [x] Existing W5/WASM gates use the UCE CLI test runner instead of `tests/run_network_tests.py`. -- [x] Python test runner/plugins are deleted after parity validation. -- [x] Full suite passes on `uce-dev` with wasm-only unit execution. - -## Current State - -- Status: verifying -- Last updated: 2026-06-13 -- Source of truth: `/root/mount_ssh/uce-dev-root-htdocs-uce` -- Runtime/live target: `uce-dev:/Code/uce.openfu.com/uce` - -## Goal Tree - -Legend: `[ ]` not started, `[~]` in progress, `[x]` done, `[!]` blocked, `[-]` superseded - -- [x] G1: Inventory Python test coverage and CLI constraints - - Why: replacement must preserve coverage before deleting Python. - - Done when: every plugin has a mapped UCE equivalent or blocker. - - Verify: coverage matrix in this document. - - [x] G1.1: Delegate design/coverage review to Spark. - - [x] G1.2: Inspect delegates and reconcile plan. -- [x] G2: Build UCE CLI test harness - - Why: Python runner features need a UCE-native replacement. - - Done when: one CLI unit can list/run tests, print pass/fail summary, and return failing CLI status. - - Verify: `scripts/run_cli_tests.sh --list` and `scripts/run_cli_tests.sh --include-wasm-kill`. - - [x] G2.1: Add reusable UCE assertion/reporting helpers. - - [x] G2.2: Add HTTP/TCP helper functions using UCE socket APIs. - - [x] G2.3: Add bash wrapper under `scripts/`. -- [x] G3: Port current plugin cases to UCE - - Why: only delete Python after equivalent UCE coverage exists. - - Done when: UCE suite covers demo, HTTP docs/starter, site suites, security, starter parity, TCP, wasm kill. - - Verify: UCE CLI full run passes and output names match coverage matrix. -- [x] G4: Replace Python gate usage and delete Python tests - - Why: user explicitly requested eliminating the Python suite. - - Done when: scripts no longer call `tests/run_network_tests.py`, Python test files removed, validation green. - - Verify: `rg 'run_network_tests|tests/plugins|python3 tests'` has no obsolete gate references except historical docs and benchmark/audit utilities. -- [~] G5: Document and validate - - Why: future agents/operators need the new test workflow. - - Done when: docs/project notes and in-repo docs mention the CLI test command and validation artifact. - - Verify: docs committed, full suite run artifact recorded. - -## Coverage Matrix - -- `uce_demo_smoke.py` → `cli_run_demo_smoke()` in `site/tests/cli_runner.uce` (43 demo pages). -- `uce_http_smoke.py` → `cli_run_http_smoke()` (docs and starter route/body checks, 14 cases). -- `uce_site_suite.py` → `cli_run_site_suite()` (manifest-driven published site suite pages, 13 cases). -- `uce_security_smoke.py` → `cli_run_security_smoke()` (direct HTTP traversal/header spoofing, CRLF header sanitization, session hardening, 4 cases). -- `uce_starter_parity.py` → `cli_run_starter_parity()` (starter view title/404 checks, 7 cases). -- `uce_tcp_smoke.py` → `cli_run_tcp_smoke()` (port 80 and 8080 reachability, 2 cases). -- `uce_wasm_kill.py` → `cli_run_wasm_kill()` gated by `--include-wasm-kill` (trap/loop/recurse + post-kill health checks, 3 cases). - -## Execution Queue - -1. Run final no-Python-suite validation after removing stale references. -2. Commit UCE and project-doc updates. - -## Decisions - -- 2026-06-13: Use UCE CLI socket invocation as the test entrypoint; bash wrappers are acceptable, Python runner/plugins are not. -- 2026-06-13: Do not delete Python tests until UCE replacement validates equivalent coverage on `uce-dev`. - -## Assumptions - -- UCE socket APIs are sufficient for HTTP/1.0 probes, TCP connect checks, and security header injection checks. -- Bash can provide filtering/list convenience if exact Python CLI parity is not needed. - -## Blockers and Risks - -- [ ] The new runner intentionally does not preserve the old Python runner's dynamic plugin/tag/regex filtering; add UCE-side selectors later if operators miss them. -- [ ] `tests/wasm_benchmark.py` and `tests/wasm_site_audit.py` remain Python utility scripts, not the network test suite; port separately if a strict no-Python tools policy is desired. - -## Evidence and Verification Log - -- 2026-06-13: Prior to this plan, `b1856c1 chore: narrow wasm backend entrypoint API` was committed after build and focused services validation. -- 2026-06-13: `scripts/run_cli_tests.sh --include-wasm-kill` on `uce-dev` passed `86 passed, 0 failed, 0 skipped` (`/tmp/uce/cli-tests-final.txt`). - -## Change Log - -- 2026-06-13: Created initial goal tree for CLI unit test port. -- 2026-06-13: Added `site/tests/cli_runner.uce`, `scripts/run_cli_tests.sh`, removed Python network runner/plugins, and repointed W5 network gates to the CLI runner. diff --git a/docs/setup.md b/docs/setup.md new file mode 100644 index 0000000..e16294f --- /dev/null +++ b/docs/setup.md @@ -0,0 +1,565 @@ +# UCE Runtime Setup + +This guide describes how to run UCE behind nginx or Apache. UCE is a FastCGI application server for `.uce` units; the web server should serve static files directly and forward dynamic `.uce` requests to the UCE runtime. + +## Deployment shape + +A typical deployment has four pieces: + +1. A checked-out or packaged UCE runtime tree. +2. `/etc/uce/settings.cfg`, read by the UCE runtime at startup. +3. `uce.service`, a systemd service that builds/starts/restarts the runtime. +4. nginx or Apache as the public HTTP server. + +Recommended filesystem layout for a source checkout: + +```text +/opt/uce/ UCE repository/runtime root +/var/www/html/ public web root served by nginx/Apache +/etc/uce/settings.cfg runtime configuration +/run/uce/fastcgi.sock FastCGI socket used by nginx/Apache +/run/uce/cli.sock local CLI/admin/test socket +/var/cache/uce/work generated source, wasm modules, caches +/var/lib/uce/uploads multipart upload scratch space +/var/lib/uce/sessions session files +``` + +For packaged installs, the runtime may live under `/usr/lib/uce` instead of `/opt/uce`. Keep the public web root at `/var/www/html` or another normal web-root path, not under the runtime source tree. + +## Build requirements + +On Debian/Ubuntu-like systems, install the distro packages first: + +```bash +apt update +apt install -y clang build-essential libpcre2-dev mariadb-client libmariadb-dev curl rsync ca-certificates +``` + +UCE also requires two toolchains that are not vendored in this repository: + +- **Wasmtime C API / C++ headers** at `/opt/wasmtime` by default. `scripts/build_linux.sh` expects: + - `/opt/wasmtime/include/wasmtime.hh` + - `/opt/wasmtime/include/wasmtime/*.h` + - `/opt/wasmtime/lib/libwasmtime.so` +- **WASI SDK** at `/opt/wasi-sdk` by default. `scripts/build_core_wasm.sh` and `scripts/compile_wasm_unit` expect: + - `/opt/wasi-sdk/bin/clang++` + - `/opt/wasi-sdk/bin/wasm-ld` + - `/opt/wasi-sdk/bin/llvm-objcopy` + +You can use different install locations by setting environment variables before building and in the systemd service environment: + +```bash +export WASMTIME_HOME=/path/to/wasmtime +export WASI_SDK=/path/to/wasi-sdk +``` + +Install one web server: + +```bash +apt install -y nginx +# or +apt install -y apache2 +``` + +Build UCE from the repository root: + +```bash +cd /opt/uce +bash scripts/build_core_wasm.sh +bash scripts/build_linux.sh +``` + +Publish the starter site or your application files into the web root: + +```bash +mkdir -p /var/www/html +rsync -a site/ /var/www/html/ +``` + +The main binary is written to: + +```text +bin/uce_fastcgi.linux.bin +``` + +### Installing Wasmtime and WASI SDK + +The UCE build does not download these dependencies for you. Install them through your distro if it provides compatible development packages, or unpack pinned release archives under `/opt/wasmtime` and `/opt/wasi-sdk`. + +Do not use `curl | sh` installers in production setup scripts. Download archives from the upstream release pages, verify checksums/signatures when available, and keep the exact versions recorded in your deployment notes. + +The expected directory shape is: + +```text +/opt/wasmtime/include/wasmtime.hh +/opt/wasmtime/include/wasmtime/store.h +/opt/wasmtime/lib/libwasmtime.so + +/opt/wasi-sdk/bin/clang++ +/opt/wasi-sdk/bin/wasm-ld +/opt/wasi-sdk/bin/llvm-objcopy +``` + +If your paths differ, export the variables for manual builds: + +```bash +WASMTIME_HOME=/usr/local/wasmtime WASI_SDK=/usr/local/wasi-sdk bash scripts/build_core_wasm.sh +WASMTIME_HOME=/usr/local/wasmtime WASI_SDK=/usr/local/wasi-sdk bash scripts/build_linux.sh +``` + +For systemd, add an override: + +```bash +systemctl edit uce.service +``` + +```ini +[Service] +Environment=WASMTIME_HOME=/usr/local/wasmtime +Environment=WASI_SDK=/usr/local/wasi-sdk +``` + +Then reload and restart: + +```bash +systemctl daemon-reload +systemctl restart uce.service +``` + +## Runtime configuration + +Create `/etc/uce/settings.cfg` from `etc/uce/settings.cfg` and adjust paths if your runtime is not under `/opt/uce`. + +Minimum useful settings: + +```ini +BIN_DIRECTORY=/var/cache/uce/work +TMP_UPLOAD_PATH=/var/lib/uce/uploads +SESSION_PATH=/var/lib/uce/sessions + +FCGI_SOCKET_PATH=/run/uce/fastcgi.sock +CLI_SOCKET_PATH=/run/uce/cli.sock + +SITE_DIRECTORY=/var/www/html +JIT_COMPILE_ON_REQUEST=1 +PROACTIVE_COMPILE_ENABLED=1 +PROACTIVE_COMPILE_CHECK_INTERVAL=60 + +WASM_COMPILE_SCRIPT=scripts/compile_wasm_unit +WASM_BACKEND_VERBOSE=0 +WASM_CORE_PATH=/opt/uce/bin/wasm/core.wasm +WASM_MEMORY_LIMIT_BYTES=536870912 +WASM_EPOCH_DEADLINE_TICKS=200 +WASM_EPOCH_PERIOD_MS=50 + +WORKER_COUNT=4 +MAX_MEMORY=16777216 +SESSION_TIME=2592000 + +HTTP_PORT=8080 +``` + +Important settings: + +- `FCGI_SOCKET_PATH` is the Unix socket used for normal `.uce` requests. +- `CLI_SOCKET_PATH` is a local HTTP-over-Unix socket used by `scripts/uce-cli` and test/admin units. +- `SITE_DIRECTORY` is the public site tree to scan for `.uce` files. Use `/var/www/html` when the web root is outside the runtime tree; relative paths are resolved from the runtime working directory. +- `BIN_DIRECTORY` stores generated C++, wasm artifacts, compile output, and runtime caches. +- `TMP_UPLOAD_PATH` and `SESSION_PATH` must be writable by the runtime. +- `HTTP_PORT` is the built-in HTTP/WebSocket listener used for WebSocket upgrade traffic and direct local probes. +- `WASM_CORE_PATH` must point at the built `core.wasm` file. + +After editing settings, restart UCE: + +```bash +systemctl restart uce.service +``` + +## systemd service + +For source-checkout deployments, install the provided service helper: + +```bash +cd /opt/uce +scripts/systemd/manage-uce-service.sh setup +``` + +That helper installs `scripts/systemd/uce.service`, creates runtime directories, enables the service, and starts it. + +Useful commands: + +```bash +scripts/systemd/manage-uce-service.sh status +scripts/systemd/manage-uce-service.sh restart +scripts/systemd/manage-uce-service.sh logs 200 +``` + +Equivalent manual systemd service for a source checkout: + +```ini +[Unit] +Description=UCE FastCGI Runtime +After=network-online.target mariadb.service memcached.service +Wants=network-online.target + +[Service] +Type=simple +WorkingDirectory=/opt/uce +RuntimeDirectory=uce +StateDirectory=uce +CacheDirectory=uce +ExecStartPre=/usr/bin/mkdir -p /var/cache/uce/work /var/lib/uce/uploads /var/lib/uce/sessions +ExecStartPre=/usr/bin/rm -f /run/uce/fastcgi.sock +ExecStartPre=/usr/bin/bash /opt/uce/scripts/build_linux.sh +ExecStart=/opt/uce/bin/uce_fastcgi.linux.bin +ExecStopPost=/usr/bin/rm -f /run/uce/fastcgi.sock +Restart=always +RestartSec=2 +TimeoutStopSec=15 +KillMode=mixed +StandardOutput=journal +StandardError=journal + +[Install] +WantedBy=multi-user.target +``` + +Install it as `/etc/systemd/system/uce.service` and run: + +```bash +systemctl daemon-reload +systemctl enable --now uce.service +``` + +## How request routing works + +### Static files + +The web server should serve ordinary static files directly from the public web root, for example `/var/www/html`. + +Examples: + +```text +/style.css +/images/logo.png +/examples/uce-starter/js/site.js +``` + +These should not touch the UCE runtime. + +### Normal `.uce` page requests + +For a request such as: + +```text +GET /doc/index.uce?p=component +``` + +nginx/Apache forwards the request to `FCGI_SOCKET_PATH` as FastCGI. The web server must provide CGI/FastCGI variables including: + +- `SCRIPT_FILENAME` — full filesystem path to the `.uce` file. +- `DOCUMENT_ROOT` — public web root, normally `/site`. +- `SCRIPT_NAME` — URL path to the script, such as `/doc/index.uce`. +- `DOCUMENT_URI` — normalized URI path without query string. +- `REQUEST_URI` — original request URI including query string. +- standard request variables such as method, query string, content type, body length, cookies, and headers. + +UCE resolves the unit, compiles it to wasm if needed, creates a request workspace, and calls: + +```cpp +RENDER(Request& context) +``` + +The unit writes output with template literals or `print()`. Response headers and status are set through `context.header` and `context.set_status()`. + +### Component and sub-render calls + +Inside a request, UCE code can call other units: + +```cpp +component("components/card", props, context); +unit_render("other-page.uce", context); +``` + +These calls stay inside the UCE runtime. They are not new HTTP requests and do not go back through nginx or Apache. + +### WebSocket pages + +Any `.uce` unit can provide both an ordinary page render and WebSocket message handling: + +```cpp +RENDER(Request& context) { ... } // normal page load +WS(Request& context) { ... } // later WebSocket messages +``` + +The nginx and Apache examples below split traffic by checking for a WebSocket upgrade request on `.uce` paths. A file such as `chat.uce` or `events.uce` can expose `WS(Request& context)`. + +Routing split: + +- Plain `GET /demo/chat.uce` should use FastCGI, just like any other page render. +- WebSocket upgrade requests for `/demo/chat.uce` should proxy to the UCE built-in HTTP/WebSocket listener at `HTTP_PORT`. + +The built-in listener owns the socket lifecycle. When a message arrives, the broker forwards a render-style invocation back to the worker pool so `WS(Request& context)` runs inside the same wasm runtime model as normal pages. + +### CLI requests + +`CLI(Request& context)` handlers are not public web endpoints. They are invoked over `CLI_SOCKET_PATH`: + +```bash +scripts/uce-cli /tests/cli.uce action=echo message=hello +curl --unix-socket /run/uce/cli.sock http://localhost/tests/cli.uce +``` + +Use CLI units for local tests, admin commands, and maintenance tools. Do not expose the CLI socket through nginx or Apache. + +### Custom runtime HTTP servers + +UCE code can start local custom HTTP listeners with `server_start_http()`. Those are runtime-managed listeners for app-specific local services. They are separate from the public nginx/Apache entry point and should be firewalled or bound locally unless you explicitly want them reachable. + +## nginx configuration + +### Required modules + +A normal nginx build includes the needed FastCGI and proxy modules. Confirm nginx is installed and can load your config: + +```bash +nginx -t +``` + +### WebSocket upgrade map + +Put this in the nginx `http` block if using WebSockets: + +```nginx +map $http_upgrade $connection_upgrade { + default upgrade; + '' close; +} +``` + +### Server block + +Example site config: + +```nginx +server { + listen 80; + server_name example.com; + + root /var/www/html; + index index.uce index.html; + + # Serve static files directly. Directory requests use index.uce when present. + location / { + try_files $uri $uri/ =404; + } + + # UCE page requests use FastCGI. If the client asks to upgrade a .uce + # request to WebSocket, send that connection to the built-in listener. + location ~ \.uce$ { + error_page 418 = @uce_websocket; + if ($http_upgrade = "websocket") { + return 418; + } + + include fastcgi_params; + fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; + fastcgi_param DOCUMENT_ROOT $document_root; + fastcgi_param SCRIPT_NAME $fastcgi_script_name; + fastcgi_param DOCUMENT_URI $uri; + fastcgi_param REQUEST_URI $request_uri; + fastcgi_pass unix:/run/uce/fastcgi.sock; + } + + location @uce_websocket { + proxy_http_version 1.1; + proxy_set_header Host $host; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection $connection_upgrade; + proxy_pass http://127.0.0.1:8080; + } + + # Defense in depth if the root is changed later. + location ~ ^/(src|scripts|etc|bin|work|dist|pkg|docs|changelog)/ { + return 404; + } +} +``` + +Notes: + +- `fastcgi_pass` must match `FCGI_SOCKET_PATH`. +- `proxy_pass` must match `HTTP_PORT`. +- The example routes WebSocket upgrades for `.uce` paths to the HTTP/WebSocket listener. +- Ordinary `.uce` page loads continue to use FastCGI. +- Keep `root` pointed at `/var/www/html`, not the runtime repository root. +- If your app uses a front controller, replace `location /` with a `try_files` rule that ends at `/index.uce`. + +Front-controller variant: + +```nginx +location / { + try_files $uri $uri/ /index.uce?$query_string; +} +``` + +Reload nginx: + +```bash +nginx -t +systemctl reload nginx +``` + +## Apache configuration + +Apache can run UCE through `mod_proxy_fcgi` for FastCGI and `mod_proxy_wstunnel` or `mod_proxy_http` for WebSocket upgrades. + +### Enable modules + +On Debian/Ubuntu: + +```bash +a2enmod proxy proxy_fcgi proxy_http proxy_wstunnel rewrite headers setenvif +systemctl restart apache2 +``` + +### VirtualHost example + +```apache + + ServerName example.com + DocumentRoot /var/www/html + + + Require all granted + Options FollowSymLinks + AllowOverride None + DirectoryIndex index.uce index.html + + + # Do not expose repository internals if DocumentRoot changes later. + + Require all denied + + + RewriteEngine On + + # WebSocket upgrade traffic for any .uce unit goes to UCE's built-in HTTP listener. + RewriteCond %{HTTP:Upgrade} =websocket [NC] + RewriteCond %{REQUEST_URI} \.uce(?:\?|$) [NC] + RewriteRule ^/(.*)$ ws://127.0.0.1:8080/$1 [P,L] + + # Normal .uce page loads go to FastCGI. + + SetHandler "proxy:unix:/run/uce/fastcgi.sock|fcgi://localhost/" + + + # Optional: make the key CGI variables explicit for UCE. + ProxyFCGISetEnvIf "true" DOCUMENT_ROOT "/var/www/html" + ProxyFCGISetEnvIf "true" DOCUMENT_URI "%{REQUEST_URI}" + +``` + +Apache notes: + +- `SetHandler "proxy:unix:/run/uce/fastcgi.sock|fcgi://localhost/"` must use the same socket path as `FCGI_SOCKET_PATH`. +- The WebSocket rewrite rule must run before the FastCGI handler. +- Plain `.uce` page loads should not be proxied as WebSockets unless the client sends `Upgrade: websocket`. +- Apache's FastCGI environment differs by version and module configuration. If UCE cannot resolve a page, inspect the request environment and make sure `SCRIPT_FILENAME` points to the target file under the web root. + +If your Apache version does not populate `SCRIPT_FILENAME` correctly through `SetHandler`, use `ProxyPassMatch` for `.uce` files instead: + +```apache +ProxyPassMatch ^/(.*\.uce)$ unix:/run/uce/fastcgi.sock|fcgi://localhost/var/www/html/$1 +``` + +Use only one FastCGI mapping style at a time (`SetHandler` or `ProxyPassMatch`) to avoid duplicate routing. + +## Permissions + +The web server needs permission to connect to `/run/uce/fastcgi.sock`. Common approaches: + +- run UCE and the web server under compatible groups; +- add the web server user (`www-data` on Debian/Ubuntu) to the socket's group; +- adjust the service or runtime socket mode if needed. + +The runtime creates the FastCGI socket and CLI socket under `/run/uce`. The CLI socket should remain local-only and should not be reachable from the public web server. + +Writable paths for the runtime: + +```text +/var/cache/uce/work +/var/lib/uce/uploads +/var/lib/uce/sessions +/run/uce +``` + +## Verification + +Check service state: + +```bash +systemctl status uce.service +journalctl -u uce.service -n 100 --no-pager +``` + +Check the FastCGI/web-server path: + +```bash +curl -i http://127.0.0.1/doc/index.uce -H 'Host: example.com' +curl -i http://127.0.0.1/examples/uce-starter/ -H 'Host: example.com' +``` + +Check the local CLI path: + +```bash +cd /opt/uce +scripts/uce-cli /tests/cli.uce action=echo message=hello +scripts/run_cli_tests.sh +``` + +Check WebSocket routing with a WebSocket client against a `.uce` endpoint that defines `WS(Request& context)` through nginx/Apache, not directly against `HTTP_PORT`. + +## Troubleshooting + +### 502 Bad Gateway + +Check: + +- `systemctl status uce.service` +- `journalctl -u uce.service -n 200 --no-pager` +- socket path in web server config equals `FCGI_SOCKET_PATH` +- web server user can connect to the Unix socket +- `SCRIPT_FILENAME` resolves to an existing `.uce` file + +### Raw `.uce` source is downloaded or displayed + +The `.uce` request did not match the FastCGI rule. Check location/order rules and confirm the public root is `/var/www/html` or your chosen web-root path. + +### Static files 404 + +Confirm the web server `root`/`DocumentRoot` is `/var/www/html` or your chosen web-root path and that `location /` or Apache directory rules allow static file reads. + +### WebSocket page renders but upgrade fails + +Check: + +- the client sends `Upgrade: websocket` +- `.uce` upgrade traffic reaches `HTTP_PORT` +- nginx/Apache preserves `Upgrade` and `Connection` headers +- firewall/network policy allows localhost access to `HTTP_PORT` + +### Page compiles fail + +Check the compile artifact paths shown in the UCE error response and service logs. Generated files and compile output live under `BIN_DIRECTORY`. A safe last-resort recovery step is to stop UCE, move the affected generated artifact directory aside, and restart so the runtime recompiles from source. + +### CLI commands fail + +Check: + +- `CLI_SOCKET_PATH` in `/etc/uce/settings.cfg` +- `/run/uce/cli.sock` exists +- `scripts/uce-cli --socket /run/uce/cli.sock /ping` works +- the target unit defines `CLI(Request& context)` diff --git a/docs/wasm-phase1-dvalue-abi.md b/docs/wasm-phase1-dvalue-abi.md deleted file mode 100644 index 13236da..0000000 --- a/docs/wasm-phase1-dvalue-abi.md +++ /dev/null @@ -1,110 +0,0 @@ -# WASM Phase 1: DValue C ABI and UCEB1 - -Phase 1 freezes the native DValue ABI that the future WASM core and units use -as their shared structured-value contract. The implementation is in -`src/lib/dvalue.{h,cpp}` and is available in the native runtime before any WASM -backend is enabled. - -## Opaque handle - -```c -typedef struct DValue uce_dvalue; -``` - -`uce_dvalue*` is a borrowed pointer owned by the active request/workspace. It -must not be freed by ABI callers and it must not be retained beyond that -workspace lifetime. - -## Accessors - -```c -uce_dvalue* uce_dv_root(void); -uce_dvalue* uce_dv_get(uce_dvalue* value, const char* key, size_t key_len); -uce_dvalue* uce_dv_find(uce_dvalue* value, const char* key, size_t key_len); -const char* uce_dv_value(uce_dvalue* value, size_t* len_out); -void uce_dv_set_value(uce_dvalue* value, const char* bytes, size_t len); -size_t uce_dv_count(uce_dvalue* value); -int uce_dv_is_list(uce_dvalue* value); -``` - -- `uce_dv_root()` returns the current native request's `context.call` root. - The WASM core will later map this to the decoded request context root. -- `uce_dv_get()` creates the child if absent. `uce_dv_find()` returns `NULL` - if absent. -- String inputs and outputs are length-delimited and binary-safe. -- `uce_dv_value()` returns a borrowed pointer valid until the next ABI value - call on the same thread. -- Bad `NULL` inputs return `NULL`, zero, or no-op rather than trapping. - -## Iteration - -```c -typedef struct uce_dv_iter { size_t position; size_t reserved[3]; } uce_dv_iter; - -uce_dv_iter uce_dv_iter_begin(uce_dvalue* value); -int uce_dv_iter_next(uce_dvalue* value, uce_dv_iter* iter, - const char** key_out, size_t* key_len_out, - uce_dvalue** child_out); -``` - -Map iteration follows DValue's native order. List-shaped maps iterate in numeric -index order (`0`, `1`, ...), matching `DValue::each()`, `dv_values()`, and the -serializers. The reserved iterator fields are caller-opaque and must be -zero-preserved by callers that copy the iterator; they provide ABI headroom for -future non-linear keyed-map iteration without changing the struct size. - -## UCEB1 wire format - -UCEB1 is the membrane/cross-instance binary DValue encoding. - -``` -document := "UCEB" version node -version := 0x01 -node := flags scalar children -flags := u8 bitset; bit0 = list-shaped map -scalar := varuint length, bytes -children := varuint count, count * (key, node) -key := varuint length, bytes -``` - -Varuint is unsigned LEB128. Strings are byte sequences; the codec does not -assume NUL termination and preserves embedded NUL bytes. The Phase 1 layout stores scalar values as their native string representation -plus child nodes and the list-shape flag. Floating-point values use -`max_digits10` precision so numeric scalars can round-trip through the string -form. Pointer/reference identity is intentionally not part of the wire contract; -pointer nodes encode as an empty scalar rather than leaking process addresses. -An empty non-list map has no wire distinction from an empty scalar in UCEB1 v1. -Documents that contain both scalar bytes and child nodes are reserved for future -use; the v1 decoder accepts the children and ignores the scalar. - -## Codec APIs - -C++/UCE-visible helpers: - -```cpp -String ucb_encode(const DValue& value); -DValue ucb_decode(const String& encoded); -bool ucb_decode(const String& encoded, DValue& out, String* error_out = 0); -``` - -C ABI helpers: - -```c -size_t uce_dv_encode(uce_dvalue* value, char* buf, size_t cap); -uce_dvalue* uce_dv_decode(const char* buf, size_t len); -const char* uce_dv_last_error(void); -``` - -`uce_dv_encode()` returns the required byte length even when `buf` is `NULL` or -`cap` is zero. `uce_dv_decode()` returns a thread-local decoded root, or `NULL` -with `uce_dv_last_error()` populated. The returned decoded root is valid until -the next `uce_dv_decode()` call on the same thread. Decoding rejects documents -deeper than 1024 nested nodes so malformed input cannot recurse without bound. - -## Test coverage - -`site/tests/core.uce` covers: - -- UCEB1 round-trip for maps, nested values, lists, empty lists, and embedded NUL - scalar bytes. -- C ABI get/find/value/count/list/iteration/encode/decode behavior. diff --git a/docs/wasm-runtime-architecture.md b/docs/wasm-runtime-architecture.md index 2ddce43..66a8611 100644 --- a/docs/wasm-runtime-architecture.md +++ b/docs/wasm-runtime-architecture.md @@ -5,9 +5,7 @@ describes the **runtime architecture as built** — the process topology, the wasm membrane, the unified request dispatch, and the central WebSocket broker. Native `.so` unit execution/dlopen fallback has been removed; the parser and preprocessor remain only as the front-end that emits C++ for wasm side-module -compilation. For the historical motivation and the phased migration plan, see -[`WASM-PROPOSAL.md`](../WASM-PROPOSAL.md); this file is the steady-state -reference that proposal points at. +compilation. The guiding principle: **request code never shares an address space or an allocator with the runtime.** Every unit runs as a WebAssembly module inside a @@ -55,9 +53,9 @@ gets invoked*. | **serve_http dispatcher** (×bind) | one custom-server bind address | no — forwards to the pool | `custom_server_http_dispatcher_loop()` | | **Proactive compiler** | nothing; pre-compiles units | no | `run_proactive_compiler()` | -The key invariant: **only workers instantiate Wasmtime and run unit code.** +**only workers instantiate Wasmtime and run unit code.** Every connection-owning process (broker, serve_http dispatcher) forwards the -real invocation back to a worker over `/run/uce.sock` using the minimal +request invocation back to a worker over `/run/uce.sock` using the minimal FastCGI client in [`src/lib/fcgi_forward.h`](../src/lib/fcgi_forward.h). This is forced by Wasmtime: an `Engine`/`Store` cannot be safely re-created across `fork()`, and the brokers fork from the parent that already touched the @@ -163,7 +161,7 @@ otherwise (page) → serve_via_wasm(entry_unit, "render") The `UCE_*` params are set by whichever broker forwarded the request: -- **Page render**: nginx → `/run/uce.sock` directly; no `UCE_*` flags → `render`. +- **Page render**: FastCGI nginx → `/run/uce.sock` directly; no `UCE_*` flags → `render`. - **CLI**: the CLI socket sets `is_cli`. - **serve_http**: the custom-server dispatcher sets `UCE_SERVE_HTTP=1` plus `UCE_SERVE_HTTP_FUNCTION` and rewrites `SCRIPT_FILENAME` to the configured @@ -181,21 +179,12 @@ code. ## 6. The central WebSocket broker -The broker is the architectural centerpiece. **One process owns the HTTP port +**One process owns the HTTP port and every WebSocket connection**, so any unit's `ws_*` call can reach any or all connections, and a unit-code crash (which happens in a worker) never drops live connections. -### 6.1 Why central, and why it forwards - -Earlier designs gave each worker its connections and captured `ws_*` output to -return inline. That was wrong: connections couldn't outlive a worker, and one -unit could only talk to connections it happened to own. The broker fixes both — -it is the sole connection registry and data broker between connected clients and -the units that handle them. It renders nothing itself; like every other -connection owner it forwards unit invocation to the worker pool. - -### 6.2 Inbound: a WS frame → a worker render (non-blocking) +### 6.1 Inbound: a WS frame → a worker render (non-blocking) `ws_broker_ws_message(request, message, opcode)` fires when a complete (reassembled) WS message arrives on a connection. It does **not** block the @@ -216,7 +205,7 @@ writing each queued request, then drains and discards the reply (the unit's output comes back via the command socket, not this reply), closing the fd when the worker closes its end. -### 6.3 Outbound: `ws_*` commands flushed back to the broker +### 6.2 Outbound: `ws_*` commands flushed back to the broker Any unit code — not just WebSocket handlers — may call `ws_send` / `ws_send_to` / `ws_close`. In the workspace these **record dispatch commands** rather than @@ -236,14 +225,14 @@ the full registry it owns: `broadcast` (by scope), `send_to` (by connection id), `close`. If the batch carries `connection_state`, it persists that onto the matching live connection's `websocket_state`. -### 6.4 Un-upgraded HTTP on the WS port +### 6.3 Un-upgraded HTTP on the WS port The WS port can also receive ordinary (non-Upgrade) HTTP requests. `ws_broker_complete()` routes by param: `UCE_WS_DISPATCH=1` → apply commands; otherwise → `forward_request_to_worker()` — the *same* shared facility the serve_http dispatcher uses, so there is no duplicated request-forwarding code. -### 6.5 The broker loop +### 6.4 The broker loop `run_ws_broker()` drops the worker listeners it inherited (`close_inherited_server_sockets`), installs permissive `on_request`/`on_data` @@ -255,9 +244,7 @@ command socket, and loops `process(50)` + `drain_outbound()`. The design is broker's single epoll loop** — the broker never blocks on a worker. The parent respawns the broker if it dies (`ws_broker_alive` / `ensure_ws_broker` -in `main()`). A broker restart loses live connections (acceptable: crashes -happen in workers, so the broker stays up in practice), but no unit state is at -risk because the broker holds none. +in `main()`). --- @@ -309,11 +296,10 @@ header free-functions are `inline`. The wasm backend exposes only declarations - **Regression gate**: `scripts/run_cli_tests.sh --include-wasm-kill` runs the in-runtime CLI suite (`site/tests/cli_runner.uce`) plus the site test pages. - Current baseline: **87 passed, 0 failed**. + - **WebSocket end-to-end**: a headless client performs a raw WS handshake to `:HTTP_PORT` with path `/site/tests/websockets.ws.uce` (self-resolving `SCRIPT_FILENAME`) and asserts the `hello-ack` frame — exercising the full broker → worker → broker → client chain across process boundaries. -All builds, runs, and installs happen on the dev host (`k-uce` / uce-dev) over -SSH; the local checkout is edit-only. + diff --git a/etc/uce/settings.cfg b/etc/uce/settings.cfg index 84205db..49085a0 100644 --- a/etc/uce/settings.cfg +++ b/etc/uce/settings.cfg @@ -11,6 +11,10 @@ FCGI_PORT=9993 # Example: curl --unix-socket /run/uce/cli.sock http://localhost/ping CLI_SOCKET_PATH=/run/uce/cli.sock +# Built-in HTTP/WebSocket listener used for WebSocket Upgrade requests. +# Keep this behind nginx/Apache on localhost or firewall it from public access. +HTTP_PORT=8080 + # OPTIONAL PROACTIVE COMPILE ROOT # Leave empty to scan SITE_DIRECTORY relative to the runtime root. PRECOMPILE_FILES_IN= diff --git a/scripts/build_core_wasm.sh b/scripts/build_core_wasm.sh index 85818d0..18c2d48 100755 --- a/scripts/build_core_wasm.sh +++ b/scripts/build_core_wasm.sh @@ -1,5 +1,5 @@ #!/bin/bash -# Build the production W1 UCE WASM core from the real runtime carve-out. +# Build the production W1 UCE WASM core from the production runtime carve-out. # Run on k-uce from any working directory. set -euo pipefail cd "$(dirname "$0")/.." diff --git a/scripts/check_unit_wasm.py b/scripts/check_unit_wasm.py new file mode 100755 index 0000000..05491fe --- /dev/null +++ b/scripts/check_unit_wasm.py @@ -0,0 +1,235 @@ +#!/usr/bin/env python3 +"""Validate a W2 UCE PIC unit wasm artifact. + +Checks intentionally stay small and explicit: section walk for dylink.0, +uce.abi, imports/exports, plus llvm-nm for allocator definitions. +""" +from __future__ import annotations + +import argparse +import shutil +import subprocess +import sys +from pathlib import Path + + +def read_u32leb(data: bytes, pos: int) -> tuple[int, int]: + result = 0 + shift = 0 + while True: + if pos >= len(data): + raise ValueError("truncated leb128") + b = data[pos] + pos += 1 + result |= (b & 0x7F) << shift + if (b & 0x80) == 0: + return result, pos + shift += 7 + if shift > 35: + raise ValueError("oversized leb128") + + +def read_name(data: bytes, pos: int) -> tuple[str, int]: + n, pos = read_u32leb(data, pos) + end = pos + n + if end > len(data): + raise ValueError("truncated name") + return data[pos:end].decode("utf-8", "replace"), end + + +def walk_sections(data: bytes): + if not data.startswith(b"\0asm\x01\0\0\0"): + raise ValueError("not a wasm v1 module") + pos = 8 + while pos < len(data): + section_id = data[pos] + pos += 1 + size, pos = read_u32leb(data, pos) + end = pos + size + if end > len(data): + raise ValueError("section extends past EOF") + payload = data[pos:end] + yield section_id, payload + pos = end + + +def parse_imports(payload: bytes): + pos = 0 + count, pos = read_u32leb(payload, pos) + imports = [] + for _ in range(count): + module, pos = read_name(payload, pos) + name, pos = read_name(payload, pos) + if pos >= len(payload): + raise ValueError("truncated import kind") + kind = payload[pos] + pos += 1 + # Skip type descriptors. We only need module/name/kind for W2 policy. + if kind == 0: # func type index + _, pos = read_u32leb(payload, pos) + elif kind == 1: # table + if pos >= len(payload): raise ValueError("truncated table import") + pos += 1 + flags, pos = read_u32leb(payload, pos) + _, pos = read_u32leb(payload, pos) + if flags & 1: _, pos = read_u32leb(payload, pos) + elif kind == 2: # memory + flags, pos = read_u32leb(payload, pos) + _, pos = read_u32leb(payload, pos) + if flags & 1: _, pos = read_u32leb(payload, pos) + elif kind == 3: # global + pos += 2 + else: + raise ValueError(f"unknown import kind {kind}") + imports.append((module, name, kind)) + return imports + + +def parse_exports(payload: bytes): + pos = 0 + count, pos = read_u32leb(payload, pos) + exports = [] + for _ in range(count): + name, pos = read_name(payload, pos) + if pos >= len(payload): + raise ValueError("truncated export kind") + kind = payload[pos] + pos += 1 + _, pos = read_u32leb(payload, pos) + exports.append((name, kind)) + return exports + + +def collect(path: Path): + data = path.read_bytes() + customs: dict[str, list[bytes]] = {} + imports = [] + exports = [] + for section_id, payload in walk_sections(data): + if section_id == 0: + name, pos = read_name(payload, 0) + customs.setdefault(name, []).append(payload[pos:]) + elif section_id == 2: + imports = parse_imports(payload) + elif section_id == 7: + exports = parse_exports(payload) + return customs, imports, exports + + +def dylink_has_valid_mem_info(payload: bytes) -> bool: + pos = 0 + while pos < len(payload): + subsection_id = payload[pos] + pos += 1 + size, pos = read_u32leb(payload, pos) + end = pos + size + if end > len(payload): + raise ValueError("dylink.0 subsection extends past section") + if subsection_id == 1: + mem_size, p = read_u32leb(payload, pos) + mem_align, p = read_u32leb(payload, p) + table_size, p = read_u32leb(payload, p) + table_align, p = read_u32leb(payload, p) + if p > end: + raise ValueError("truncated dylink.0 mem_info") + return mem_align < 32 and table_align < 32 and mem_size < (1 << 31) and table_size < (1 << 31) + pos = end + return False + + +def defined_symbols(path: Path, llvm_nm: str) -> list[str]: + proc = subprocess.run([llvm_nm, "--defined-only", str(path)], text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + if proc.returncode != 0: + # llvm-nm (wasi-sdk) can SIGSEGV on degenerate-but-valid modules — e.g. a + # unit with no exported handlers (`empty.uce`). A toolchain crash is not + # evidence of a forbidden allocator, so skip this defense-in-depth scan + # rather than fail the unit; forbidden allocator *exports* are still + # rejected by the export-section check above. + crashed = proc.returncode < 0 or "Stack dump:" in proc.stderr or "PLEASE submit a bug report" in proc.stderr + if crashed: + return [] + raise RuntimeError(proc.stderr.strip() or "llvm-nm failed") + symbols = [] + for line in proc.stdout.splitlines(): + parts = line.split() + if parts: + symbols.append(parts[-1]) + return symbols + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("wasm", type=Path) + ap.add_argument("--abi-version", default="6") + ap.add_argument("--llvm-nm", default=None) + ap.add_argument("--verbose", action="store_true") + args = ap.parse_args() + + try: + customs, imports, exports = collect(args.wasm) + errors = [] + dylink_payloads = customs.get("dylink.0", []) + if not dylink_payloads: + errors.append("missing dylink.0 custom section") + elif not any(dylink_has_valid_mem_info(payload) for payload in dylink_payloads): + errors.append("dylink.0 missing valid mem_info subsection") + abi_payloads = customs.get("uce.abi", []) + if not abi_payloads: + errors.append("missing uce.abi custom section") + else: + abi_text = abi_payloads[-1].decode("utf-8", "replace") + required = ["format=uce-wasm-unit-abi-v1", f"unit_abi_version={args.abi_version}", "toolchain="] + for needle in required: + if needle not in abi_text: + errors.append(f"uce.abi missing {needle!r}") + export_names = {name for name, _ in exports} + forbidden_exports = {"uce_alloc", "uce_free"} + for name in sorted(export_names & forbidden_exports): + errors.append(f"forbidden allocator export {name}") + import_map = {(module, name): kind for module, name, kind in imports} + required_imports = { + ("env", "memory"): 2, + ("env", "__memory_base"): 3, + } + # units without indirect calls / stack spills / table needs + # legitimately omit these; if present, the kind must be right + optional_imports = { + ("env", "__indirect_function_table"): 1, + ("env", "__stack_pointer"): 3, + ("env", "__table_base"): 3, + } + for key, kind in required_imports.items(): + if import_map.get(key) != kind: + errors.append(f"missing required import {key[0]}.{key[1]}") + for key, kind in optional_imports.items(): + if key in import_map and import_map[key] != kind: + errors.append(f"wrong kind for import {key[0]}.{key[1]}") + for module, name, kind in imports: + if module.startswith("wasi_") or module == "wasi_snapshot_preview1": + errors.append(f"forbidden WASI import {module}.{name}") + if module not in {"env", "GOT.mem", "GOT.func"} and not module.startswith("GOT."): + errors.append(f"unexpected import module {module}.{name}") + if module.startswith("GOT.") and kind != 3: + errors.append(f"GOT import is not a global: {module}.{name}") + llvm_nm = args.llvm_nm or shutil.which("llvm-nm") or "/opt/wasi-sdk/bin/llvm-nm" + if Path(llvm_nm).exists(): + bad_prefixes = ("_Znwm", "_Znam", "_ZdlPv", "_ZdaPv", "_ZdlPvm", "_ZdaPvm") + for sym in defined_symbols(args.wasm, llvm_nm): + if sym in {"uce_alloc", "uce_free"} or sym.startswith(bad_prefixes): + errors.append(f"forbidden allocator definition {sym}") + else: + errors.append("llvm-nm not found; cannot verify allocator definitions") + if errors: + for e in errors: + print(f"ERROR: {e}", file=sys.stderr) + return 1 + if args.verbose: + print(f"UCE W2 unit check PASS: {args.wasm}") + return 0 + except Exception as exc: + print(f"ERROR: {exc}", file=sys.stderr) + return 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/compile_wasm_unit b/scripts/compile_wasm_unit index 6d6a783..b2f16dd 100755 --- a/scripts/compile_wasm_unit +++ b/scripts/compile_wasm_unit @@ -92,6 +92,6 @@ fi "$SDK/bin/llvm-objcopy" --add-section=uce.abi="$ABI_TMP" "$DEST_DIR/$WASM_FN" -python3 scripts/wasm/check_unit_wasm.py "$DEST_DIR/$WASM_FN" --abi-version "$ABI_VERSION" --llvm-nm "$SDK/bin/llvm-nm" +python3 scripts/check_unit_wasm.py "$DEST_DIR/$WASM_FN" --abi-version "$ABI_VERSION" --llvm-nm "$SDK/bin/llvm-nm" rm -f "$OBJ_FN" "$ABI_TMP" diff --git a/site/demo/index.uce b/site/demo/index.uce index 9b68240..26a56ff 100644 --- a/site/demo/index.uce +++ b/site/demo/index.uce @@ -70,7 +70,7 @@ RENDER(Request& context) - + diff --git a/site/demo/markdown-example.md b/site/demo/markdown-example.md index 174cc5a..def7a6e 100644 --- a/site/demo/markdown-example.md +++ b/site/demo/markdown-example.md @@ -1,6 +1,6 @@ # Markdown Demo -This page exercises **strong**, *emphasis*, ~~strikethrough~~, `code spans`, and a bare URL: https://uce.openfu.com/doc/index.uce +This page exercises **strong**, *emphasis*, ~~strikethrough~~, `code spans`, and a bare URL: https://example.com/doc/index.uce ## Task List diff --git a/site/doc/pages/1_WS.txt b/site/doc/pages/1_WS.txt index e66891f..9a994de 100644 --- a/site/doc/pages/1_WS.txt +++ b/site/doc/pages/1_WS.txt @@ -12,9 +12,9 @@ WS(Request& context) >1_RENDER :content -Defines the WebSocket message handler for the current `.ws.uce` page. +Defines the WebSocket message handler for the current `.uce` page. -The same page may expose both `RENDER(Request& context)` and `WS(Request& context)`. `RENDER(Request& context)` serves the initial HTTP response, while `WS(Request& context)` is called whenever a complete WebSocket message arrives for that page. +Any `.uce` unit may expose both `RENDER(Request& context)` and `WS(Request& context)`. `RENDER(Request& context)` serves the initial HTTP response, while `WS(Request& context)` is called whenever a complete WebSocket message arrives for that page. Configure nginx/Apache to route WebSocket upgrade requests for `.uce` paths to the runtime's HTTP/WebSocket listener. UCE reassembles fragmented messages before calling `WS(Request& context)`. Text and binary frames are both delivered. The current payload is available directly on `context.in`, message metadata is mirrored into `context.params["WS_..."]`, and connection-local state lives on `context.connection`. diff --git a/site/doc/pages/cwd_set.txt b/site/doc/pages/cwd_set.txt index f9cb19d..062d442 100644 --- a/site/doc/pages/cwd_set.txt +++ b/site/doc/pages/cwd_set.txt @@ -8,7 +8,7 @@ path : the new working directory >sys :content -Sets the host worker process current directory. In wasm this is a real hostcall, so restore the previous directory when using it inside request code. +Sets the host worker process current directory. In wasm this is a hostcall, so restore the previous directory when using it inside request code. ## Related Concepts diff --git a/site/doc/pages/ws_scope.txt b/site/doc/pages/ws_scope.txt index 1420515..003752d 100644 --- a/site/doc/pages/ws_scope.txt +++ b/site/doc/pages/ws_scope.txt @@ -12,7 +12,7 @@ Returns the runtime's scope identifier for the current WebSocket endpoint. This is the same default scope used by `ws_send()`, `ws_connections()`, and `ws_connection_count()` when no explicit scope is supplied. -In the current runtime implementation this scope is the page's internal endpoint identifier, typically the absolute `SCRIPT_FILENAME` of the `.ws.uce` file. +In the current runtime implementation this scope is the page's internal endpoint identifier, typically the absolute `SCRIPT_FILENAME` of the `.uce` file that accepted the WebSocket upgrade. Related: diff --git a/site/errors/compiling.uce b/site/errors/compiling.uce index 9cc66e3..50898eb 100644 --- a/site/errors/compiling.uce +++ b/site/errors/compiling.uce @@ -1,7 +1,7 @@ // Example page_compiling handler. Enable in /etc/uce/settings.cfg: // page_compiling=site/errors/compiling.uce // Served with status 503 while the requested unit is being (re)built; the -// page refreshes itself until the build finishes and the real page renders. +// page refreshes itself until the build finishes and the requested page renders. RENDER(Request& context) { diff --git a/site/examples/uce-starter/js/u-howler-demo.html b/site/examples/uce-starter/js/u-howler-demo.html index 7d86387..d2a718e 100755 --- a/site/examples/uce-starter/js/u-howler-demo.html +++ b/site/examples/uce-starter/js/u-howler-demo.html @@ -548,7 +548,7 @@ howl.updateEffectsChain(); // Apply changes

Frequency Analysis

- enableEQ(bands, callback, boost) - Enable real-time frequency analysis
+ enableEQ(bands, callback, boost) - Enable live frequency analysis
• bands: Number of frequency bands (default: 16)
• callback: Function called with frequency data array
• boost: Gain multiplier for visualization (default: 4.0)

diff --git a/site/examples/uce-starter/views/auth/callback.uce b/site/examples/uce-starter/views/auth/callback.uce index 09ec87f..4b468b0 100644 --- a/site/examples/uce-starter/views/auth/callback.uce +++ b/site/examples/uce-starter/views/auth/callback.uce @@ -25,7 +25,7 @@ COMPONENT(Request& context)
✓

Authentication Successful

-

OAuth callback received successfully! Demo mode is active until real provider credentials are configured.

+

OAuth callback received successfully! Demo mode is active until provider credentials are configured.

Service:

Code: ...

diff --git a/site/examples/uce-starter/views/workspace/index.uce b/site/examples/uce-starter/views/workspace/index.uce index 512bddb..484b271 100644 --- a/site/examples/uce-starter/views/workspace/index.uce +++ b/site/examples/uce-starter/views/workspace/index.uce @@ -102,7 +102,7 @@ COMPONENT(Request& context) DValue empty_props; empty_props["icon_class"] = "fas fa-clock-rotate-left"; empty_props["title"] = "No live stream wired yet"; - empty_props["text"] = "The shell is generic. Add your own websocket, polling, or event-driven runtime behind it when a real product needs one."; + empty_props["text"] = "The shell is generic. Add your own websocket, polling, or event-driven runtime behind it when a product needs one."; empty_props["action_html"] = "Open dashboard demo"; main_body += component("../../components/workspace/primitives:EMPTY_STATE", empty_props, context); } diff --git a/site/info/index.uce b/site/info/index.uce index e3ac28c..45b7c10 100644 --- a/site/info/index.uce +++ b/site/info/index.uce @@ -29,7 +29,7 @@ void render_link_card(String href, String title, String body, String meta) } -void render_code_example(Request& context, String eyebrow, String title, String summary, String code, String note, String language = "uce") +DValue code_example_props(String eyebrow, String title, String summary, String code, String note, String language = "uce") { DValue props; props["eyebrow"] = eyebrow; @@ -38,7 +38,7 @@ void render_code_example(Request& context, String eyebrow, String title, String props["code"] = code; props["note"] = note; props["language"] = language; - <> + return(props); } RENDER(Request& context) @@ -123,7 +123,7 @@ RENDER(Request& context) "}\n"; String snippet_nginx = - "location ~ \\.ws\\.uce$ {\n" + "location ~ \\.uce$ {\n" "\terror_page 418 = @uce_websocket;\n" "\tif ($http_upgrade = \"websocket\") { return 418; }\n" "\tinclude fastcgi_params;\n" @@ -212,7 +212,7 @@ RENDER(Request& context)
- +
@@ -248,12 +248,12 @@ RENDER(Request& context)

- - - - - - + + + + + +
@@ -266,9 +266,9 @@ RENDER(Request& context)

The intended shape

    -
  • nginx serves static files directly from `site/`
    • -
  • ordinary `.uce` and plain `.ws.uce` page loads go through FastCGI
    • -
  • real websocket upgrade traffic goes to the built-in HTTP listener
    • +
  • nginx serves static files directly from `/var/www/html`
    • +
  • ordinary `.uce` page loads go through FastCGI
    • +
  • websocket upgrade traffic for `.uce` paths goes to the built-in HTTP listener
  • systemd manages the runtime binary and restart policy
@@ -276,9 +276,9 @@ RENDER(Request& context)

The practical settings

  • `FCGI_SOCKET_PATH=/run/uce/fastcgi.sock` for normal requests
    • -
  • `HTTP_PORT=8080` for `.ws.uce` websocket upgrades
    • +
  • `HTTP_PORT=8080` for `.uce` websocket upgrades
  • `scripts/systemd/manage-uce-service.sh` for build and service control
    • -
  • published root should be `site/`, not the repo root
    • +
  • published root should be `/var/www/html`, not the repo root
@@ -291,7 +291,7 @@ RENDER(Request& context)
- + diff --git a/site/info/js/u-howler-demo.html b/site/info/js/u-howler-demo.html index 7d86387..d2a718e 100755 --- a/site/info/js/u-howler-demo.html +++ b/site/info/js/u-howler-demo.html @@ -548,7 +548,7 @@ howl.updateEffectsChain(); // Apply changes

Frequency Analysis

- enableEQ(bands, callback, boost) - Enable real-time frequency analysis
+ enableEQ(bands, callback, boost) - Enable live frequency analysis
• bands: Number of frequency bands (default: 16)
• callback: Function called with frequency data array
• boost: Gain multiplier for visualization (default: 4.0)

diff --git a/site/tests/sqlite.uce b/site/tests/sqlite.uce index 258f239..72aaf22 100644 --- a/site/tests/sqlite.uce +++ b/site/tests/sqlite.uce @@ -25,7 +25,7 @@ RENDER(Request& context) SQLite* db = sqlite_connect(db_path); check("sqlite_connect()", db && db->connection != 0 && sqlite_error(db) == "connected", sqlite_error(db)); - sqlite_query(db, "create table users(id integer primary key autoincrement, email text not null unique, visits integer not null, rating real not null)"); + sqlite_query(db, "create table users(id integer primary key autoincrement, email text not null unique, visits integer not null, rating numeric not null)"); check("sqlite create table", sqlite_error(db) == "ok", sqlite_error(db)); sqlite_query(db, "create table dropped(id integer); insert into dropped values(1);"); diff --git a/src/fastcgi/src/fcgicc.cc b/src/fastcgi/src/fcgicc.cc index 3aa622d..51dbb2a 100644 --- a/src/fastcgi/src/fcgicc.cc +++ b/src/fastcgi/src/fcgicc.cc @@ -826,6 +826,8 @@ FastCGIServer::process_http_request(FastCGIRequest& request, String& data) request.params["SCRIPT_FILENAME"] = real_candidate; } + // Any .uce unit may expose WS(Request&). The runtime accepts WebSocket + // upgrades based on HTTP headers and dispatches messages to that handler. if(to_lower(request.params["HTTP_UPGRADE"]) == "websocket") { Connection* connection = client_sockets[request.resources.client_socket]; diff --git a/src/lib/functionlib.cpp b/src/lib/functionlib.cpp index 9bdf06c..86abb58 100644 --- a/src/lib/functionlib.cpp +++ b/src/lib/functionlib.cpp @@ -645,7 +645,7 @@ StringList regex_split(String pattern, String subject, String flags) // PCRE2 is not compiled into the wasm core; regex runs host-side (the host // already links libpcre2). One UCEB1-marshalled hostcall carries the request // {op,pattern,subject,flags,replacement} in and the result tree out — the host -// runs the real regex_* and packs the answer. See uce_host_regex in +// runs the native regex_* and packs the answer. See uce_host_regex in // src/wasm/worker.cpp. extern "C" size_t uce_host_regex(const char* in, size_t in_len, char* out, size_t cap); diff --git a/src/lib/sys.cpp b/src/lib/sys.cpp index bdefb5e..7c30b8a 100644 --- a/src/lib/sys.cpp +++ b/src/lib/sys.cpp @@ -298,7 +298,7 @@ StringList ws_connections(String scope) { (void)scope; return(context ? context- u64 ws_connection_count(String scope) { return(ws_connections(scope).size()); } // The wasm workspace owns no connections; ws_send/ws_close record dispatch // commands (same shape as the native websocket_exec capture) that the host -// carries back to the broker, which sends them over the real connections. +// carries back to the broker, which sends them over the client connections. bool ws_send(String message, bool binary, String scope) { if(!context) diff --git a/src/wasm/backend.cpp b/src/wasm/backend.cpp index 78c3548..80b447f 100644 --- a/src/wasm/backend.cpp +++ b/src/wasm/backend.cpp @@ -6,7 +6,7 @@ // wasm workspace. The legacy native dlopen execution path has been removed. #include "../lib/wasm_trace.h" -// The native server TU has the real connectors (sqlite/mysql) compiled in, so +// The native server TU has the native connectors (sqlite/mysql) compiled in, so // the worker's host-side connector hostcalls are available here. The W3 CLI // driver does not define this and gets a named-trap stub for those imports. #define UCE_WASM_HOST_CONNECTORS 1 diff --git a/src/wasm/core.cpp b/src/wasm/core.cpp index adcc117..80a614f 100644 --- a/src/wasm/core.cpp +++ b/src/wasm/core.cpp @@ -1,6 +1,6 @@ // Production WASM W1 core entrypoint. // -// This file deliberately includes the real UCE runtime amalgamation with +// This file deliberately includes the production UCE runtime amalgamation with // __UCE_WASM_CORE__ enabled. Native-only pieces are carved out in the runtime // sources, while the workspace-owned DValue ABI and output plumbing are built // into core.wasm. diff --git a/src/wasm/w3_driver.cpp b/src/wasm/w3_driver.cpp index 467d152..9626bee 100644 --- a/src/wasm/w3_driver.cpp +++ b/src/wasm/w3_driver.cpp @@ -1,7 +1,7 @@ // W3 CLI driver — the exit gate for WASM-PROPOSAL §9.1 W3. // -// Serves real requests through the production workspace runtime -// (src/wasm/worker.cpp): UCEB1 context in → core + lazily loaded real +// Serves requests through the production workspace runtime +// (src/wasm/worker.cpp): UCEB1 context in → core + lazily loaded // generated units → body/response-meta out. Each --repeat gets a fresh // workspace, proving birth/drop. An epoch ticker thread enforces the CPU // budget; the store limiter enforces memory; traps come back as collapsed diff --git a/src/wasm/worker.cpp b/src/wasm/worker.cpp index 8aab1c4..5595f29 100644 --- a/src/wasm/worker.cpp +++ b/src/wasm/worker.cpp @@ -1,7 +1,7 @@ // W3 — production wasm workspace runtime + membrane (WASM-PROPOSAL §6, §7). // // One workspace per request: a fresh Wasmtime store holding the core module -// instance (real uce_lib carve-out, owns memory/allocator/DValue) plus unit +// instance (production uce_lib carve-out, owns memory/allocator/DValue) plus unit // PIC side modules loaded lazily — including mid-request via the // uce_host_component_resolve hostcall, which is how component()/unit_render() // inside the guest trigger dynamic loading. @@ -1691,7 +1691,7 @@ private: #ifdef UCE_WASM_HOST_CONNECTORS if(mod == "env" && name == "uce_host_sqlite") return(add([self](Caller, Span args, Span results) -> Result { - // {op,handle,path,query,params} in → result out. The real native + // {op,handle,path,query,params} in → result out. The native // connector runs host-side; connections live in the workspace // handle table (handle = 1-based index). String encoded; @@ -2030,7 +2030,7 @@ private: if(mod == "env" && name == "uce_host_regex") return(add([self](Caller, Span args, Span results) -> Result { // {op,pattern,subject,flags,replacement} in (UCEB1) → result out. - // PCRE2 lives host-side; this runs the real native regex_*. + // PCRE2 lives host-side; this runs the native regex_*. String encoded; self->hostcall_read(args[0].i32(), args[1].i32(), encoded); DValue request;