udo/uce
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Parallelize proactive unit compilation

Browse Source
This commit is contained in:
udo
2026-07-18 20:19:37 +00:00
parent d4ae51f54b
commit 33a7b00c78
7 changed files with 303 additions and 64 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/setup.md
+2
View File
@@ -190,6 +190,7 @@ SITE_DIRECTORY=/var/www/html
HTTP_DOCUMENT_ROOT=/var/www/html
JIT_COMPILE_ON_REQUEST=1
PROACTIVE_COMPILE_ENABLED=1
PROACTIVE_COMPILE_JOBS=2
PROACTIVE_COMPILE_CHECK_INTERVAL=60
WASM_COMPILE_SCRIPT=scripts/compile_wasm_unit
@@ -224,6 +225,7 @@ Important settings:
- `HTTP_PORT` is the built-in HTTP/WebSocket listener used for WebSocket upgrade traffic and direct local probes. Bind/firewall it for local access only; nginx/Apache should be the public entry point.
- `WS_BROKER_OUTBOUND_TIMEOUT_SECONDS` controls how long a forwarded WS message can remain queued in the broker before being dropped (default `30`). Set to `0` to disable the timeout.
- `WASM_COMPILE_SCRIPT` must point to `scripts/compile_wasm_unit` unless you provide an equivalent compiler. Relative paths are resolved from the runtime root/`COMPILER_SYS_PATH`. That script calls `scripts/check_unit_wasm.py` after linking each unit and uses the pinned WASI SDK on every deployment host.
- `PROACTIVE_COMPILE_JOBS` selects 1–16 low-priority full-site scanner processes (default `2`). Each canonical unit path has one scanner owner. The separate higher-priority demand compiler remains reserved for stale units requested over HTTP, so total background compile concurrency can reach this value plus one.
- `WASM_CORE_PATH` must point at the built `core.wasm` file.
After editing settings, restart UCE:
docs/wasm-runtime-architecture.md
+10 -8
View File
@@ -246,14 +246,16 @@ publication and the shared generation PCH use separate advisory locks, and any
worker/reporting failure rejects the candidate generation.
The proactive compiler and request workers coordinate through per-unit file
locks and a lock-protected demand-priority queue under `BIN_DIRECTORY`. A small
priority-only compiler process drains that queue independently of the full-site
scanner. This matters when the scanner is already inside a long transitive C++
compile: a requested stale component can rebuild immediately instead of waiting
for that unrelated compile to finish. Both compiler processes use the same
per-unit lock, so concurrent demand and scan discovery cannot publish duplicate
artifacts. The priority worker is idle when there is no demand and never scans
the site on its own. Unit
locks and a lock-protected demand-priority queue under `BIN_DIRECTORY`. The
full-site scan uses two low-priority processes by default, bounded from 1–16 by
`PROACTIVE_COMPILE_JOBS`; a stable canonical-path partition gives every unit one
scanner owner and keeps retry/backoff state local. A separate higher-priority
compiler process exclusively drains the demand queue. This matters when the
scanners are already inside long transitive C++ compiles: a requested stale
component can rebuild immediately instead of waiting for an unrelated compile.
All compiler processes use the same per-unit lock, so concurrent demand and scan
discovery cannot publish duplicate artifacts. The priority worker is idle when
there is no demand and never scans the site on its own. Unit
compilation writes and validates a process-unique temporary wasm file, then
publishes it with an atomic rename. When proactive compilation is enabled,
read-only HTTP requests keep using the last complete artifact while requesting