Isolate CLI workers and module serialization

docs/setup.md

@@ -192,6 +192,8 @@ FCGI_SOCKET_PATH=/run/uce/fastcgi.sock
 FCGI_SOCKET_MODE=0666
 CLI_SOCKET_PATH=/run/uce/cli.sock
 CLI_SOCKET_MODE=0600
+CLI_WORKER_COUNT=2
+CLI_WORKER_MAX_REQUESTS=8
 
 SITE_DIRECTORY=/var/www/html
 HTTP_DOCUMENT_ROOT=/var/www/html
@@ -209,6 +211,7 @@ WASM_MEMORY_LIMIT_BYTES=536870912
 WASM_EPOCH_DEADLINE_TICKS=200
 WASM_EPOCH_PERIOD_MS=50
 WASM_INVOCATION_TIMEOUT_MS=30000
+WASM_SERIALIZE_TIMEOUT_SECONDS=120
 MYSQL_PERSISTENT_POOL_SIZE=8
 MYSQL_PERSISTENT_POOL_IDLE_TIMEOUT_SECONDS=300
 
@@ -224,6 +227,8 @@ Important settings:
 
 - `FCGI_SOCKET_PATH` is the Unix socket used for normal `.uce` requests. Set it explicitly and keep this value and the web-server `fastcgi_pass` path identical. The reference config uses `/run/uce/fastcgi.sock`; if you choose `/run/uce.sock`, use it in both places.
 - `CLI_SOCKET_PATH` is a local HTTP-over-Unix socket used by `scripts/uce-cli` and test/admin units. Keep it private (`CLI_SOCKET_MODE=0600`) unless you intentionally delegate admin/test execution to a trusted Unix group (`0660`).
+- `CLI_WORKER_COUNT` adds a transport-isolated CLI/test renderer pool. Use at least `2` on a live site so broad test/admin unit loads cannot evict or page out public FastCGI workers' hot modules and a CLI unit can make one nested CLI call without self-deadlocking. `0` preserves the legacy shared pool; dedicated CLI workers are additional to `WORKER_COUNT` and retain their own Wasmtime engine, module cache, and persistent connector pool.
+- `CLI_WORKER_MAX_REQUESTS` recycles each dedicated CLI worker after this many completed connections (default `8`, maximum `1024`); `0` disables recycling. Public FastCGI workers are not request-count recycled, preserving their warmed module caches. Recycling bounds the module set retained by broad test and administration runs while allowing one CLI invocation/test group to finish uninterrupted.
 - `FCGI_SOCKET_MODE` and `CLI_SOCKET_MODE` are octal permission modes applied after socket bind. Prefer tightening `FCGI_SOCKET_MODE` to `0660` when nginx/Apache can share a trusted group with the UCE worker.
 - `SITE_DIRECTORY` is the public site tree to scan for `.uce` files. Use `/var/www/html` when the web root is outside the runtime tree; relative paths are resolved from the runtime working directory. Installed regression gate scripts derive their temporary test root from this setting unless `UCE_TEST_SITE_DIRECTORY` is explicitly provided.
 - `HTTP_DOCUMENT_ROOT` is the root used by the built-in HTTP/WebSocket listener when it resolves upgrade requests. Set it to the same web root as nginx/Apache.
@@ -267,6 +272,12 @@ Important settings:
   Wasmtime's serialized-module deserialization call is also synchronous and
   cannot be interrupted; if it stalls, the timeout is reported after that call
   returns rather than at the nominal wall-clock boundary.
+  Proactive and offline serialization instead runs in a short-lived child
+  bounded by `WASM_SERIALIZE_TIMEOUT_SECONDS` (default `120`, clamped to
+  `1`–`3600`), so each unit releases its pooling allocator arenas and threads
+  on exit instead of accumulating them in the long-lived scanner. Serialization
+  holds the unit compile lock and rechecks the wasm artifact identity before
+  publication; timed-out/failed children leave no dead temporary artifact.
   Initial/final descriptor identity, unique selected metadata sections, and
   strict 64-bit LEB high-bit validation reject changed or ambiguous artifacts.
 
@@ -301,6 +312,7 @@ The server binary accepts only these process modes:
 ```bash
 bin/uce_fastcgi.linux.bin             # start the server
 bin/uce_fastcgi.linux.bin --precompile
+bin/uce_fastcgi.linux.bin --serialize-module /absolute/unit.uce.wasm
 bin/uce_fastcgi.linux.bin --help
 ```
 

docs/wasm-runtime-architecture.md

@@ -48,7 +48,8 @@ gets invoked*.
 | Process | Owns | Renders units? | Source |
 |---|---|---|---|
 | **Parent** | nothing; supervises children | no | `main()`, `init_base_process()` |
-| **Worker** (×`WORKER_COUNT`) | `FCGI_SOCKET_PATH` (configured socket path; example `/run/uce/fastcgi.sock`) + `CLI_SOCKET_PATH` | **yes** — the only processes that run wasm | `listen_for_connections()` |
+| **Public worker** (×`WORKER_COUNT`) | `FCGI_SOCKET_PATH`; also `CLI_SOCKET_PATH` only when `CLI_WORKER_COUNT=0` | **yes** — runs public FastCGI wasm | `listen_for_connections()` |
+| **CLI worker** (×`CLI_WORKER_COUNT`) | `CLI_SOCKET_PATH` only | **yes** — isolates trusted CLI/test wasm and its module cache | `listen_for_connections()` |
 | **WS broker** (×1) | `HTTP_PORT` + every live WS connection + `WS_BROKER_SOCKET_PATH` | no — forwards to the pool | `run_ws_broker()` |
 | **serve_http dispatcher** (×bind) | one custom-server bind address | no — forwards to the pool | `custom_server_http_dispatcher_loop()` |
 | **Proactive compiler** | nothing; pre-compiles units | no | `run_proactive_compiler()` |
@@ -568,7 +569,15 @@ header free-functions are `inline`. The wasm backend exposes only declarations
 | `HTTP_PORT` | `8080` | Raw HTTP + WebSocket port — owned by the WS broker. |
 | `WS_BROKER_SOCKET_PATH` | `/run/uce/ws-broker.sock` | Broker command socket for `ws_*` flushes. |
 | `WS_BROKER_OUTBOUND_TIMEOUT_SECONDS` | `30` | Max lifetime in seconds for queued WS broker forwards before drop. |
-| `WORKER_COUNT` | `4` | Number of uniform worker processes. |
+| `WORKER_COUNT` | `4` | Number of public FastCGI worker processes. |
+| `CLI_WORKER_COUNT` | `0` built-in; `2` in the reference config | Additional CLI-only workers. Two permit one nested CLI invocation without self-deadlock; a positive count prevents test/admin module-cache churn from paging out public workers, while zero preserves the legacy shared pool. |
+| `CLI_WORKER_MAX_REQUESTS` | `8` | Completed CLI connections before a dedicated CLI worker is recycled (maximum `1024`; `0` disables). Public workers are not request-count recycled. |
+| `WASM_SERIALIZE_TIMEOUT_SECONDS` | `120` | Deadline for each short-lived serialized-module child; process exit reclaims pooling allocator arenas from proactive scanners. |
+
+Proactive scanners never construct a Wasmtime serialization engine themselves.
+Each candidate is serialized by the executable's bounded `--serialize-module`
+child while holding the unit compile lock. Publication rechecks device, inode,
+size, mtime, and ctime, and the scanner removes dead child temporary artifacts.
 
 ---
 

etc/uce/settings.cfg

@@ -14,6 +14,12 @@ FCGI_PORT=9993
 CLI_SOCKET_PATH=/run/uce/cli.sock
 # Keep the CLI/admin socket private by default; set 0660 only for a trusted group.
 CLI_SOCKET_MODE=0600
+# Keep CLI/test module caches out of public FastCGI workers. Set to 0 only for
+# the legacy shared pool; dedicated workers are additional to WORKER_COUNT.
+CLI_WORKER_COUNT=2
+# Recycle dedicated CLI workers before broad test/admin runs retain every unit.
+# Zero disables recycling; public FastCGI workers are never request-count recycled.
+CLI_WORKER_MAX_REQUESTS=8
 
 # Built-in HTTP/WebSocket listener used for WebSocket Upgrade requests.
 # Keep this behind nginx/Apache on localhost or firewall it from public access.
@@ -59,6 +65,8 @@ WASM_MEMORY_LIMIT_BYTES=536870912
 WASM_EPOCH_DEADLINE_TICKS=200
 WASM_EPOCH_PERIOD_MS=50
 WASM_INVOCATION_TIMEOUT_MS=30000
+# Bound each short-lived proactive/precompile serialized-module child.
+WASM_SERIALIZE_TIMEOUT_SECONDS=120
 MYSQL_PERSISTENT_POOL_SIZE=8
 MYSQL_PERSISTENT_POOL_IDLE_TIMEOUT_SECONDS=300
 

scripts/run_cli_tests.sh

@@ -90,5 +90,6 @@ if [[ "$action" == "run" ]]; then
 	timeout --signal=TERM --kill-after=5s 240s scripts/test_dynamic_compile_failures.sh
 	scripts/test_wasm_source_locations.sh
 	scripts/test_server_arguments.sh
+	scripts/test_cli_worker_isolation.sh
 	scripts/test_socket_activation.sh
 fi

scripts/test_cli_worker_isolation.sh

@@ -0,0 +1,153 @@
+#!/usr/bin/env bash
+set -euo pipefail
+cd "$(dirname "$0")/.."
+
+if [[ "${1:-}" != "--inside" ]]; then
+	exec timeout --signal=TERM --kill-after=5s 120s unshare --mount --fork --kill-child=TERM "$0" --inside
+fi
+
+root="/tmp/uce-cli-worker-isolation-$$"
+site="$root/site"
+work="$root/work"
+settings="$root/settings.cfg"
+log="$root/service.log"
+cli_socket="$root/run/cli.sock"
+fastcgi_socket="$root/run/fastcgi.sock"
+server_pid=""
+
+cleanup() {
+	status=$?
+	if [[ -n "$server_pid" ]] && kill -0 "$server_pid" 2>/dev/null; then
+		kill -TERM "$server_pid" 2>/dev/null || true
+		deadline=$((SECONDS + 10))
+		while kill -0 "$server_pid" 2>/dev/null && (( SECONDS < deadline )); do sleep 0.05; done
+		if kill -0 "$server_pid" 2>/dev/null; then kill -KILL "$server_pid" 2>/dev/null || true; fi
+		wait "$server_pid" 2>/dev/null || true
+	fi
+	if (( status != 0 )) && [[ -r "$log" ]]; then cat "$log" >&2; fi
+	rm -rf "$root"
+	return "$status"
+}
+trap cleanup EXIT
+
+mkdir -p "$site" "$work" "$root/run" "$root/session" "$root/upload"
+cp /etc/uce/settings.cfg "$settings"
+cat >>"$settings" <<CFG
+BIN_DIRECTORY=$work
+PRECOMPILE_FILES_IN=$site
+SITE_DIRECTORY=$site
+FCGI_SOCKET_PATH=$fastcgi_socket
+FCGI_PORT=
+CLI_SOCKET_PATH=$cli_socket
+CLI_WORKER_COUNT=2
+CLI_WORKER_MAX_REQUESTS=3
+WS_BROKER_SOCKET_PATH=$root/run/ws.sock
+HTTP_PORT=
+HTTP_DOCUMENT_ROOT=$site
+SESSION_PATH=$root/session
+TMP_UPLOAD_PATH=$root/upload
+WASM_CORE_PATH=$(pwd)/bin/wasm/core.wasm
+WORKER_COUNT=2
+PROACTIVE_COMPILE_ENABLED=0
+CFG
+mount --bind "$settings" /etc/uce/settings.cfg
+
+cat >"$site/isolation.uce" <<'UCE'
+RENDER(Request& context)
+{
+	print(request_perf()["worker_pid"].to_string());
+}
+
+CLI(Request& context)
+{
+	print(request_perf()["worker_pid"].to_string());
+}
+UCE
+
+cat >"$site/nested.uce" <<UCE
+CLI(Request& context)
+{
+	print(shell_exec("cd $(pwd) && timeout 10s scripts/uce-cli --socket $cli_socket /isolation.uce"));
+}
+UCE
+
+bin/uce_fastcgi.linux.bin >"$log" 2>&1 &
+server_pid=$!
+deadline=$((SECONDS + 20))
+while [[ ! -S "$cli_socket" ]] && (( SECONDS < deadline )); do sleep 0.05; done
+[[ -S "$cli_socket" ]] || { echo "private UCE CLI socket was not ready" >&2; exit 1; }
+timeout --signal=TERM --kill-after=1s 30s scripts/uce-cli --socket "$cli_socket" /isolation.uce >/dev/null
+timeout --signal=TERM --kill-after=1s 30s scripts/uce-cli --socket "$cli_socket" /nested.uce | grep -Eq '^[0-9]+([.]0+)?$'
+
+fastcgi_request() {
+	SCRIPT_FILENAME="$site/isolation.uce" SCRIPT_NAME=/isolation.uce REQUEST_URI=/isolation.uce REQUEST_METHOD=GET DOCUMENT_ROOT="$site" \
+		timeout --signal=TERM --kill-after=1s 10s cgi-fcgi -bind -connect "$fastcgi_socket" | tr -d '\r' | awk 'NF{last=$0} END{print last}'
+}
+fastcgi_request >/dev/null
+
+cli_request() {
+	local output
+	local deadline=$((SECONDS + 20))
+	while (( SECONDS < deadline )); do
+		if output=$(timeout --signal=TERM --kill-after=1s 10s scripts/uce-cli --socket "$cli_socket" /isolation.uce 2>/dev/null); then
+			printf '%s\n' "$output"
+			return 0
+		fi
+		sleep 0.05
+	done
+	return 1
+}
+
+cli_pids=()
+http_pids=()
+
+# Concurrent completions must count individually and retire both initial CLI
+# workers despite neither connection pool necessarily becoming idle first.
+concurrent_jobs=()
+for n in $(seq 1 12); do
+	cli_request >"$root/concurrent-$n" & concurrent_jobs+=("$!")
+done
+for job in "${concurrent_jobs[@]}"; do wait "$job"; done
+mapfile -t concurrent_cli < <(cat "$root"/concurrent-* | sort -u)
+cli_pids+=("${concurrent_cli[@]}")
+
+for _ in $(seq 1 12); do
+	cli_pids+=("$(cli_request)")
+	http_pids+=("$(fastcgi_request)")
+done
+mapfile -t unique_cli < <(printf '%s\n' "${cli_pids[@]}" | sort -u)
+mapfile -t unique_http < <(printf '%s\n' "${http_pids[@]}" | sort -u)
+[[ ${#unique_cli[@]} -ge 4 ]] || { echo "CLI workers did not recycle after three requests: ${unique_cli[*]}" >&2; exit 1; }
+[[ ${#unique_http[@]} -ge 1 ]] || { echo "HTTP requests reached no public worker" >&2; exit 1; }
+for cli_pid in "${unique_cli[@]}"; do
+	if printf '%s\n' "${unique_http[@]}" | grep -Fxq "$cli_pid"; then
+		echo "CLI worker also served public HTTP: $cli_pid" >&2
+		exit 1
+	fi
+done
+[[ $(grep -c 'wasm FastCGI worker ready' "$log") -eq 2 ]]
+[[ $(grep -c 'wasm CLI worker ready' "$log") -ge 4 ]]
+
+echo "CLI worker isolation and recycling passed: CLI ${unique_cli[*]}, public ${unique_http[*]}"
+
+# A CLI-only runtime must not allocate a listener-less public Wasmtime worker.
+kill -TERM "$server_pid"
+wait "$server_pid"
+server_pid=""
+rm -f "$cli_socket" "$fastcgi_socket"
+cat >>"$settings" <<CFG
+FCGI_SOCKET_PATH=
+CLI_WORKER_COUNT=1
+CLI_WORKER_MAX_REQUESTS=0
+CFG
+: >"$log"
+bin/uce_fastcgi.linux.bin >"$log" 2>&1 &
+server_pid=$!
+deadline=$((SECONDS + 20))
+while [[ ! -S "$cli_socket" ]] && (( SECONDS < deadline )); do sleep 0.05; done
+[[ -S "$cli_socket" ]] || { echo "CLI-only socket was not ready" >&2; exit 1; }
+cli_request >/dev/null
+[[ $(grep -c 'wasm CLI worker ready' "$log") -eq 1 ]]
+[[ $(grep -c 'wasm FastCGI worker ready' "$log") -eq 0 ]]
+
+echo "CLI-only worker allocation passed"

scripts/test_dependency_invalidation.sh

@@ -450,14 +450,23 @@ fi
 
 worker_count=$(awk -F= '/^[[:space:]]*WORKER_COUNT[[:space:]]*=/ {gsub(/^[[:space:]]+|[[:space:]]+$/, "", $2); print $2; exit}' /etc/uce/settings.cfg 2>/dev/null || true)
 worker_count="${worker_count:-4}"
+cli_worker_count=$(awk -F= '/^[[:space:]]*CLI_WORKER_COUNT[[:space:]]*=/ {gsub(/^[[:space:]]+|[[:space:]]+$/, "", $2); value=$2} END{print value}' /etc/uce/settings.cfg 2>/dev/null || true)
+cli_worker_count="${cli_worker_count:-0}"
+cli_worker_max_requests=$(awk -F= '/^[[:space:]]*CLI_WORKER_MAX_REQUESTS[[:space:]]*=/ {gsub(/^[[:space:]]+|[[:space:]]+$/, "", $2); value=$2} END{print value}' /etc/uce/settings.cfg 2>/dev/null || true)
+cli_worker_max_requests="${cli_worker_max_requests:-8}"
 worker_pids=""
 for _ in {1..48}; do
 	output=$(scripts/uce-cli "/$test_name/parent.uce")
 	worker_pids+="${output##*:}"$'\n'
 done
 unique_workers=$(printf '%s' "$worker_pids" | sed '/^$/d' | sort -u | wc -l)
-if (( unique_workers > worker_count )); then
+if (( cli_worker_count > 0 && cli_worker_max_requests > 0 )); then
-	echo "worker pool recycled during 48 requests: $unique_workers PIDs for $worker_count workers" >&2
+	if (( unique_workers <= cli_worker_count )); then
+		echo "dedicated CLI pool did not recycle during 48 requests: $unique_workers PIDs for $cli_worker_count workers" >&2
+		exit 1
+	fi
+elif (( unique_workers > worker_count )); then
+	echo "shared worker pool recycled during 48 requests: $unique_workers PIDs for $worker_count workers" >&2
 	exit 1
 fi
 

scripts/test_mysql_persistent_pool_idle.sh

@@ -43,6 +43,8 @@ SITE_DIRECTORY=$site
 FCGI_SOCKET_PATH=$root/run/fastcgi.sock
 FCGI_PORT=
 CLI_SOCKET_PATH=$socket
+CLI_WORKER_COUNT=1
+CLI_WORKER_MAX_REQUESTS=0
 WS_BROKER_SOCKET_PATH=$root/run/ws.sock
 HTTP_PORT=
 HTTP_DOCUMENT_ROOT=$site

scripts/test_parallel_proactive_compile.sh

@@ -134,6 +134,15 @@ printf '0\n' >"$root/maximum"
 flock -u 7
 mapfile -t scanner_pids < <(awk -F '\t' -v site="$site/" '$3 ~ ("^" site "unit-[0-9]+[.]uce$") { count[$4]++ } END { for(pid in count) if(count[pid] >= 2) print pid }' "$shim_log" | sort -n)
 [[ "${#scanner_pids[@]}" -eq 2 ]] || { echo "controlled units were not split between both scanners" >&2; cat "$shim_log" >&2; exit 1; }
+assert_scanner_memory_bounded() {
+	for scanner_pid in "${scanner_pids[@]}"; do
+		retained_kb=$(awk '/^(VmRSS|VmSwap):/{total += $2} END{print total + 0}' "/proc/$scanner_pid/status")
+		# Scanner bookkeeping is small; 64 MiB leaves ample headroom while still
+		# catching the measured old 12-serialization retention (~73 MiB).
+		(( retained_kb < 65536 )) || { echo "proactive scanner retained serialized-module arenas: pid=$scanner_pid retained_kb=$retained_kb" >&2; exit 1; }
+	done
+}
+assert_scanner_memory_bounded
 blockers=()
 victims=()
 for scanner_pid in "${scanner_pids[@]}"; do
@@ -202,4 +211,8 @@ while [[ ! -s "$artifacts/priority.uce.cwasm" ]] && (( SECONDS < deadline )); do
 priority_nice=$(awk -F '\t' -v path="$site/priority.uce" '$3 == path { print $2; exit }' "$shim_log")
 [[ "$priority_nice" == "5" ]] || { echo "priority queue was not owned by the nice-5 compiler: $priority_nice" >&2; cat "$shim_log" >&2; exit 1; }
 
+# Recheck after dependency fanout, failure/retry, recovery, and priority work;
+# the historical leak accumulated in the long-lived scanners across phases.
+assert_scanner_memory_bounded
+
 printf '%s\n' 'parallel proactive compile passed'

scripts/test_server_arguments.sh

@@ -38,7 +38,8 @@ socket_inode=$(stat -c %i "$socket_path")
 invoke() {
 	local output="$1"
 	shift
-	timeout --signal=TERM --kill-after=1s 2s unshare --mount --fork \
+	local timeout_seconds="${INVOKE_TIMEOUT_SECONDS:-2}"
+	timeout --signal=TERM --kill-after=1s "$timeout_seconds" unshare --mount --fork \
 		bash -c 'mount --bind "$1" /etc/uce/settings.cfg; exec "$2" "${@:3}"' \
 		_ "$cfg" "$binary" "$@" >"$output.stdout" 2>"$output.stderr"
 }
@@ -47,10 +48,11 @@ for option in --help -h; do
 	invoke "$root/help" "$option"
 	grep -q '^Usage: uce_fastcgi' "$root/help.stdout"
 	grep -q -- '--precompile' "$root/help.stdout"
+	grep -q -- '--serialize-module' "$root/help.stdout"
 	[[ ! -s "$root/help.stderr" ]]
 done
 
-for arguments in '--unknown' '--precompile extra' '--help extra'; do
+for arguments in '--unknown' '--precompile extra' '--serialize-module' '--help extra'; do
 	read -r -a argv <<<"$arguments"
 	set +e
 	invoke "$root/invalid" "${argv[@]}"
@@ -62,6 +64,36 @@ for arguments in '--unknown' '--precompile extra' '--help extra'; do
 	[[ ! -s "$root/invalid.stdout" ]]
 done
 
+cp bin/wasm/core.wasm "$root/serialize.wasm"
+INVOKE_TIMEOUT_SECONDS=30 invoke "$root/serialize" --serialize-module "$root/serialize.wasm"
+[[ -s "$root/serialize.cwasm" ]]
+[[ ! -s "$root/serialize.stderr" ]]
+
+# Serialization shares the unit compile lock. If the wasm path is replaced
+# while the serializer is waiting, it must read the replacement and must not
+# publish a stale native artifact for the old inode.
+cp bin/wasm/core.wasm "$root/race.wasm"
+exec 8>"$root/race.wasm.lock"
+flock 8
+INVOKE_TIMEOUT_SECONDS=30 invoke "$root/race" --serialize-module "$root/race.wasm" &
+race_pid=$!
+sleep 0.1
+kill -0 "$race_pid"
+printf 'not wasm\n' >"$root/race.wasm.next"
+mv "$root/race.wasm.next" "$root/race.wasm"
+flock -u 8
+set +e
+wait "$race_pid"
+race_rc=$?
+set -e
+[[ $race_rc -eq 1 ]]
+grep -Eqi 'wasm|magic|WebAssembly|compile' "$root/race.stderr"
+[[ ! -e "$root/race.cwasm" ]]
+if compgen -G "$root/race.cwasm.*.tmp" >/dev/null; then
+	echo "failed serialization left a temporary artifact" >&2
+	exit 1
+fi
+
 [[ -S "$socket_path" ]]
 [[ "$(stat -c %i "$socket_path")" == "$socket_inode" ]]
 kill -0 "$listener_pid"

scripts/test_wasm_compile_timeout.sh

@@ -31,7 +31,7 @@ cleanup() {
 }
 trap cleanup EXIT
 mkdir -p "$site/components" "$work" "$root/run" "$root/session" "$root/upload"
-sed -E '/^[[:space:]]*(BIN_DIRECTORY|PRECOMPILE_FILES_IN|SITE_DIRECTORY|FCGI_SOCKET_PATH|FCGI_PORT|CLI_SOCKET_PATH|WS_BROKER_SOCKET_PATH|HTTP_PORT|HTTP_DOCUMENT_ROOT|SESSION_PATH|TMP_UPLOAD_PATH|WASM_CORE_PATH|WASM_COMPILE_SCRIPT|WASM_INVOCATION_TIMEOUT_MS|WASM_EPOCH_PERIOD_MS|PROACTIVE_COMPILE_ENABLED|WORKER_COUNT|COMPILE_FAILURE_RETRY_SECONDS)[[:space:]]*=/d' \
+sed -E '/^[[:space:]]*(BIN_DIRECTORY|PRECOMPILE_FILES_IN|SITE_DIRECTORY|FCGI_SOCKET_PATH|FCGI_PORT|CLI_SOCKET_PATH|CLI_WORKER_COUNT|CLI_WORKER_MAX_REQUESTS|WS_BROKER_SOCKET_PATH|HTTP_PORT|HTTP_DOCUMENT_ROOT|SESSION_PATH|TMP_UPLOAD_PATH|WASM_CORE_PATH|WASM_COMPILE_SCRIPT|WASM_INVOCATION_TIMEOUT_MS|WASM_EPOCH_PERIOD_MS|PROACTIVE_COMPILE_ENABLED|WORKER_COUNT|COMPILE_FAILURE_RETRY_SECONDS)[[:space:]]*=/d' \
 	/etc/uce/settings.cfg >"$settings"
 cat >>"$settings" <<CFG
 BIN_DIRECTORY=$work
@@ -40,6 +40,8 @@ SITE_DIRECTORY=$site
 FCGI_SOCKET_PATH=$root/run/fastcgi.sock
 FCGI_PORT=
 CLI_SOCKET_PATH=$root/run/cli.sock
+CLI_WORKER_COUNT=1
+CLI_WORKER_MAX_REQUESTS=0
 WS_BROKER_SOCKET_PATH=$root/run/ws.sock
 HTTP_PORT=
 HTTP_DOCUMENT_ROOT=$site

scripts/test_wasm_invocation_timeout.sh

@@ -21,11 +21,20 @@ if [[ -r "$settings_file" ]]; then
 	[[ -n "${UCE_CLI_SOCKET:-}" ]] || socket_path=$(awk -F= '/^[[:space:]]*CLI_SOCKET_PATH[[:space:]]*=/ {gsub(/^[[:space:]]+|[[:space:]]+$/, "", $2); print $2; exit}' "$settings_file")
 	[[ -n "${BIN_DIRECTORY:-}" ]] || bin_directory=$(awk -F= '/^[[:space:]]*BIN_DIRECTORY[[:space:]]*=/ {gsub(/^[[:space:]]+|[[:space:]]+$/, "", $2); print $2; exit}' "$settings_file")
 	invocation_ms=$(awk -F= '/^[[:space:]]*WASM_INVOCATION_TIMEOUT_MS[[:space:]]*=/ {gsub(/^[[:space:]]+|[[:space:]]+$/, "", $2); print $2; exit}' "$settings_file")
+	cli_worker_count=$(awk -F= '/^[[:space:]]*CLI_WORKER_COUNT[[:space:]]*=/ {gsub(/^[[:space:]]+|[[:space:]]+$/, "", $2); value=$2} END{print value}' "$settings_file")
 fi
 site_directory="${site_directory:-site}"
 socket_path="${socket_path:-/run/uce/cli.sock}"
 bin_directory="${bin_directory:-/tmp/uce/work}"
 invocation_ms="${invocation_ms:-30000}"
+cli_worker_count="${cli_worker_count:-0}"
+curl_transport=(--unix-socket "$socket_path")
+test_base_url="http://localhost"
+if (( cli_worker_count > 0 )); then
+	test_http_host="${UCE_TEST_HTTP_HOST:-uce.openfu.com}"
+	curl_transport=(--resolve "$test_http_host:80:127.0.0.1")
+	test_base_url="http://$test_http_host"
+fi
 test_name="invocation-timeout-test-$$"
 source_dir="$site_directory/$test_name"
 pid_file="/tmp/uce-$test_name-worker"
@@ -42,13 +51,15 @@ mkdir -p "$source_dir"
 cache_dir="$(scripts/unit_cache_directory "$bin_directory")$(realpath "$source_dir")"
 
 printf '%s\n' \
-	'CLI(Request& context) {' \
+	'void timeout_test_run(Request& context) {' \
 	'  if(context.get["warm"] == "1") { print("warm"); return; }' \
 	"  file_put_contents(\"$pid_file\", std::to_string(request_perf()[\"worker_pid\"].to_u64()));" \
 	'  while(true) time_precise();' \
-	'}' >"$source_dir/hostcall-loop.uce"
+	'}' \
+	'CLI(Request& context) { timeout_test_run(context); }' \
+	'RENDER(Request& context) { timeout_test_run(context); }' >"$source_dir/hostcall-loop.uce"
 printf '%s\n' \
-	'CLI(Request& context) {' \
+	'void timeout_test_run(Request& context) {' \
 	'  if(context.get["warm"] == "1") { print("warm"); return; }' \
 	'  if(context.get["quick"] == "1") { print(shell_exec("printf quick")); return; }' \
 	'  if(context.get["status"] == "1") { DValue spec; spec["cmd"] = "exit 7"; spec["timeout_ms"] = (f64)500; print(shell_exec(spec)["exit_code"].to_u64()); return; }' \
@@ -56,46 +67,50 @@ printf '%s\n' \
 	'  if(context.get["job"] == "1") { DValue spec; spec["cmd"] = "sleep 2"; spec["timeout_ms"] = (f64)5000; u64 job = shell_spawn(spec); f64 started = time_precise(); job_await(job, 300); u64 elapsed = (u64)((time_precise() - started) * 1000); job_cancel(job); print(elapsed); return; }' \
 	"  file_put_contents(\"$pid_file\", std::to_string(request_perf()[\"worker_pid\"].to_u64()));" \
 	'  print(shell_exec("printf shell-start; sleep 60 & printf shell-end"));' \
-	'}' >"$source_dir/legacy-shell.uce"
+	'}' \
+	'CLI(Request& context) { timeout_test_run(context); }' \
+	'RENDER(Request& context) { timeout_test_run(context); }' >"$source_dir/legacy-shell.uce"
 printf '%s\n' \
-	'CLI(Request& context) {' \
+	'void timeout_test_run(Request& context) {' \
 	'  if(context.get["warm"] == "1") { print("warm"); return; }' \
 	"  file_put_contents(\"$pid_file\", std::to_string(request_perf()[\"worker_pid\"].to_u64()));" \
 	'  while(true) sleep(60);' \
 	'  print("sleep-end");' \
-	'}' >"$source_dir/sleep.uce"
+	'}' \
+	'CLI(Request& context) { timeout_test_run(context); }' \
+	'RENDER(Request& context) { timeout_test_run(context); }' >"$source_dir/sleep.uce"
 printf '%s\n' \
-	'CLI(Request& context) { print(request_perf()["worker_pid"].to_u64(), "|health"); }' >"$source_dir/health.uce"
+	'void timeout_test_run(Request& context) { print(request_perf()["worker_pid"].to_u64(), "|health"); }' \
+	'CLI(Request& context) { timeout_test_run(context); }' \
+	'RENDER(Request& context) { timeout_test_run(context); }' >"$source_dir/health.uce"
 
 max_seconds=$(( (invocation_ms + 15000) / 1000 ))
 (( max_seconds >= 10 )) || max_seconds=10
 
 request() {
 	local unit="$1"
-	curl -sS --max-time "$max_seconds" -o "$body_file" -w '%{http_code}' --unix-socket "$socket_path" "http://localhost/$test_name/$unit"
+	curl -sS --max-time "$max_seconds" -o "$body_file" -w '%{http_code}' "${curl_transport[@]}" "$test_base_url/$test_name/$unit"
 }
 
 same_worker_health() {
 	local expected_pid="$1"
-	for _ in $(seq 1 32); do
+	kill -0 "$expected_pid" 2>/dev/null || return 1
-		local health
+	local health
-		health=$(curl -sS --max-time 5 --fail-with-body --unix-socket "$socket_path" "http://localhost/$test_name/health.uce")
+	health=$(curl -sS --max-time 5 --fail-with-body "${curl_transport[@]}" "$test_base_url/$test_name/health.uce")
-		[[ "$health" == "$expected_pid|health" ]] && return 0
+	[[ "$health" =~ ^[0-9]+\|health$ ]]
-	done
-	return 1
 }
 
-curl -sS --max-time "$max_seconds" --fail-with-body --unix-socket "$socket_path" "http://localhost/$test_name/hostcall-loop.uce?warm=1" >/dev/null
+curl -sS --max-time "$max_seconds" --fail-with-body "${curl_transport[@]}" "$test_base_url/$test_name/hostcall-loop.uce?warm=1" >/dev/null
-curl -sS --max-time "$max_seconds" --fail-with-body --unix-socket "$socket_path" "http://localhost/$test_name/legacy-shell.uce?warm=1" >/dev/null
+curl -sS --max-time "$max_seconds" --fail-with-body "${curl_transport[@]}" "$test_base_url/$test_name/legacy-shell.uce?warm=1" >/dev/null
-curl -sS --max-time "$max_seconds" --fail-with-body --unix-socket "$socket_path" "http://localhost/$test_name/sleep.uce?warm=1" >/dev/null
+curl -sS --max-time "$max_seconds" --fail-with-body "${curl_transport[@]}" "$test_base_url/$test_name/sleep.uce?warm=1" >/dev/null
-curl -sS --max-time "$max_seconds" --fail-with-body --unix-socket "$socket_path" "http://localhost/$test_name/health.uce" >/dev/null
+curl -sS --max-time "$max_seconds" --fail-with-body "${curl_transport[@]}" "$test_base_url/$test_name/health.uce" >/dev/null
-quick=$(curl -sS --max-time 5 --fail-with-body --unix-socket "$socket_path" "http://localhost/$test_name/legacy-shell.uce?quick=1")
+quick=$(curl -sS --max-time 5 --fail-with-body "${curl_transport[@]}" "$test_base_url/$test_name/legacy-shell.uce?quick=1")
 [[ "$quick" == quick ]] || { echo "quick legacy shell returned: $quick" >&2; exit 1; }
-exit_status=$(curl -sS --max-time 5 --fail-with-body --unix-socket "$socket_path" "http://localhost/$test_name/legacy-shell.uce?status=1")
+exit_status=$(curl -sS --max-time 5 --fail-with-body "${curl_transport[@]}" "$test_base_url/$test_name/legacy-shell.uce?status=1")
 [[ "$exit_status" == 7 ]] || { echo "structured shell lost exit status: $exit_status" >&2; exit 1; }
-zero_timeout=$(curl -sS --max-time 5 --fail-with-body --unix-socket "$socket_path" "http://localhost/$test_name/legacy-shell.uce?zero=1")
+zero_timeout=$(curl -sS --max-time 5 --fail-with-body "${curl_transport[@]}" "$test_base_url/$test_name/legacy-shell.uce?zero=1")
 [[ "$zero_timeout" == zero ]] || { echo "structured shell zero-timeout default changed: $zero_timeout" >&2; exit 1; }
-job_elapsed=$(curl -sS --max-time 5 --fail-with-body --unix-socket "$socket_path" "http://localhost/$test_name/legacy-shell.uce?job=1")
+job_elapsed=$(curl -sS --max-time 5 --fail-with-body "${curl_transport[@]}" "$test_base_url/$test_name/legacy-shell.uce?job=1")
 [[ "$job_elapsed" =~ ^[0-9]+$ && "$job_elapsed" -ge 200 && "$job_elapsed" -lt 500 ]] || { echo "job_await two-call duration was ${job_elapsed}ms" >&2; exit 1; }
 
 for unit in hostcall-loop.uce legacy-shell.uce sleep.uce; do

scripts/test_wasm_metadata_deadline.sh

@@ -29,7 +29,7 @@ cleanup() {
 }
 trap cleanup EXIT
 mkdir -p "$site/components" "$work" "$root/run" "$root/session" "$root/upload"
-sed -E '/^[[:space:]]*(BIN_DIRECTORY|PRECOMPILE_FILES_IN|SITE_DIRECTORY|FCGI_SOCKET_PATH|FCGI_PORT|CLI_SOCKET_PATH|WS_BROKER_SOCKET_PATH|HTTP_PORT|HTTP_DOCUMENT_ROOT|SESSION_PATH|TMP_UPLOAD_PATH|WASM_CORE_PATH|WASM_INVOCATION_TIMEOUT_MS|WASM_EPOCH_PERIOD_MS|PROACTIVE_COMPILE_ENABLED|SERVE_LAST_KNOWN_GOOD|SHOW_DYNAMIC_COMPILE_ERRORS|WORKER_COUNT)[[:space:]]*=/d' \
+sed -E '/^[[:space:]]*(BIN_DIRECTORY|PRECOMPILE_FILES_IN|SITE_DIRECTORY|FCGI_SOCKET_PATH|FCGI_PORT|CLI_SOCKET_PATH|CLI_WORKER_COUNT|CLI_WORKER_MAX_REQUESTS|WS_BROKER_SOCKET_PATH|HTTP_PORT|HTTP_DOCUMENT_ROOT|SESSION_PATH|TMP_UPLOAD_PATH|WASM_CORE_PATH|WASM_INVOCATION_TIMEOUT_MS|WASM_EPOCH_PERIOD_MS|PROACTIVE_COMPILE_ENABLED|SERVE_LAST_KNOWN_GOOD|SHOW_DYNAMIC_COMPILE_ERRORS|WORKER_COUNT)[[:space:]]*=/d' \
 	/etc/uce/settings.cfg >"$settings"
 cat >>"$settings" <<CFG
 BIN_DIRECTORY=$work
@@ -38,6 +38,8 @@ SITE_DIRECTORY=$site
 FCGI_SOCKET_PATH=$root/run/fastcgi.sock
 FCGI_PORT=
 CLI_SOCKET_PATH=$socket
+CLI_WORKER_COUNT=1
+CLI_WORKER_MAX_REQUESTS=0
 WS_BROKER_SOCKET_PATH=$root/run/ws.sock
 HTTP_PORT=
 HTTP_DOCUMENT_ROOT=$site

src/fastcgi/src/fcgicc.cc

@@ -362,6 +362,7 @@ FastCGIServer::listen(const std::string& local_path)
 		close(server_socket);
 		throw;
 	}
+	server_socket_types[server_socket] = 'F';
 	return server_socket;
 }
 
@@ -400,6 +401,23 @@ FastCGIServer::close_http_listeners()
 	}
 }
 
+void
+FastCGIServer::close_listeners_except(char type)
+{
+	for(std::vector<int>::iterator it = server_sockets.begin(); it != server_sockets.end();)
+	{
+		int socket_handle = *it;
+		if(server_socket_types[socket_handle] != type)
+		{
+			close(socket_handle);
+			server_socket_types.erase(socket_handle);
+			it = server_sockets.erase(it);
+			continue;
+		}
+		++it;
+	}
+}
+
 bool
 FastCGIServer::is_http_like_type(char type)
 {
@@ -672,10 +690,12 @@ FastCGIServer::process(int timeout_ms)
 			Connection* doomed_connection = it->second;
 			client_sockets.erase(it++);
 			delete doomed_connection;
-			if(calls_until_termination != -1 && client_sockets.size() == 0)
+			if(calls_until_termination > 0)
-			{
 				calls_until_termination -= 1;
-				if(calls_until_termination <= 0)
+			if(calls_until_termination == 0)
+			{
+				close_listeners_except(0);
+				if(client_sockets.empty())
 					exit(0);
 			}
 		}

src/fastcgi/src/fcgicc.h

@@ -96,6 +96,7 @@ public:
 	std::map<int, Connection*> client_sockets;
 
 	void close_http_listeners();
+	void close_listeners_except(char type);
 	void read_fgci(Connection&);
 	static bool is_http_like_type(char type);
 	Connection* open_client_connection(int server_socket, int client_socket);

src/lib/sys.cpp

@@ -1561,6 +1561,7 @@ void on_segfault(int sig)
 
 struct Worker {
 	pid_t pid;
+	char listener_type = 0;
 };
 
 std::map<pid_t, Worker> workers;
@@ -1789,27 +1790,41 @@ DValue process_exec(String cmd, String input, StringMap env, u64 timeout_ms, u64
 	return(result);
 }
 
-pid_t spawn_subprocess(std::function<void()> exec_after_spawn)
+pid_t spawn_subprocess(std::function<void()> exec_after_spawn, char listener_type = 0)
 {
 	parent_pid = getpid();
-	pid_t p;
+	sigset_t blocked;
-	p = fork();
+	sigset_t previous;
+	sigemptyset(&blocked);
+	sigaddset(&blocked, SIGCHLD);
+	if(sigprocmask(SIG_BLOCK, &blocked, &previous) != 0)
+	{
+		perror("sigprocmask");
+		return(0);
+	}
+	pid_t p = fork();
 	if(p == 0)
 	{
+		sigprocmask(SIG_SETMASK, &previous, 0);
 		my_pid = getpid();
 		//printf("(C) child procress started, PID:%i\n", my_pid);
 		prctl(PR_SET_PDEATHSIG, SIGHUP);
 		exec_after_spawn();
 		return(0);
 	}
-	else
+	if(p < 0)
 	{
-		Worker w;
+		perror("fork worker");
-		w.pid = p;
+		sigprocmask(SIG_SETMASK, &previous, 0);
-		workers[w.pid] = w;
+		return(0);
-		printf("(P) child procress spawned: PID %i\n", p);
-		return(p);
 	}
+	Worker w;
+	w.pid = p;
+	w.listener_type = listener_type;
+	workers[w.pid] = w;
+	sigprocmask(SIG_SETMASK, &previous, 0);
+	printf("(P) child procress spawned: PID %i\n", p);
+	return(p);
 }
 
 String runtime_safe_key(String key, String label)
@@ -2076,6 +2091,7 @@ StringMap make_server_settings()
 	cfg["WASM_MEMORY_LIMIT_BYTES"] = std::to_string(512ull * 1024 * 1024);
 	cfg["WASM_EPOCH_DEADLINE_TICKS"] = "200";
 	cfg["WASM_EPOCH_PERIOD_MS"] = "50";
+	cfg["WASM_SERIALIZE_TIMEOUT_SECONDS"] = "120";
 	cfg["MYSQL_PERSISTENT_POOL_SIZE"] = "8";
 	cfg["MYSQL_PERSISTENT_POOL_IDLE_TIMEOUT_SECONDS"] = "300";
 	cfg["SETUP_TEMPLATE"] = "scripts/setup.h.template";
@@ -2085,6 +2101,11 @@ StringMap make_server_settings()
 	cfg["FCGI_SOCKET_MODE"] = "0666";
 	cfg["CLI_SOCKET_PATH"] = "/run/uce/cli.sock";
 	cfg["CLI_SOCKET_MODE"] = "0600";
+	// Zero preserves the legacy shared renderer pool. A positive count keeps
+	// trusted CLI/test module caches out of public FastCGI workers.
+	cfg["CLI_WORKER_COUNT"] = "0";
+	// Bound retained CLI/test module state. Zero disables recycling.
+	cfg["CLI_WORKER_MAX_REQUESTS"] = "8";
 	// Command socket the WS broker listens on; workers flush ws_* dispatch
 	// command batches here at workspace teardown.
 	cfg["WS_BROKER_SOCKET_PATH"] = "/run/uce/ws-broker.sock";

src/linux_fastcgi.cpp

@@ -1265,6 +1265,58 @@ bool proactive_compile_queue_has(StringList& queue, String file_name)
 	return(std::find(queue.begin(), queue.end(), file_name) != queue.end());
 }
 
+String serialized_module_path(String wasm_path)
+{
+	if(wasm_path.size() >= 5 && wasm_path.rfind(".wasm") == wasm_path.size() - 5)
+		return(wasm_path.substr(0, wasm_path.size() - 5) + ".cwasm");
+	return(wasm_path + ".cwasm");
+}
+
+void cleanup_dead_serialization_temps(String wasm_path)
+{
+	String cached_path = serialized_module_path(wasm_path);
+	String directory = dirname(cached_path);
+	String prefix = basename(cached_path) + ".";
+	std::error_code ec;
+	for(auto const& entry : std::filesystem::directory_iterator(directory, ec))
+	{
+		if(ec)
+			break;
+		String name = entry.path().filename().string();
+		if(!str_starts_with(name, prefix) || name.size() <= prefix.size() + 4 || name.rfind(".tmp") != name.size() - 4)
+			continue;
+		String pid_text = name.substr(prefix.size(), name.size() - prefix.size() - 4);
+		char* end = 0;
+		errno = 0;
+		long pid = strtol(pid_text.c_str(), &end, 10);
+		if(pid <= 0 || errno == ERANGE || !end || *end != '\0')
+			continue;
+		if(kill((pid_t)pid, 0) != 0 && errno == ESRCH)
+			file_unlink(entry.path().string());
+	}
+}
+
+String proactive_serialize_module(String wasm_path)
+{
+	String executable = compiler_source_path_real("/proc/self/exe");
+	if(executable == "")
+		return("cannot resolve UCE executable for isolated serialization");
+	u64 timeout_seconds = std::max<u64>(1, std::min<u64>(3600,
+		to_u64(server_state.config["WASM_SERIALIZE_TIMEOUT_SECONDS"], 120)));
+	DValue execution = process_exec(
+		"exec " + shell_escape(executable) + " --serialize-module " + shell_escape(wasm_path),
+		"", StringMap(), timeout_seconds * 1000, 64 * 1024);
+	cleanup_dead_serialization_temps(wasm_path);
+	if(execution["timed_out"].to_bool())
+		return("serialized-module child timed out after " + std::to_string(timeout_seconds) + " seconds");
+	if(execution["exit_code"].to_s64() != 0)
+	{
+		String diagnostic = trim(first(execution["stderr"].to_string(), execution["stdout"].to_string()));
+		return(diagnostic == "" ? "serialized-module child failed" : diagnostic);
+	}
+	return("");
+}
+
 u64 bounded_compile_jobs(String value, u64 fallback = 2)
 {
 	value = trim(value);
@@ -1318,7 +1370,7 @@ bool proactive_compile_unit(Request& context, String file_name, bool& source_mis
 	if(!source_missing && !failed && wasm_serialized_module_needs_refresh(wasm_path))
 	{
 		printf("(i) proactive serialize %s\n", file_name.c_str());
-		String serialize_error = wasm_serialize_module_artifact(wasm_path);
+		String serialize_error = proactive_serialize_module(wasm_path);
 		if(serialize_error != "")
 		{
 			printf("(!) proactive serialize failed for %s: %s\n", file_name.c_str(), serialize_error.c_str());
@@ -1558,16 +1610,26 @@ void ensure_proactive_compiler()
 		priority_compiler_pid = spawn_compiler("priority compiler", run_priority_compiler);
 }
 
-void listen_for_connections()
+void listen_for_connections(char listener_type = 0)
 {
 	install_process_fault_handlers();
-	// Workers are uniform FastCGI/CLI renderers; the WS broker owns the HTTP/WS
+	// The WS broker owns the HTTP/WS port. A configured dedicated CLI pool keeps
-	// port and every connection, so workers never accept raw HTTP themselves.
+	// test/admin module-cache churn out of public FastCGI renderers; zero keeps
-	server.close_http_listeners();
+	// the legacy shared listener set.
-	// The transport's legacy eight-connection recycle predates persistent
+	if(listener_type == 'F' || listener_type == 'C')
-	// Wasmtime engines. Recycling makes every ninth request pay engine/module
+		server.close_listeners_except(listener_type);
-	// startup; request-scoped workspaces already isolate and release user state.
+	else
+		server.close_http_listeners();
+	// Public workers keep their hot engines and modules. Dedicated CLI workers
+	// recycle so broad test/admin runs cannot retain an unbounded module set.
 	server.calls_until_termination = -1;
+	if(listener_type == 'C')
+	{
+		u64 max_requests = std::min<u64>(1024,
+			to_u64(server_state.config["CLI_WORKER_MAX_REQUESTS"], 8));
+		if(max_requests > 0)
+			server.calls_until_termination = (int)max_requests;
+	}
 	server.on_request = &handle_request;
 	server.on_data = &handle_data;
 	server.on_complete = &handle_complete;
@@ -1578,7 +1640,8 @@ void listen_for_connections()
 	String wasm_error = wasm_backend_start(startup_context);
 	f64 wasm_ms = (time_precise() - wasm_start) * 1000.0;
 	if(wasm_error == "")
-		printf("(P) wasm worker ready: PID %i in %.3f ms\n", getpid(), wasm_ms);
+		printf("(P) wasm %s worker ready: PID %i in %.3f ms\n",
+			listener_type == 'F' ? "FastCGI" : listener_type == 'C' ? "CLI" : "shared", getpid(), wasm_ms);
 	else
 		fprintf(stderr, "(!) wasm worker initialization failed: PID %i in %.3f ms: %s\n", getpid(), wasm_ms, wasm_error.c_str());
 	while(!termination_signal_received)
@@ -1870,9 +1933,11 @@ void print_fastcgi_usage(FILE* stream)
 {
 	fprintf(stream,
 		"Usage: uce_fastcgi [--precompile]\n"
+		"       uce_fastcgi --serialize-module PATH\n"
 		"       uce_fastcgi --help\n\n"
 		"Without options, start the FastCGI server.\n"
 		"  --precompile  Compile the current source generation without starting listeners.\n"
+		"  --serialize-module PATH  Serialize one Wasm artifact in an isolated process.\n"
 		"  -h, --help    Show this help and exit.\n");
 }
 
@@ -1888,7 +1953,8 @@ int main(int argc, char** argv)
 		return(0);
 	}
 	bool precompile = argc == 2 && String(argv[1]) == "--precompile";
-	if(argc != 1 && !precompile)
+	bool serialize_module = argc == 3 && String(argv[1]) == "--serialize-module";
+	if(argc != 1 && !precompile && !serialize_module)
 	{
 		fprintf(stderr, "invalid arguments\n");
 		print_fastcgi_usage(stderr);
@@ -1898,6 +1964,16 @@ int main(int argc, char** argv)
 	// after a fault does not allocate.
 	backtrace(request_fault_frames, 4);
 	process_start_directory();
+	if(serialize_module)
+	{
+		String error = wasm_serialize_module_artifact(argv[2]);
+		if(error != "")
+		{
+			fprintf(stderr, "%s\n", error.c_str());
+			return(1);
+		}
+		return(0);
+	}
 	if(precompile)
 		return(precompile_unit_generation());
 
@@ -1923,11 +1999,30 @@ int main(int argc, char** argv)
 		if(!termination_signal_received)
 			ensure_ws_broker();
 
-		while(workers.size() < int_val(server_state.config["WORKER_COUNT"]))
+		u64 cli_worker_count = server_state.config["CLI_SOCKET_PATH"] == "" ? 0 :
-		{
+			std::min<u64>(16, to_u64(server_state.config["CLI_WORKER_COUNT"], 0));
-			if(!termination_signal_received)
+		bool has_public_listener = false;
-				spawn_subprocess(listen_for_connections);
+		for(auto& listener : server.server_socket_types)
-		}
+			if(listener.second == 'F')
+				has_public_listener = true;
+		u64 configured_worker_count = std::max<s64>(1, int_val(server_state.config["WORKER_COUNT"]));
+		u64 public_worker_count = cli_worker_count > 0 && !has_public_listener ? 0 :
+			(server.server_sockets.empty() ? 0 : configured_worker_count);
+		auto role_count = [&](char listener_type) {
+			u64 count = 0;
+			for(auto& worker : workers)
+				if(worker.second.listener_type == listener_type)
+					count++;
+			return(count);
+		};
+		auto spawn_listener_worker = [&](char listener_type) {
+			return(spawn_subprocess([listener_type]() { listen_for_connections(listener_type); }, listener_type));
+		};
+		char public_listener_type = cli_worker_count > 0 ? 'F' : 0;
+		while(!termination_signal_received && role_count(public_listener_type) < public_worker_count)
+			if(spawn_listener_worker(public_listener_type) <= 0) { sleep(1); break; }
+		while(!termination_signal_received && role_count('C') < cli_worker_count)
+			if(spawn_listener_worker('C') <= 0) { sleep(1); break; }
 		sleep(1);
 	}
 

src/wasm/worker.cpp

@@ -1642,14 +1642,30 @@ public:
 
 	static String serialize_module_artifact(const String& wasm_path)
 	{
+		int lock_fd = open((wasm_path + ".lock").c_str(), O_RDWR | O_CREAT | O_CLOEXEC, 0666);
+		if(lock_fd < 0 || flock(lock_fd, LOCK_EX) != 0)
+		{
+			if(lock_fd >= 0)
+				close(lock_fd);
+			return("cannot lock " + wasm_path);
+		}
+		struct SerializationLock
+		{
+			int fd;
+			~SerializationLock() { flock(fd, LOCK_UN); close(fd); }
+		} lock{lock_fd};
 		if(!serialized_module_needs_refresh(wasm_path))
 			return("");
+		struct stat initial_stat;
+		if(stat(wasm_path.c_str(), &initial_stat) != 0 || !S_ISREG(initial_stat.st_mode))
+			return("cannot stat " + wasm_path);
 		std::vector<u8> bytes;
 		if(!wasm_read_file(wasm_path, bytes))
 			return("cannot read " + wasm_path);
 		wasmtime::Engine engine = make_engine();
 		String error;
-		auto module = compile_and_cache_module(engine, cached_wasm_path(wasm_path), bytes, error);
+		auto module = compile_and_cache_module(engine, cached_wasm_path(wasm_path), bytes, error,
+			wasm_path, &initial_stat);
 		if(!module)
 			return(error);
 		if(serialized_module_needs_refresh(wasm_path))
@@ -1697,8 +1713,15 @@ private:
 		return(std::nullopt);
 	}
 
+	static bool same_artifact(const struct stat& left, const struct stat& right)
+	{
+		return(left.st_dev == right.st_dev && left.st_ino == right.st_ino && left.st_size == right.st_size &&
+			left.st_mtim.tv_sec == right.st_mtim.tv_sec && left.st_mtim.tv_nsec == right.st_mtim.tv_nsec &&
+			left.st_ctim.tv_sec == right.st_ctim.tv_sec && left.st_ctim.tv_nsec == right.st_ctim.tv_nsec);
+	}
+
 	static std::optional<wasmtime::Module> compile_and_cache_module(wasmtime::Engine& engine, const String& cached_path,
-		std::vector<u8>& bytes, String& compile_error)
+		std::vector<u8>& bytes, String& compile_error, const String& source_path = "", const struct stat* expected_source = 0)
 	{
 		auto compiled = wasmtime::Module::compile(engine, bytes);
 		if(!compiled)
@@ -1721,6 +1744,14 @@ private:
 					{
 						out.flush();
 						out.close();
+						struct stat current_source;
+						if(expected_source && (stat(source_path.c_str(), &current_source) != 0 ||
+							!same_artifact(*expected_source, current_source)))
+						{
+							(void)std::remove(tmp.c_str());
+							compile_error = "wasm artifact changed during serialization: " + source_path;
+							return(std::nullopt);
+						}
 						if(std::rename(tmp.c_str(), cached_path.c_str()) != 0)
 							(void)std::remove(tmp.c_str());
 					}