Compare commits

12 Commits
59 changed files with 2562 additions and 195 deletions
+12 -3
View File
@@ -1,6 +1,11 @@
# DOCS-TODO — `/doc` documentation overhaul
# Completed `/doc` documentation overhaul (historical plan)
Briefing for the doc-system overhaul. Read this fully before touching anything.
This June 2026 plan is complete and is retained only as implementation history.
It is not the current task queue and its one-off execution rules do not govern
new UCE work. Current documentation has 269 rendered pages and is covered by
the sharded doc gate in `scripts/run_cli_tests.sh`.
The original briefing follows.
The approved design and the two locked decisions are: **(1) examples are LIVE and
SELF-VERIFYING** — real UCE code executed at render time, showing source + actual
captured output, gated by the test suite; **(2) FULL SWEEP** of all 257 pages.
@@ -163,7 +168,11 @@ and the suite stays green.
```
Expect the current 91 to grow by the new doc gate(s), 0 failed.
## Hard rules (non-negotiable)
## Historical execution rules (no longer active)
These rules applied only while the overhaul was being carried out in June 2026.
They are preserved to explain that work and must not override current repository
or operator instructions.
- **NEVER `git commit`/push/tag.** Not pi, not its subagents, not anyone. Leave a clean
working tree and report.
+7 -6
View File
@@ -14,9 +14,9 @@ Implement the DOCS-TODO.md overhaul for the UCE `/doc` system on the sshfs sourc
## Current State
- Status: verifying; Phase 1 + gate complete, StringList conversion complete, Phase 2 example/filler sweep complete.
- Last updated: 2026-06-16
- Source of truth: `/root/mount_ssh/uce-dev-root-htdocs-uce`
- Status: complete; retained as the implementation record for the June 2026 overhaul.
- Last updated: 2026-07-20
- Source of truth: `/root/mount_ssh/uce-upstream`
- Runtime/live target: `root@10.4.2.110:/Code/uce.openfu.com/uce`
## Goal Tree
@@ -38,10 +38,10 @@ Legend: `[ ]` not started, `[~]` in progress, `[x]` done, `[!]` blocked, `[-]` s
- Done when: CLI runner checks every doc page and examples.
- Verify: full host gate passes.
## Execution Queue
## Completed Execution Queue
1. G2 area batching/delegation now that the mechanism is proven.
2. Keep the host suite green after each area batch.
1. Phase 2 area batching and delegation completed.
2. The host suite and sharded 269-page documentation gate remain the ongoing regression boundary.
## Decisions
@@ -55,6 +55,7 @@ Legend: `[ ]` not started, `[~]` in progress, `[x]` done, `[!]` blocked, `[-]` s
- 2026-06-16: Converted retiring `list_unique/list_sort/list_some/list_every/list_find` free functions to `StringList` methods, updated callers and method docs; full host gate passed: `Summary: 353 passed, 0 failed, 0 skipped`.
- 2026-06-16: Added live examples across types, string, regex, time, markup, noise, uri, and session area pages; full host gate passed: `Summary: 353 passed, 0 failed, 0 skipped`.
- 2026-06-16: Added `:example` blocks to all remaining doc pages, removed full PHP/JS Related Concepts filler, removed duplicate content code fences except resource-bound pages; full host gate passed: `Summary: 353 passed, 0 failed, 0 skipped`.
- 2026-07-20: Marked the finished plan historical so it no longer presents completed work or one-off execution rules as current instructions.
## Change Log
+44
View File
@@ -169,6 +169,13 @@ systemctl daemon-reload
systemctl restart uce.service
```
When `uce.socket` is active, restart only `uce.service` (or use
`scripts/systemd/manage-uce-service.sh restart`). The socket unit must retain
ownership of the FastCGI listener while workers change. Restarting
`uce.socket` alongside the service removes that listener briefly and can expose
HTTP 502 responses through nginx. Restart the socket unit only to repair the
socket itself, with an explicit outage window.
## Runtime configuration
Create `/etc/uce/settings.cfg` from `etc/uce/settings.cfg` and adjust paths to your checkout root. Replace checkout-specific settings such as `WASM_CORE_PATH` with `<UCE_REPO_ROOT>/bin/wasm/core.wasm` or leave it relative to `COMPILER_SYS_PATH` as appropriate.
@@ -185,6 +192,8 @@ FCGI_SOCKET_PATH=/run/uce/fastcgi.sock
FCGI_SOCKET_MODE=0666
CLI_SOCKET_PATH=/run/uce/cli.sock
CLI_SOCKET_MODE=0600
CLI_WORKER_COUNT=2
CLI_WORKER_MAX_REQUESTS=8
SITE_DIRECTORY=/var/www/html
HTTP_DOCUMENT_ROOT=/var/www/html
@@ -202,7 +211,9 @@ WASM_MEMORY_LIMIT_BYTES=536870912
WASM_EPOCH_DEADLINE_TICKS=200
WASM_EPOCH_PERIOD_MS=50
WASM_INVOCATION_TIMEOUT_MS=30000
WASM_SERIALIZE_TIMEOUT_SECONDS=120
MYSQL_PERSISTENT_POOL_SIZE=8
MYSQL_PERSISTENT_POOL_IDLE_TIMEOUT_SECONDS=300
WORKER_COUNT=4
MAX_MEMORY=16777216
@@ -216,6 +227,8 @@ Important settings:
- `FCGI_SOCKET_PATH` is the Unix socket used for normal `.uce` requests. Set it explicitly and keep this value and the web-server `fastcgi_pass` path identical. The reference config uses `/run/uce/fastcgi.sock`; if you choose `/run/uce.sock`, use it in both places.
- `CLI_SOCKET_PATH` is a local HTTP-over-Unix socket used by `scripts/uce-cli` and test/admin units. Keep it private (`CLI_SOCKET_MODE=0600`) unless you intentionally delegate admin/test execution to a trusted Unix group (`0660`).
- `CLI_WORKER_COUNT` adds a transport-isolated CLI/test renderer pool. Use at least `2` on a live site so broad test/admin unit loads cannot evict or page out public FastCGI workers' hot modules and a CLI unit can make one nested CLI call without self-deadlocking. `0` preserves the legacy shared pool; dedicated CLI workers are additional to `WORKER_COUNT` and retain their own Wasmtime engine, module cache, and persistent connector pool.
- `CLI_WORKER_MAX_REQUESTS` recycles each dedicated CLI worker after this many completed connections (default `8`, maximum `1024`); `0` disables recycling. Public FastCGI workers are not request-count recycled, preserving their warmed module caches. Recycling bounds the module set retained by broad test and administration runs while allowing one CLI invocation/test group to finish uninterrupted.
- `FCGI_SOCKET_MODE` and `CLI_SOCKET_MODE` are octal permission modes applied after socket bind. Prefer tightening `FCGI_SOCKET_MODE` to `0660` when nginx/Apache can share a trusted group with the UCE worker.
- `SITE_DIRECTORY` is the public site tree to scan for `.uce` files. Use `/var/www/html` when the web root is outside the runtime tree; relative paths are resolved from the runtime working directory. Installed regression gate scripts derive their temporary test root from this setting unless `UCE_TEST_SITE_DIRECTORY` is explicitly provided.
- `HTTP_DOCUMENT_ROOT` is the root used by the built-in HTTP/WebSocket listener when it resolves upgrade requests. Set it to the same web root as nginx/Apache.
@@ -225,6 +238,7 @@ Important settings:
- `TMP_UPLOAD_PATH` and `SESSION_PATH` must be writable by the runtime.
- `SESSION_COOKIE_SECURE=1` adds the `Secure` attribute to UCE-managed session cookies and should be used for HTTPS-only deployments. Leave it `0` only for local/plain-HTTP development.
- `MYSQL_PERSISTENT_POOL_SIZE` caps credential-keyed connections retained by each Wasm worker. The default `8` is clamped to `64`; set it to `0` to restore request-lifetime connections. Cached sessions are reset before reuse.
- `MYSQL_PERSISTENT_POOL_IDLE_TIMEOUT_SECONDS` retires a pooled connection at the first request boundary after it has been idle for this interval (default `300`, maximum `86400`). Set it to `0` to retain idle connections until capacity eviction, reset failure, worker exit, or database-side closure. Keep a positive value below the database server's idle timeout to avoid attempting stale sessions after traffic resumes. A completely idle UCE worker has no request boundary and therefore does not proactively close sockets.
- `HTTP_PORT` is the built-in HTTP/WebSocket listener used for WebSocket upgrade traffic and direct local probes. Bind/firewall it for local access only; nginx/Apache should be the public entry point.
- `WS_BROKER_OUTBOUND_TIMEOUT_SECONDS` controls how long a forwarded WS message can remain queued in the broker before being dropped (default `30`). Set to `0` to disable the timeout.
- `WASM_COMPILE_SCRIPT` must point to `scripts/compile_wasm_unit` unless you provide an equivalent compiler. Relative paths are resolved from the runtime root/`COMPILER_SYS_PATH`. That script calls `scripts/check_unit_wasm.py` after linking each unit and uses the pinned WASI SDK on every deployment host.
@@ -255,6 +269,15 @@ Important settings:
4 KiB positional read. Cache-miss full-artifact reads use the same checks
around 64 KiB positional chunks. A single in-progress regular-file syscall
cannot be cooperatively interrupted; later reads and parsing cannot overrun the budget.
Wasmtime's serialized-module deserialization call is also synchronous and
cannot be interrupted; if it stalls, the timeout is reported after that call
returns rather than at the nominal wall-clock boundary.
Proactive and offline serialization instead runs in a short-lived child
bounded by `WASM_SERIALIZE_TIMEOUT_SECONDS` (default `120`, clamped to
`1``3600`), so each unit releases its pooling allocator arenas and threads
on exit instead of accumulating them in the long-lived scanner. Serialization
holds the unit compile lock and rechecks the wasm artifact identity before
publication; timed-out/failed children leave no dead temporary artifact.
Initial/final descriptor identity, unique selected metadata sections, and
strict 64-bit LEB high-bit validation reject changed or ambiguous artifacts.
@@ -289,6 +312,7 @@ The server binary accepts only these process modes:
```bash
bin/uce_fastcgi.linux.bin # start the server
bin/uce_fastcgi.linux.bin --precompile
bin/uce_fastcgi.linux.bin --serialize-module /absolute/unit.uce.wasm
bin/uce_fastcgi.linux.bin --help
```
@@ -312,6 +336,12 @@ workers continue to use `/etc/uce/settings.cfg`. The parallel precompile
regression uses private roots so a running proactive compiler cannot consume
or publish its controlled race fixtures.
Direct `--precompile` is a listener-free whole-generation gate, not a
single-unit readiness probe. It scans the configured source root, waits for
unit publication locks, and has no internal wall-clock timeout. Automation
must wrap it in a finite external timeout, as the managed restart does, before
starting separately deadline-bounded CLI or HTTP invocations.
Prefer the managed restart when the service runs as an unprivileged user: it
precompiles as that same user. Request-time publication still accepts readable
artifacts produced by a trusted administrator. When Linux protected-hardlink
@@ -715,6 +745,20 @@ if(valid && password_needs_rehash(encoded))
`password_hash()` returns a self-contained `$uce$scrypt$...` encoding with a random 16-byte salt and the bounded scrypt parameters `N=65536`, `r=8`, `p=1`. `password_verify()` accepts only structurally valid encodings with bounded cost parameters and compares the derived key in constant time. `password_needs_rehash()` reports malformed, legacy, or non-current parameters so applications can upgrade a credential after a successful legacy verification. Treat an empty hash as an operational failure and never store it. Application-level password length policy, rate limiting, and legacy-format verification remain the application's responsibility.
## Structured cryptographic operations
`crypto_operation(request)` is the bounded, algorithm-selected API for structured
asymmetric operations. The initial allowlist supports `operation=key_generate`
and `operation=jwt_sign` with `algorithm=ES256`. Key generation returns P-256
public/private JWKs plus the RFC 7638 thumbprint (`kid`). JWT signing accepts only
a consistent P-256 private JWK, forces `alg=ES256`, and emits a compact JWT with
a 64-byte JOSE signature. It also supports `cbor_decode`, `cose_es256_parse`, and `es256_verify` with `algorithm=ES256`: inputs are canonical unpadded base64url fields; requests are capped at 32 KiB, while CBOR input is definite-length only and capped at 16 KiB decoded (21,846 base64url characters), 256 nodes, and depth 16; it rejects duplicate keys and trailing bytes, and returns an explicit typed tree (`unsigned`, `negative`, `bytes`, `text`, `array`, `map`) rather than DValue-coerced map keys. COSE accepts only EC2/-7/P-256 with exact 32-byte coordinates and an OpenSSL-valid point. Verification accepts only complete canonical DER ECDSA over `message_base64url` and returns `valid`; malformed material fails closed. V1 has no attestation, Ed25519, or RS256. Unknown operations and algorithms fail closed.
Existing typed digest, HMAC, password, randomness, and constant-time comparison
functions remain separate. `crypto_operation()` exposes no raw signing, arbitrary
digest/curve selection, encryption, network fetch, or token exchange. Store
private JWKs as secrets.
## Operational footguns
- Keep the FastCGI socket path consistent: `FCGI_SOCKET_PATH` and the web-server `fastcgi_pass` must match exactly. The reference config uses `/run/uce/fastcgi.sock`; if you choose `/run/uce.sock`, use it in both places.
+36 -7
View File
@@ -1,8 +1,8 @@
# UCE WASM Runtime Architecture
Status: current as of the W7e native-pipeline removal (June 2026). This document
describes the **runtime architecture as built** — the process topology, the
wasm membrane, the unified request dispatch, and the central WebSocket broker.
Status: current as of the worker-role isolation and bounded serializer changes
(July 2026). This document describes the **runtime architecture as built**. It
covers the process topology, wasm membrane, request dispatch, and WebSocket broker.
Native `.so` unit execution/dlopen fallback has been removed; the parser and
preprocessor remain only as the front-end that emits C++ for wasm side-module
compilation.
@@ -26,8 +26,8 @@ gets invoked*.
```
┌────────────────────────────┐
nginx ──FastCGI──► worker pool (N processes) │ $FCGI_SOCKET_PATH (example `/run/uce/fastcgi.sock`)
(port 80 etc.) │ uniform unit renderers (FastCGI + CLI)
nginx ──FastCGI──► public workers (N processes)│ $FCGI_SOCKET_PATH (example `/run/uce/fastcgi.sock`)
(port 80 etc.) │ public unit renderers
└─────────────▲──────────────┘
│ forward render (FastCGI, FCGI_SOCKET_PATH)
@@ -42,13 +42,19 @@ gets invoked*.
│ dispatcher(s) │ bind addr; forwards to pool
└──────────────────┘
trusted CLI/test calls ──► ┌──────────────────┐
(CLI socket) │ CLI workers │ isolated module and
│ (M processes) │ connector caches
└──────────────────┘
parent process: spawns/respawns all of the above + the proactive compiler.
```
| Process | Owns | Renders units? | Source |
|---|---|---|---|
| **Parent** | nothing; supervises children | no | `main()`, `init_base_process()` |
| **Worker** (×`WORKER_COUNT`) | `FCGI_SOCKET_PATH` (configured socket path; example `/run/uce/fastcgi.sock`) + `CLI_SOCKET_PATH` | **yes** — the only processes that run wasm | `listen_for_connections()` |
| **Public worker** (×`WORKER_COUNT`) | `FCGI_SOCKET_PATH`; also `CLI_SOCKET_PATH` only when `CLI_WORKER_COUNT=0` | **yes** — runs public FastCGI wasm | `listen_for_connections()` |
| **CLI worker** (×`CLI_WORKER_COUNT`) | `CLI_SOCKET_PATH` only | **yes** — isolates trusted CLI/test wasm and its module cache | `listen_for_connections()` |
| **WS broker** (×1) | `HTTP_PORT` + every live WS connection + `WS_BROKER_SOCKET_PATH` | no — forwards to the pool | `run_ws_broker()` |
| **serve_http dispatcher** (×bind) | one custom-server bind address | no — forwards to the pool | `custom_server_http_dispatcher_loop()` |
| **Proactive compiler** | nothing; pre-compiles units | no | `run_proactive_compiler()` |
@@ -192,6 +198,9 @@ the service and its post-fork workers restart. Direct launches without systemd
activation retain the existing configured Unix/TCP listener behavior.
The service preserves its runtime directory across service restarts because the
socket unit, not the service unit, owns a listener path inside that directory.
Normal deployments therefore restart only `uce.service`; restarting
`uce.socket` at the same time removes the listener and forfeits the zero-gap
handoff.
On termination the parent asks render workers to close their listeners, finish
accepted connections within the bounded worker drain interval, and only then
exits. This prevents an accepted FastCGI request from being reset at handoff;
@@ -211,6 +220,14 @@ hostcall loop therefore cannot renew an invocation indefinitely. Blocking host
helpers retain operation-specific limits and cap them to the remaining
invocation budget where the underlying operation is cancellable. Forked task
callbacks receive a fresh invocation deadline capped by the task lifetime.
The Memcached ASCII hostcall applies one deadline to the complete command write
and protocol-framed response read. Retrieval payload lengths are parsed, so a
payload containing `END` cannot terminate the response; multi-value and `stats`
responses may span socket chunks up to the 8 MiB response bound. Timeout,
malformed framing, or overflow shuts down the retained descriptor before
returning an empty result, preventing unread response bytes from poisoning the
next command on that connection. The native fallback uses the same parser and
deadline contract.
Synchronous compiler locks, transitive `#load` compilation, and compiler child
processes consume that same deadline. Compiler children run in a dedicated
process group; timeout kills the group, retains the previous generation, and
@@ -266,6 +283,10 @@ Cold module compilation and deserialization are host work, so `load_unit()`
refreshes the epoch deadline before its first guest call. Otherwise a component
whose compilation outlasted the guest CPU budget would immediately trap in the
following allocator/relocation call even though no guest loop consumed it.
Wasmtime's serialized-module deserialization API is synchronous and has no
cooperative cancellation point. The absolute invocation timeout is checked
immediately after it returns, so a stalled deserialization may overrun the
nominal wall boundary before UCE reports the timeout.
Unit artifacts live beneath an ABI-generation directory such as
`BIN_DIRECTORY/units-c13-w7`: `c13` is the compiler/unit-metadata ABI and `w7`
@@ -553,7 +574,15 @@ header free-functions are `inline`. The wasm backend exposes only declarations
| `HTTP_PORT` | `8080` | Raw HTTP + WebSocket port — owned by the WS broker. |
| `WS_BROKER_SOCKET_PATH` | `/run/uce/ws-broker.sock` | Broker command socket for `ws_*` flushes. |
| `WS_BROKER_OUTBOUND_TIMEOUT_SECONDS` | `30` | Max lifetime in seconds for queued WS broker forwards before drop. |
| `WORKER_COUNT` | `4` | Number of uniform worker processes. |
| `WORKER_COUNT` | `4` | Number of public FastCGI worker processes. |
| `CLI_WORKER_COUNT` | `0` built-in; `2` in the reference config | Additional CLI-only workers. Two permit one nested CLI invocation without self-deadlock; a positive count prevents test/admin module-cache churn from paging out public workers, while zero preserves the legacy shared pool. |
| `CLI_WORKER_MAX_REQUESTS` | `8` | Completed CLI connections before a dedicated CLI worker is recycled (maximum `1024`; `0` disables). Public workers are not request-count recycled. |
| `WASM_SERIALIZE_TIMEOUT_SECONDS` | `120` | Deadline for each short-lived serialized-module child; process exit reclaims pooling allocator arenas from proactive scanners. |
Proactive scanners never construct a Wasmtime serialization engine themselves.
Each candidate is serialized by the executable's bounded `--serialize-module`
child while holding the unit compile lock. Publication rechecks device, inode,
size, mtime, and ctime, and the scanner removes dead child temporary artifacts.
---
+9
View File
@@ -14,6 +14,12 @@ FCGI_PORT=9993
CLI_SOCKET_PATH=/run/uce/cli.sock
# Keep the CLI/admin socket private by default; set 0660 only for a trusted group.
CLI_SOCKET_MODE=0600
# Keep CLI/test module caches out of public FastCGI workers. Set to 0 only for
# the legacy shared pool; dedicated workers are additional to WORKER_COUNT.
CLI_WORKER_COUNT=2
# Recycle dedicated CLI workers before broad test/admin runs retain every unit.
# Zero disables recycling; public FastCGI workers are never request-count recycled.
CLI_WORKER_MAX_REQUESTS=8
# Built-in HTTP/WebSocket listener used for WebSocket Upgrade requests.
# Keep this behind nginx/Apache on localhost or firewall it from public access.
@@ -59,7 +65,10 @@ WASM_MEMORY_LIMIT_BYTES=536870912
WASM_EPOCH_DEADLINE_TICKS=200
WASM_EPOCH_PERIOD_MS=50
WASM_INVOCATION_TIMEOUT_MS=30000
# Bound each short-lived proactive/precompile serialized-module child.
WASM_SERIALIZE_TIMEOUT_SECONDS=120
MYSQL_PERSISTENT_POOL_SIZE=8
MYSQL_PERSISTENT_POOL_IDLE_TIMEOUT_SECONDS=300
# ENABLE THE BACKGROUND PROACTIVE COMPILER LOOP
PROACTIVE_COMPILE_ENABLED=1
+1
View File
@@ -39,6 +39,7 @@ PUBLIC_APIS = [
("hmac_sha256", True, "public"), ("hmac_sha256_hex", True, "public"), ("random_bytes", True, "public"),
("crypto_equal", True, "public"), ("password_hash", True, "public"),
("password_verify", True, "public"), ("password_needs_rehash", True, "public"),
("crypto_operation", True, "public"),
("gen_noise32", True, "public"), ("gen_noise64", True, "public"),
("gen_noise01", True, "public"), ("gen_int", True, "public"), ("gen_float", True, "public"),
("draw_int", True, "public"), ("draw_float", True, "public"),
+5
View File
@@ -67,6 +67,7 @@ if [[ "$action" == "run" ]]; then
curl -sS --max-time "$curl_timeout" --fail-with-body --unix-socket "$socket_path" "${base_url}&group=${group}"
done
scripts/test_dependency_invalidation.sh
scripts/test_compiler_lock_directory.sh
scripts/test_abi_generation_rollout.sh
scripts/test_parallel_precompile.sh
timeout --signal=TERM --kill-after=5s 175s scripts/test_parallel_proactive_compile.sh
@@ -77,8 +78,11 @@ if [[ "$action" == "run" ]]; then
scripts/test_component_once_prefetch.sh
scripts/test_relative_component_cache.sh
scripts/test_password_hashing.sh
scripts/test_crypto_operation_native.sh
scripts/test_hardened_http_native.sh
scripts/test_mysql_epoch_refresh.sh
scripts/test_mysql_persistent_pool.sh
scripts/test_mysql_persistent_pool_idle.sh
scripts/test_log_timeliness.sh
scripts/test_raw_http_request_log.sh
scripts/test_component_resolution_ttl.sh
@@ -89,5 +93,6 @@ if [[ "$action" == "run" ]]; then
timeout --signal=TERM --kill-after=5s 240s scripts/test_dynamic_compile_failures.sh
scripts/test_wasm_source_locations.sh
scripts/test_server_arguments.sh
scripts/test_cli_worker_isolation.sh
scripts/test_socket_activation.sh
fi
+153
View File
@@ -0,0 +1,153 @@
#!/usr/bin/env bash
set -euo pipefail
cd "$(dirname "$0")/.."
if [[ "${1:-}" != "--inside" ]]; then
exec timeout --signal=TERM --kill-after=5s 120s unshare --mount --fork --kill-child=TERM "$0" --inside
fi
root="/tmp/uce-cli-worker-isolation-$$"
site="$root/site"
work="$root/work"
settings="$root/settings.cfg"
log="$root/service.log"
cli_socket="$root/run/cli.sock"
fastcgi_socket="$root/run/fastcgi.sock"
server_pid=""
cleanup() {
status=$?
if [[ -n "$server_pid" ]] && kill -0 "$server_pid" 2>/dev/null; then
kill -TERM "$server_pid" 2>/dev/null || true
deadline=$((SECONDS + 10))
while kill -0 "$server_pid" 2>/dev/null && (( SECONDS < deadline )); do sleep 0.05; done
if kill -0 "$server_pid" 2>/dev/null; then kill -KILL "$server_pid" 2>/dev/null || true; fi
wait "$server_pid" 2>/dev/null || true
fi
if (( status != 0 )) && [[ -r "$log" ]]; then cat "$log" >&2; fi
rm -rf "$root"
return "$status"
}
trap cleanup EXIT
mkdir -p "$site" "$work" "$root/run" "$root/session" "$root/upload"
cp /etc/uce/settings.cfg "$settings"
cat >>"$settings" <<CFG
BIN_DIRECTORY=$work
PRECOMPILE_FILES_IN=$site
SITE_DIRECTORY=$site
FCGI_SOCKET_PATH=$fastcgi_socket
FCGI_PORT=
CLI_SOCKET_PATH=$cli_socket
CLI_WORKER_COUNT=2
CLI_WORKER_MAX_REQUESTS=3
WS_BROKER_SOCKET_PATH=$root/run/ws.sock
HTTP_PORT=
HTTP_DOCUMENT_ROOT=$site
SESSION_PATH=$root/session
TMP_UPLOAD_PATH=$root/upload
WASM_CORE_PATH=$(pwd)/bin/wasm/core.wasm
WORKER_COUNT=2
PROACTIVE_COMPILE_ENABLED=0
CFG
mount --bind "$settings" /etc/uce/settings.cfg
cat >"$site/isolation.uce" <<'UCE'
RENDER(Request& context)
{
print(request_perf()["worker_pid"].to_string());
}
CLI(Request& context)
{
print(request_perf()["worker_pid"].to_string());
}
UCE
cat >"$site/nested.uce" <<UCE
CLI(Request& context)
{
print(shell_exec("cd $(pwd) && timeout 10s scripts/uce-cli --socket $cli_socket /isolation.uce"));
}
UCE
bin/uce_fastcgi.linux.bin >"$log" 2>&1 &
server_pid=$!
deadline=$((SECONDS + 20))
while [[ ! -S "$cli_socket" ]] && (( SECONDS < deadline )); do sleep 0.05; done
[[ -S "$cli_socket" ]] || { echo "private UCE CLI socket was not ready" >&2; exit 1; }
timeout --signal=TERM --kill-after=1s 30s scripts/uce-cli --socket "$cli_socket" /isolation.uce >/dev/null
timeout --signal=TERM --kill-after=1s 30s scripts/uce-cli --socket "$cli_socket" /nested.uce | grep -Eq '^[0-9]+([.]0+)?$'
fastcgi_request() {
SCRIPT_FILENAME="$site/isolation.uce" SCRIPT_NAME=/isolation.uce REQUEST_URI=/isolation.uce REQUEST_METHOD=GET DOCUMENT_ROOT="$site" \
timeout --signal=TERM --kill-after=1s 10s cgi-fcgi -bind -connect "$fastcgi_socket" | tr -d '\r' | awk 'NF{last=$0} END{print last}'
}
fastcgi_request >/dev/null
cli_request() {
local output
local deadline=$((SECONDS + 20))
while (( SECONDS < deadline )); do
if output=$(timeout --signal=TERM --kill-after=1s 10s scripts/uce-cli --socket "$cli_socket" /isolation.uce 2>/dev/null); then
printf '%s\n' "$output"
return 0
fi
sleep 0.05
done
return 1
}
cli_pids=()
http_pids=()
# Concurrent completions must count individually and retire both initial CLI
# workers despite neither connection pool necessarily becoming idle first.
concurrent_jobs=()
for n in $(seq 1 12); do
cli_request >"$root/concurrent-$n" & concurrent_jobs+=("$!")
done
for job in "${concurrent_jobs[@]}"; do wait "$job"; done
mapfile -t concurrent_cli < <(cat "$root"/concurrent-* | sort -u)
cli_pids+=("${concurrent_cli[@]}")
for _ in $(seq 1 12); do
cli_pids+=("$(cli_request)")
http_pids+=("$(fastcgi_request)")
done
mapfile -t unique_cli < <(printf '%s\n' "${cli_pids[@]}" | sort -u)
mapfile -t unique_http < <(printf '%s\n' "${http_pids[@]}" | sort -u)
[[ ${#unique_cli[@]} -ge 4 ]] || { echo "CLI workers did not recycle after three requests: ${unique_cli[*]}" >&2; exit 1; }
[[ ${#unique_http[@]} -ge 1 ]] || { echo "HTTP requests reached no public worker" >&2; exit 1; }
for cli_pid in "${unique_cli[@]}"; do
if printf '%s\n' "${unique_http[@]}" | grep -Fxq "$cli_pid"; then
echo "CLI worker also served public HTTP: $cli_pid" >&2
exit 1
fi
done
[[ $(grep -c 'wasm FastCGI worker ready' "$log") -eq 2 ]]
[[ $(grep -c 'wasm CLI worker ready' "$log") -ge 4 ]]
echo "CLI worker isolation and recycling passed: CLI ${unique_cli[*]}, public ${unique_http[*]}"
# A CLI-only runtime must not allocate a listener-less public Wasmtime worker.
kill -TERM "$server_pid"
wait "$server_pid"
server_pid=""
rm -f "$cli_socket" "$fastcgi_socket"
cat >>"$settings" <<CFG
FCGI_SOCKET_PATH=
CLI_WORKER_COUNT=1
CLI_WORKER_MAX_REQUESTS=0
CFG
: >"$log"
bin/uce_fastcgi.linux.bin >"$log" 2>&1 &
server_pid=$!
deadline=$((SECONDS + 20))
while [[ ! -S "$cli_socket" ]] && (( SECONDS < deadline )); do sleep 0.05; done
[[ -S "$cli_socket" ]] || { echo "CLI-only socket was not ready" >&2; exit 1; }
cli_request >/dev/null
[[ $(grep -c 'wasm CLI worker ready' "$log") -eq 1 ]]
[[ $(grep -c 'wasm FastCGI worker ready' "$log") -eq 0 ]]
echo "CLI-only worker allocation passed"
+55
View File
@@ -0,0 +1,55 @@
#!/usr/bin/env bash
set -euo pipefail
cd "$(dirname "$0")/.."
if [[ "${1:-}" != "--inside" ]]; then
exec timeout --signal=TERM --kill-after=5s 90s unshare --mount --fork --kill-child=TERM "$0" --inside
fi
root="/tmp/uce-compiler-lock-directory-$$"
site="$root/site"
work="$root/missing/parents/work"
settings="$root/settings.cfg"
log="$root/service.log"
server_pid=""
cleanup() {
if [[ -n "$server_pid" ]] && kill -0 "$server_pid" 2>/dev/null; then
kill -TERM "$server_pid" 2>/dev/null || true
wait "$server_pid" 2>/dev/null || true
fi
rm -rf "$root"
}
trap cleanup EXIT
mkdir -p "$site" "$root/run" "$root/session" "$root/upload"
printf '%s\n' 'CLI(Request& context) { print("recursive-lock-directory-ok"); }' >"$site/test.uce"
cp /etc/uce/settings.cfg "$settings"
cat >>"$settings" <<CFG
BIN_DIRECTORY=$work
PRECOMPILE_FILES_IN=$site
SITE_DIRECTORY=$site
FCGI_SOCKET_PATH=$root/run/fastcgi.sock
FCGI_PORT=
CLI_SOCKET_PATH=$root/run/cli.sock
WS_BROKER_SOCKET_PATH=$root/run/ws.sock
HTTP_PORT=
HTTP_DOCUMENT_ROOT=$site
SESSION_PATH=$root/session
TMP_UPLOAD_PATH=$root/upload
WASM_CORE_PATH=$(pwd)/bin/wasm/core.wasm
WORKER_COUNT=1
PROACTIVE_COMPILE_ENABLED=0
CFG
mount --bind "$settings" /etc/uce/settings.cfg
bin/uce_fastcgi.linux.bin >"$log" 2>&1 &
server_pid=$!
deadline=$((SECONDS + 20))
while [[ ! -S "$root/run/cli.sock" ]] && (( SECONDS < deadline )); do sleep 0.05; done
[[ -S "$root/run/cli.sock" ]] || { cat "$log" >&2; exit 1; }
rm -rf "$work"
response=$(curl -sS --max-time 45 --fail-with-body --unix-socket "$root/run/cli.sock" http://localhost/test.uce) || { cat "$log" >&2; exit 1; }
[[ "$response" == *"recursive-lock-directory-ok"* ]] || { printf '%s\n' "$response" >&2; cat "$log" >&2; exit 1; }
find "$work" -type f -name '*.lock' -print -quit | grep -q . || { echo "compiler did not recreate a nested lock path" >&2; cat "$log" >&2; exit 1; }
echo "compiler recursive lock directory passed"
+176
View File
@@ -0,0 +1,176 @@
#!/usr/bin/env bash
set -euo pipefail
cd "$(dirname "$0")/.."
test_source="/tmp/uce-oauth-es256-native-$$.cpp"
test_binary="/tmp/uce-oauth-es256-native-$$"
cleanup() { rm -f "$test_source" "$test_binary"; }
trap cleanup EXIT
cat >"$test_source" <<'EOF'
#include "src/lib/types.cpp"
#include "src/lib/dvalue.cpp"
#include "src/lib/functionlib.cpp"
#include <openssl/bn.h>
#include <openssl/core_names.h>
#include <openssl/ecdsa.h>
#include <openssl/evp.h>
#include <openssl/params.h>
#include <memory>
String base64_encode(String raw)
{
if(raw.empty()) return("");
String out(4 * ((raw.size() + 2) / 3), 0);
int size = EVP_EncodeBlock((unsigned char*)out.data(), (const unsigned char*)raw.data(), (int)raw.size());
out.resize(size > 0 ? (size_t)size : 0);
return(out);
}
String base64_decode(String raw, bool& ok)
{
ok = false;
if(raw.empty() || raw.size() % 4) return("");
String out(3 * raw.size() / 4, 0);
int size = EVP_DecodeBlock((unsigned char*)out.data(), (const unsigned char*)raw.data(), (int)raw.size());
if(size < 0) return("");
while(!raw.empty() && raw.back() == '=') { size--; raw.pop_back(); }
out.resize((size_t)size); ok = true; return(out);
}
#include "src/lib/hash.cpp"
static String b64url_encode(String text)
{
String out=replace(replace(base64_encode(text),"+","-"),"/","_"); while(!out.empty()&&out.back()=='=') out.pop_back(); return out;
}
static String b64url_decode(String text)
{
text = replace(replace(text, "-", "+"), "_", "/");
while(text.size() % 4) text += "=";
bool ok = false;
String result = base64_decode(text, ok);
return(ok ? result : String(""));
}
static bool verify(DValue public_jwk, String jwt)
{
StringList parts = split(jwt, ".");
if(parts.size() != 3) return(false);
String x = b64url_decode(public_jwk["x"].to_string());
String y = b64url_decode(public_jwk["y"].to_string());
String raw = b64url_decode(parts[2]);
if(x.size() != 32 || y.size() != 32 || raw.size() != 64) return(false);
unsigned char point[65] = {4}; memcpy(point + 1, x.data(), 32); memcpy(point + 33, y.data(), 32);
OSSL_PARAM params[] = { OSSL_PARAM_construct_utf8_string(OSSL_PKEY_PARAM_GROUP_NAME, (char*)"prime256v1", 0), OSSL_PARAM_construct_octet_string(OSSL_PKEY_PARAM_PUB_KEY, point, sizeof(point)), OSSL_PARAM_construct_end() };
EVP_PKEY_CTX* build = EVP_PKEY_CTX_new_from_name(0, "EC", 0); EVP_PKEY* key = 0;
if(!build || EVP_PKEY_fromdata_init(build) <= 0 || EVP_PKEY_fromdata(build, &key, EVP_PKEY_PUBLIC_KEY, params) <= 0) { EVP_PKEY_CTX_free(build); return(false); }
EVP_PKEY_CTX_free(build);
BIGNUM* r = BN_bin2bn((const unsigned char*)raw.data(), 32, 0); BIGNUM* s = BN_bin2bn((const unsigned char*)raw.data() + 32, 32, 0);
ECDSA_SIG* sig = ECDSA_SIG_new(); int der_size = r && s && sig && ECDSA_SIG_set0(sig, r, s) ? i2d_ECDSA_SIG(sig, 0) : 0; r = s = 0;
String der(der_size > 0 ? (size_t)der_size : 0, 0); unsigned char* out = (unsigned char*)der.data();
bool ok = der_size > 0 && i2d_ECDSA_SIG(sig, &out) == der_size;
EVP_MD_CTX* verify_ctx = EVP_MD_CTX_new();
String signing_input = parts[0] + "." + parts[1];
ok = ok && verify_ctx && EVP_DigestVerifyInit(verify_ctx, 0, EVP_sha256(), 0, key) > 0 && EVP_DigestVerify(verify_ctx, (const unsigned char*)der.data(), der.size(), (const unsigned char*)signing_input.data(), signing_input.size()) == 1;
EVP_MD_CTX_free(verify_ctx); ECDSA_SIG_free(sig); EVP_PKEY_free(key); return(ok);
}
int main()
{
DValue key_request; key_request["operation"] = "key_generate"; key_request["algorithm"] = "ES256";
DValue key = crypto_operation_native(key_request);
DValue header; header["alg"] = "none"; header["kid"] = key["kid"]; DValue claims; claims["iss"] = "https://client.example";
auto sign = [&](DValue private_jwk) { DValue request; request["operation"] = "jwt_sign"; request["algorithm"] = "ES256"; request["private_jwk"] = private_jwk; request["protected_header"] = header; request["claims"] = claims; return(crypto_operation_native(request)); };
DValue signed_result = sign(key["private_jwk"]); String jwt = signed_result["jwt"].to_string();
DValue wrong_curve = key["private_jwk"]; wrong_curve["crv"] = "P-384";
DValue malformed = key["private_jwk"]; malformed["x"] = "bad=";
DValue mismatch = key["private_jwk"]; String d = mismatch["d"].to_string(); d[0] = d[0] == 'A' ? 'B' : 'A'; mismatch["d"] = d;
const String cose="pQECAyYgASFYIGsX0fLhLEJH-Lzm5WOkQPJ3A32BLeszoPShOUXYmMKWIlggT-NC4v4af5uO5-tKfA-eFivOM1drMV7Oy7ZAaDe_UfU", message="d2ViYXV0aG4gZml4ZWQgbWVzc2FnZQ", signature="MEUCIQCkatZK1VVsjk17uvyzyhjdAkMNWXPjxSOMqWcjmM_8XAIgDaSk3Qufyd0_6r9Dm9A8RQbFco-FdTBulq7bvRGoBC4";
auto op = [&](String operation) { DValue r; r["operation"]=operation; r["algorithm"]="ES256"; return r; };
auto parse = [&](String key) { DValue r=op("cose_es256_parse"); r["cose_key_base64url"]=key; return crypto_operation_native(r); };
auto verify_cose = [&](String key,String msg,String sig) { DValue r=op("es256_verify"); r["cose_key_base64url"]=key; r["message_base64url"]=msg; r["signature_der_base64url"]=sig; return crypto_operation_native(r); };
auto decode_cbor = [&](String bytes) { DValue r=op("cbor_decode"); r["cbor_base64url"]=b64url_encode(bytes); return crypto_operation_native(r); };
DValue unsupported; unsupported["operation"] = "encrypt"; unsupported["algorithm"] = "ES256";
DValue unknown_algorithm; unknown_algorithm["operation"] = "key_generate"; unknown_algorithm["algorithm"] = "none";
DValue untyped_algorithm = key_request; untyped_algorithm["algorithm"] = (f64)256;
DValue list_header; list_header.set_array(); DValue list_item; list_item = "not-an-object"; list_header.push(list_item); DValue list_request; list_request["operation"] = "jwt_sign"; list_request["algorithm"] = "ES256"; list_request["private_jwk"] = key["private_jwk"]; list_request["protected_header"] = list_header; list_request["claims"] = claims;
DValue control_request = list_request; control_request["protected_header"] = header; control_request["claims"] = claims; control_request["claims"]["bad"] = String("control\nbyte");
DValue oversized = key_request; oversized["ignored"] = String(33000, 'x');
DValue nonfinite = key_request; nonfinite["ignored"] = std::numeric_limits<f64>::quiet_NaN();
String tampered = jwt; if(tampered.size() > 2) tampered[tampered.size() - 2] = tampered[tampered.size() - 2] == 'A' ? 'B' : 'A';
bool kid_ok = key["ok"].to_bool() && key["kid"].to_string() == key["thumbprint"].to_string();
bool signed_ok = signed_result["ok"].to_bool() && jwt != "" && verify(key["public_jwk"], jwt);
bool tamper_ok = !verify(key["public_jwk"], tampered);
auto cbor_error = [&](String bytes) { return(decode_cbor(bytes)["error"].to_string() == "invalid_cbor"); };
auto cose_error = [&](String encoded) { return(parse(encoded)["error"].to_string() == "invalid_cose_key"); };
String cose_raw = b64url_decode(cose);
String wrong_kty = cose_raw; wrong_kty[2] = 1;
String wrong_alg = cose_raw; wrong_alg[4] = 0x27;
String wrong_cose_curve = cose_raw; wrong_cose_curve[6] = 2;
String missing_label = cose_raw; missing_label[0] = 0xa4; missing_label.resize(missing_label.size() - 35);
String short_x = String("\xa5\x01\x02\x03\x26\x20\x01\x21\x58\x1f",10) + String(31, 'x') + String("\x22\x58\x20", 3) + String(32, 'y');
String duplicate_label = cose_raw; duplicate_label[0] = 0xa6; duplicate_label += String("\x01\x02", 2);
String invalid_point = String("\xa5\x01\x02\x03\x26\x20\x01\x21\x58\x20", 10) + String(32, 'x') + String("\x22\x58\x20", 3) + String(32, 'y');
String der = b64url_decode(signature);
String tampered_der = der; tampered_der[10] ^= 1;
String trailing_der = der + String("\x00", 1);
String noncanonical_der = der; noncanonical_der[1]++; noncanonical_der[3]++; noncanonical_der.insert(4, 1, '\0');
DValue valid_verify = verify_cose(cose, message, signature);
DValue tampered_signature = verify_cose(cose, message, b64url_encode(tampered_der));
DValue tampered_message = verify_cose(cose, b64url_encode("tampered"), signature);
DValue malformed_der = verify_cose(cose, message, b64url_encode(String("\x30\x00", 2)));
DValue noncanonical_der_result = verify_cose(cose, message, b64url_encode(noncanonical_der));
DValue trailing_der_result = verify_cose(cose, message, b64url_encode(trailing_der));
DValue invalid_point_result = verify_cose(b64url_encode(invalid_point), message, signature);
String large_cbor = String("\x58\x81", 2) + String(129, 'x');
String oversized_cbor = String("\x5a\x00\x00\x40\x00", 5) + String(16384, 'x');
String duplicate_compound = String("\xa2\x82\x41[\x41]\x01\x82\x41[\x41]\x02", 13);
String distinct_compound = String("\xa2\x82\x41[\x41]\x01\x82\x41[\x41[\x02", 13);
String node_overflow = String("\x99\x01\x00", 3) + String(256, 0);
String decoded;
bool cbor_valid = decode_cbor(String("\x82\x01\x62ok", 5))["ok"].to_bool();
bool cbor_large = decode_cbor(large_cbor)["ok"].to_bool();
bool cbor_control_text = decode_cbor(String("\x61\x01", 2))["ok"].to_bool();
bool cbor_duplicate = cbor_error(String("\xa2\x01\x02\x01\x03", 5));
bool cbor_compound_duplicate = cbor_error(duplicate_compound);
bool cbor_compound_distinct = decode_cbor(distinct_compound)["ok"].to_bool();
bool cbor_invalid_utf8 = cbor_error(String("\x61\x80", 2));
bool cbor_truncated = cbor_error(String("\xa1", 1));
bool cbor_trailing = cbor_error(String("\x01\x02", 2));
bool cbor_depth = cbor_error(String(17, '\x81') + "\x00");
bool cbor_nodes = cbor_error(node_overflow);
bool cbor_size = cbor_error(oversized_cbor);
bool cbor_indefinite = cbor_error(String("\x9f\x01\xff", 3));
bool cbor_integer_overflow = cbor_error(String("\x5b\xff\xff\xff\xff\xff\xff\xff\xff", 9));
bool cbor_nonminimal = cbor_error(String("\x18\x17", 2));
bool b64_padding = !uce_base64url_decode(cose + "=", decoded, UCE_CBOR_MAX_BASE64URL);
bool b64_truncated = !uce_base64url_decode("A", decoded, UCE_CBOR_MAX_BASE64URL);
bool b64_trailing_bits = !uce_base64url_decode("AB", decoded, UCE_CBOR_MAX_BASE64URL);
bool cose_valid = parse(cose)["ok"].to_bool();
bool cose_wrong_kty = cose_error(b64url_encode(wrong_kty));
bool cose_wrong_alg = cose_error(b64url_encode(wrong_alg));
bool cose_wrong_curve = cose_error(b64url_encode(wrong_cose_curve));
bool cose_short_coordinate = cose_error(b64url_encode(short_x));
bool cose_missing_label = cose_error(b64url_encode(missing_label));
bool cose_duplicate_label = cose_error(b64url_encode(duplicate_label));
bool cose_invalid_point = invalid_point_result["error"].to_string() == "invalid_key_or_payload";
bool verify_valid = valid_verify["ok"].to_bool() && valid_verify["valid"].to_bool();
bool verify_tampered_signature = tampered_signature["ok"].to_bool() && !tampered_signature["valid"].to_bool();
bool verify_tampered_message = tampered_message["ok"].to_bool() && !tampered_message["valid"].to_bool();
bool verify_malformed_der = malformed_der["error"].to_string() == "invalid_signature";
bool verify_noncanonical_der = noncanonical_der_result["error"].to_string() == "invalid_signature";
bool verify_trailing_der = trailing_der_result["error"].to_string() == "invalid_signature";
bool cbor_ok = cbor_valid && cbor_large && cbor_control_text && cbor_duplicate && cbor_compound_duplicate && cbor_compound_distinct && cbor_invalid_utf8 && cbor_truncated && cbor_trailing && cbor_depth && cbor_nodes && cbor_size && cbor_indefinite && cbor_integer_overflow && cbor_nonminimal;
bool cose_ok = cose_valid && cose_wrong_kty && cose_wrong_alg && cose_wrong_curve && cose_short_coordinate && cose_missing_label && cose_duplicate_label && cose_invalid_point;
bool cose_negative = verify_valid && verify_tampered_signature && verify_tampered_message && verify_malformed_der && verify_noncanonical_der && verify_trailing_der;
bool b64_ok = b64_padding && b64_truncated && b64_trailing_bits;
bool negatives_ok = !sign(wrong_curve)["ok"].to_bool() && !sign(malformed)["ok"].to_bool() && !sign(mismatch)["ok"].to_bool() && crypto_operation_native(unsupported)["error"].to_string() == "unsupported_operation" && crypto_operation_native(unknown_algorithm)["error"].to_string() == "unsupported_algorithm" && crypto_operation_native(untyped_algorithm)["error"].to_string() == "invalid_request" && crypto_operation_native(list_request)["error"].to_string() == "invalid_key_or_payload" && crypto_operation_native(control_request)["error"].to_string() == "invalid_request" && crypto_operation_native(oversized)["error"].to_string() == "invalid_request" && crypto_operation_native(nonfinite)["error"].to_string() == "invalid_request";
if(!(kid_ok && signed_ok && tamper_ok && negatives_ok && cose_ok && cbor_ok && cose_negative && b64_ok)) std::cerr << "kid=" << kid_ok << " signed=" << signed_ok << " tamper=" << tamper_ok << " negatives=" << negatives_ok << " cbor=" << cbor_ok << " cose=" << cose_ok << " verify=" << cose_negative << " b64=" << b64_ok << " large=" << cbor_large << " compound=" << cbor_compound_duplicate << "/" << cbor_compound_distinct << " der=" << verify_malformed_der << "/" << verify_noncanonical_der << "/" << verify_trailing_der << " cborparts=" << cbor_valid << cbor_control_text << cbor_duplicate << cbor_invalid_utf8 << cbor_truncated << cbor_trailing << cbor_depth << cbor_nodes << cbor_size << cbor_indefinite << cbor_integer_overflow << cbor_nonminimal << "\\n";
return(kid_ok && signed_ok && tamper_ok && negatives_ok && cose_ok && cbor_ok && cose_negative && b64_ok ? 0 : 1);
}
EOF
"${CXX:-c++}" -std=c++20 -fpermissive -I. "$test_source" -lpcre2-8 -lcrypto -o "$test_binary"
"$test_binary"
echo "native structured crypto operation passed"
+11 -2
View File
@@ -450,14 +450,23 @@ fi
worker_count=$(awk -F= '/^[[:space:]]*WORKER_COUNT[[:space:]]*=/ {gsub(/^[[:space:]]+|[[:space:]]+$/, "", $2); print $2; exit}' /etc/uce/settings.cfg 2>/dev/null || true)
worker_count="${worker_count:-4}"
cli_worker_count=$(awk -F= '/^[[:space:]]*CLI_WORKER_COUNT[[:space:]]*=/ {gsub(/^[[:space:]]+|[[:space:]]+$/, "", $2); value=$2} END{print value}' /etc/uce/settings.cfg 2>/dev/null || true)
cli_worker_count="${cli_worker_count:-0}"
cli_worker_max_requests=$(awk -F= '/^[[:space:]]*CLI_WORKER_MAX_REQUESTS[[:space:]]*=/ {gsub(/^[[:space:]]+|[[:space:]]+$/, "", $2); value=$2} END{print value}' /etc/uce/settings.cfg 2>/dev/null || true)
cli_worker_max_requests="${cli_worker_max_requests:-8}"
worker_pids=""
for _ in {1..48}; do
output=$(scripts/uce-cli "/$test_name/parent.uce")
worker_pids+="${output##*:}"$'\n'
done
unique_workers=$(printf '%s' "$worker_pids" | sed '/^$/d' | sort -u | wc -l)
if (( unique_workers > worker_count )); then
echo "worker pool recycled during 48 requests: $unique_workers PIDs for $worker_count workers" >&2
if (( cli_worker_count > 0 && cli_worker_max_requests > 0 )); then
if (( unique_workers <= cli_worker_count )); then
echo "dedicated CLI pool did not recycle during 48 requests: $unique_workers PIDs for $cli_worker_count workers" >&2
exit 1
fi
elif (( unique_workers > worker_count )); then
echo "shared worker pool recycled during 48 requests: $unique_workers PIDs for $worker_count workers" >&2
exit 1
fi
+37
View File
@@ -0,0 +1,37 @@
#include "src/lib/types.cpp"
#include "src/lib/dvalue.cpp"
#include "src/lib/functionlib.cpp"
#include "src/wasm/hardened_http_internal.h"
#include <iostream>
struct Fake { std::vector<String> answers, argv, env; String input; HardenedHttpExecResult result; };
static DValue request() { DValue r; r["method"]="POST"; r["url"]="https://api.example.test/token"; r["headers"]["Accept"]="application/json"; r["body"]="client_secret=SECRET"; DValue& s=r["security"]; s["https_only"].set_bool(true); s["public_dns_only"].set_bool(true); s["pin_dns"].set_bool(true); s["isolated_curl"].set_bool(true); s["no_redirects"].set_bool(true); return r; }
static DValue run(Fake& f,DValue r) { HardenedHttpHooks h; h.resolve=[&](String){return f.answers;}; h.execute=[&](std::vector<String> a,String i,std::vector<String> e,u64,size_t){f.argv=a;f.input=i;f.env=e;return f.result;}; return hardened_http_request_internal(r,9000,h); }
static String headers(int status=200) { return "HTTP/1.1 "+std::to_string(status)+" OK\r\nContent-Type: application/json\r\n\r\n"; }
static bool exists(String path) { return access(path.c_str(),F_OK)==0; }
static void wait_for(String path) { for(int n=0;n<500&&!exists(path);n++) usleep(1000); }
static bool process_live(pid_t pid) { char state=0; String path="/proc/"+std::to_string((long long)pid)+"/stat"; int fd=open(path.c_str(),O_RDONLY); if(fd<0) return false; char text[256]{}; ssize_t n=read(fd,text,sizeof(text)-1); close(fd); if(n<=0) return false; char* close_paren=strrchr(text,')'); return close_paren&&close_paren[2]!='Z'; }
int main(int argc,char** argv) {
if(argc==3&&String(argv[1])=="--child-timeout") { pid_t child=fork(); if(child==0) { usleep(100000); int fd=open(argv[2],O_WRONLY|O_CREAT|O_TRUNC,0600); if(fd>=0) { write(fd,"leaked",6); close(fd); } _exit(0); } for(;;) pause(); }
if(argc==2&&String(argv[1])=="--child-output") { String x(4096,'x'); for(int n=0;n<64;n++) write(1,x.data(),x.size()); return 0; }
if(argc==3&&String(argv[1])=="--child-fds") { int inherited=atoi(argv[2]); bool closed=fcntl(inherited,F_GETFD)==-1&&errno==EBADF; bool header=fcntl(3,F_GETFD)!=-1; write(1,closed&&header?"closed":"open",closed&&header?6:4); return closed&&header?0:1; }
if(argc==5&&String(argv[1])=="--child-job") { int ready=open(argv[3],O_WRONLY|O_CREAT|O_TRUNC,0600); if(ready>=0) { write(ready,"ready",5); close(ready); } pid_t child=fork(); if(child==0) { int pidfile=open(argv[4],O_WRONLY|O_CREAT|O_TRUNC,0600); if(pidfile>=0) { String pid=std::to_string((long long)getpid()); write(pidfile,pid.data(),pid.size()); close(pidfile); } usleep(150000); int marker=open(argv[2],O_WRONLY|O_CREAT|O_TRUNC,0600); if(marker>=0) { write(marker,"leaked",6); close(marker); } _exit(0); } for(;;) pause(); }
if(argc==3&&String(argv[1])=="--child-async-timeout") { setsid(); HardenedHttpExecResult result=hardened_http_exec_argv_capture({"/bin/sleep","1"},"",20,4096,false,false); if(result.timed_out&&result.exit_code==137) { int fd=open(argv[2],O_WRONLY|O_CREAT|O_TRUNC,0600); if(fd>=0) { write(fd,"typed",5); close(fd); } } return 0; }
if(argc==3&&String(argv[1])=="--child-async-output") { setsid(); HardenedHttpExecResult result=hardened_http_exec_argv_capture({"/proc/self/exe","--child-output"},"",1000,1024,false,false); if(result.output_limited&&result.exit_code==137) { int fd=open(argv[2],O_WRONLY|O_CREAT|O_TRUNC,0600); if(fd>=0) { write(fd,"typed",5); close(fd); } } return 0; }
bool ok=true; auto need=[&](bool x,const char* n){if(!x){std::cerr<<n<<"\n";ok=false;}};
Fake f; f.answers={"8.8.8.8"}; f.result.exit_code=0; f.result.stderr_text="SECRET"; f.result.headers_text=headers(); f.result.body_text="one\r\n\r\ntwo"; DValue out=run(f,request());
need(out["error"].to_string()==""&&out["error"].to_string().find("SECRET")==String::npos&&out["body"].to_string()=="one\r\n\r\ntwo","body framing / non-secret errors"); need(f.argv.size()>2&&f.argv[0]=="/usr/bin/curl"&&f.argv[1]=="--disable","absolute curl disable"); need(f.env.size()==1&&f.env[0]=="PATH=/usr/bin:/bin","clean env"); need(std::find(f.argv.begin(),f.argv.end(),"SECRET")==f.argv.end()&&f.input.find("SECRET")!=String::npos,"secret stdin only"); need(std::find(f.argv.begin(),f.argv.end(),"--resolve")!=f.argv.end(),"pinned dns");
Fake interim; interim.answers={"8.8.8.8"}; interim.result.exit_code=0; interim.result.headers_text="HTTP/1.1 100 Continue\r\n\r\n"+headers(201); interim.result.body_text="{}"; need(run(interim,request())["status"].to_u64()==201,"interim headers");
for(String bad:{"127.0.0.1","10.0.0.1","169.254.1.1","192.168.1.1","::1","2001:db8::1"}) { Fake x; x.answers={"8.8.8.8",bad}; need(run(x,request())["error"].to_string()=="unsafe_dns","dns matrix"); }
for(auto pair:std::vector<std::pair<int,String>>{{302,headers(302)},{500,headers(500)},{200,"bad\r\n\r\n"}}) { Fake x; x.answers={"8.8.8.8"};x.result.exit_code=0;x.result.headers_text=pair.second;need(run(x,request())["error"].to_string()==(pair.first==302?"redirect_not_allowed":pair.first==500?"http_status":"malformed_output"),"status errors"); }
DValue bad=request(); bad["method"]="TRACE"; Fake x; need(run(x,bad)["error"].to_string()=="invalid_request"&&x.argv.empty(),"method validation"); bad=request();bad["headers"]["Host"]="evil";need(run(x,bad)["error"].to_string()=="invalid_request","header validation"); bad=request();bad["headers"]["Accept"]="ok\r\nInjected: x";need(run(x,bad)["error"].to_string()=="invalid_request","header CRLF validation");bad=request();bad["url"]="https://127.0.0.1/";need(run(x,bad)["error"].to_string()=="invalid_request","url validation");bad=request();bad["follow_redirects"].set_bool(true);need(run(x,bad)["error"].to_string()=="invalid_request","redirect composition");
for(String key:{"https_only","public_dns_only","pin_dns","isolated_curl","no_redirects"}) { bad=request(); bad["security"].remove(key); need(run(x,bad)["error"].to_string()=="invalid_request","partial hardening fails closed"); bad=request(); bad["security"][key].set_bool(false); need(run(x,bad)["error"].to_string()=="invalid_request","false hardening fails closed"); bad=request(); bad["security"][key]="true"; need(run(x,bad)["error"].to_string()=="invalid_request","non-boolean hardening fails closed"); }
DValue legacy=request(); legacy["security"].clear(); legacy["security"]["unrelated"]="value"; need(!hardened_http_security_requested(legacy.key("security")),"unknown security object remains legacy");
Fake large;large.answers={"8.8.8.8"};large.result.exit_code=0;large.result.headers_text=headers();large.result.body_text=String(65537,'x');need(run(large,request())["error"].to_string()=="response_too_large","body cap");
String base="/tmp/hardened-http-"+std::to_string((long long)getpid()), marker=base+"-marker"; unlink(marker.c_str()); HardenedHttpExecResult timeout=hardened_http_exec_argv_capture({"/proc/self/exe","--child-timeout",marker},"",30,4096,false); usleep(150000); need(timeout.timed_out&&timeout.exit_code==137&&!exists(marker),"timeout kills descendants"); unlink(marker.c_str());
HardenedHttpExecResult overflow=hardened_http_exec_argv_capture({"/proc/self/exe","--child-output"},"",1000,1024,false);need(overflow.output_limited&&overflow.body_text.size()<=1024,"output cap/reap");
int inherited=open("/dev/null",O_RDONLY), high_inherited=fcntl(inherited,F_DUPFD,10); close(inherited); HardenedHttpExecResult fds=hardened_http_exec_argv_capture({"/proc/self/exe","--child-fds",std::to_string(high_inherited)},"",1000,4096,false); close(high_inherited); need(fds.exit_code==0&&fds.body_text=="closed","only stdio and header fd inherited");
String ready=base+"-ready", descendant=base+"-descendant", sentinel=base+"-sentinel"; unlink(marker.c_str()); unlink(ready.c_str()); unlink(descendant.c_str()); unlink(sentinel.c_str()); pid_t worker=fork(); if(worker==0) { setsid(); hardened_http_exec_argv_capture({"/proc/self/exe","--child-job",marker,ready,descendant},"",5000,4096,false,false); _exit(0); } wait_for(ready); wait_for(descendant); pid_t unrelated=fork(); if(unrelated==0) { setsid(); usleep(150000); int fd=open(sentinel.c_str(),O_WRONLY|O_CREAT|O_TRUNC,0600); if(fd>=0) { write(fd,"alive",5); close(fd); } _exit(0); } pid_t descendant_pid=exists(descendant)?(pid_t)strtol([](String path){ int fd=open(path.c_str(),O_RDONLY); char text[32]{}; ssize_t n=fd<0?-1:read(fd,text,sizeof(text)-1); if(fd>=0) close(fd); return String(text,n>0?(size_t)n:0); }(descendant).c_str(),0,10):0; need(exists(ready)&&descendant_pid>0,"async worker and descendant started"); kill(-worker,SIGKILL); waitpid(worker,0,0); usleep(250000); waitpid(unrelated,0,0); need(!process_live(descendant_pid)&&!exists(marker)&&exists(sentinel),"async cancellation leaves no live descendant and kills only its worker group"); unlink(marker.c_str()); unlink(ready.c_str()); unlink(descendant.c_str()); unlink(sentinel.c_str());
String async_timeout=base+"-async-timeout"; unlink(async_timeout.c_str()); pid_t timeout_worker=fork(); if(timeout_worker==0) { execl("/proc/self/exe","test_hardened_http_native","--child-async-timeout",async_timeout.c_str(),(char*)0); _exit(127); } int timeout_status=0; waitpid(timeout_worker,&timeout_status,0); need(WIFEXITED(timeout_status)&&WEXITSTATUS(timeout_status)==0&&exists(async_timeout),"async timeout records a typed terminal result before worker exit"); unlink(async_timeout.c_str());
String async_output=base+"-async-output"; unlink(async_output.c_str()); pid_t output_worker=fork(); if(output_worker==0) { execl("/proc/self/exe","test_hardened_http_native","--child-async-output",async_output.c_str(),(char*)0); _exit(127); } int output_status=0; waitpid(output_worker,&output_status,0); need(WIFEXITED(output_status)&&WEXITSTATUS(output_status)==0&&exists(async_output),"async output cap records a typed terminal result before worker exit"); unlink(async_output.c_str());
return ok?0:1;
}
+8
View File
@@ -0,0 +1,8 @@
#!/usr/bin/env bash
set -euo pipefail
repo=$(cd "$(dirname "$0")/.." && pwd)
bin="${TMPDIR:-/tmp}/test_hardened_http_native.$$"
trap 'rm -f "$bin"' EXIT
compiler=$(command -v clang++ || command -v g++)
"$compiler" -std=c++20 -fpermissive -I"$repo" "$repo/scripts/test_hardened_http_native.cpp" -lpcre2-8 -o "$bin"
timeout --signal=TERM --kill-after=2s 15s "$bin"
+145
View File
@@ -0,0 +1,145 @@
#!/usr/bin/env bash
set -euo pipefail
cd "$(dirname "$0")/.."
if [[ "${1:-}" != "--inside" ]]; then
exec timeout --signal=TERM --kill-after=5s 150s unshare --mount --fork --kill-child=TERM "$0" --inside
fi
name="mysql-pool-idle-test-$$"
root="/tmp/$name"
site="$root/site"
work="$root/work"
settings="$root/settings.cfg"
log="$root/service.log"
socket="$root/run/cli.sock"
test_user="uce_pool_idle_$$"
test_database="uce_pool_idle_$$"
test_password=$(printf '%s' "$name-$(date +%s%N)" | sha256sum | cut -c1-32)
server_pid=""
cleanup() {
status=$?
if [[ -n "$server_pid" ]] && kill -0 "$server_pid" 2>/dev/null; then
kill -TERM "$server_pid" 2>/dev/null || true
deadline=$((SECONDS + 10))
while kill -0 "$server_pid" 2>/dev/null && (( SECONDS < deadline )); do sleep 0.05; done
if kill -0 "$server_pid" 2>/dev/null; then kill -KILL "$server_pid" 2>/dev/null || true; fi
wait "$server_pid" 2>/dev/null || true
fi
mariadb -e "DROP DATABASE IF EXISTS \`$test_database\`; DROP USER IF EXISTS '$test_user'@'127.0.0.1'" >/dev/null 2>&1 || true
if (( status != 0 )) && [[ -r "$log" ]]; then cat "$log" >&2; fi
rm -rf "$root"
return "$status"
}
trap cleanup EXIT
mkdir -p "$site" "$work" "$root/run" "$root/session" "$root/upload"
cp /etc/uce/settings.cfg "$settings"
cat >>"$settings" <<CFG
BIN_DIRECTORY=$work
PRECOMPILE_FILES_IN=$site
SITE_DIRECTORY=$site
FCGI_SOCKET_PATH=$root/run/fastcgi.sock
FCGI_PORT=
CLI_SOCKET_PATH=$socket
CLI_WORKER_COUNT=1
CLI_WORKER_MAX_REQUESTS=0
WS_BROKER_SOCKET_PATH=$root/run/ws.sock
HTTP_PORT=
HTTP_DOCUMENT_ROOT=$site
SESSION_PATH=$root/session
TMP_UPLOAD_PATH=$root/upload
WASM_CORE_PATH=$(pwd)/bin/wasm/core.wasm
WORKER_COUNT=1
PROACTIVE_COMPILE_ENABLED=0
MYSQL_PERSISTENT_POOL_SIZE=8
MYSQL_PERSISTENT_POOL_IDLE_TIMEOUT_SECONDS=2
CFG
mount --bind "$settings" /etc/uce/settings.cfg
mariadb -e "DROP DATABASE IF EXISTS \`$test_database\`; DROP USER IF EXISTS '$test_user'@'127.0.0.1'; CREATE DATABASE \`$test_database\`; CREATE USER '$test_user'@'127.0.0.1' IDENTIFIED BY '$test_password'; GRANT ALL ON \`$test_database\`.* TO '$test_user'@'127.0.0.1'"
cat >"$site/database.uce" <<UCE
CLI(Request& context)
{
MySQL* db = mysql_connect("127.0.0.1", "$test_user", "$test_password", "$test_database");
if(!mysql_connected(db)) { print("connect-failed"); return; }
String connection_id;
mysql_query(db, "SELECT CONNECTION_ID() AS id").each([&](DValue row, String key) { connection_id = row["id"].to_string(); });
String source;
DValue perf = request_perf();
perf["mysql_operations"].each([&](DValue operation, String key) {
if(operation["op"].to_string() == "connect") source = operation["source"].to_string();
});
print(perf["worker_pid"].to_string(), "|", connection_id, "|", source);
mysql_disconnect(db);
}
UCE
printf '%s\n' 'CLI(Request& context) { print(request_perf()["worker_pid"].to_string(), "|plain"); }' >"$site/plain.uce"
timeout --signal=TERM --kill-after=5s 120s bin/uce_fastcgi.linux.bin >"$log" 2>&1 &
server_pid=$!
deadline=$((SECONDS + 20))
while [[ ! -S "$socket" ]] && (( SECONDS < deadline )); do sleep 0.05; done
[[ -S "$socket" ]] || { echo "private UCE CLI socket was not ready" >&2; exit 1; }
request() {
timeout --signal=TERM --kill-after=1s 30s scripts/uce-cli --socket "$socket" "$1"
}
first=$(request /database.uce)
second=$(request /database.uce)
IFS='|' read -r first_pid first_id first_source <<<"$first"
IFS='|' read -r second_pid second_id second_source <<<"$second"
[[ "$first_source" == "new" && "$second_source" == "worker" ]]
[[ "$first_pid" == "$second_pid" && "$first_id" == "$second_id" ]]
sleep 3
plain=$(request /plain.uce)
[[ "$plain" == "$first_pid|plain" ]]
deadline=$((SECONDS + 5))
while (( SECONDS < deadline )); do
connections=$(mariadb --batch --skip-column-names -e "SELECT COUNT(*) FROM information_schema.PROCESSLIST WHERE USER='$test_user'")
[[ "$connections" == "0" ]] && break
sleep 0.05
done
[[ "$connections" == "0" ]] || { echo "expired pooled connection remained after a request boundary" >&2; exit 1; }
third=$(request /database.uce)
IFS='|' read -r third_pid third_id third_source <<<"$third"
[[ "$third_pid" == "$first_pid" && "$third_source" == "new" && "$third_id" != "$first_id" ]]
sleep 3
plain=$(request /plain.uce)
[[ "$plain" == "$first_pid|plain" ]]
deadline=$((SECONDS + 5))
while (( SECONDS < deadline )); do
connections=$(mariadb --batch --skip-column-names -e "SELECT COUNT(*) FROM information_schema.PROCESSLIST WHERE USER='$test_user'")
[[ "$connections" == "0" ]] && break
sleep 0.05
done
[[ "$connections" == "0" ]] || { echo "final pooled fixture connection remained after idle eviction" >&2; exit 1; }
retained=$(request /database.uce)
IFS='|' read -r retained_pid retained_id retained_source <<<"$retained"
[[ "$retained_pid" == "$first_pid" && "$retained_source" == "new" ]]
connections=$(mariadb --batch --skip-column-names -e "SELECT COUNT(*) FROM information_schema.PROCESSLIST WHERE USER='$test_user'")
[[ "$connections" == "1" ]] || { echo "shutdown fixture connection was not retained in the worker pool" >&2; exit 1; }
journal_cursor=$(journalctl -u mariadb.service -n 0 --show-cursor --no-pager | sed -n 's/^-- cursor: //p')
[[ -n "$journal_cursor" ]] || { echo "could not capture MariaDB journal cursor" >&2; exit 1; }
kill -TERM "$server_pid"
deadline=$((SECONDS + 10))
while kill -0 "$server_pid" 2>/dev/null && (( SECONDS < deadline )); do sleep 0.05; done
[[ ! -e "/proc/$server_pid" ]] || { echo "private UCE server did not stop cleanly" >&2; exit 1; }
wait "$server_pid" 2>/dev/null || true
server_pid=""
sleep 1
shutdown_log=$(journalctl -u mariadb.service --after-cursor "$journal_cursor" --no-pager)
if grep -F "$test_user" <<<"$shutdown_log" | grep -Fq "Aborted connection"; then
echo "clean UCE shutdown aborted a retained MySQL connection" >&2
exit 1
fi
echo "Persistent MySQL idle eviction preserved hot reuse, retired expired connections, and closed a retained connection on clean shutdown"
@@ -134,6 +134,15 @@ printf '0\n' >"$root/maximum"
flock -u 7
mapfile -t scanner_pids < <(awk -F '\t' -v site="$site/" '$3 ~ ("^" site "unit-[0-9]+[.]uce$") { count[$4]++ } END { for(pid in count) if(count[pid] >= 2) print pid }' "$shim_log" | sort -n)
[[ "${#scanner_pids[@]}" -eq 2 ]] || { echo "controlled units were not split between both scanners" >&2; cat "$shim_log" >&2; exit 1; }
assert_scanner_memory_bounded() {
for scanner_pid in "${scanner_pids[@]}"; do
retained_kb=$(awk '/^(VmRSS|VmSwap):/{total += $2} END{print total + 0}' "/proc/$scanner_pid/status")
# Scanner bookkeeping is small; 64 MiB leaves ample headroom while still
# catching the measured old 12-serialization retention (~73 MiB).
(( retained_kb < 65536 )) || { echo "proactive scanner retained serialized-module arenas: pid=$scanner_pid retained_kb=$retained_kb" >&2; exit 1; }
done
}
assert_scanner_memory_bounded
blockers=()
victims=()
for scanner_pid in "${scanner_pids[@]}"; do
@@ -202,4 +211,8 @@ while [[ ! -s "$artifacts/priority.uce.cwasm" ]] && (( SECONDS < deadline )); do
priority_nice=$(awk -F '\t' -v path="$site/priority.uce" '$3 == path { print $2; exit }' "$shim_log")
[[ "$priority_nice" == "5" ]] || { echo "priority queue was not owned by the nice-5 compiler: $priority_nice" >&2; cat "$shim_log" >&2; exit 1; }
# Recheck after dependency fanout, failure/retry, recovery, and priority work;
# the historical leak accumulated in the long-lived scanners across phases.
assert_scanner_memory_bounded
printf '%s\n' 'parallel proactive compile passed'
+34 -2
View File
@@ -38,7 +38,8 @@ socket_inode=$(stat -c %i "$socket_path")
invoke() {
local output="$1"
shift
timeout --signal=TERM --kill-after=1s 2s unshare --mount --fork \
local timeout_seconds="${INVOKE_TIMEOUT_SECONDS:-2}"
timeout --signal=TERM --kill-after=1s "$timeout_seconds" unshare --mount --fork \
bash -c 'mount --bind "$1" /etc/uce/settings.cfg; exec "$2" "${@:3}"' \
_ "$cfg" "$binary" "$@" >"$output.stdout" 2>"$output.stderr"
}
@@ -47,10 +48,11 @@ for option in --help -h; do
invoke "$root/help" "$option"
grep -q '^Usage: uce_fastcgi' "$root/help.stdout"
grep -q -- '--precompile' "$root/help.stdout"
grep -q -- '--serialize-module' "$root/help.stdout"
[[ ! -s "$root/help.stderr" ]]
done
for arguments in '--unknown' '--precompile extra' '--help extra'; do
for arguments in '--unknown' '--precompile extra' '--serialize-module' '--help extra'; do
read -r -a argv <<<"$arguments"
set +e
invoke "$root/invalid" "${argv[@]}"
@@ -62,6 +64,36 @@ for arguments in '--unknown' '--precompile extra' '--help extra'; do
[[ ! -s "$root/invalid.stdout" ]]
done
cp bin/wasm/core.wasm "$root/serialize.wasm"
INVOKE_TIMEOUT_SECONDS=30 invoke "$root/serialize" --serialize-module "$root/serialize.wasm"
[[ -s "$root/serialize.cwasm" ]]
[[ ! -s "$root/serialize.stderr" ]]
# Serialization shares the unit compile lock. If the wasm path is replaced
# while the serializer is waiting, it must read the replacement and must not
# publish a stale native artifact for the old inode.
cp bin/wasm/core.wasm "$root/race.wasm"
exec 8>"$root/race.wasm.lock"
flock 8
INVOKE_TIMEOUT_SECONDS=30 invoke "$root/race" --serialize-module "$root/race.wasm" &
race_pid=$!
sleep 0.1
kill -0 "$race_pid"
printf 'not wasm\n' >"$root/race.wasm.next"
mv "$root/race.wasm.next" "$root/race.wasm"
flock -u 8
set +e
wait "$race_pid"
race_rc=$?
set -e
[[ $race_rc -eq 1 ]]
grep -Eqi 'wasm|magic|WebAssembly|compile' "$root/race.stderr"
[[ ! -e "$root/race.cwasm" ]]
if compgen -G "$root/race.cwasm.*.tmp" >/dev/null; then
echo "failed serialization left a temporary artifact" >&2
exit 1
fi
[[ -S "$socket_path" ]]
[[ "$(stat -c %i "$socket_path")" == "$socket_inode" ]]
kill -0 "$listener_pid"
+3
View File
@@ -24,6 +24,9 @@ inode_before=$(stat -c %i "$socket_path")
load_pid=$!
sleep 0.05
systemctl restart "$service_name"
systemctl is-active --quiet "$socket_name"
inode_during=$(stat -c %i "$socket_path")
[[ "$inode_during" = "$inode_before" ]]
wait "$load_pid"
inode_after=$(stat -c %i "$socket_path")
+4 -2
View File
@@ -31,7 +31,7 @@ cleanup() {
}
trap cleanup EXIT
mkdir -p "$site/components" "$work" "$root/run" "$root/session" "$root/upload"
sed -E '/^[[:space:]]*(BIN_DIRECTORY|PRECOMPILE_FILES_IN|SITE_DIRECTORY|FCGI_SOCKET_PATH|FCGI_PORT|CLI_SOCKET_PATH|WS_BROKER_SOCKET_PATH|HTTP_PORT|HTTP_DOCUMENT_ROOT|SESSION_PATH|TMP_UPLOAD_PATH|WASM_CORE_PATH|WASM_COMPILE_SCRIPT|WASM_INVOCATION_TIMEOUT_MS|WASM_EPOCH_PERIOD_MS|PROACTIVE_COMPILE_ENABLED|WORKER_COUNT|COMPILE_FAILURE_RETRY_SECONDS)[[:space:]]*=/d' \
sed -E '/^[[:space:]]*(BIN_DIRECTORY|PRECOMPILE_FILES_IN|SITE_DIRECTORY|FCGI_SOCKET_PATH|FCGI_PORT|CLI_SOCKET_PATH|CLI_WORKER_COUNT|CLI_WORKER_MAX_REQUESTS|WS_BROKER_SOCKET_PATH|HTTP_PORT|HTTP_DOCUMENT_ROOT|SESSION_PATH|TMP_UPLOAD_PATH|WASM_CORE_PATH|WASM_COMPILE_SCRIPT|WASM_INVOCATION_TIMEOUT_MS|WASM_EPOCH_PERIOD_MS|PROACTIVE_COMPILE_ENABLED|WORKER_COUNT|COMPILE_FAILURE_RETRY_SECONDS)[[:space:]]*=/d' \
/etc/uce/settings.cfg >"$settings"
cat >>"$settings" <<CFG
BIN_DIRECTORY=$work
@@ -40,12 +40,14 @@ SITE_DIRECTORY=$site
FCGI_SOCKET_PATH=$root/run/fastcgi.sock
FCGI_PORT=
CLI_SOCKET_PATH=$root/run/cli.sock
CLI_WORKER_COUNT=1
CLI_WORKER_MAX_REQUESTS=0
WS_BROKER_SOCKET_PATH=$root/run/ws.sock
HTTP_PORT=
HTTP_DOCUMENT_ROOT=$site
SESSION_PATH=$root/session
TMP_UPLOAD_PATH=$root/upload
WASM_CORE_PATH=$(pwd)/bin/wasm/core.wasm
WASM_CORE_PATH=$(pwd -P)/bin/wasm/core.wasm
WASM_COMPILE_SCRIPT=$root/compile
WASM_INVOCATION_TIMEOUT_MS=2000
WASM_EPOCH_PERIOD_MS=20
+37 -22
View File
@@ -21,11 +21,20 @@ if [[ -r "$settings_file" ]]; then
[[ -n "${UCE_CLI_SOCKET:-}" ]] || socket_path=$(awk -F= '/^[[:space:]]*CLI_SOCKET_PATH[[:space:]]*=/ {gsub(/^[[:space:]]+|[[:space:]]+$/, "", $2); print $2; exit}' "$settings_file")
[[ -n "${BIN_DIRECTORY:-}" ]] || bin_directory=$(awk -F= '/^[[:space:]]*BIN_DIRECTORY[[:space:]]*=/ {gsub(/^[[:space:]]+|[[:space:]]+$/, "", $2); print $2; exit}' "$settings_file")
invocation_ms=$(awk -F= '/^[[:space:]]*WASM_INVOCATION_TIMEOUT_MS[[:space:]]*=/ {gsub(/^[[:space:]]+|[[:space:]]+$/, "", $2); print $2; exit}' "$settings_file")
cli_worker_count=$(awk -F= '/^[[:space:]]*CLI_WORKER_COUNT[[:space:]]*=/ {gsub(/^[[:space:]]+|[[:space:]]+$/, "", $2); value=$2} END{print value}' "$settings_file")
fi
site_directory="${site_directory:-site}"
socket_path="${socket_path:-/run/uce/cli.sock}"
bin_directory="${bin_directory:-/tmp/uce/work}"
invocation_ms="${invocation_ms:-30000}"
cli_worker_count="${cli_worker_count:-0}"
curl_transport=(--unix-socket "$socket_path")
test_base_url="http://localhost"
if (( cli_worker_count > 0 )); then
test_http_host="${UCE_TEST_HTTP_HOST:-uce.openfu.com}"
curl_transport=(--resolve "$test_http_host:80:127.0.0.1")
test_base_url="http://$test_http_host"
fi
test_name="invocation-timeout-test-$$"
source_dir="$site_directory/$test_name"
pid_file="/tmp/uce-$test_name-worker"
@@ -42,13 +51,15 @@ mkdir -p "$source_dir"
cache_dir="$(scripts/unit_cache_directory "$bin_directory")$(realpath "$source_dir")"
printf '%s\n' \
'CLI(Request& context) {' \
'void timeout_test_run(Request& context) {' \
' if(context.get["warm"] == "1") { print("warm"); return; }' \
" file_put_contents(\"$pid_file\", std::to_string(request_perf()[\"worker_pid\"].to_u64()));" \
' while(true) time_precise();' \
'}' >"$source_dir/hostcall-loop.uce"
'}' \
'CLI(Request& context) { timeout_test_run(context); }' \
'RENDER(Request& context) { timeout_test_run(context); }' >"$source_dir/hostcall-loop.uce"
printf '%s\n' \
'CLI(Request& context) {' \
'void timeout_test_run(Request& context) {' \
' if(context.get["warm"] == "1") { print("warm"); return; }' \
' if(context.get["quick"] == "1") { print(shell_exec("printf quick")); return; }' \
' if(context.get["status"] == "1") { DValue spec; spec["cmd"] = "exit 7"; spec["timeout_ms"] = (f64)500; print(shell_exec(spec)["exit_code"].to_u64()); return; }' \
@@ -56,46 +67,50 @@ printf '%s\n' \
' if(context.get["job"] == "1") { DValue spec; spec["cmd"] = "sleep 2"; spec["timeout_ms"] = (f64)5000; u64 job = shell_spawn(spec); f64 started = time_precise(); job_await(job, 300); u64 elapsed = (u64)((time_precise() - started) * 1000); job_cancel(job); print(elapsed); return; }' \
" file_put_contents(\"$pid_file\", std::to_string(request_perf()[\"worker_pid\"].to_u64()));" \
' print(shell_exec("printf shell-start; sleep 60 & printf shell-end"));' \
'}' >"$source_dir/legacy-shell.uce"
'}' \
'CLI(Request& context) { timeout_test_run(context); }' \
'RENDER(Request& context) { timeout_test_run(context); }' >"$source_dir/legacy-shell.uce"
printf '%s\n' \
'CLI(Request& context) {' \
'void timeout_test_run(Request& context) {' \
' if(context.get["warm"] == "1") { print("warm"); return; }' \
" file_put_contents(\"$pid_file\", std::to_string(request_perf()[\"worker_pid\"].to_u64()));" \
' while(true) sleep(60);' \
' print("sleep-end");' \
'}' >"$source_dir/sleep.uce"
'}' \
'CLI(Request& context) { timeout_test_run(context); }' \
'RENDER(Request& context) { timeout_test_run(context); }' >"$source_dir/sleep.uce"
printf '%s\n' \
'CLI(Request& context) { print(request_perf()["worker_pid"].to_u64(), "|health"); }' >"$source_dir/health.uce"
'void timeout_test_run(Request& context) { print(request_perf()["worker_pid"].to_u64(), "|health"); }' \
'CLI(Request& context) { timeout_test_run(context); }' \
'RENDER(Request& context) { timeout_test_run(context); }' >"$source_dir/health.uce"
max_seconds=$(( (invocation_ms + 15000) / 1000 ))
(( max_seconds >= 10 )) || max_seconds=10
request() {
local unit="$1"
curl -sS --max-time "$max_seconds" -o "$body_file" -w '%{http_code}' --unix-socket "$socket_path" "http://localhost/$test_name/$unit"
curl -sS --max-time "$max_seconds" -o "$body_file" -w '%{http_code}' "${curl_transport[@]}" "$test_base_url/$test_name/$unit"
}
same_worker_health() {
local expected_pid="$1"
for _ in $(seq 1 32); do
local health
health=$(curl -sS --max-time 5 --fail-with-body --unix-socket "$socket_path" "http://localhost/$test_name/health.uce")
[[ "$health" == "$expected_pid|health" ]] && return 0
done
return 1
kill -0 "$expected_pid" 2>/dev/null || return 1
local health
health=$(curl -sS --max-time 5 --fail-with-body "${curl_transport[@]}" "$test_base_url/$test_name/health.uce")
[[ "$health" =~ ^[0-9]+\|health$ ]]
}
curl -sS --max-time "$max_seconds" --fail-with-body --unix-socket "$socket_path" "http://localhost/$test_name/hostcall-loop.uce?warm=1" >/dev/null
curl -sS --max-time "$max_seconds" --fail-with-body --unix-socket "$socket_path" "http://localhost/$test_name/legacy-shell.uce?warm=1" >/dev/null
curl -sS --max-time "$max_seconds" --fail-with-body --unix-socket "$socket_path" "http://localhost/$test_name/sleep.uce?warm=1" >/dev/null
curl -sS --max-time "$max_seconds" --fail-with-body --unix-socket "$socket_path" "http://localhost/$test_name/health.uce" >/dev/null
quick=$(curl -sS --max-time 5 --fail-with-body --unix-socket "$socket_path" "http://localhost/$test_name/legacy-shell.uce?quick=1")
curl -sS --max-time "$max_seconds" --fail-with-body "${curl_transport[@]}" "$test_base_url/$test_name/hostcall-loop.uce?warm=1" >/dev/null
curl -sS --max-time "$max_seconds" --fail-with-body "${curl_transport[@]}" "$test_base_url/$test_name/legacy-shell.uce?warm=1" >/dev/null
curl -sS --max-time "$max_seconds" --fail-with-body "${curl_transport[@]}" "$test_base_url/$test_name/sleep.uce?warm=1" >/dev/null
curl -sS --max-time "$max_seconds" --fail-with-body "${curl_transport[@]}" "$test_base_url/$test_name/health.uce" >/dev/null
quick=$(curl -sS --max-time 5 --fail-with-body "${curl_transport[@]}" "$test_base_url/$test_name/legacy-shell.uce?quick=1")
[[ "$quick" == quick ]] || { echo "quick legacy shell returned: $quick" >&2; exit 1; }
exit_status=$(curl -sS --max-time 5 --fail-with-body --unix-socket "$socket_path" "http://localhost/$test_name/legacy-shell.uce?status=1")
exit_status=$(curl -sS --max-time 5 --fail-with-body "${curl_transport[@]}" "$test_base_url/$test_name/legacy-shell.uce?status=1")
[[ "$exit_status" == 7 ]] || { echo "structured shell lost exit status: $exit_status" >&2; exit 1; }
zero_timeout=$(curl -sS --max-time 5 --fail-with-body --unix-socket "$socket_path" "http://localhost/$test_name/legacy-shell.uce?zero=1")
zero_timeout=$(curl -sS --max-time 5 --fail-with-body "${curl_transport[@]}" "$test_base_url/$test_name/legacy-shell.uce?zero=1")
[[ "$zero_timeout" == zero ]] || { echo "structured shell zero-timeout default changed: $zero_timeout" >&2; exit 1; }
job_elapsed=$(curl -sS --max-time 5 --fail-with-body --unix-socket "$socket_path" "http://localhost/$test_name/legacy-shell.uce?job=1")
job_elapsed=$(curl -sS --max-time 5 --fail-with-body "${curl_transport[@]}" "$test_base_url/$test_name/legacy-shell.uce?job=1")
[[ "$job_elapsed" =~ ^[0-9]+$ && "$job_elapsed" -ge 200 && "$job_elapsed" -lt 500 ]] || { echo "job_await two-call duration was ${job_elapsed}ms" >&2; exit 1; }
for unit in hostcall-loop.uce legacy-shell.uce sleep.uce; do
+10 -6
View File
@@ -29,7 +29,7 @@ cleanup() {
}
trap cleanup EXIT
mkdir -p "$site/components" "$work" "$root/run" "$root/session" "$root/upload"
sed -E '/^[[:space:]]*(BIN_DIRECTORY|PRECOMPILE_FILES_IN|SITE_DIRECTORY|FCGI_SOCKET_PATH|FCGI_PORT|CLI_SOCKET_PATH|WS_BROKER_SOCKET_PATH|HTTP_PORT|HTTP_DOCUMENT_ROOT|SESSION_PATH|TMP_UPLOAD_PATH|WASM_CORE_PATH|WASM_INVOCATION_TIMEOUT_MS|WASM_EPOCH_PERIOD_MS|PROACTIVE_COMPILE_ENABLED|WORKER_COUNT)[[:space:]]*=/d' \
sed -E '/^[[:space:]]*(BIN_DIRECTORY|PRECOMPILE_FILES_IN|SITE_DIRECTORY|FCGI_SOCKET_PATH|FCGI_PORT|CLI_SOCKET_PATH|CLI_WORKER_COUNT|CLI_WORKER_MAX_REQUESTS|WS_BROKER_SOCKET_PATH|HTTP_PORT|HTTP_DOCUMENT_ROOT|SESSION_PATH|TMP_UPLOAD_PATH|WASM_CORE_PATH|WASM_INVOCATION_TIMEOUT_MS|WASM_EPOCH_PERIOD_MS|PROACTIVE_COMPILE_ENABLED|SERVE_LAST_KNOWN_GOOD|SHOW_DYNAMIC_COMPILE_ERRORS|WORKER_COUNT)[[:space:]]*=/d' \
/etc/uce/settings.cfg >"$settings"
cat >>"$settings" <<CFG
BIN_DIRECTORY=$work
@@ -38,15 +38,19 @@ SITE_DIRECTORY=$site
FCGI_SOCKET_PATH=$root/run/fastcgi.sock
FCGI_PORT=
CLI_SOCKET_PATH=$socket
CLI_WORKER_COUNT=1
CLI_WORKER_MAX_REQUESTS=0
WS_BROKER_SOCKET_PATH=$root/run/ws.sock
HTTP_PORT=
HTTP_DOCUMENT_ROOT=$site
SESSION_PATH=$root/session
TMP_UPLOAD_PATH=$root/upload
WASM_CORE_PATH=$(pwd)/bin/wasm/core.wasm
WASM_INVOCATION_TIMEOUT_MS=100
WASM_INVOCATION_TIMEOUT_MS=1000
WASM_EPOCH_PERIOD_MS=10
PROACTIVE_COMPILE_ENABLED=0
SERVE_LAST_KNOWN_GOOD=0
SHOW_DYNAMIC_COMPILE_ERRORS=1
WORKER_COUNT=1
CFG
mount --bind "$settings" /etc/uce/settings.cfg
@@ -259,14 +263,14 @@ wait_ready() {
prepare_deadline
deadline_wasm="$cache/components/deadline.uce.wasm"
deadline_counter="$root/deadline.counter"
printf '%s delay 150 - - %s\n' "$deadline_wasm" "$deadline_counter" >"$control"
printf '%s delay 1100 - - %s\n' "$deadline_wasm" "$deadline_counter" >"$control"
started=$(date +%s%N)
set +e
deadline_output=$(request target=components/deadline.uce 2>&1)
set -e
elapsed_ms=$(( ($(date +%s%N) - started) / 1000000 ))
[[ "$deadline_output" == *UCE_INVOCATION_TIMEOUT:* ]] || { echo "metadata scan lacked canonical timeout: $deadline_output" >&2; exit 1; }
(( elapsed_ms >= 100 && elapsed_ms < 2000 )) || { echo "metadata timeout took ${elapsed_ms}ms" >&2; exit 1; }
(( elapsed_ms >= 1000 && elapsed_ms < 3000 )) || { echo "metadata timeout took ${elapsed_ms}ms" >&2; exit 1; }
[[ $(wc -l <"$deadline_counter") -eq 1 ]] || { echo "metadata scanner read past the first delayed refill" >&2; exit 1; }
rm -f "$control"
[[ "$(request health=1)" == "$worker_pid|health" ]]
@@ -275,14 +279,14 @@ restore deadline
rm -f "$cache/components/deadline.uce.cwasm"
touch "$deadline_wasm"
printf '%s delay 150 - - %s\n' "$deadline_wasm" "$root/full-deadline.counter" >"$control"
printf '%s delay 1100 - - %s\n' "$deadline_wasm" "$root/full-deadline.counter" >"$control"
started=$(date +%s%N)
set +e
full_deadline_output=$(request target=components/deadline.uce 2>&1)
set -e
elapsed_ms=$(( ($(date +%s%N) - started) / 1000000 ))
[[ "$full_deadline_output" == *UCE_INVOCATION_TIMEOUT:* ]] || { echo "full artifact read lacked canonical timeout: $full_deadline_output" >&2; exit 1; }
(( elapsed_ms >= 100 && elapsed_ms < 2000 )) || { echo "full artifact timeout took ${elapsed_ms}ms" >&2; exit 1; }
(( elapsed_ms >= 1000 && elapsed_ms < 3000 )) || { echo "full artifact timeout took ${elapsed_ms}ms" >&2; exit 1; }
[[ $(wc -l <"$root/full-deadline.counter") -eq 1 ]] || { echo "full artifact reader continued after the first delayed chunk" >&2; exit 1; }
rm -f "$control"
[[ "$(request health=1)" == "$worker_pid|health" ]]
+1
View File
@@ -14,6 +14,7 @@ hmac_sha256
hmac_sha256_hex
random_bytes
crypto_equal
crypto_operation
password_hash
password_verify
password_needs_rehash
+28
View File
@@ -0,0 +1,28 @@
:sig
DValue crypto_operation(DValue request)
:params
request : structured operation, algorithm, and operation-specific fields
return value : map with ok, bounded error code, and operation-specific output
:content
Runs one explicitly supported structured asymmetric cryptographic operation. The initial allowlist is `key_generate` with `ES256` and `jwt_sign` with `ES256`. Unknown operations and algorithms fail closed.
ES256 signing validates that `x`, `y`, and `d` form one P-256 key, forces the protected `alg` to `ES256`, and emits a compact JWT with a 64-byte JOSE signature. Requests are capped at 32 KiB; `cbor_decode` accepts at most 16 KiB decoded CBOR (21,846 canonical base64url characters), 256 nodes, and depth 16. CBOR is definite-length only, validates UTF-8 text, preserves typed map keys, rejects structurally duplicate keys and trailing bytes, and reports malformed CBOR as `invalid_cbor`. Header and claims roots must be JSON objects containing valid UTF-8 without raw control bytes.
This function does not replace typed digest, HMAC, password, randomness, or constant-time comparison APIs. It exposes no raw signing, arbitrary curve/digest selection, encryption, or generic OpenSSL access. Keep returned private JWKs secret.
:example
DValue key_request;
key_request["operation"] = "key_generate";
key_request["algorithm"] = "ES256";
DValue key = crypto_operation(key_request);
DValue sign_request;
sign_request["operation"] = "jwt_sign";
sign_request["algorithm"] = "ES256";
sign_request["private_jwk"] = key["private_jwk"];
sign_request["protected_header"]["typ"] = "JWT";
sign_request["claims"]["iss"] = "https://client.example";
DValue signed_jwt = crypto_operation(sign_request);
print(signed_jwt["ok"].to_bool() ? "signed" : "failed", "\n");
+19 -2
View File
@@ -1,8 +1,18 @@
# http_request
Performs a bounded outbound HTTP(S) request using the runtime `curl` binary. Request fields: `method`, `url`, `headers`, `body`, `timeout_ms`, `follow_redirects`.
Performs an outbound HTTP(S) request using the runtime `curl` binary. Existing request fields remain backward-compatible: `method`, `url`, `headers`, `body`, `timeout_ms`, and `follow_redirects`.
Returns `{ status, headers, body, error }`. `headers` is a name/value map. A missing `curl` binary returns a clear `error` string.
An absent `security` map, or a map containing none of these recognized keys, keeps the legacy request behavior. If a `security` map contains any recognized key, it selects hardening and **every** recognized field below must be explicitly boolean `true`; missing, partial, non-true, or `false` fields fail closed with `invalid_request`.
- `https_only` rejects non-HTTPS URLs, IP-literal hosts, and URL userinfo.
- `public_dns_only` validates **every** DNS answer against the public IPv4 policy. IPv6 answers currently fail closed.
- `pin_dns` pins curl to one validated answer with `--resolve`, retaining the URL hostname for TLS/SNI.
- `isolated_curl` uses absolute `/usr/bin/curl`, `--disable` as argv[1], cleared environment, no proxy/config/netrc/HSTS/Alt-Svc inheritance, no redirects, a three-second connect bound, and a ten-second total bound.
- `no_redirects` makes redirect following and `follow_redirects=true` invalid composition.
Hardened requests bound body input/output to 64 KiB and response headers to 8 KiB. Async hardened requests keep curl and its descendants in the job worker process group, so cancelling that job kills that group only. Methods are limited to `GET`, `POST`, `PUT`, `PATCH`, `DELETE`, `HEAD`, and `OPTIONS`; header names/values are validated and caller-controlled `Host`, framing, connection, and expectation headers are rejected. Sensitive request bodies go to curl stdin, never argv. Errors are typed non-secret values such as `invalid_request`, `unsafe_dns`, `timeout`, `response_too_large`, `redirect_not_allowed`, `http_status`, and `network_failure`.
Returns `{ status, headers, body, error }`.
:see
>socket
@@ -12,3 +22,10 @@ DValue req; req["method"] = "GET"; req["url"] = "http://127.0.0.1/doc/index.uce"
req["headers"]["Host"] = "uce.openfu.com"; req["timeout_ms"] = (f64)2000;
DValue resp = http_request(req);
print("HTTP ", resp["status"].to_u64(), ", ", resp["body"].to_string().length(), " bytes returned\n");
:example
// Hardened requests reject IP-literal hosts before they start curl.
DValue hardened; hardened["method"]="GET"; hardened["url"]="https://127.0.0.1/";
for(String key:{"https_only","public_dns_only","pin_dns","isolated_curl","no_redirects"}) hardened["security"][key].set_bool(true);
DValue rejected=http_request(hardened);
print(rejected["error"].to_string(), "\n");
+1 -1
View File
@@ -1,6 +1,6 @@
# http_request_async
Starts the same bounded curl-backed request as `http_request()` in the file-backed async job registry and returns a job id. Use `job_await()` or `job_result()` to retrieve the HTTP result.
Starts the same request shape as `http_request()`, including its opt-in `security` hardening fields, in the file-backed async job registry and returns a job id. An unknown `security` object remains legacy; any recognized hardening key requires all five keys to be explicit `true` or the request fails closed. Hardened curl remains in this job's worker process group, so `job_cancel()` kills curl and its descendants without signalling other jobs. Use `job_await()` or `job_result()` to retrieve the HTTP result.
:see
>socket
+1
View File
@@ -13,6 +13,7 @@ return value : string containing the Memcache server's response
Executes a raw command on an open Memcache connection and returns the server response as a string.
This is the low-level escape hatch for Memcache operations that are not covered by the dedicated helpers.
UCE reads the complete framed ASCII response, including multi-value retrievals and `stats`, rather than returning a socket chunk. The complete write and read share a one-second deadline and responses are limited to 8 MiB. A timeout, malformed frame, or oversized response returns an empty string and makes that connection unusable so unread bytes cannot affect a later command. Commands ending in `noreply` and `quit` return immediately with an empty string.
:example
u64 conn = memcache_connect();
+1 -1
View File
@@ -16,7 +16,7 @@ Establishes a connection to a MySQL server and returns a request-owned pointer t
This connection handle is then used with helpers such as `mysql_query()`, `mysql_error()`, and `mysql_disconnect()`.
MySQL handles are request-scoped framework resources. Repeated `mysql_connect()` calls with the same host, credentials, and database reuse one server connection within the current request. Database identity is part of both request-local and worker-persistent pool keys. Each call creates a lease and `mysql_disconnect()` releases that lease. At request cleanup UCE returns the server connection to the current worker's persistent pool, then resets it before cross-request reuse. The pool holds up to `MYSQL_PERSISTENT_POOL_SIZE` connections per worker (default 8); set the size to 0 to disable cross-request reuse. Never store a `MySQL*` in globals, sessions, or other state that can outlive the current request.
MySQL handles are request-scoped framework resources. Repeated `mysql_connect()` calls with the same host, credentials, and database reuse one server connection within the current request. Database identity is part of both request-local and worker-persistent pool keys. Each call creates a lease and `mysql_disconnect()` releases that lease. At request cleanup UCE returns the server connection to the current worker's persistent pool, then resets it before cross-request reuse. The pool holds up to `MYSQL_PERSISTENT_POOL_SIZE` connections per worker (default 8); set the size to 0 to disable cross-request reuse. `MYSQL_PERSISTENT_POOL_IDLE_TIMEOUT_SECONDS` retires idle entries at the first later request boundary (default 300, 0 keeps unlimited idle retention). A completely idle worker has no boundary and does not proactively close sockets. Never store a `MySQL*` in globals, sessions, or other state that can outlive the current request.
:example
MySQL* db = mysql_connect();
+4 -1
View File
@@ -16,6 +16,8 @@ Executes a MySQL query and returns the resulting data, if any.
`params` provides the query parameter values used by the statement. Use named `:name` placeholders only; positional `?` placeholders are rejected.
Ordinary placeholders are escaped and quoted as SQL string values. For grammar positions that require an unquoted non-negative integer, such as `LIMIT` and `OFFSET`, append `!` to the placeholder (`:limit!`). Unsigned placeholders fail before query execution unless their value is a non-empty sequence of decimal digits. They never accept signs, whitespace, expressions, identifiers, or other SQL fragments.
The result is returned as a `DValue`, which makes it easy to iterate through rows and read fields with the usual `DValue` accessors.
After an insert, update, or delete, use `mysql_affected_rows()` to inspect how many rows changed.
@@ -24,7 +26,8 @@ After an insert, update, or delete, use `mysql_affected_rows()` to inspect how m
MySQL* db = mysql_connect();
if(db != 0)
{
DValue rows = mysql_query(db, "select 'ada@example.test' as email, 1 + 1 as total");
StringMap params; params["limit"] = "1";
DValue rows = mysql_query(db, "select 'ada@example.test' as email, 1 + 1 as total limit :limit!", params);
String email = "none"; String total = "?";
rows.each([&](DValue r, String key) { email = r["email"].to_string(); total = r["total"].to_string(); });
print(email, " / total=", total, "\n");
+1 -1
View File
@@ -27,7 +27,7 @@ The request log's `wasm-ready`, `wasm`, `workspace`, `invoke`, `collect`, and `p
Workspace birth is subdivided into `birth_policy_us`, `birth_import_us`, `birth_instantiate_us`, `birth_exports_us`, and `birth_initialize_us`. Request-context transfer reports `context_bytes`, `context_encode_us`, `context_allocate_us`, `context_write_us`, `context_guest_apply_us`, and `context_free_us`. These bounded aggregate fields expose sizes and timing only, never request values.
Wasm FastCGI workers retain up to `MYSQL_PERSISTENT_POOL_SIZE` credential-keyed MySQL connections (default `8`; set `0` to disable). UCE calls the client library's connection-reset operation before another request receives a cached connection, clearing transactions, temporary tables, session variables, and selected databases while avoiding a new authentication handshake. Same-request leases continue to share state until request cleanup.
Wasm FastCGI workers retain up to `MYSQL_PERSISTENT_POOL_SIZE` credential-keyed MySQL connections (default `8`; set `0` to disable). `MYSQL_PERSISTENT_POOL_IDLE_TIMEOUT_SECONDS` retires an idle entry at the first later request boundary (default `300`; `0` disables age eviction). UCE calls the client library's connection-reset operation before another request receives a cached connection, clearing transactions, temporary tables, session variables, and selected databases while avoiding a new authentication handshake. Same-request leases continue to share state until request cleanup.
:example
DValue perf = request_perf();
+1 -1
View File
@@ -21,7 +21,7 @@ Starts a repeating background worker process.
If a process with the same `key` is already running anywhere in the runtime instance, `task_repeat()` does not start a second worker and instead returns the PID of the existing one. Coordination is through the same shared task state used by `task()`.
`timeout` bounds the lifetime of the repeating worker. The default is ten minutes. Pass `0` only for workers that have another shutdown path.
`timeout` bounds the lifetime of the repeating worker. The default is ten minutes. Pass `0` only for workers that have another shutdown path. In Wasm, each individual callback invocation remains bounded by the runtime invocation timeout even when the repeating process lifetime is unbounded.
:example
task_repeat("doc-demo-repeat", 60.0, []() { usleep(10000); });
+6
View File
@@ -33,3 +33,9 @@ The example uses query-string routing in the same style as the PHP starter, but
Direct requests to `/examples/uce-starter/index.uce` still work, but self-links are canonicalized back to `/examples/uce-starter/`.
The demo account pages use a small file-backed user store under `/tmp/uce-starter-data/` with session-based login state.
## Optional Datastar Assets
This starter vendors Datastar v1.0.2 as `js/datastar.js` for future server-rendered interactive islands. It is not loaded by the default page shell. `lib/datastar.uce` contains small Datastar SSE formatting helpers, and `views/datastar-example.uce` is an unlinked opt-in example route.
File diff suppressed because one or more lines are too long
@@ -0,0 +1,38 @@
String datastar_data_lines(String field, String value)
{
String out = "";
for(String line : split(value, "\n"))
out += "data: " + field + " " + line + "\n";
return(out);
}
void datastar_sse_headers(Request& context)
{
context.call["app"]["page_type"] = "blank";
context.header["Content-Type"] = "text/event-stream; charset=utf-8";
context.header["Cache-Control"] = "no-cache, no-store, must-revalidate";
context.header["X-Accel-Buffering"] = "no";
}
String datastar_event(String event_name, StringMap fields)
{
String out = "event: " + event_name + "\n";
for(const auto& field : fields)
out += datastar_data_lines(field.first, field.second);
out += "\n";
return(out);
}
String datastar_patch_elements(String html)
{
StringMap fields;
fields["elements"] = html;
return(datastar_event("datastar-patch-elements", fields));
}
String datastar_patch_signals(String signals_json)
{
StringMap fields;
fields["signals"] = signals_json;
return(datastar_event("datastar-patch-signals", fields));
}
@@ -0,0 +1,20 @@
#load "../lib/app.uce"
#load "../lib/datastar.uce"
COMPONENT(Request& context)
{
if(context.get["stream"] == "1")
{
datastar_sse_headers(context);
print(datastar_patch_elements("<div id=\"datastar-example-result\">Server-rendered UCE Datastar fragment</div>"));
return;
}
<>
<h1>Datastar optional example</h1>
<p>Datastar is vendored beside the starter client libraries for future server-rendered islands. It is not loaded globally.</p>
<script type="module" src="<?= app_asset_url("js/datastar.js", context) ?>"></script>
<button class="btn" data-on:click="@get('<?= app_link("datastar-example", context) ?>&amp;stream=1')">Patch a fragment</button>
<div id="datastar-example-result">Waiting for a server patch.</div>
</>;
}
File diff suppressed because one or more lines are too long
+31
View File
@@ -20,6 +20,37 @@ CLI(Request& context)
print(cli_arg(context, "missing", "fallback"), "\n");
return;
}
if(action == "task_repeat_unbounded")
{
String marker = "/tmp/uce-task-repeat-unbounded.txt";
file_unlink(marker);
pid_t pid = task_repeat("uce-task-repeat-unbounded", 0.05, [marker]() { file_put_contents(marker, "ran"); }, 0);
usleep(200000);
bool ran = file_get_contents(marker) == "ran";
if(pid > 0) task_kill(pid, 15);
file_unlink(marker);
print(pid > 0 && ran ? "task repeat unbounded ok\n" : "task repeat unbounded failed\n");
return;
}
if(action == "mysql_unsigned_params")
{
MySQL db;
StringMap params; params["limit"] = "1";
String parsed = db.parse_query_parameters("SELECT 7 LIMIT :limit!", params);
StringMap comparison; comparison["left"] = "7"; comparison["right"] = "8";
String inequality = db.parse_query_parameters("SELECT :left!=:right", comparison);
StringMap empty_name; empty_name[""] = "1";
db.parameter_error = false; db.statement_info = "";
db.parse_query_parameters("SELECT 1 LIMIT :!", empty_name);
bool empty_rejected = db.parameter_error;
StringMap invalid; invalid["limit"] = "1;SELECT 9";
db.parameter_error = false;
db.statement_info = "";
db.parse_query_parameters("SELECT 1 LIMIT :limit!", invalid);
bool ok = parsed.find("LIMIT 1") != String::npos && inequality.find("'7'!='8'") != String::npos && empty_rejected && db.parameter_error && db.statement_info.find("must contain only decimal digits") != String::npos;
print(ok ? "mysql unsigned params ok\n" : "mysql unsigned params failed: parsed=" + parsed + ", parameter_error=" + (db.parameter_error ? "1" : "0") + ", statement=" + db.statement_info + "\n");
return;
}
context.set_status(400, "CLI Error");
print("unknown cli action: ", action, "\n");
}
+11 -3
View File
@@ -31,7 +31,7 @@ RENDER(Request& context)
StringMap colon_uri_headers = split_http_headers("GET /clock.uce?t=12:30 HTTP/1.1\r\nHost: colon.example\r\n");
check("split_http_headers() request line with colon in URI", colon_uri_headers["REQUEST_METHOD"] == "GET" && colon_uri_headers["DOCUMENT_URI"] == "/clock.uce" && colon_uri_headers["QUERY_STRING"] == "t=12:30" && colon_uri_headers["HTTP_HOST"] == "colon.example", var_dump(colon_uri_headers));
check("trim() / split_kv() / split_http_headers()", trim(" padded value ") == "padded value" && kv["alpha"] == "one" && kv["empty"] == "" && http_headers["REQUEST_METHOD"] == "GET" && http_headers["DOCUMENT_URI"] == "/demo.uce" && http_headers["QUERY_STRING"] == "x=1" && http_headers["HTTP_X_EMPTY"] == "" && leading_crlf_headers["REQUEST_METHOD"] == "GET" && leading_crlf_headers["DOCUMENT_URI"] == "/lead.uce" && leading_crlf_headers["HTTP_HOST"] == "lead.example" && header_only["REQUEST_METHOD"] == "" && header_only["HTTP_HOST"] == "example.test" && header_only["HTTP_X_TOKEN"] == "abc", trim(" padded value ") + " / " + var_dump(kv) + " / " + var_dump(http_headers) + " / " + var_dump(leading_crlf_headers) + " / " + var_dump(header_only));
check("replace()", replace("alpha-beta-beta", "beta", "done") == "alpha-done-done", replace("alpha-beta-beta", "beta", "done"));
check("replace()", replace("alpha-beta-beta", "beta", "done") == "alpha-done-done" && replace("hello", "", "X") == "hello", replace("alpha-beta-beta", "beta", "done"));
check("html_escape() attribute-safe quotes", html_escape("<&>\"Don't") == "&lt;&amp;&gt;&quot;Don&#39;t", html_escape("<&>\"Don't"));
check("regex_match()", regex_match("[A-Z][a-z]+", "Alice") && !regex_match("[A-Z][a-z]+", "Alice!"), "full-string validation");
@@ -212,11 +212,19 @@ RENDER(Request& context)
String decoded_base64 = base64_decode(encoded_base64, base64_ok);
bool invalid_base64_ok = true;
base64_decode("AA=A", invalid_base64_ok);
check("base64_encode() / base64_decode() binary-safe", base64_ok && decoded_base64 == binary_payload && decoded_base64.size() == binary_payload.size() && !invalid_base64_ok, encoded_base64 + " bytes=" + std::to_string((u64)decoded_base64.size()));
bool early_padding_base64_ok = true;
base64_decode("A=AA", early_padding_base64_ok);
check("base64_encode() / base64_decode() binary-safe", base64_ok && decoded_base64 == binary_payload && decoded_base64.size() == binary_payload.size() && !invalid_base64_ok && !early_padding_base64_ok, encoded_base64 + " bytes=" + std::to_string((u64)decoded_base64.size()));
String utf8_sample = "A\xC3\xA9";
auto utf8_parts = split_utf8(utf8_sample);
check("split_utf8()", utf8_parts.size() == 2, "count=" + std::to_string(utf8_parts.size()));
String truncated_utf8 = "abc";
truncated_utf8.push_back((char)0xE2);
truncated_utf8.push_back((char)0x82);
auto truncated_utf8_parts = split_utf8(truncated_utf8);
String lone_utf8_lead(1, (char)0xF0);
auto compound_lone_utf8_parts = split_utf8(lone_utf8_lead, true);
check("split_utf8()", utf8_parts.size() == 2 && truncated_utf8_parts.size() == 4 && truncated_utf8_parts[3].size() == 2 && compound_lone_utf8_parts.size() == 1 && compound_lone_utf8_parts[0] == lone_utf8_lead, "count=" + std::to_string(utf8_parts.size()) + " truncated=" + std::to_string(truncated_utf8_parts.size()));
DValue payload;
payload["name"] = "uce";
+75
View File
@@ -0,0 +1,75 @@
#include "testlib.h"
RENDER(Request& context)
{
u64 passed = 0;
u64 failed = 0;
u64 skipped = 0;
auto check = [&](String name, bool ok, String detail)
{
site_tests_case(name, ok ? "pass" : "fail", detail);
if(ok) passed++; else failed++;
};
site_tests_page_start("Structured crypto operations", "Algorithm-selected P-256 JWK creation and ES256 JWT signing.");
DValue key_request;
key_request["operation"] = "key_generate";
key_request["algorithm"] = "ES256";
DValue key = crypto_operation(key_request);
DValue header;
header["alg"] = "none";
header["kid"] = key["kid"];
DValue claims;
claims["iss"] = "https://client.example";
claims["sub"] = "client.example";
DValue sign_request;
sign_request["operation"] = "jwt_sign";
sign_request["algorithm"] = "ES256";
sign_request["private_jwk"] = key["private_jwk"];
sign_request["protected_header"] = header;
sign_request["claims"] = claims;
DValue signed_result = crypto_operation(sign_request);
String jwt = signed_result["jwt"].to_string();
StringList parts = split(jwt, ".");
String header_b64 = parts.size() == 3 ? parts[0] : "";
while(header_b64.size() % 4) header_b64 += "=";
String decoded_header = base64_decode(header_b64);
DValue wrong_curve = key["private_jwk"];
wrong_curve["crv"] = "P-384";
DValue malformed = key["private_jwk"];
malformed["x"] = "not_base64url=";
DValue wrong_request = sign_request;
wrong_request["private_jwk"] = wrong_curve;
DValue malformed_request = sign_request;
malformed_request["private_jwk"] = malformed;
DValue cose_request;
cose_request["operation"] = "cose_es256_parse";
cose_request["algorithm"] = "ES256";
cose_request["cose_key_base64url"] = "pQECAyYgASFYIGsX0fLhLEJH-Lzm5WOkQPJ3A32BLeszoPShOUXYmMKWIlggT-NC4v4af5uO5-tKfA-eFivOM1drMV7Oy7ZAaDe_UfU";
DValue cose = crypto_operation(cose_request);
DValue verify_request;
verify_request["operation"] = "es256_verify";
verify_request["algorithm"] = "ES256";
verify_request["cose_key_base64url"] = cose_request["cose_key_base64url"];
verify_request["message_base64url"] = "d2ViYXV0aG4gZml4ZWQgbWVzc2FnZQ";
verify_request["signature_der_base64url"] = "MEUCIQCkatZK1VVsjk17uvyzyhjdAkMNWXPjxSOMqWcjmM_8XAIgDaSk3Qufyd0_6r9Dm9A8RQbFco-FdTBulq7bvRGoBC4";
DValue unsupported;
unsupported["operation"] = "encrypt";
unsupported["algorithm"] = "ES256";
DValue list_header;
list_header.set_array();
DValue list_item;
list_item = "not-an-object";
list_header.push(list_item);
DValue list_request = sign_request;
list_request["protected_header"] = list_header;
DValue control_request = sign_request;
control_request["claims"]["bad"] = String("control\nbyte");
check("crypto_operation() generates an ES256 key", key["ok"].to_bool() && key["private_jwk"]["kty"].to_string() == "EC" && key["private_jwk"]["crv"].to_string() == "P-256" && key["public_jwk"]["d"].to_string() == "" && key["kid"].to_string() == key["thumbprint"].to_string(), key["kid"].to_string());
check("crypto_operation() signs ES256 JWTs", signed_result["ok"].to_bool() && parts.size() == 3 && parts[2].size() == 86 && decoded_header.find("ES256") != String::npos, signed_result["error"].to_string());
check("crypto_operation() rejects invalid signing requests", !crypto_operation(wrong_request)["ok"].to_bool() && !crypto_operation(malformed_request)["ok"].to_bool() && !crypto_operation(list_request)["ok"].to_bool() && crypto_operation(control_request)["error"].to_string() == "invalid_request", "signing validation");
check("crypto_operation() parses an ES256 COSE key", cose["ok"].to_bool(), cose["error"].to_string());
check("crypto_operation() verifies ES256 DER signatures", crypto_operation(verify_request)["ok"].to_bool() && crypto_operation(verify_request)["valid"].to_bool(), crypto_operation(verify_request)["error"].to_string());
check("crypto_operation() rejects unsupported operations", crypto_operation(unsupported)["error"].to_string() == "unsupported_operation", crypto_operation(unsupported)["error"].to_string());
site_tests_summary(passed, failed, skipped, "Structured crypto tests generate ephemeral P-256 keys and retain no key material.");
site_tests_page_end();
}
+1
View File
@@ -8,6 +8,7 @@ markdown.uce|Markdown|Markdown parsing, rendering, and component hooks.|http sui
units.uce|Units|unit_call(), lifecycle hooks, and unit metadata.|http suite uce public|Units|1|1
websockets.ws.uce|WebSockets|Browser-driven WebSocket helper checks.|http suite uce public websocket|WebSockets|1|1
io.uce|Filesystem|Filesystem helpers that are restricted outside trusted networks.|http suite uce internal|Filesystem|1|1
crypto_operation.uce|Structured Crypto|Algorithm-selected key generation and JWT signing coverage.|http suite uce public crypto|Structured crypto operations|1|1
sqlite.uce|SQLite|SQLite connector with prepared named parameters and DValue rows.|http suite uce internal sqlite|SQLite|1|1
zip.uce|ZIP|Archive helpers that create and extract temporary server-side files.|http suite uce internal|ZIP|1|1
services.uce|Sockets And Services|Network/service helpers that are restricted outside trusted networks.|http suite uce internal|Sockets And Services|1|1
+4
View File
@@ -31,6 +31,8 @@ RENDER(Request& context)
};
String nested = preprocessor_nested_literal();
String trailing_backslash = "C:\\Users\\";
String escaped_quote = "before\"after";
site_tests_page_start("Preprocessor", "Regression coverage for literal output rewriting and parser edge cases.");
?>
@@ -43,6 +45,8 @@ RENDER(Request& context)
check("raw string terminator in nested literal", contains(nested, "nested )\" marker"), nested);
check("entrypoint @fragment attribute captures output", context.call["fragments"]["preprocessor-test"].to_string() == "fragment attr once", context.call["fragments"]["preprocessor-test"].to_string());
check("inline code island after dangerous literal", true, "parser returned to C++ after rendering literal content containing )\"");
check("quote scanner handles trailing escaped backslash", trailing_backslash == "C:\\Users\\", trailing_backslash);
check("quote scanner retains escaped quote content", escaped_quote == "before\"after", escaped_quote);
site_tests_summary(passed, failed, skipped, "Literal content containing the C++ raw-string terminator sequence must compile and render unchanged.");
site_tests_page_end();
+43 -5
View File
@@ -28,14 +28,20 @@ RENDER(Request& context)
u64 sockfd = socket_connect("127.0.0.1", 80);
if(sockfd != 0)
{
u64 closed_handle = socket_connect("127.0.0.1", 80);
bool opaque_handles = sockfd == 1 && closed_handle == 2;
socket_close(closed_handle);
socket_close(closed_handle);
socket_close(999999);
bool closed_write_rejected = !socket_write(closed_handle, "invalid");
bool write_ok = socket_write(sockfd, "GET /tests/index.uce HTTP/1.0\r\nHost: uce.openfu.com\r\n\r\n");
String response = socket_read(sockfd, 4096, 2);
socket_close(sockfd);
bool has_nul = response.find(String("\0", 1)) != String::npos;
mark(
"socket_connect() / socket_write() / socket_read()",
(write_ok && response.find("200 OK") != String::npos && !has_nul) ? "pass" : "fail",
response.substr(0, response.length() > 220 ? 220 : response.length()) + (has_nul ? " [unexpected NUL]" : "")
"socket handles reject invalid and closed indices",
(opaque_handles && closed_write_rejected && write_ok && response.find("200 OK") != String::npos && !has_nul) ? "pass" : "fail",
"handles=" + std::to_string(sockfd) + "/" + std::to_string(closed_handle) + " " + response.substr(0, response.length() > 180 ? 180 : response.length()) + (has_nul ? " [unexpected NUL]" : "")
);
}
else
@@ -90,10 +96,25 @@ RENDER(Request& context)
StringMap mem_multi = memcache_get_multiple(memfd, {"site-tests-key", "site-tests-key2"});
memcache_delete(memfd, "site-tests-key");
memcache_delete(memfd, "site-tests-key2");
String large_mem_value(200 * 1024, 'x');
large_mem_value.replace(1024, 7, "\r\nEND\r\n");
bool large_mem_stored = memcache_set(memfd, "site-tests-large", large_mem_value);
String large_mem_loaded = memcache_get(memfd, "site-tests-large");
String second_large_mem_value(160 * 1024, 'y');
bool second_large_mem_stored = memcache_set(memfd, "site-tests-large-2", second_large_mem_value);
StringMap large_mem_multi = memcache_get_multiple(memfd, {"site-tests-large", "site-tests-large-2"});
String no_reply = memcache_command(memfd, "set site-tests-noreply 0 60 5 noreply\r\nvalue");
String no_reply_value = memcache_get(memfd, "site-tests-noreply");
String stats_mem = memcache_command(memfd, "stats");
String stats_reset_mem = memcache_command(memfd, "stats reset");
String after_stats_reset = memcache_get(memfd, "site-tests-noreply");
bool large_mem_deleted = memcache_delete(memfd, "site-tests-large");
bool second_large_mem_deleted = memcache_delete(memfd, "site-tests-large-2");
bool no_reply_deleted = memcache_delete(memfd, "site-tests-noreply");
mark(
"memcache_connect() / memcache_command() / memcache_get_multiple()",
mem_value == "value-1" && contains(raw_mem, "VALUE site-tests-key") && mem_multi["site-tests-key"] == "value-1" && mem_multi["site-tests-key2"] == "value-2" ? "pass" : "fail",
"memcache value=" + mem_value + " raw=" + raw_mem + " multi=" + var_dump(mem_multi)
mem_value == "value-1" && contains(raw_mem, "VALUE site-tests-key") && mem_multi["site-tests-key"] == "value-1" && mem_multi["site-tests-key2"] == "value-2" && large_mem_stored && large_mem_loaded == large_mem_value && second_large_mem_stored && large_mem_multi["site-tests-large"] == large_mem_value && large_mem_multi["site-tests-large-2"] == second_large_mem_value && no_reply == "" && no_reply_value == "value" && contains(stats_mem, "STAT ") && trim(stats_reset_mem) == "RESET" && after_stats_reset == "value" && large_mem_deleted && second_large_mem_deleted && no_reply_deleted ? "pass" : "fail",
"memcache value=" + mem_value + " raw=" + raw_mem + " multi=" + var_dump(mem_multi) + " large=" + std::to_string(large_mem_loaded.size()) + " multi-large=" + std::to_string(large_mem_multi["site-tests-large"].size() + large_mem_multi["site-tests-large-2"].size()) + " noreply=" + no_reply_value + " stats=" + std::to_string(stats_mem.size()) + "/" + trim(stats_reset_mem) + " after-reset=" + after_stats_reset + " deleted=" + (large_mem_deleted && second_large_mem_deleted && no_reply_deleted ? "true" : "false")
);
}
@@ -115,6 +136,15 @@ RENDER(Request& context)
parsed_underscore.find("'Ada'") != String::npos && parsed_underscore.find("_name") == String::npos ? "pass" : "fail",
parsed_underscore
);
StringMap quoted_params;
quoted_params["name_looking_text"] = "SUBSTITUTED";
String escaped_quote_query = "SELECT 'a\\':name_looking_text'";
String parsed_escaped_quote = placeholder_guard.parse_query_parameters(escaped_quote_query, quoted_params);
mark(
"mysql named placeholders stay opaque after escaped quotes",
parsed_escaped_quote.find(":name_looking_text") != String::npos && parsed_escaped_quote.find("SUBSTITUTED") == String::npos ? "pass" : "fail",
parsed_escaped_quote
);
MySQL unavailable_mysql;
unavailable_mysql.connect("127.0.0.1", "__uce_intentionally_missing__", "not-a-real-password");
mark(
@@ -158,6 +188,14 @@ RENDER(Request& context)
inserted == 2 && updated == 1 && selected == 0 ? "pass" : "fail",
"inserted=" + std::to_string((u64)inserted) + " updated=" + std::to_string((u64)updated) + " selected=" + std::to_string((u64)selected)
);
mysql.query("INSERT INTO uce_affected_rows_test VALUES (1,30)");
u32 duplicate_error_code = mysql._preload_next_error_code;
String duplicate_error = mysql.error();
mark(
"mysql query failures retain server diagnostics",
duplicate_error_code > 1 && duplicate_error != "" && duplicate_error != "Unknown server error" ? "pass" : "fail",
"code=" + std::to_string((u64)duplicate_error_code) + " error=" + duplicate_error
);
}
DValue mysql_perf = request_perf();
String mysql_operations = json_encode(mysql_perf["mysql_operations"]);
+12
View File
@@ -18,6 +18,16 @@ RENDER(Request& context)
auto unit_paths = units_list();
DValue info = unit_info("call_helpers.uce");
DValue relative_info = unit_info("components/../relative-child.uce");
String outside_unit = "/tmp/uce-site-tests-outside-unit.uce";
file_put_contents(outside_unit, "RENDER(Request& context) {}");
DValue outside_info = unit_info(outside_unit);
file_unlink(outside_unit);
String generated_dir = path_join(context.params["UCE_BIN_DIRECTORY"], "site-tests-generated");
mkdir(generated_dir);
String generated_unit = path_join(generated_dir, "generated.uce");
file_put_contents(generated_unit, "RENDER(Request& context) {}");
DValue generated_info = unit_info(generated_unit);
file_unlink(generated_unit);
ob_start();
unit_call("call_helpers.uce", "emit_marker");
@@ -32,6 +42,8 @@ RENDER(Request& context)
check("units_list()", unit_paths.size() > 0, "count=" + std::to_string(unit_paths.size()));
check("unit_info()", info["path"].to_string() != "", json_encode(info));
check("compiler canonicalizes relative unit paths", relative_info["path"].to_string().find("/../") == String::npos && str_ends_with(relative_info["path"].to_string(), "/site/tests/relative-child.uce"), json_encode(relative_info));
check("compiler rejects units outside document root", outside_info["path"].to_string() == "", json_encode(outside_info));
check("compiler accepts generated units in BIN_DIRECTORY", generated_info["path"].to_string() == generated_unit, json_encode(generated_info));
check("unit_compile()", unit_compile("call_helpers.uce"), "call_helpers.uce");
check("unit_call()", call_output.find("UNIT_CALL_EXPORT_OK") != String::npos, call_output);
check("unit_render()", render_output.find("data-unit-render=\"ok\"") != String::npos, render_output);
+23 -3
View File
@@ -362,6 +362,7 @@ FastCGIServer::listen(const std::string& local_path)
close(server_socket);
throw;
}
server_socket_types[server_socket] = 'F';
return server_socket;
}
@@ -400,6 +401,23 @@ FastCGIServer::close_http_listeners()
}
}
void
FastCGIServer::close_listeners_except(char type)
{
for(std::vector<int>::iterator it = server_sockets.begin(); it != server_sockets.end();)
{
int socket_handle = *it;
if(server_socket_types[socket_handle] != type)
{
close(socket_handle);
server_socket_types.erase(socket_handle);
it = server_sockets.erase(it);
continue;
}
++it;
}
}
bool
FastCGIServer::is_http_like_type(char type)
{
@@ -672,10 +690,12 @@ FastCGIServer::process(int timeout_ms)
Connection* doomed_connection = it->second;
client_sockets.erase(it++);
delete doomed_connection;
if(calls_until_termination != -1 && client_sockets.size() == 0)
{
if(calls_until_termination > 0)
calls_until_termination -= 1;
if(calls_until_termination <= 0)
if(calls_until_termination == 0)
{
close_listeners_except(0);
if(client_sockets.empty())
exit(0);
}
}
+1
View File
@@ -96,6 +96,7 @@ public:
std::map<int, Connection*> client_sockets;
void close_http_listeners();
void close_listeners_except(char type);
void read_fgci(Connection&);
static bool is_http_like_type(char type);
Connection* open_client_connection(int server_socket, int client_socket);
+8 -2
View File
@@ -74,8 +74,14 @@ void compiler_code_state_consume(CompilerCodeState& state, String& buffer, const
if(state.inside_quote)
{
if(state.quote_char == c && (i == 0 || content[i-1] != '\\'))
state.inside_quote = false;
if(state.quote_char == c)
{
u32 backslashes = 0;
for(u32 j = i; j > 0 && content[j - 1] == '\\'; j -= 1)
backslashes += 1;
if(backslashes % 2 == 0)
state.inside_quote = false;
}
return;
}
+39 -6
View File
@@ -646,7 +646,10 @@ int compiler_open_lock_file(String file_name, String purpose, bool nonblocking =
(void)purpose;
auto lock_dir = dirname(file_name);
if(lock_dir != "")
mkdir(lock_dir);
{
std::error_code error;
std::filesystem::create_directories(lock_dir, error);
}
int fdlock = open(file_name.c_str(), O_RDWR | O_CREAT, 0666);
if(fdlock == -1 && (errno == EACCES || errno == EPERM))
fdlock = open(file_name.c_str(), O_RDONLY | O_CLOEXEC);
@@ -676,7 +679,10 @@ int compiler_open_lock_file_bounded(String file_name, String purpose, CompilerDe
return(compiler_open_lock_file(file_name, purpose));
auto lock_dir = dirname(file_name);
if(lock_dir != "")
mkdir(lock_dir);
{
std::error_code error;
std::filesystem::create_directories(lock_dir, error);
}
int fdlock = open(file_name.c_str(), O_RDWR | O_CREAT, 0666);
if(fdlock == -1 && (errno == EACCES || errno == EPERM))
fdlock = open(file_name.c_str(), O_RDONLY | O_CLOEXEC);
@@ -734,14 +740,41 @@ static void compiler_mark_source_generation_nonblocking(Request* context)
String compiler_normalize_unit_path(Request* context, String file_name)
{
file_name = trim(file_name);
if(file_name == "")
if(file_name == "" || !context || !context->server)
return("");
if(file_name[0] != '/')
file_name = expand_path(file_name, context->server->config["COMPILER_SYS_PATH"]);
String canonical = path_real(file_name);
if(canonical != "")
return(canonical);
return(file_name);
if(canonical == "")
{
std::error_code error;
canonical = std::filesystem::weakly_canonical(file_name, error).string();
if(error || canonical == "")
return("");
}
StringList allowed_roots = {
context->params["DOCUMENT_ROOT"],
context->server->config["PRECOMPILE_FILES_IN"],
context->server->config["HTTP_DOCUMENT_ROOT"],
path_join(context->server->config["COMPILER_SYS_PATH"], context->server->config["SITE_DIRECTORY"]),
context->server->config["BIN_DIRECTORY"]
};
for(String allowed_root : allowed_roots)
{
allowed_root = trim(allowed_root);
if(allowed_root == "")
continue;
if(allowed_root[0] != '/')
allowed_root = expand_path(allowed_root, context->server->config["COMPILER_SYS_PATH"]);
allowed_root = path_real(allowed_root);
if(allowed_root == "")
continue;
if(allowed_root[allowed_root.length() - 1] != '/')
allowed_root += "/";
if(canonical + "/" == allowed_root || str_starts_with(canonical, allowed_root))
return(canonical);
}
return("");
}
bool compiler_is_known_unit_file(String file_name)
+8 -6
View File
@@ -140,6 +140,8 @@ bool contains(String haystack, String needle)
String replace(String s, String search, String replace_with)
{
if(search == "")
return(s);
s64 last_spos = 0;
auto spos = s.find(search);
if(spos == std::string::npos)
@@ -791,7 +793,7 @@ DValue array_merge(DValue a, DValue b)
StringList split_utf8(String s, bool compound_characters)
{
StringList result;
auto len = s.size();
s64 len = (s64)s.size();
String codepoint = "";
for(s64 i = 0; i < len; i++)
{
@@ -800,13 +802,13 @@ StringList split_utf8(String s, bool compound_characters)
{
codepoint = "";
codepoint.append(1, c);
if(is_bit_set(c, 6))
if(is_bit_set(c, 6) && i + 1 < len)
{
codepoint.append(1, s[++i]);
if(is_bit_set(c, 5))
if(is_bit_set(c, 5) && i + 1 < len)
{
codepoint.append(1, s[++i]);
if(is_bit_set(c, 4))
if(is_bit_set(c, 4) && i + 1 < len)
{
codepoint.append(1, s[++i]);
}
@@ -837,7 +839,7 @@ StringList split_utf8(String s, bool compound_characters)
join_next = true;
last_was_regional = false;
}
else if(s[0] == '\xF0' && s[1] == '\x9F' && s[2] == '\x87' && s[3] >= '\xA6' && s[3] <= '\xBF') // Regional indicator letters
else if(s.size() == 4 && s[0] == '\xF0' && s[1] == '\x9F' && s[2] == '\x87' && s[3] >= '\xA6' && s[3] <= '\xBF') // Regional indicator letters
{
if(last_was_regional)
{
@@ -850,7 +852,7 @@ StringList split_utf8(String s, bool compound_characters)
last_was_regional = true;
}
}
else if(s[0] == '\xEF' && s[1] == '\xB8' && s[2] >= '\x80' && s[2] <= '\x8F') // Variation selector
else if(s.size() == 3 && s[0] == '\xEF' && s[1] == '\xB8' && s[2] >= '\x80' && s[2] <= '\x8F') // Variation selector
{
compound_result[compound_result.size()-1] += s;
last_was_regional = false;
+443
View File
@@ -41,6 +41,7 @@ void SHA1Final(unsigned char digest[20], SHA1_CTX* context);
#include <string.h>
#include <sys/types.h> /* for u_int*_t */
#include "hash.h"
#include "uri.h"
#ifndef BYTE_ORDER
#if (BSD >= 199103)
@@ -422,8 +423,69 @@ bool crypto_equal_native(String a, String b)
return(diff == 0);
}
static bool uce_crypto_utf8_string(String value)
{
for(size_t i = 0; i < value.size();)
{
u8 c = (u8)value[i];
if(c < 0x80) { i++; continue; }
size_t need = c >= 0xC2 && c <= 0xDF ? 1 : (c >= 0xE0 && c <= 0xEF ? 2 : (c >= 0xF0 && c <= 0xF4 ? 3 : 0));
if(need == 0 || i + need >= value.size()) return(false);
for(size_t j = 1; j <= need; j++) if(((u8)value[i + j] & 0xC0) != 0x80) return(false);
u8 second = (u8)value[i + 1];
if((c == 0xE0 && second < 0xA0) || (c == 0xED && second >= 0xA0) || (c == 0xF0 && second < 0x90) || (c == 0xF4 && second >= 0x90)) return(false);
i += need + 1;
}
return(true);
}
static bool uce_crypto_utf8_json_string(String value)
{
if(!uce_crypto_utf8_string(value)) return(false);
for(unsigned char c : value) if(c < 0x20) return(false);
return(true);
}
static bool uce_crypto_value_valid(const DValue& value, size_t depth, size_t& nodes, size_t& bytes)
{
if(depth > 16 || ++nodes > 256) return(false);
const DValue& item = value.deref();
if(item.type == 'S')
{
bytes += item._String.size();
return(bytes <= 32768 && uce_crypto_utf8_json_string(item._String));
}
if(item.type == 'F') return(std::isfinite(item._float));
if(item.type == 'B') return(true);
if(item.type != 'M') return(false);
bool list = item.is_list();
for(const auto& child : item._map)
{
if(!list)
{
bytes += child.first.size();
if(bytes > 32768 || !uce_crypto_utf8_json_string(child.first)) return(false);
}
if(!uce_crypto_value_valid(child.second, depth + 1, nodes, bytes)) return(false);
}
return(true);
}
bool crypto_operation_request_valid(DValue request)
{
const DValue& root = request.deref();
if(root.type != 'M' || root.is_list()) return(false);
size_t nodes = 0, bytes = 0;
return(uce_crypto_value_valid(root, 0, nodes, bytes));
}
#ifndef __UCE_WASM_CORE__
#include <openssl/bn.h>
#include <openssl/core_names.h>
#include <openssl/ecdsa.h>
#include <openssl/evp.h>
#include <openssl/params.h>
#include <openssl/param_build.h>
#include <openssl/rand.h>
namespace {
@@ -525,4 +587,385 @@ bool password_needs_rehash_native(String encoded)
String salt, digest;
return(!uce_password_parts(encoded, n, r, p, salt, digest) || n != UCE_PASSWORD_SCRYPT_N || r != UCE_PASSWORD_SCRYPT_R || p != UCE_PASSWORD_SCRYPT_P);
}
namespace {
static constexpr size_t UCE_ES256_COORDINATE_BYTES = 32;
static constexpr size_t UCE_ES256_SIGNATURE_BYTES = 64;
static constexpr size_t UCE_ES256_JSON_MAX = 16 * 1024;
static constexpr size_t UCE_ES256_VALUE_MAX = 256;
static constexpr size_t UCE_ES256_DEPTH_MAX = 16;
static constexpr size_t UCE_ES256_COORDINATE_BASE64URL_MAX = 43;
static constexpr size_t UCE_CBOR_MAX_BYTES = 16 * 1024;
static constexpr size_t UCE_CBOR_MAX_BASE64URL = 21846;
static constexpr size_t UCE_ES256_DER_MAX_BYTES = 144;
static constexpr size_t UCE_ES256_DER_BASE64URL_MAX = 192;
struct UcePkeyDeleter { void operator()(EVP_PKEY* value) const { EVP_PKEY_free(value); } };
struct UcePkeyCtxDeleter { void operator()(EVP_PKEY_CTX* value) const { EVP_PKEY_CTX_free(value); } };
struct UceEcdsaSigDeleter { void operator()(ECDSA_SIG* value) const { ECDSA_SIG_free(value); } };
struct UceBnDeleter { void operator()(BIGNUM* value) const { BN_clear_free(value); } };
struct UceParamBldDeleter { void operator()(OSSL_PARAM_BLD* value) const { OSSL_PARAM_BLD_free(value); } };
struct UceParamsDeleter { void operator()(OSSL_PARAM* value) const { OSSL_PARAM_free(value); } };
static String uce_base64url_encode(const unsigned char* bytes, size_t size)
{
String encoded = base64_encode(String((const char*)bytes, size));
encoded = replace(replace(encoded, "+", "-"), "/", "_");
while(!encoded.empty() && encoded.back() == '=') encoded.pop_back();
return(encoded);
}
static bool uce_base64url_decode(String encoded, String& decoded, size_t max_encoded)
{
if(encoded.empty() || encoded.size() > max_encoded || encoded.find('=') != String::npos || encoded.size() % 4 == 1)
return(false);
for(char c : encoded)
if(!(c >= 'A' && c <= 'Z') && !(c >= 'a' && c <= 'z') && !(c >= '0' && c <= '9') && c != '-' && c != '_')
return(false);
String padded = replace(replace(encoded, "-", "+"), "_", "/");
while(padded.size() % 4) padded += "=";
bool ok = false;
decoded = base64_decode(padded, ok);
return(ok && uce_base64url_encode((const unsigned char*)decoded.data(), decoded.size()) == encoded);
}
static bool uce_es256_json_value(const DValue& value, size_t depth, size_t& values, size_t& bytes)
{
if(depth > UCE_ES256_DEPTH_MAX || ++values > UCE_ES256_VALUE_MAX)
return(false);
const DValue& item = value.deref();
if(item.type == 'S')
return((bytes += item._String.size()) <= UCE_ES256_JSON_MAX);
if(item.type == 'F' || item.type == 'B')
return(true);
if(item.type != 'M')
return(false);
for(const auto& child : item._map)
{
if((bytes += child.first.size()) > UCE_ES256_JSON_MAX || !uce_es256_json_value(child.second, depth + 1, values, bytes))
return(false);
}
return(true);
}
static bool uce_es256_json_map(const DValue& value)
{
if(value.deref().type != 'M' || value.deref().is_list())
return(false);
size_t values = 0, bytes = 0;
return(uce_es256_json_value(value, 0, values, bytes));
}
static bool uce_es256_jwk_string(const DValue& jwk, const String& field, String& value)
{
const DValue* found = jwk.key(field);
if(!found)
return(false);
const DValue& item = found->deref();
if(item.type != 'S' || item._String.empty() || item._String.size() > 32768)
return(false);
value = item._String;
return(true);
}
static std::unique_ptr<EVP_PKEY, UcePkeyDeleter> uce_es256_key_from_jwk(const DValue& jwk)
{
if(jwk.deref().type != 'M')
return(nullptr);
String kty, crv, x64, y64, d64, x, y, d;
if(!uce_es256_jwk_string(jwk, "kty", kty) || !uce_es256_jwk_string(jwk, "crv", crv) || !uce_es256_jwk_string(jwk, "x", x64) || !uce_es256_jwk_string(jwk, "y", y64) || !uce_es256_jwk_string(jwk, "d", d64) ||
kty != "EC" || crv != "P-256" || !uce_base64url_decode(x64, x, UCE_ES256_COORDINATE_BASE64URL_MAX) || !uce_base64url_decode(y64, y, UCE_ES256_COORDINATE_BASE64URL_MAX) || !uce_base64url_decode(d64, d, UCE_ES256_COORDINATE_BASE64URL_MAX) ||
x.size() != UCE_ES256_COORDINATE_BYTES || y.size() != UCE_ES256_COORDINATE_BYTES || d.size() != UCE_ES256_COORDINATE_BYTES)
return(nullptr);
unsigned char public_key[65];
public_key[0] = 4;
memcpy(public_key + 1, x.data(), x.size());
memcpy(public_key + 33, y.data(), y.size());
std::unique_ptr<BIGNUM, UceBnDeleter> private_bn(BN_bin2bn((const unsigned char*)d.data(), d.size(), 0));
std::unique_ptr<OSSL_PARAM_BLD, UceParamBldDeleter> builder(OSSL_PARAM_BLD_new());
if(!private_bn || !builder || OSSL_PARAM_BLD_push_utf8_string(builder.get(), OSSL_PKEY_PARAM_GROUP_NAME, "prime256v1", 0) <= 0 ||
OSSL_PARAM_BLD_push_octet_string(builder.get(), OSSL_PKEY_PARAM_PUB_KEY, public_key, sizeof(public_key)) <= 0 ||
OSSL_PARAM_BLD_push_BN(builder.get(), OSSL_PKEY_PARAM_PRIV_KEY, private_bn.get()) <= 0)
return(nullptr);
std::unique_ptr<OSSL_PARAM, UceParamsDeleter> params(OSSL_PARAM_BLD_to_param(builder.get()));
std::unique_ptr<EVP_PKEY_CTX, UcePkeyCtxDeleter> ctx(EVP_PKEY_CTX_new_from_name(0, "EC", 0));
EVP_PKEY* raw = 0;
if(!params || !ctx || EVP_PKEY_fromdata_init(ctx.get()) <= 0 || EVP_PKEY_fromdata(ctx.get(), &raw, EVP_PKEY_KEYPAIR, params.get()) <= 0)
return(nullptr);
std::unique_ptr<EVP_PKEY, UcePkeyDeleter> key(raw);
std::unique_ptr<EVP_PKEY_CTX, UcePkeyCtxDeleter> check(EVP_PKEY_CTX_new(key.get(), 0));
return(check && EVP_PKEY_public_check(check.get()) > 0 && EVP_PKEY_private_check(check.get()) > 0 && EVP_PKEY_pairwise_check(check.get()) > 0 ? std::move(key) : nullptr);
}
static bool uce_es256_key_coordinates(EVP_PKEY* key, String& x, String& y, String& d)
{
unsigned char public_key[65]; size_t public_key_size = sizeof(public_key);
BIGNUM* private_bn = 0;
if(EVP_PKEY_get_octet_string_param(key, OSSL_PKEY_PARAM_PUB_KEY, public_key, sizeof(public_key), &public_key_size) <= 0 || public_key_size != sizeof(public_key) || public_key[0] != 4 ||
EVP_PKEY_get_bn_param(key, OSSL_PKEY_PARAM_PRIV_KEY, &private_bn) <= 0)
return(false);
std::unique_ptr<BIGNUM, UceBnDeleter> private_key(private_bn);
if(BN_bn2binpad(private_key.get(), (unsigned char*)d.data(), UCE_ES256_COORDINATE_BYTES) != UCE_ES256_COORDINATE_BYTES)
return(false);
x.assign((const char*)public_key + 1, UCE_ES256_COORDINATE_BYTES);
y.assign((const char*)public_key + 33, UCE_ES256_COORDINATE_BYTES);
return(true);
}
enum class UceCborKind { Unsigned, Negative, Bytes, Text, Array, Map };
struct UceCbor { UceCborKind kind; u64 number = 0; String bytes; std::vector<UceCbor> items; };
static constexpr size_t UCE_CBOR_MAX_NODES = 256, UCE_CBOR_MAX_DEPTH = 16;
static bool uce_cbor_uint(const String& in, size_t& p, u8 ai, u64& n)
{
if(ai < 24) { n = ai; return(true); }
size_t count = ai == 24 ? 1 : ai == 25 ? 2 : ai == 26 ? 4 : ai == 27 ? 8 : 0;
if(!count || p > in.size() || count > in.size() - p) return(false);
n = 0;
for(size_t i = 0; i < count; i++)
{
if(n > (UINT64_MAX - (u8)in[p + i]) / 256) return(false);
n = n * 256 + (u8)in[p + i];
}
p += count;
return((count == 1 && n >= 24) || (count == 2 && n > 0xff) || (count == 4 && n > 0xffff) || (count == 8 && n > 0xffffffff));
}
static bool uce_cbor_equal(const UceCbor& left, const UceCbor& right)
{
if(left.kind != right.kind || left.number != right.number || left.bytes != right.bytes || left.items.size() != right.items.size()) return(false);
for(size_t i = 0; i < left.items.size(); i++) if(!uce_cbor_equal(left.items[i], right.items[i])) return(false);
return(true);
}
static bool uce_cbor_read(const String& in, size_t& p, UceCbor& out, size_t depth, size_t& nodes)
{
if(p >= in.size() || depth > UCE_CBOR_MAX_DEPTH || ++nodes > UCE_CBOR_MAX_NODES) return(false);
u8 header = (u8)in[p++], major = header >> 5, ai = header & 31;
u64 n = 0;
if(ai == 31 || !uce_cbor_uint(in, p, ai, n)) return(false);
if(major == 0 || major == 1)
{
out.kind = major ? UceCborKind::Negative : UceCborKind::Unsigned;
out.number = n;
return(true);
}
if(major == 2 || major == 3)
{
if(n > UCE_CBOR_MAX_BYTES || n > in.size() - p || (major == 3 && !uce_crypto_utf8_string(String(in.data() + p, (size_t)n)))) return(false);
out.kind = major == 2 ? UceCborKind::Bytes : UceCborKind::Text;
out.bytes.assign(in.data() + p, (size_t)n);
p += (size_t)n;
return(true);
}
if((major != 4 && major != 5) || n > UCE_CBOR_MAX_NODES) return(false);
size_t item_count = (size_t)n * (major == 5 ? 2 : 1);
if(item_count > UCE_CBOR_MAX_NODES - nodes) return(false);
out.kind = major == 4 ? UceCborKind::Array : UceCborKind::Map;
out.items.reserve(item_count);
for(size_t i = 0; i < item_count; i++)
{
UceCbor child;
if(!uce_cbor_read(in, p, child, depth + 1, nodes)) return(false);
if(major == 5 && !(i & 1))
for(size_t previous = 0; previous < out.items.size(); previous += 2)
if(uce_cbor_equal(out.items[previous], child)) return(false);
out.items.push_back(std::move(child));
}
return(true);
}
static DValue uce_cbor_value(const UceCbor& value)
{
DValue out; switch(value.kind) { case UceCborKind::Unsigned: out["type"]="unsigned"; out["value"]=std::to_string(value.number); break; case UceCborKind::Negative: out["type"]="negative"; out["value"]=value.number==UINT64_MAX?"-18446744073709551616":"-"+std::to_string(value.number+1); break; case UceCborKind::Bytes: out["type"]="bytes"; out["base64url"]=uce_base64url_encode((const unsigned char*)value.bytes.data(),value.bytes.size()); break; case UceCborKind::Text: out["type"]="text"; out["value"]=value.bytes; break; case UceCborKind::Array: out["type"]="array"; out["items"].set_array(); for(const auto& x:value.items) out["items"].push(uce_cbor_value(x)); break; case UceCborKind::Map: out["type"]="map"; out["entries"].set_array(); for(size_t i=0;i<value.items.size();i+=2) { DValue entry; entry["key"]=uce_cbor_value(value.items[i]); entry["value"]=uce_cbor_value(value.items[i+1]); out["entries"].push(entry); } } return out;
}
static bool uce_cbor_es256(const UceCbor& cose,String& x,String& y)
{
if(cose.kind!=UceCborKind::Map) return false; const UceCbor *kty=0,*alg=0,*crv=0,*xx=0,*yy=0;
for(size_t i=0;i<cose.items.size();i+=2) { const UceCbor& k=cose.items[i]; const UceCbor& v=cose.items[i+1]; if(k.kind!=UceCborKind::Unsigned&&k.kind!=UceCborKind::Negative) continue; bool neg=k.kind==UceCborKind::Negative; if(!neg&&k.number==1) kty=&v; else if(!neg&&k.number==3) alg=&v; else if(neg&&k.number==0) crv=&v; else if(neg&&k.number==1) xx=&v; else if(neg&&k.number==2) yy=&v; }
return(kty&&alg&&crv&&xx&&yy&&kty->kind==UceCborKind::Unsigned&&kty->number==2&&alg->kind==UceCborKind::Negative&&alg->number==6&&crv->kind==UceCborKind::Unsigned&&crv->number==1&&xx->kind==UceCborKind::Bytes&&yy->kind==UceCborKind::Bytes&&xx->bytes.size()==32&&yy->bytes.size()==32&&(x=xx->bytes,true)&&(y=yy->bytes,true));
}
static std::unique_ptr<EVP_PKEY,UcePkeyDeleter> uce_es256_public_key(String x,String y)
{
if(x.size()!=32||y.size()!=32) return nullptr; unsigned char point[65]={4}; memcpy(point+1,x.data(),32); memcpy(point+33,y.data(),32); OSSL_PARAM p[]={OSSL_PARAM_construct_utf8_string(OSSL_PKEY_PARAM_GROUP_NAME,(char*)"prime256v1",0),OSSL_PARAM_construct_octet_string(OSSL_PKEY_PARAM_PUB_KEY,point,sizeof(point)),OSSL_PARAM_construct_end()}; std::unique_ptr<EVP_PKEY_CTX,UcePkeyCtxDeleter> ctx(EVP_PKEY_CTX_new_from_name(0,"EC",0)); EVP_PKEY* raw=0; if(!ctx||EVP_PKEY_fromdata_init(ctx.get())<=0||EVP_PKEY_fromdata(ctx.get(),&raw,EVP_PKEY_PUBLIC_KEY,p)<=0) return nullptr; std::unique_ptr<EVP_PKEY,UcePkeyDeleter> key(raw); std::unique_ptr<EVP_PKEY_CTX,UcePkeyCtxDeleter> check(EVP_PKEY_CTX_new(key.get(),0)); return check&&EVP_PKEY_public_check(check.get())>0?std::move(key):nullptr;
}
static bool uce_es256_cose(String encoded,String& x,String& y)
{
String raw; UceCbor cose; size_t p=0,nodes=0; return uce_base64url_decode(encoded, raw, UCE_CBOR_MAX_BASE64URL) && raw.size() <= UCE_CBOR_MAX_BYTES && uce_cbor_read(raw, p, cose, 0, nodes) && p == raw.size() && uce_cbor_es256(cose, x, y) && uce_es256_public_key(x, y);
}
static DValue uce_es256_jwk(String x, String y, String d = "")
{
DValue jwk;
jwk["kty"] = "EC";
jwk["crv"] = "P-256";
jwk["x"] = uce_base64url_encode((const unsigned char*)x.data(), x.size());
jwk["y"] = uce_base64url_encode((const unsigned char*)y.data(), y.size());
if(d != "") jwk["d"] = uce_base64url_encode((const unsigned char*)d.data(), d.size());
return(jwk);
}
static String uce_es256_thumbprint(const DValue& public_jwk)
{
const DValue* x = public_jwk.key("x");
const DValue* y = public_jwk.key("y");
if(!x || !y)
return("");
String canonical = "{\"crv\":\"P-256\",\"kty\":\"EC\",\"x\":\"" + x->to_string() + "\",\"y\":\"" + y->to_string() + "\"}";
String digest = sha256_native(canonical);
return(uce_base64url_encode((const unsigned char*)digest.data(), digest.size()));
}
}
static DValue uce_es256_key_create()
{
std::unique_ptr<EVP_PKEY_CTX, UcePkeyCtxDeleter> ctx(EVP_PKEY_CTX_new_from_name(0, "EC", 0));
EVP_PKEY* raw = 0;
OSSL_PARAM params[] = { OSSL_PARAM_construct_utf8_string(OSSL_PKEY_PARAM_GROUP_NAME, (char*)"prime256v1", 0), OSSL_PARAM_construct_end() };
if(!ctx || EVP_PKEY_keygen_init(ctx.get()) <= 0 || EVP_PKEY_CTX_set_params(ctx.get(), params) <= 0 || EVP_PKEY_generate(ctx.get(), &raw) <= 0)
return(DValue());
std::unique_ptr<EVP_PKEY, UcePkeyDeleter> key(raw);
String x(UCE_ES256_COORDINATE_BYTES, 0), y(UCE_ES256_COORDINATE_BYTES, 0), d(UCE_ES256_COORDINATE_BYTES, 0);
if(!uce_es256_key_coordinates(key.get(), x, y, d))
return(DValue());
DValue result;
result["public_jwk"] = uce_es256_jwk(x, y);
String kid = uce_es256_thumbprint(result["public_jwk"]);
result["public_jwk"]["kid"] = kid;
result["private_jwk"] = uce_es256_jwk(x, y, d);
result["private_jwk"]["kid"] = kid;
result["kid"] = kid;
result["thumbprint"] = kid;
return(result);
}
static String uce_es256_jwt(DValue private_jwk, DValue protected_header, DValue claims)
{
if(!uce_es256_json_map(protected_header) || !uce_es256_json_map(claims))
return("");
std::unique_ptr<EVP_PKEY, UcePkeyDeleter> key = uce_es256_key_from_jwk(private_jwk);
if(!key)
return("");
protected_header["alg"] = "ES256";
String header_json = json_encode(protected_header);
String claims_json = json_encode(claims);
if(header_json.size() > UCE_ES256_JSON_MAX || claims_json.size() > UCE_ES256_JSON_MAX)
return("");
String signing_input = uce_base64url_encode((const unsigned char*)header_json.data(), header_json.size()) + "." + uce_base64url_encode((const unsigned char*)claims_json.data(), claims_json.size());
std::unique_ptr<EVP_MD_CTX, decltype(&EVP_MD_CTX_free)> ctx(EVP_MD_CTX_new(), EVP_MD_CTX_free);
size_t der_size = 0;
if(!ctx || EVP_DigestSignInit(ctx.get(), 0, EVP_sha256(), 0, key.get()) <= 0 || EVP_DigestSign(ctx.get(), 0, &der_size, (const unsigned char*)signing_input.data(), signing_input.size()) <= 0 || der_size == 0 || der_size > 256)
return("");
String der(der_size, 0);
if(EVP_DigestSign(ctx.get(), (unsigned char*)der.data(), &der_size, (const unsigned char*)signing_input.data(), signing_input.size()) <= 0)
return("");
const unsigned char* cursor = (const unsigned char*)der.data();
std::unique_ptr<ECDSA_SIG, UceEcdsaSigDeleter> signature(d2i_ECDSA_SIG(0, &cursor, der_size));
const BIGNUM *r = 0, *s = 0;
if(!signature || cursor != (const unsigned char*)der.data() + der_size)
return("");
ECDSA_SIG_get0(signature.get(), &r, &s);
String jose(UCE_ES256_SIGNATURE_BYTES, 0);
if(!r || !s || BN_bn2binpad(r, (unsigned char*)jose.data(), UCE_ES256_COORDINATE_BYTES) != UCE_ES256_COORDINATE_BYTES || BN_bn2binpad(s, (unsigned char*)jose.data() + UCE_ES256_COORDINATE_BYTES, UCE_ES256_COORDINATE_BYTES) != UCE_ES256_COORDINATE_BYTES)
return("");
return(signing_input + "." + uce_base64url_encode((const unsigned char*)jose.data(), jose.size()));
}
DValue crypto_operation_native(DValue request)
{
DValue result;
result["ok"].set_bool(false);
if(!crypto_operation_request_valid(request))
{
result["error"] = "invalid_request";
return(result);
}
String operation, algorithm;
if(!uce_es256_jwk_string(request, "operation", operation) || !uce_es256_jwk_string(request, "algorithm", algorithm))
{
result["error"] = "invalid_request";
return(result);
}
if(algorithm != "ES256")
{
result["error"] = "unsupported_algorithm";
return(result);
}
if(operation == "key_generate")
{
DValue key = uce_es256_key_create();
if(key["private_jwk"]["d"].to_string() == "")
{
result["error"] = "operation_failed";
return(result);
}
key["ok"].set_bool(true);
key["operation"] = operation;
key["algorithm"] = algorithm;
return(key);
}
if(operation == "cbor_decode")
{
String encoded, raw; UceCbor value; size_t p=0,nodes=0;
if(!uce_es256_jwk_string(request, "cbor_base64url", encoded) || !uce_base64url_decode(encoded, raw, UCE_CBOR_MAX_BASE64URL) || raw.size() > UCE_CBOR_MAX_BYTES || !uce_cbor_read(raw, p, value, 0, nodes) || p != raw.size()) { result["error"] = "invalid_cbor"; return(result); }
result["ok"].set_bool(true); result["operation"]=operation; result["value"]=uce_cbor_value(value); return result;
}
if(operation == "cose_es256_parse")
{
String encoded,x,y; if(!uce_es256_jwk_string(request,"cose_key_base64url",encoded)||!uce_es256_cose(encoded,x,y)) { result["error"]="invalid_cose_key"; return result; }
result["ok"].set_bool(true); result["operation"]=operation; result["algorithm"]=algorithm; result["x_base64url"]=uce_base64url_encode((const unsigned char*)x.data(),x.size()); result["y_base64url"]=uce_base64url_encode((const unsigned char*)y.data(),y.size()); return result;
}
if(operation == "es256_verify")
{
String encoded, message64, signature64, x, y, message, der;
if(!uce_es256_jwk_string(request, "cose_key_base64url", encoded) || !uce_es256_jwk_string(request, "message_base64url", message64) || !uce_es256_jwk_string(request, "signature_der_base64url", signature64) || !uce_es256_cose(encoded, x, y) || !uce_base64url_decode(message64, message, UCE_CBOR_MAX_BASE64URL) || !uce_base64url_decode(signature64, der, UCE_ES256_DER_BASE64URL_MAX) || message.size() > UCE_CBOR_MAX_BYTES || der.empty() || der.size() > UCE_ES256_DER_MAX_BYTES)
{
result["error"] = "invalid_key_or_payload";
return(result);
}
const unsigned char* cursor = (const unsigned char*)der.data();
std::unique_ptr<ECDSA_SIG, UceEcdsaSigDeleter> signature(d2i_ECDSA_SIG(0, &cursor, der.size()));
int canonical = signature ? i2d_ECDSA_SIG(signature.get(), 0) : 0;
String reencoded(canonical > 0 ? (size_t)canonical : 0, 0);
unsigned char* dest = (unsigned char*)reencoded.data();
if(!signature || cursor != (const unsigned char*)der.data() + der.size() || canonical <= 0 || i2d_ECDSA_SIG(signature.get(), &dest) != canonical || reencoded != der)
{
result["error"] = "invalid_signature";
return(result);
}
std::unique_ptr<EVP_PKEY, UcePkeyDeleter> key = uce_es256_public_key(x, y);
std::unique_ptr<EVP_MD_CTX, decltype(&EVP_MD_CTX_free)> ctx(EVP_MD_CTX_new(), EVP_MD_CTX_free);
if(!key || !ctx || EVP_DigestVerifyInit(ctx.get(), 0, EVP_sha256(), 0, key.get()) <= 0)
{
result["error"] = "operation_failed";
return(result);
}
int verified = EVP_DigestVerify(ctx.get(), (const unsigned char*)der.data(), der.size(), (const unsigned char*)message.data(), message.size());
if(verified < 0)
{
result["error"] = "operation_failed";
return(result);
}
result["ok"].set_bool(true);
result["operation"] = operation;
result["algorithm"] = algorithm;
result["valid"].set_bool(verified == 1);
return(result);
}
if(operation == "jwt_sign")
{
String jwt = uce_es256_jwt(request["private_jwk"], request["protected_header"], request["claims"]);
if(jwt == "")
{
result["error"] = "invalid_key_or_payload";
return(result);
}
result["ok"].set_bool(true);
result["operation"] = operation;
result["algorithm"] = algorithm;
result["jwt"] = jwt;
return(result);
}
result["error"] = "unsupported_operation";
return(result);
}
#endif
+4
View File
@@ -1,5 +1,7 @@
#pragma once
struct DValue;
/* ================ sha1.h ================ */
/*
SHA-1 in C
@@ -27,6 +29,8 @@ bool crypto_equal_native(String a, String b);
String password_hash_native(String password);
bool password_verify_native(String password, String encoded);
bool password_needs_rehash_native(String encoded);
bool crypto_operation_request_valid(DValue request);
DValue crypto_operation_native(DValue request);
String sha256(String data);
String sha256_hex(String data);
String hmac_sha256(String key, String data);
+48 -10
View File
@@ -246,9 +246,10 @@ DValue MySQL::query(String q)
statement_info = "mysql connection is not open";
return(DValue());
}
_preload_next_error_code = mysql_query((MYSQL*)connection, q.c_str());
int query_status = mysql_query((MYSQL*)connection, q.c_str());
_preload_next_error_code = query_status == 0 ? 0 : mysql_errno((MYSQL*)connection);
DValue result;
if(_preload_next_error_code == 0)
if(query_status == 0)
result = get_pending_result();
return(result);
}
@@ -291,11 +292,16 @@ static bool mysql_has_unquoted_positional_placeholder(String query)
DValue MySQL::query(String q, StringMap params)
{
// Positional ? placeholders survive named substitution (values are always
// quoted by escape()), so the check in query(String) covers this path too.
return(query(
parse_query_parameters(q, params).c_str()
));
// Positional ? placeholders survive named substitution (ordinary values
// are quoted by escape()), so the check in query(String) covers this path.
parameter_error = false;
String parsed = parse_query_parameters(q, params);
if(parameter_error)
{
_preload_next_error_code = CR_UNKNOWN_ERROR;
return(DValue());
}
return(query(parsed));
}
String MySQL::parse_query_parameters(String query, StringMap map)
@@ -305,6 +311,7 @@ String MySQL::parse_query_parameters(String query, StringMap map)
u8 mode = 0;
char quote;
bool escaped = false;
String identifier;
for(u32 i = 0; i < query.length(); i++)
{
@@ -321,6 +328,7 @@ String MySQL::parse_query_parameters(String query, StringMap map)
result.append(1, c);
mode = 2;
quote = c;
escaped = false;
}
else
{
@@ -329,10 +337,25 @@ String MySQL::parse_query_parameters(String query, StringMap map)
}
else if(mode == 1) // identifier mode
{
if(isalnum(c) || c == '_')
if(isalnum((unsigned char)c) || c == '_')
{
identifier.append(1, c);
}
else if(c == '!' && query[i + 1] != '=')
{
String value = map[identifier];
bool valid = identifier != "" && value != "";
for(char digit : value)
if(!isdigit((unsigned char)digit)) valid = false;
if(!valid)
{
parameter_error = true;
statement_info = "mysql unsigned parameter :" + identifier + "! must contain only decimal digits";
return("");
}
result.append(value);
mode = 0;
}
else
{
result.append(escape(map[identifier]));
@@ -342,9 +365,19 @@ String MySQL::parse_query_parameters(String query, StringMap map)
}
else if(mode == 2) // quoted mode
{
result.append(1, c);
if(escaped)
{
escaped = false;
continue;
}
if(c == '\\')
{
escaped = true;
continue;
}
if(c == quote)
mode = 0;
result.append(1, c);
}
}
@@ -390,10 +423,15 @@ String MySQL::error()
case(CR_OUT_OF_MEMORY):
p = "Out of memory";
break;
default:
case(CR_UNKNOWN_ERROR):
p = "Unknown server error";
break;
default:
if(connection && mysql_error((MYSQL*)connection)[0] != '\0')
p = mysql_error((MYSQL*)connection);
else
p = "Unknown server error";
break;
}
_preload_next_error_code = 0;
return(p);
+1
View File
@@ -20,6 +20,7 @@ struct MySQL {
u32 row_count = 0;
u64 insert_id = 0;
String statement_info = ""; //
bool parameter_error = false;
bool request_cleanup_delete = false;
bool request_pooled = false;
bool worker_persistent = false;
+240 -21
View File
@@ -64,6 +64,7 @@ int uce_host_crypto_equal(const char* a, size_t a_len, const char* b, size_t b_l
size_t uce_host_password_hash(const char* password, size_t password_len, char* out, size_t cap);
int uce_host_password_verify(const char* password, size_t password_len, const char* encoded, size_t encoded_len);
int uce_host_password_needs_rehash(const char* encoded, size_t encoded_len);
size_t uce_host_crypto_operation(const char* in, size_t in_len, char* out, size_t cap);
size_t uce_host_http_request(const char* in, size_t in_len, char* out, size_t cap);
uint64_t uce_host_http_request_async(const char* in, size_t in_len);
size_t uce_host_shell_exec_dv(const char* in, size_t in_len, char* out, size_t cap);
@@ -284,6 +285,25 @@ String password_hash(String password)
}
bool password_verify(String password, String encoded) { return(uce_host_password_verify(password.data(), password.size(), encoded.data(), encoded.size()) != 0); }
bool password_needs_rehash(String encoded) { return(uce_host_password_needs_rehash(encoded.data(), encoded.size()) != 0); }
DValue crypto_operation(DValue request)
{
if(!crypto_operation_request_valid(request))
{
DValue result;
result["ok"].set_bool(false);
result["error"] = "invalid_request";
return(result);
}
String encoded = ucb_encode(request);
size_t required = uce_host_crypto_operation(encoded.data(), encoded.size(), 0, 0);
if(required == 0 || required > 64 * 1024)
return(DValue());
String out(required, 0);
size_t got = uce_host_crypto_operation(encoded.data(), encoded.size(), &out[0], required);
DValue result;
String error;
return(got <= required && ucb_decode(String(out.data(), got), result, &error) ? result : DValue());
}
String base64_decode(String raw) { return(wasm_string_hostcall_1(uce_host_base64_decode, raw)); }
String random_bytes(u64 n)
{
@@ -305,6 +325,7 @@ u64 http_request_async(DValue req)
return((u64)uce_host_http_request_async(encoded.data(), encoded.size()));
}
DValue shell_exec(DValue spec)
{
String encoded = ucb_encode(spec);
@@ -737,6 +758,7 @@ StringMap default_config()
#include <fcntl.h>
#include <limits.h>
#include <sys/file.h>
#include <poll.h>
#include <errno.h>
#include "sys.h"
#include "hash.h"
@@ -752,10 +774,12 @@ bool crypto_equal(String a, String b) { return(crypto_equal_native(a, b)); }
String password_hash(String password) { return(password_hash_native(password)); }
bool password_verify(String password, String encoded) { return(password_verify_native(password, encoded)); }
bool password_needs_rehash(String encoded) { return(password_needs_rehash_native(encoded)); }
DValue crypto_operation(DValue request) { return(crypto_operation_native(request)); }
// Single definitions for the native split build (declared extern in sys.h).
pid_t parent_pid = 0;
pid_t my_pid = 0;
bool task_child_process = false;
namespace {
@@ -1291,6 +1315,185 @@ String socket_read(u64 sockfd, u32 max_length, u32 timeout)
return("");
}
static bool memcache_decimal(String value, u64& result)
{
if(value == "")
return(false);
result = 0;
for(char c : value)
{
if(c < '0' || c > '9' || result > (UINT64_MAX - (u64)(c - '0')) / 10)
return(false);
result = result * 10 + (u64)(c - '0');
}
return(true);
}
static String memcache_command_line(const String& command)
{
size_t end = command.find("\r\n");
return(trim(command.substr(0, end)));
}
bool uce_memcache_command_has_no_reply(const String& command)
{
StringList words = split(memcache_command_line(command), " ");
if(words.size() == 0)
return(false);
String verb = to_lower(words[0]);
return(verb == "quit" || to_lower(words.back()) == "noreply");
}
MemcacheResponseState uce_memcache_response_advance(const String& command, const String& response, MemcacheResponseParser& parser)
{
while(true)
{
if(parser.mode == MemcacheResponseParser::Mode::FirstLine)
{
size_t line_end = response.find("\r\n");
if(line_end == String::npos)
return(MemcacheResponseState::Incomplete);
String line = response.substr(0, line_end);
if(line == "ERROR" || line.rfind("CLIENT_ERROR", 0) == 0 || line.rfind("SERVER_ERROR", 0) == 0 || line == "RESET" || line == "OK")
return(MemcacheResponseState::Complete);
if(line.rfind("VALUE ", 0) == 0)
parser.mode = MemcacheResponseParser::Mode::RetrievalHeader;
else if(line.rfind("VA ", 0) == 0)
{
StringList fields = split(line, " ");
u64 length = 0;
if(fields.size() < 2 || !memcache_decimal(fields[1], length) || length > SIZE_MAX - line_end - 4)
return(MemcacheResponseState::Malformed);
parser.data_end = line_end + 2 + (size_t)length;
parser.mode = MemcacheResponseParser::Mode::MetaData;
}
else
{
StringList request_words = split(memcache_command_line(command), " ");
String request_verb = request_words.size() ? to_lower(request_words[0]) : String("");
bool end_terminated = line.rfind("STAT ", 0) == 0 || line.rfind("ITEM ", 0) == 0 ||
request_verb == "stats" || (request_verb == "lru_crawler" && request_words.size() > 1 && to_lower(request_words[1]) == "metadump");
if(!end_terminated)
return(MemcacheResponseState::Complete);
parser.mode = MemcacheResponseParser::Mode::Lines;
}
}
if(parser.mode == MemcacheResponseParser::Mode::RetrievalHeader)
{
size_t line_end = response.find("\r\n", parser.offset);
if(line_end == String::npos)
return(MemcacheResponseState::Incomplete);
String line = response.substr(parser.offset, line_end - parser.offset);
if(line == "END")
return(MemcacheResponseState::Complete);
StringList fields = split(line, " ");
u64 length = 0;
if(fields.size() < 4 || fields[0] != "VALUE" || !memcache_decimal(fields[3], length) || length > SIZE_MAX - line_end - 4)
return(MemcacheResponseState::Malformed);
parser.data_end = line_end + 2 + (size_t)length;
parser.mode = MemcacheResponseParser::Mode::RetrievalData;
}
if(parser.mode == MemcacheResponseParser::Mode::RetrievalData)
{
if(response.size() < parser.data_end + 2)
return(MemcacheResponseState::Incomplete);
if(response.compare(parser.data_end, 2, "\r\n") != 0)
return(MemcacheResponseState::Malformed);
parser.offset = parser.data_end + 2;
parser.mode = MemcacheResponseParser::Mode::RetrievalHeader;
}
if(parser.mode == MemcacheResponseParser::Mode::MetaData)
{
if(response.size() < parser.data_end + 2)
return(MemcacheResponseState::Incomplete);
return(response.compare(parser.data_end, 2, "\r\n") == 0 ? MemcacheResponseState::Complete : MemcacheResponseState::Malformed);
}
if(parser.mode == MemcacheResponseParser::Mode::Lines)
{
size_t line_end = response.find("\r\n", parser.offset);
if(line_end == String::npos)
return(MemcacheResponseState::Incomplete);
if(response.compare(parser.offset, line_end - parser.offset, "END") == 0)
return(MemcacheResponseState::Complete);
parser.offset = line_end + 2;
}
}
}
static bool native_socket_wait(int fd, short events, u64 deadline)
{
while(true)
{
u64 now = monotonic_ms();
if(now >= deadline)
return(false);
pollfd item = {fd, events, 0};
int timeout = (int)std::min<u64>(deadline - now, INT_MAX);
int ready = poll(&item, 1, timeout);
if(ready > 0)
return((item.revents & events) != 0);
if(ready == 0 || errno != EINTR)
return(false);
}
}
static String native_memcache_exchange(u64 connection, String command)
{
static constexpr size_t max_response = 8 * 1024 * 1024;
int fd = (int)connection;
u64 deadline = monotonic_ms() + 1000;
String request = command + "\r\n";
size_t sent = 0;
while(sent < request.size())
{
if(!native_socket_wait(fd, POLLOUT, deadline))
break;
ssize_t count = send(fd, request.data() + sent, request.size() - sent, MSG_DONTWAIT | MSG_NOSIGNAL);
if(count > 0)
sent += (size_t)count;
else if(count < 0 && (errno == EINTR || errno == EAGAIN || errno == EWOULDBLOCK))
continue;
else
break;
}
if(sent != request.size())
{
shutdown(fd, SHUT_RDWR);
return("");
}
if(uce_memcache_command_has_no_reply(command))
return("");
String response;
response.reserve(16 * 1024);
MemcacheResponseParser parser;
while(response.size() < max_response)
{
MemcacheResponseState state = uce_memcache_response_advance(command, response, parser);
if(state == MemcacheResponseState::Complete)
return(response);
if(state == MemcacheResponseState::Malformed || !native_socket_wait(fd, POLLIN, deadline))
break;
char buffer[64 * 1024];
size_t remaining = max_response - response.size();
ssize_t count = recv(fd, buffer, std::min(remaining, sizeof(buffer)), MSG_DONTWAIT);
if(count > 0)
response.append(buffer, (size_t)count);
else if(count < 0 && (errno == EINTR || errno == EAGAIN || errno == EWOULDBLOCK))
continue;
else
break;
}
if(uce_memcache_response_advance(command, response, parser) == MemcacheResponseState::Complete)
return(response);
shutdown(fd, SHUT_RDWR);
return("");
}
String memcache_escape_key(String key)
{
String result;
@@ -1320,26 +1523,18 @@ u64 memcache_connect(String host, u16 port)
String memcache_command(u64 connection, String command)
{
socket_write(connection, command+"\r\n");
return(socket_read(connection)); // FIXME: do multi-chunk until END line is received!
return(native_memcache_exchange(connection, command));
}
bool memcache_set(u64 connection, String key, String value, u64 expires_in)
{
socket_write(connection,
// set KEY META_DATA EXPIRY_TIME LENGTH_IN_BYTES
String("set ") + memcache_escape_key(key) + " 0 " + std::to_string(expires_in) + " " + std::to_string(value.length()) + "\r\n" +
value + "\r\n");
return("STORED" == trim(socket_read(connection)));
return("STORED" == trim(memcache_command(connection,
String("set ") + memcache_escape_key(key) + " 0 " + std::to_string(expires_in) + " " + std::to_string(value.length()) + "\r\n" + value)));
}
bool memcache_delete(u64 connection, String key)
{
socket_write(connection,
// set KEY META_DATA EXPIRY_TIME LENGTH_IN_BYTES
String("delete ") + memcache_escape_key(key) + "\r\n"
);
return("DELETED" == trim(socket_read(connection)));
return("DELETED" == trim(memcache_command(connection, String("delete ") + memcache_escape_key(key))));
}
String memcache_get(u64 connection, String key, String default_value)
@@ -1389,6 +1584,7 @@ void on_segfault(int sig)
struct Worker {
pid_t pid;
char listener_type = 0;
};
std::map<pid_t, Worker> workers;
@@ -1617,27 +1813,41 @@ DValue process_exec(String cmd, String input, StringMap env, u64 timeout_ms, u64
return(result);
}
pid_t spawn_subprocess(std::function<void()> exec_after_spawn)
pid_t spawn_subprocess(std::function<void()> exec_after_spawn, char listener_type = 0)
{
parent_pid = getpid();
pid_t p;
p = fork();
sigset_t blocked;
sigset_t previous;
sigemptyset(&blocked);
sigaddset(&blocked, SIGCHLD);
if(sigprocmask(SIG_BLOCK, &blocked, &previous) != 0)
{
perror("sigprocmask");
return(0);
}
pid_t p = fork();
if(p == 0)
{
sigprocmask(SIG_SETMASK, &previous, 0);
my_pid = getpid();
//printf("(C) child procress started, PID:%i\n", my_pid);
prctl(PR_SET_PDEATHSIG, SIGHUP);
exec_after_spawn();
return(0);
}
else
if(p < 0)
{
Worker w;
w.pid = p;
workers[w.pid] = w;
printf("(P) child procress spawned: PID %i\n", p);
return(p);
perror("fork worker");
sigprocmask(SIG_SETMASK, &previous, 0);
return(0);
}
Worker w;
w.pid = p;
w.listener_type = listener_type;
workers[w.pid] = w;
sigprocmask(SIG_SETMASK, &previous, 0);
printf("(P) child procress spawned: PID %i\n", p);
return(p);
}
String runtime_safe_key(String key, String label)
@@ -1788,6 +1998,7 @@ pid_t task(String key, std::function<void()> exec_after_spawn, u64 timeout)
{
close_locked_file(lock_fd);
my_pid = getpid();
task_child_process = true;
// The FastCGI worker handles termination to drain accepted requests.
// Generic task children do not run that drain loop, so inheriting those
// handlers would turn task_kill(SIGTERM) into a no-op.
@@ -1904,6 +2115,9 @@ StringMap make_server_settings()
cfg["WASM_MEMORY_LIMIT_BYTES"] = std::to_string(512ull * 1024 * 1024);
cfg["WASM_EPOCH_DEADLINE_TICKS"] = "200";
cfg["WASM_EPOCH_PERIOD_MS"] = "50";
cfg["WASM_SERIALIZE_TIMEOUT_SECONDS"] = "120";
cfg["MYSQL_PERSISTENT_POOL_SIZE"] = "8";
cfg["MYSQL_PERSISTENT_POOL_IDLE_TIMEOUT_SECONDS"] = "300";
cfg["SETUP_TEMPLATE"] = "scripts/setup.h.template";
cfg["LIT_ESC"] = "3d5b5_1";
cfg["CONTENT_TYPE"] = "text/html; charset=utf-8";
@@ -1911,6 +2125,11 @@ StringMap make_server_settings()
cfg["FCGI_SOCKET_MODE"] = "0666";
cfg["CLI_SOCKET_PATH"] = "/run/uce/cli.sock";
cfg["CLI_SOCKET_MODE"] = "0600";
// Zero preserves the legacy shared renderer pool. A positive count keeps
// trusted CLI/test module caches out of public FastCGI workers.
cfg["CLI_WORKER_COUNT"] = "0";
// Bound retained CLI/test module state. Zero disables recycling.
cfg["CLI_WORKER_MAX_REQUESTS"] = "8";
// Command socket the WS broker listens on; workers flush ws_* dispatch
// command batches here at workspace teardown.
cfg["WS_BROKER_SOCKET_PATH"] = "/run/uce/ws-broker.sock";
+12
View File
@@ -51,6 +51,7 @@ bool crypto_equal(String a, String b);
String password_hash(String password);
bool password_verify(String password, String encoded);
bool password_needs_rehash(String encoded);
DValue crypto_operation(DValue request);
String shell_escape(String raw);
String basename(String fn);
String dirname(String fn);
@@ -128,6 +129,15 @@ String signal_name(s32 sig);
String memcache_escape_key(String key);
StringList memcache_escape_keys(StringList keys);
u64 memcache_connect(String host = "127.0.0.1", u16 port = 11211);
enum class MemcacheResponseState { Incomplete, Complete, Malformed };
struct MemcacheResponseParser
{
enum class Mode { FirstLine, RetrievalHeader, RetrievalData, Lines, MetaData } mode = Mode::FirstLine;
size_t offset = 0;
size_t data_end = 0;
};
__attribute__((visibility("hidden"))) bool uce_memcache_command_has_no_reply(const String& command);
__attribute__((visibility("hidden"))) MemcacheResponseState uce_memcache_response_advance(const String& command, const String& response, MemcacheResponseParser& parser);
String memcache_command(u64 connection, String command);
bool memcache_set(u64 connection, String key, String value, u64 expires_in = 60*60);
bool memcache_delete(u64 connection, String key);
@@ -139,9 +149,11 @@ StringMap memcache_get_multiple(u64 connection, StringList keys);
#if defined(__UCE_WASM_CORE__) || defined(__UCE_WASM_UNIT__)
pid_t parent_pid = 0;
pid_t my_pid = 0;
bool task_child_process = false;
#else
extern pid_t parent_pid;
extern pid_t my_pid;
extern bool task_child_process;
#endif
void on_segfault(int sig);
+5 -1
View File
@@ -97,7 +97,11 @@ String base64_decode(String raw, bool& ok)
return("");
if(padding > 0 && i + 4 != cleaned.length())
return("");
if(cleaned[i + 2] == '=' && cleaned[i + 3] != '=')
if(cleaned[i] == '=' || cleaned[i + 1] == '=')
return("");
if(padding == 1 && (cleaned[i + 2] == '=' || cleaned[i + 3] != '='))
return("");
if(padding == 2 && (cleaned[i + 2] != '=' || cleaned[i + 3] != '='))
return("");
result.append(1, (char)((values[0] << 2) | (values[1] >> 4)));
+112 -16
View File
@@ -1132,7 +1132,8 @@ void run_ws_broker()
if(server_state.config["WS_BROKER_SOCKET_PATH"] != "")
{
ws_broker.listen(server_state.config["WS_BROKER_SOCKET_PATH"]);
chmod(server_state.config["WS_BROKER_SOCKET_PATH"].c_str(), S_IRWXU | S_IRGRP | S_IWGRP);
if(chmod(server_state.config["WS_BROKER_SOCKET_PATH"].c_str(), S_IRWXU | S_IRGRP | S_IWGRP) != 0)
fprintf(stderr, "(!) Could not chmod socket %s to %04o: %s\n", server_state.config["WS_BROKER_SOCKET_PATH"].c_str(), 0760, strerror(errno));
}
while(!termination_signal_received)
{
@@ -1265,6 +1266,58 @@ bool proactive_compile_queue_has(StringList& queue, String file_name)
return(std::find(queue.begin(), queue.end(), file_name) != queue.end());
}
String serialized_module_path(String wasm_path)
{
if(wasm_path.size() >= 5 && wasm_path.rfind(".wasm") == wasm_path.size() - 5)
return(wasm_path.substr(0, wasm_path.size() - 5) + ".cwasm");
return(wasm_path + ".cwasm");
}
void cleanup_dead_serialization_temps(String wasm_path)
{
String cached_path = serialized_module_path(wasm_path);
String directory = dirname(cached_path);
String prefix = basename(cached_path) + ".";
std::error_code ec;
for(auto const& entry : std::filesystem::directory_iterator(directory, ec))
{
if(ec)
break;
String name = entry.path().filename().string();
if(!str_starts_with(name, prefix) || name.size() <= prefix.size() + 4 || name.rfind(".tmp") != name.size() - 4)
continue;
String pid_text = name.substr(prefix.size(), name.size() - prefix.size() - 4);
char* end = 0;
errno = 0;
long pid = strtol(pid_text.c_str(), &end, 10);
if(pid <= 0 || errno == ERANGE || !end || *end != '\0')
continue;
if(kill((pid_t)pid, 0) != 0 && errno == ESRCH)
file_unlink(entry.path().string());
}
}
String proactive_serialize_module(String wasm_path)
{
String executable = compiler_source_path_real("/proc/self/exe");
if(executable == "")
return("cannot resolve UCE executable for isolated serialization");
u64 timeout_seconds = std::max<u64>(1, std::min<u64>(3600,
to_u64(server_state.config["WASM_SERIALIZE_TIMEOUT_SECONDS"], 120)));
DValue execution = process_exec(
"exec " + shell_escape(executable) + " --serialize-module " + shell_escape(wasm_path),
"", StringMap(), timeout_seconds * 1000, 64 * 1024);
cleanup_dead_serialization_temps(wasm_path);
if(execution["timed_out"].to_bool())
return("serialized-module child timed out after " + std::to_string(timeout_seconds) + " seconds");
if(execution["exit_code"].to_s64() != 0)
{
String diagnostic = trim(first(execution["stderr"].to_string(), execution["stdout"].to_string()));
return(diagnostic == "" ? "serialized-module child failed" : diagnostic);
}
return("");
}
u64 bounded_compile_jobs(String value, u64 fallback = 2)
{
value = trim(value);
@@ -1318,7 +1371,7 @@ bool proactive_compile_unit(Request& context, String file_name, bool& source_mis
if(!source_missing && !failed && wasm_serialized_module_needs_refresh(wasm_path))
{
printf("(i) proactive serialize %s\n", file_name.c_str());
String serialize_error = wasm_serialize_module_artifact(wasm_path);
String serialize_error = proactive_serialize_module(wasm_path);
if(serialize_error != "")
{
printf("(!) proactive serialize failed for %s: %s\n", file_name.c_str(), serialize_error.c_str());
@@ -1558,16 +1611,26 @@ void ensure_proactive_compiler()
priority_compiler_pid = spawn_compiler("priority compiler", run_priority_compiler);
}
void listen_for_connections()
void listen_for_connections(char listener_type = 0)
{
install_process_fault_handlers();
// Workers are uniform FastCGI/CLI renderers; the WS broker owns the HTTP/WS
// port and every connection, so workers never accept raw HTTP themselves.
server.close_http_listeners();
// The transport's legacy eight-connection recycle predates persistent
// Wasmtime engines. Recycling makes every ninth request pay engine/module
// startup; request-scoped workspaces already isolate and release user state.
// The WS broker owns the HTTP/WS port. A configured dedicated CLI pool keeps
// test/admin module-cache churn out of public FastCGI renderers; zero keeps
// the legacy shared listener set.
if(listener_type == 'F' || listener_type == 'C')
server.close_listeners_except(listener_type);
else
server.close_http_listeners();
// Public workers keep their hot engines and modules. Dedicated CLI workers
// recycle so broad test/admin runs cannot retain an unbounded module set.
server.calls_until_termination = -1;
if(listener_type == 'C')
{
u64 max_requests = std::min<u64>(1024,
to_u64(server_state.config["CLI_WORKER_MAX_REQUESTS"], 8));
if(max_requests > 0)
server.calls_until_termination = (int)max_requests;
}
server.on_request = &handle_request;
server.on_data = &handle_data;
server.on_complete = &handle_complete;
@@ -1578,7 +1641,8 @@ void listen_for_connections()
String wasm_error = wasm_backend_start(startup_context);
f64 wasm_ms = (time_precise() - wasm_start) * 1000.0;
if(wasm_error == "")
printf("(P) wasm worker ready: PID %i in %.3f ms\n", getpid(), wasm_ms);
printf("(P) wasm %s worker ready: PID %i in %.3f ms\n",
listener_type == 'F' ? "FastCGI" : listener_type == 'C' ? "CLI" : "shared", getpid(), wasm_ms);
else
fprintf(stderr, "(!) wasm worker initialization failed: PID %i in %.3f ms: %s\n", getpid(), wasm_ms, wasm_error.c_str());
while(!termination_signal_received)
@@ -1870,9 +1934,11 @@ void print_fastcgi_usage(FILE* stream)
{
fprintf(stream,
"Usage: uce_fastcgi [--precompile]\n"
" uce_fastcgi --serialize-module PATH\n"
" uce_fastcgi --help\n\n"
"Without options, start the FastCGI server.\n"
" --precompile Compile the current source generation without starting listeners.\n"
" --serialize-module PATH Serialize one Wasm artifact in an isolated process.\n"
" -h, --help Show this help and exit.\n");
}
@@ -1888,7 +1954,8 @@ int main(int argc, char** argv)
return(0);
}
bool precompile = argc == 2 && String(argv[1]) == "--precompile";
if(argc != 1 && !precompile)
bool serialize_module = argc == 3 && String(argv[1]) == "--serialize-module";
if(argc != 1 && !precompile && !serialize_module)
{
fprintf(stderr, "invalid arguments\n");
print_fastcgi_usage(stderr);
@@ -1898,6 +1965,16 @@ int main(int argc, char** argv)
// after a fault does not allocate.
backtrace(request_fault_frames, 4);
process_start_directory();
if(serialize_module)
{
String error = wasm_serialize_module_artifact(argv[2]);
if(error != "")
{
fprintf(stderr, "%s\n", error.c_str());
return(1);
}
return(0);
}
if(precompile)
return(precompile_unit_generation());
@@ -1923,11 +2000,30 @@ int main(int argc, char** argv)
if(!termination_signal_received)
ensure_ws_broker();
while(workers.size() < int_val(server_state.config["WORKER_COUNT"]))
{
if(!termination_signal_received)
spawn_subprocess(listen_for_connections);
}
u64 cli_worker_count = server_state.config["CLI_SOCKET_PATH"] == "" ? 0 :
std::min<u64>(16, to_u64(server_state.config["CLI_WORKER_COUNT"], 0));
bool has_public_listener = false;
for(auto& listener : server.server_socket_types)
if(listener.second == 'F')
has_public_listener = true;
u64 configured_worker_count = std::max<s64>(1, int_val(server_state.config["WORKER_COUNT"]));
u64 public_worker_count = cli_worker_count > 0 && !has_public_listener ? 0 :
(server.server_sockets.empty() ? 0 : configured_worker_count);
auto role_count = [&](char listener_type) {
u64 count = 0;
for(auto& worker : workers)
if(worker.second.listener_type == listener_type)
count++;
return(count);
};
auto spawn_listener_worker = [&](char listener_type) {
return(spawn_subprocess([listener_type]() { listen_for_connections(listener_type); }, listener_type));
};
char public_listener_type = cli_worker_count > 0 ? 'F' : 0;
while(!termination_signal_received && role_count(public_listener_type) < public_worker_count)
if(spawn_listener_worker(public_listener_type) <= 0) { sleep(1); break; }
while(!termination_signal_received && role_count('C') < cli_worker_count)
if(spawn_listener_worker('C') <= 0) { sleep(1); break; }
sleep(1);
}
+11 -2
View File
@@ -121,6 +121,13 @@ static String wasm_backend_ensure_started(Request* context)
return(g_wasm_init_error);
}
wc.mysql_persistent_pool_size = std::min<u64>(to_u64(cfg["MYSQL_PERSISTENT_POOL_SIZE"], 8), 64);
u64 mysql_idle_timeout = to_u64(first(cfg["MYSQL_PERSISTENT_POOL_IDLE_TIMEOUT_SECONDS"], "300"), UINT64_MAX);
if(mysql_idle_timeout == UINT64_MAX || mysql_idle_timeout > 86400)
{
g_wasm_init_error = "MYSQL_PERSISTENT_POOL_IDLE_TIMEOUT_SECONDS must be an integer no greater than 86400";
return(g_wasm_init_error);
}
wc.mysql_persistent_pool_idle_timeout_seconds = mysql_idle_timeout;
wc.profile_hostcall_cpu = to_bool(cfg["WASM_PROFILE_HOSTCALL_CPU"], false);
wc.profile_thread_runtime = to_bool(cfg["WASM_PROFILE_THREAD_RUNTIME"], false);
wc.verbose = to_bool(cfg["WASM_BACKEND_VERBOSE"], false);
@@ -422,8 +429,8 @@ String wasm_backend_serve(Request& request, const String& entry_unit, const Stri
return("");
}
// Stop the ticker before the worker process exits (best-effort; forked workers
// are usually killed, but a clean ager-out path should join the thread).
// Stop the ticker before destroying the worker state it references. Deleting
// the worker also closes retained connector resources on a clean process exit.
void wasm_backend_shutdown()
{
if(g_wasm_epoch_ticker)
@@ -434,4 +441,6 @@ void wasm_backend_shutdown()
delete g_wasm_epoch_ticker;
g_wasm_epoch_ticker = 0;
}
delete g_wasm_worker;
g_wasm_worker = 0;
}
+42 -3
View File
@@ -207,6 +207,7 @@ String MySQL::parse_query_parameters(String query, StringMap map)
u8 mode = 0;
char quote = 0;
bool escaped = false;
String identifier;
for(u32 i = 0; i < query.length(); i++)
{
@@ -223,14 +224,30 @@ String MySQL::parse_query_parameters(String query, StringMap map)
result.append(1, c);
mode = 2;
quote = c;
escaped = false;
}
else
result.append(1, c);
}
else if(mode == 1)
{
if(isalnum(c) || c == '_')
if(isalnum((unsigned char)c) || c == '_')
identifier.append(1, c);
else if(c == '!' && query[i + 1] != '=')
{
String value = map[identifier];
bool valid = identifier != "" && value != "";
for(char digit : value)
if(!isdigit((unsigned char)digit)) valid = false;
if(!valid)
{
parameter_error = true;
statement_info = "mysql unsigned parameter :" + identifier + "! must contain only decimal digits";
return("");
}
result.append(value);
mode = 0;
}
else
{
result.append(escape(map[identifier]));
@@ -240,9 +257,19 @@ String MySQL::parse_query_parameters(String query, StringMap map)
}
else if(mode == 2)
{
result.append(1, c);
if(escaped)
{
escaped = false;
continue;
}
if(c == '\\')
{
escaped = true;
continue;
}
if(c == quote)
mode = 0;
result.append(1, c);
}
}
@@ -315,7 +342,17 @@ DValue MySQL::query(String q)
return(result ? *result : DValue());
}
DValue MySQL::query(String q, StringMap params) { return(query(parse_query_parameters(q, params))); }
DValue MySQL::query(String q, StringMap params)
{
parameter_error = false;
String parsed = parse_query_parameters(q, params);
if(parameter_error)
{
_preload_next_error_code = 2000;
return(DValue());
}
return(query(parsed));
}
DValue MySQL::get_pending_result() { return(DValue()); }
// sqlite runs host-side (the host links libsqlite and owns the connections in
@@ -461,6 +498,8 @@ extern "C" void uce_wasm_link_anchors()
(void*)(double (*)(double))&cos,
(void*)(double (*)(double))&sin,
(void*)(double (*)(double))&round,
// structured crypto dispatcher is hostcall-backed and otherwise unreferenced by core
(void*)&crypto_operation,
};
(void)libc_anchors;
}
+1
View File
@@ -13,6 +13,7 @@ uce_host_crypto_equal
uce_host_password_hash
uce_host_password_verify
uce_host_password_needs_rehash
uce_host_crypto_operation
uce_host_task_spawn
uce_host_task_pid
uce_host_task_kill
+146
View File
@@ -0,0 +1,146 @@
// Internal worker-local implementation for opt-in generic hardened HTTP.
#pragma once
#include <arpa/inet.h>
#include <cctype>
#include <cerrno>
#include <fcntl.h>
#include <functional>
#include <signal.h>
#include <sys/wait.h>
#include <sys/syscall.h>
#include <unistd.h>
static u64 hardened_http_monotonic_ms() { timespec ts{}; clock_gettime(CLOCK_MONOTONIC,&ts); return (u64)ts.tv_sec*1000ull+(u64)ts.tv_nsec/1000000ull; }
static void hardened_http_close_inherited_fds()
{
#ifdef SYS_close_range
if(syscall(SYS_close_range,4u,~0u,0u)==0) return;
#endif
long max_fd=sysconf(_SC_OPEN_MAX); if(max_fd<4) max_fd=4;
for(int fd=4;fd<max_fd;fd++) close(fd);
}
struct HardenedHttpExecResult { int exit_code=-1; bool timed_out=false, output_limited=false; String headers_text, body_text, stderr_text; };
struct HardenedHttpHooks {
std::function<std::vector<String>(String)> resolve;
std::function<HardenedHttpExecResult(std::vector<String>, String, std::vector<String>, u64, size_t)> execute;
};
// Hardened requests deliberately support public IPv4 answers only. Every IPv6
// answer fails closed until this generic transport has a reviewed IPv6 policy.
// IPv4 must be globally unicast: reject IANA special-purpose 0/8, 10/8,
// 100.64/10, 127/8, 169.254/16, 172.16/12, 192.0.0/24, 192.0.2/24,
// 192.31.196/24, 192.52.193/24, 192.88.99/24, 192.175.48/24, 192.168/16,
// 198.18/15, 198.51.100/24, 203.0.113/24, and all 224/4 multicast/reserved.
static bool hardened_http_public_address(String text)
{
in_addr v4{};
if(inet_pton(AF_INET,text.c_str(),&v4)!=1) return false; // Includes every AAAA candidate.
u32 x=ntohl(v4.s_addr); u8 a=x>>24,b=x>>16,c=x>>8;
if(a==0||a==10||a==127||a>=224||(a==100&&b>=64&&b<=127)||(a==169&&b==254)||(a==172&&b>=16&&b<=31)||(a==192&&(b==0||b==2||b==168))||(a==192&&b==31&&c==196)||(a==192&&b==52&&c==193)||(a==192&&b==88&&c==99)||(a==192&&b==175&&c==48)||(a==198&&(b==18||b==19||b==51))||(a==203&&b==0&&c==113)) return false;
return true;
}
// Three private pipes keep curl's -D headers and -o body distinct. The child
// inherits only stdin/stdout/stderr and the header pipe on fd 3. Async workers
// retain their own process group so job_cancel can kill curl and its descendants.
static HardenedHttpExecResult hardened_http_exec_argv_capture(std::vector<String> argv, String input, u64 timeout_ms, size_t output_limit, bool clean_env=true, bool own_process_group=true)
{
HardenedHttpExecResult r; if(argv.empty()) return r;
int in[2], body[2], headers[2], err[2];
if(pipe(in)||pipe(body)||pipe(headers)||pipe(err)) return r;
pid_t pid=fork();
if(pid==0) {
if(own_process_group) setpgid(0,0);
dup2(in[0],0); dup2(body[1],1); dup2(err[1],2); dup2(headers[1],3);
hardened_http_close_inherited_fds();
std::vector<char*> args; for(String& a:argv) args.push_back((char*)a.c_str()); args.push_back(0);
if(clean_env) { clearenv(); setenv("PATH","/usr/bin:/bin",1); execv(args[0],args.data()); } else execvp(args[0],args.data());
_exit(127);
}
auto close_all=[&](){ for(int fd: {in[0],in[1],body[0],body[1],headers[0],headers[1],err[0],err[1]}) close(fd); };
if(pid<0) { close_all(); return r; }
if(own_process_group) setpgid(pid,pid);
close(in[0]); close(body[1]); close(headers[1]); close(err[1]);
for(int fd: {in[1],body[0],headers[0],err[0]}) fcntl(fd,F_SETFL,fcntl(fd,F_GETFL,0)|O_NONBLOCK);
size_t input_off=0, total=0; bool in_open=true,body_open=true,headers_open=true,err_open=true,exited=false,killed=false; int status=0; u64 deadline=hardened_http_monotonic_ms()+timeout_ms;
auto terminate=[&](bool timeout){ if(timeout) r.timed_out=true; if(!killed) { if(own_process_group) kill(-pid,SIGKILL); kill(pid,SIGKILL); killed=true; } };
auto drain=[&](int fd,bool& open,String& dst) { char buf[4096]; ssize_t n; while((n=read(fd,buf,sizeof(buf)))>0) { if(total+(size_t)n>output_limit) { r.output_limited=true; terminate(false); } else { dst+=String(buf,n); total+=(size_t)n; } } if(n==0) { close(fd); open=false; } };
while(in_open||body_open||headers_open||err_open||!exited) {
if(!exited) { pid_t w=waitpid(pid,&status,WNOHANG); if(w==pid) exited=true; }
if(in_open) { if(input_off<input.size()) { ssize_t n=write(in[1],input.data()+input_off,input.size()-input_off); if(n>0) input_off+=(size_t)n; else if(n<0&&errno!=EINTR&&errno!=EAGAIN&&errno!=EWOULDBLOCK) { close(in[1]); in_open=false; } } else { close(in[1]); in_open=false; } }
drain(body[0],body_open,r.body_text); drain(headers[0],headers_open,r.headers_text); drain(err[0],err_open,r.stderr_text);
if(!killed&&hardened_http_monotonic_ms()>=deadline) terminate(true);
if(killed&&!exited) { while(waitpid(pid,&status,0)<0&&errno==EINTR) {} exited=true; }
if(in_open||body_open||headers_open||err_open||!exited) usleep(1000);
}
if(WIFEXITED(status)) r.exit_code=WEXITSTATUS(status); else if(WIFSIGNALED(status)) r.exit_code=128+WTERMSIG(status);
return r;
}
static bool hardened_http_parse_headers(String text, u64& status, DValue& filtered)
{
status=0; filtered.set_array(); size_t pos=0; bool found=false;
while(pos<text.size()) {
size_t end=text.find("\r\n\r\n",pos); size_t sep=4; if(end==String::npos) { end=text.find("\n\n",pos); sep=2; } if(end==String::npos) return false;
String block=replace(text.substr(pos,end-pos),"\r",""); size_t nl=block.find('\n'); String first=nl==String::npos?block:block.substr(0,nl);
if(first.size()<12||first.rfind("HTTP/",0)!=0||first[8]!=' '||!isdigit((unsigned char)first[9])||!isdigit((unsigned char)first[10])||!isdigit((unsigned char)first[11])) return false;
status=strtoull(first.substr(9,3).c_str(),0,10); DValue current; current.set_array();
if(nl!=String::npos) for(String line:split(block.substr(nl+1),"\n")) { size_t c=line.find(':'); if(c==String::npos) return false; String k=to_lower(trim(line.substr(0,c))); if(k=="content-type"||k=="cache-control") current[k]=trim(line.substr(c+1)); }
filtered=current; found=true; pos=end+sep;
}
return found;
}
static bool hardened_http_token(String text)
{
if(text=="") return false;
for(unsigned char c:text) if(!(isalnum(c)||c=='-'||c=='_')) return false;
return true;
}
static bool hardened_http_header_value(String text)
{
if(text.size()>8192||text.find('\0')!=String::npos||text.find('\r')!=String::npos||text.find('\n')!=String::npos) return false;
return true;
}
static bool hardened_http_ip_literal(String text)
{
in_addr v4{}; in6_addr v6{};
return inet_pton(AF_INET,text.c_str(),&v4)==1 || inet_pton(AF_INET6,text.c_str(),&v6)==1;
}
static bool hardened_http_url(String url, String& host, String& port)
{
if(url.find('\0')!=String::npos||url.find_first_of("\\ \r\n\t")!=String::npos||url.rfind("https://",0)!=0) return false;
String rest=url.substr(8); size_t slash=rest.find('/'); String authority=slash==String::npos?rest:rest.substr(0,slash);
if(authority==""||authority.find('@')!=String::npos||authority.find('[')!=String::npos||authority.find(']')!=String::npos) return false;
size_t colon=authority.rfind(':'); host=colon==String::npos?authority:authority.substr(0,colon); port=colon==String::npos?"443":authority.substr(colon+1);
if(host==""||hardened_http_ip_literal(host)||!hardened_http_token(port)) return false;
for(unsigned char c:host) if(!(isalnum(c)||c=='.'||c=='-')) return false;
char* end=0; unsigned long n=strtoul(port.c_str(),&end,10); return end!=port.c_str()&&*end==0&&n>0&&n<=65535;
}
static bool hardened_http_security_requested(const DValue* security)
{
return security && (security->key("https_only") || security->key("public_dns_only") || security->key("pin_dns") || security->key("isolated_curl") || security->key("no_redirects"));
}
static bool hardened_http_true(const DValue* value)
{
return value && value->get_type_name()=="bool" && value->to_bool();
}
static DValue hardened_http_request_internal(const DValue& req, u64 timeout_ms, const HardenedHttpHooks& hooks)
{
DValue r; r["status"]=(f64)0; r["headers"].set_array(); r["body"]=""; r["error"]="";
const DValue* sec=req.key("security");
if(req.key("follow_redirects") && req.key("follow_redirects")->to_bool()) { r["error"]="invalid_request"; return r; }
timeout_ms=std::min<u64>(std::max<u64>(1,timeout_ms),10000);
if(!sec || !hardened_http_true(sec->key("https_only")) || !hardened_http_true(sec->key("public_dns_only")) || !hardened_http_true(sec->key("pin_dns")) || !hardened_http_true(sec->key("isolated_curl")) || !hardened_http_true(sec->key("no_redirects"))) { r["error"]="invalid_request"; return r; }
String method=req.key("method")?req.key("method")->to_string():"GET"; String url=req.key("url")?req.key("url")->to_string():""; String host,port;
if((method!="GET"&&method!="POST"&&method!="PUT"&&method!="PATCH"&&method!="DELETE"&&method!="HEAD"&&method!="OPTIONS")||!hardened_http_url(url,host,port)) { r["error"]="invalid_request"; return r; }
String body=req.key("body")?req.key("body")->to_string():""; if(body.size()>65536||body.find('\0')!=String::npos) { r["error"]="invalid_request"; return r; }
std::vector<String> argv={"/usr/bin/curl","--disable","-sS","--http1.1","--proto","=https","--proto-redir","=https","--noproxy","*","--proxy","","--alt-svc","","--hsts","","--cacert","/etc/ssl/certs/ca-certificates.crt","--connect-timeout","3","--max-time",std::to_string(std::max<u64>(1,timeout_ms/1000)),"--max-filesize","65536","-X",method,"-D","/proc/self/fd/3","-o","/proc/self/fd/1"};
const DValue* hs=req.key("headers"); if(hs) { bool valid=true; hs->each([&](const DValue& v,String k) { String value=v.to_string(), lower=to_lower(k); if(!hardened_http_token(k)||!hardened_http_header_value(value)||lower=="host"||lower=="content-length"||lower=="transfer-encoding"||lower=="connection"||lower=="proxy-connection"||lower=="expect") valid=false; else { argv.push_back("-H"); argv.push_back(k+": "+value); } }); if(!valid) { r["error"]="invalid_request"; return r; } }
std::vector<String> answers=hooks.resolve(host); String address; for(String a:answers) { if(!hardened_http_public_address(a)) { r["error"]="unsafe_dns"; return r; } if(address=="") address=a; } if(address=="") { r["error"]="unsafe_dns"; return r; }
argv.push_back("--resolve"); argv.push_back(host+":"+port+":"+address); if(req.key("body")) { argv.push_back("--data-binary"); argv.push_back("@-"); } argv.push_back(url);
HardenedHttpExecResult pr=hooks.execute(argv,body,{"PATH=/usr/bin:/bin"},timeout_ms,72*1024); if(pr.output_limited||pr.body_text.size()>65536||pr.headers_text.size()>8192) { r["error"]="response_too_large"; return r; }
u64 status; DValue headers; if(!hardened_http_parse_headers(pr.headers_text,status,headers)) { r["error"]=pr.timed_out?"timeout":"malformed_output"; return r; }
r["status"]=(f64)status; r["headers"]=headers; r["body"]=pr.body_text;
if(status/100==3) r["error"]="redirect_not_allowed"; else if(pr.exit_code!=0) r["error"]=pr.timed_out?"timeout":"network_failure"; else if(status/100!=2) r["error"]="http_status"; return r;
}
+302 -60
View File
@@ -39,6 +39,7 @@
#include <string>
#include <arpa/inet.h>
#include <netinet/in.h>
#include <netdb.h>
#include <sys/socket.h>
#include <sys/stat.h>
#include <sys/file.h>
@@ -55,6 +56,7 @@
#include <sys/wait.h>
#include <signal.h>
#include <poll.h>
#include "hardened_http_internal.h"
struct WasmDylinkInfo
{
@@ -153,6 +155,7 @@ struct WasmWorkerConfig
u64 epoch_period_ms = 50;
u64 invocation_timeout_ms = 30000;
u64 mysql_persistent_pool_size = 8;
u64 mysql_persistent_pool_idle_timeout_seconds = 300;
bool profile_hostcall_cpu = false;
bool profile_thread_runtime = false;
bool verbose = false;
@@ -389,15 +392,12 @@ static u64 wasm_socket_connect_bounded(const String& host, u16 port, u64 timeout
}
if(fd <= 0)
return(0);
if(context)
context->resources.sockets.push_back(fd);
return((u64)fd);
}
static bool wasm_socket_write_bounded(u64 socket_fd, const String& data, u64 timeout_ms)
static bool wasm_socket_write_until(u64 socket_fd, const String& data, u64 deadline)
{
int fd = (int)socket_fd;
u64 deadline = wasm_deadline_after_ms(timeout_ms);
size_t offset = 0;
while(offset < data.size())
{
@@ -412,6 +412,11 @@ static bool wasm_socket_write_bounded(u64 socket_fd, const String& data, u64 tim
return(true);
}
static bool wasm_socket_write_bounded(u64 socket_fd, const String& data, u64 timeout_ms)
{
return(wasm_socket_write_until(socket_fd, data, wasm_deadline_after_ms(timeout_ms)));
}
static String wasm_socket_read_bounded(u64 socket_fd, u32 max_length, u64 timeout_ms)
{
if(max_length == 0 || !wasm_socket_wait((int)socket_fd, POLLIN, wasm_deadline_after_ms(timeout_ms)))
@@ -421,6 +426,42 @@ static String wasm_socket_read_bounded(u64 socket_fd, u32 max_length, u64 timeou
return(count > 0 ? String(buffer.data(), (size_t)count) : String(""));
}
static String wasm_memcache_exchange(u64 socket_fd, const String& command, u64 timeout_ms)
{
static constexpr size_t max_response = 8 * 1024 * 1024;
int fd = (int)socket_fd;
u64 deadline = wasm_deadline_after_ms(timeout_ms);
if(!wasm_socket_write_until(socket_fd, command + "\r\n", deadline))
{
shutdown(fd, SHUT_RDWR);
return("");
}
if(uce_memcache_command_has_no_reply(command))
return("");
String response;
response.reserve(16 * 1024);
MemcacheResponseParser parser;
while(response.size() < max_response)
{
MemcacheResponseState state = uce_memcache_response_advance(command, response, parser);
if(state == MemcacheResponseState::Complete)
return(response);
if(state == MemcacheResponseState::Malformed || !wasm_socket_wait(fd, POLLIN, deadline))
break;
char buffer[64 * 1024];
size_t remaining = max_response - response.size();
ssize_t count = recv(fd, buffer, std::min(remaining, sizeof(buffer)), MSG_DONTWAIT);
if(count > 0)
response.append(buffer, (size_t)count);
else if(count == 0 || (errno != EINTR && errno != EAGAIN && errno != EWOULDBLOCK))
break;
}
if(uce_memcache_response_advance(command, response, parser) == MemcacheResponseState::Complete)
return(response);
shutdown(fd, SHUT_RDWR);
return("");
}
static f64 wasm_thread_cpu_time()
{
struct timespec ts;
@@ -581,7 +622,7 @@ static u64 uce_shell_spawn_spec(const DValue& spec)
}
static DValue uce_exec_argv_capture(std::vector<String> argv, String input, u64 timeout_ms)
static DValue uce_exec_argv_capture(std::vector<String> argv, String input, u64 timeout_ms, size_t output_limit=0, bool clean_env=false)
{
DValue r; r["exit_code"]=(f64)-1; r["stdout"]=""; r["stderr"]=""; r["timed_out"].set_bool(false);
if(argv.empty()) { r["stderr"]="empty argv"; return(r); }
@@ -598,7 +639,9 @@ static DValue uce_exec_argv_capture(std::vector<String> argv, String input, u64
dup2(inpipe[0],0); dup2(outpipe[1],1); dup2(errpipe[1],2);
close(inpipe[0]); close(inpipe[1]); close(outpipe[0]); close(outpipe[1]); close(errpipe[0]); close(errpipe[1]);
std::vector<char*> args; for(auto& a: argv) args.push_back((char*)a.c_str()); args.push_back(0);
execvp(args[0], args.data()); _exit(127);
if(clean_env) { clearenv(); setenv("PATH", "/usr/bin:/bin", 1); execv(args[0], args.data()); }
else execvp(args[0], args.data());
_exit(127);
}
if(pid < 0)
{
@@ -614,8 +657,8 @@ static DValue uce_exec_argv_capture(std::vector<String> argv, String input, u64
{
if(!exited) { pid_t w=waitpid(pid,&status,WNOHANG); if(w==pid) { exited=true; status_valid=true; } else if(w<0&&errno==ECHILD) { u64 transfer_deadline=wasm_deadline_after_ms(50); do { status_valid=child_exit_status_take(pid,status,child_status_snapshot); if(!status_valid)sched_yield(); } while(!status_valid&&wasm_monotonic_ms()<transfer_deadline); exited=true; if(!status_valid)r["stderr"]=r["stderr"].to_string()+"lost child exit status"; } }
if(in_open) { if(input_off<input.size()) { ssize_t n=write(inpipe[1], input.data()+input_off, input.size()-input_off); if(n>0) input_off+=(size_t)n; else if(n<0 && errno!=EINTR && errno!=EAGAIN && errno!=EWOULDBLOCK) { close(inpipe[1]); in_open=false; } } else { close(inpipe[1]); in_open=false; } }
char buf[4096]; ssize_t n; while((n=read(outpipe[0],buf,sizeof(buf)))>0) r["stdout"] = r["stdout"].to_string()+String(buf,n); if(n==0&&out_open){close(outpipe[0]);out_open=false;}
while((n=read(errpipe[0],buf,sizeof(buf)))>0) r["stderr"] = r["stderr"].to_string()+String(buf,n); if(n==0&&err_open){close(errpipe[0]);err_open=false;}
char buf[4096]; ssize_t n; while((n=read(outpipe[0],buf,sizeof(buf)))>0) { if(output_limit && r["stdout"].to_string().size()+(size_t)n>output_limit) { r["output_limited"].set_bool(true); kill(-pid,SIGKILL); kill(pid,SIGKILL); } else r["stdout"] = r["stdout"].to_string()+String(buf,n); } if(n==0&&out_open){close(outpipe[0]);out_open=false;}
while((n=read(errpipe[0],buf,sizeof(buf)))>0) { if(output_limit && r["stderr"].to_string().size()+(size_t)n>output_limit) { r["output_limited"].set_bool(true); kill(-pid,SIGKILL); kill(pid,SIGKILL); } else r["stderr"] = r["stderr"].to_string()+String(buf,n); } if(n==0&&err_open){close(errpipe[0]);err_open=false;}
if((out_open || err_open || !exited) && wasm_monotonic_ms() >= deadline) { r["timed_out"].set_bool(true); kill(-pid,SIGKILL); kill(pid,SIGKILL); if(!exited) status_valid=waitpid(pid,&status,0)==pid; exited=true; if(in_open){close(inpipe[1]);in_open=false;} if(out_open){close(outpipe[0]);out_open=false;} if(err_open){close(errpipe[0]);err_open=false;} }
if(out_open || err_open || !exited) usleep(10000);
}
@@ -630,8 +673,11 @@ static bool uce_header_name_safe(String name)
return(true);
}
static DValue uce_hardened_http_request_value(const DValue& req, u64 timeout_ms, bool keep_worker_process_group=false);
static DValue uce_http_request_value(const DValue& req)
{
const DValue* security=req.key("security"); if(hardened_http_security_requested(security)) { u64 requested=req.key("timeout_ms")?req.key("timeout_ms")->to_u64(5000):5000; return(uce_hardened_http_request_value(req,std::max<u64>(1,requested))); }
DValue r; r["status"]=(f64)0; r["headers"].set_array(); r["body"]=""; r["error"]="";
const DValue* method_value = req.key("method");
const DValue* url_value = req.key("url");
@@ -665,19 +711,35 @@ static DValue uce_http_request_value(const DValue& req)
return(r);
}
static DValue uce_hardened_http_request_value(const DValue& req, u64 timeout_ms, bool keep_worker_process_group)
{
HardenedHttpHooks hooks;
hooks.resolve=[](String host) { std::vector<String> answers; addrinfo hints{}; hints.ai_socktype=SOCK_STREAM; hints.ai_family=AF_UNSPEC; addrinfo* result=0; if(getaddrinfo(host.c_str(), "443", &hints, &result)!=0) return answers; for(addrinfo* p=result;p;p=p->ai_next) { char text[INET6_ADDRSTRLEN]; if(p->ai_family==AF_INET && inet_ntop(AF_INET,&((sockaddr_in*)p->ai_addr)->sin_addr,text,sizeof(text))) answers.push_back(text); else if(p->ai_family==AF_INET6 && inet_ntop(AF_INET6,&((sockaddr_in6*)p->ai_addr)->sin6_addr,text,sizeof(text))) answers.push_back(text); else answers.push_back(""); } freeaddrinfo(result); return answers; };
hooks.execute=[keep_worker_process_group](std::vector<String> argv, String input, std::vector<String>, u64 deadline, size_t limit) { return hardened_http_exec_argv_capture(argv,input,deadline,limit,true,!keep_worker_process_group); };
return hardened_http_request_internal(req,timeout_ms,hooks);
}
static u64 uce_http_spawn_spec(const DValue& req)
{
uce_job_reap(); u64 id=uce_job_new("http"); if(!id) return(0);
int ready[2]; if(pipe(ready)) { DValue r; r["error"]="pipe failed"; uce_job_finish(id,r,"failed"); return(id); }
pid_t pid=fork();
if(pid==0) { setsid(); uce_write_text(uce_job_path(id)+"/worker_pid", std::to_string((long long)getpid())); uce_write_text(uce_job_path(id)+"/state", "running"); DValue result=uce_http_request_value(req); uce_job_finish(id,result,result["error"].to_string()==""?"done":"failed"); _exit(0); }
if(pid<0) { DValue r; r["error"]="fork failed"; uce_job_finish(id,r,"failed"); return(id); }
if(pid==0) { close(ready[0]); if(setsid()<0) _exit(127); char ok='1'; if(write(ready[1],&ok,1)!=1) _exit(127); close(ready[1]); uce_write_text(uce_job_path(id)+"/worker_pid", std::to_string((long long)getpid())); uce_write_text(uce_job_path(id)+"/state", "running"); const DValue* security=req.key("security"); bool hardened=hardened_http_security_requested(security); DValue result=hardened ? uce_hardened_http_request_value(req,req.key("timeout_ms")?std::max<u64>(1,req.key("timeout_ms")->to_u64(5000)):5000,true) : uce_http_request_value(req); uce_job_finish(id,result,result["error"].to_string()==""?"done":"failed"); _exit(0); }
close(ready[1]); char ok=0; ssize_t started; do { started=read(ready[0],&ok,1); } while(started<0&&errno==EINTR); close(ready[0]);
if(pid<0 || started!=1 || ok!='1') { DValue r; r["error"]="async worker start failed"; uce_job_finish(id,r,"failed"); return(id); }
uce_write_text(uce_job_path(id)+"/worker_pid", std::to_string((long long)pid)); uce_write_text(uce_job_path(id)+"/state", "running"); return(id);
}
static DValue uce_job_status_value(u64 id)
{
DValue r; String dir=uce_job_path(id); r["job_id"]=(f64)id;
if(id==0 || !std::filesystem::is_directory(dir)) { r["state"]="missing"; return(r); }
bool is_directory = false;
if(id != 0)
{
try { is_directory = std::filesystem::is_directory(dir); }
catch(...) { is_directory = false; }
}
if(!is_directory) { r["state"]="missing"; return(r); }
String state=trim(uce_read_text(dir+"/state")); if(state=="") state="pending"; r["state"]=state;
r["kind"]=trim(uce_read_text(dir+"/kind")); r["pid"]=(f64)strtoull(uce_read_text(dir+"/worker_pid").c_str(),0,10);
r["done"].set_bool(state=="done"||state=="failed"||state=="cancelled");
@@ -1271,6 +1333,14 @@ static bool wasm_read_metadata_file(const String& path, std::vector<u8>& metadat
class WasmWorkspace;
#ifdef UCE_WASM_HOST_CONNECTORS
struct WasmMySQLPersistentConnection
{
MySQL* db = 0;
std::chrono::steady_clock::time_point idle_since;
};
#endif
class WasmWorker
{
public:
@@ -1283,25 +1353,59 @@ public:
#ifdef UCE_WASM_HOST_CONNECTORS
~WasmWorker()
{
for(auto* db : mysql_persistent_pool)
delete db;
for(auto& entry : mysql_persistent_pool)
delete entry.db;
}
void mysql_evict_idle(std::chrono::steady_clock::time_point now = std::chrono::steady_clock::now())
{
if(cfg.mysql_persistent_pool_idle_timeout_seconds == 0)
return;
auto timeout = std::chrono::seconds(cfg.mysql_persistent_pool_idle_timeout_seconds);
for(size_t i = 0; i < mysql_persistent_pool.size();)
{
auto& entry = mysql_persistent_pool[i];
if(entry.idle_since != std::chrono::steady_clock::time_point() && now - entry.idle_since >= timeout)
{
delete entry.db;
mysql_persistent_pool.erase(mysql_persistent_pool.begin() + i);
continue;
}
i++;
}
}
void mysql_release(MySQL* db)
{
if(!db)
return;
db->request_leases = 0;
for(auto& entry : mysql_persistent_pool)
if(entry.db == db)
{
entry.idle_since = std::chrono::steady_clock::now();
return;
}
}
MySQL* mysql_checkout(const String& host, const String& username, const String& password, const String& database, bool& reused, bool& persistent)
{
mysql_evict_idle();
reused = false;
persistent = false;
for(size_t i = 0; i < mysql_persistent_pool.size(); i++)
{
MySQL* db = mysql_persistent_pool[i];
MySQL* db = mysql_persistent_pool[i].db;
if(!db || !db->connection || db->request_host != host || db->request_username != username || db->request_password != password || db->request_database != database)
continue;
if(db->reset_connection())
{
mysql_persistent_pool[i].idle_since = std::chrono::steady_clock::time_point();
if(i + 1 < mysql_persistent_pool.size())
{
auto entry = mysql_persistent_pool[i];
mysql_persistent_pool.erase(mysql_persistent_pool.begin() + i);
mysql_persistent_pool.push_back(db);
mysql_persistent_pool.push_back(entry);
}
reused = true;
persistent = true;
@@ -1326,15 +1430,17 @@ public:
{
while(mysql_persistent_pool.size() >= cfg.mysql_persistent_pool_size)
{
delete mysql_persistent_pool.front();
delete mysql_persistent_pool.front().db;
mysql_persistent_pool.erase(mysql_persistent_pool.begin());
}
mysql_persistent_pool.push_back(db);
WasmMySQLPersistentConnection entry;
entry.db = db;
mysql_persistent_pool.push_back(entry);
}
return(db);
}
std::vector<MySQL*> mysql_persistent_pool;
std::vector<WasmMySQLPersistentConnection> mysql_persistent_pool;
#endif
String init()
@@ -1557,14 +1663,30 @@ public:
static String serialize_module_artifact(const String& wasm_path)
{
int lock_fd = open((wasm_path + ".lock").c_str(), O_RDWR | O_CREAT | O_CLOEXEC, 0666);
if(lock_fd < 0 || flock(lock_fd, LOCK_EX) != 0)
{
if(lock_fd >= 0)
close(lock_fd);
return("cannot lock " + wasm_path);
}
struct SerializationLock
{
int fd;
~SerializationLock() { flock(fd, LOCK_UN); close(fd); }
} lock{lock_fd};
if(!serialized_module_needs_refresh(wasm_path))
return("");
struct stat initial_stat;
if(stat(wasm_path.c_str(), &initial_stat) != 0 || !S_ISREG(initial_stat.st_mode))
return("cannot stat " + wasm_path);
std::vector<u8> bytes;
if(!wasm_read_file(wasm_path, bytes))
return("cannot read " + wasm_path);
wasmtime::Engine engine = make_engine();
String error;
auto module = compile_and_cache_module(engine, cached_wasm_path(wasm_path), bytes, error);
auto module = compile_and_cache_module(engine, cached_wasm_path(wasm_path), bytes, error,
wasm_path, &initial_stat);
if(!module)
return(error);
if(serialized_module_needs_refresh(wasm_path))
@@ -1612,8 +1734,15 @@ private:
return(std::nullopt);
}
static bool same_artifact(const struct stat& left, const struct stat& right)
{
return(left.st_dev == right.st_dev && left.st_ino == right.st_ino && left.st_size == right.st_size &&
left.st_mtim.tv_sec == right.st_mtim.tv_sec && left.st_mtim.tv_nsec == right.st_mtim.tv_nsec &&
left.st_ctim.tv_sec == right.st_ctim.tv_sec && left.st_ctim.tv_nsec == right.st_ctim.tv_nsec);
}
static std::optional<wasmtime::Module> compile_and_cache_module(wasmtime::Engine& engine, const String& cached_path,
std::vector<u8>& bytes, String& compile_error)
std::vector<u8>& bytes, String& compile_error, const String& source_path = "", const struct stat* expected_source = 0)
{
auto compiled = wasmtime::Module::compile(engine, bytes);
if(!compiled)
@@ -1636,6 +1765,14 @@ private:
{
out.flush();
out.close();
struct stat current_source;
if(expected_source && (stat(source_path.c_str(), &current_source) != 0 ||
!same_artifact(*expected_source, current_source)))
{
(void)std::remove(tmp.c_str());
compile_error = "wasm artifact changed during serialization: " + source_path;
return(std::nullopt);
}
if(std::rename(tmp.c_str(), cached_path.c_str()) != 0)
(void)std::remove(tmp.c_str());
}
@@ -1752,6 +1889,7 @@ public:
bool writable = false;
};
std::vector<FileHandle> file_handles;
std::vector<int> socket_fds;
struct RequestPerfSnapshot
{
@@ -1909,7 +2047,9 @@ public:
// the wasm-side enforcement of request-scoped DB lifecycle; app code should
// never cache these opaque handles across requests.
std::vector<SQLite*> sqlite_handles;
std::vector<MySQL*> mysql_handles;
std::map<u64, MySQL*> mysql_handles;
std::set<u64> mysql_task_handles;
u64 mysql_next_handle = 1;
std::vector<MySQL*> mysql_request_pool;
std::vector<MySQL*> mysql_request_owned;
#endif
@@ -1926,13 +2066,20 @@ public:
h.fd = -1;
}
}
for(auto& fd : socket_fds)
{
if(fd >= 0)
{
::socket_close((u64)fd);
fd = -1;
}
}
#ifdef UCE_WASM_HOST_CONNECTORS
for(auto* db : sqlite_handles)
if(db)
delete db; // ~SQLite disconnects
for(auto* db : mysql_request_pool)
if(db)
db->request_leases = 0;
worker.mysql_release(db);
for(auto* db : mysql_request_owned)
if(db)
delete db; // ~MySQL disconnects
@@ -3559,6 +3706,33 @@ private:
return(add([self](Caller, Span<const Val> args, Span<Val> results) -> Result<std::monostate, Trap> { String password,encoded; self->hostcall_read(args[0].i32(), args[1].i32(), password); self->hostcall_read(args[2].i32(), args[3].i32(), encoded); bool valid=password_verify_native(password,encoded); results[0]=Val((int32_t)(valid?1:0)); return(std::monostate()); }));
if(mod == "env" && name == "uce_host_password_needs_rehash")
return(add([self](Caller, Span<const Val> args, Span<Val> results) -> Result<std::monostate, Trap> { String encoded; self->hostcall_read(args[0].i32(), args[1].i32(), encoded); results[0]=Val((int32_t)(password_needs_rehash_native(encoded)?1:0)); return(std::monostate()); }));
if(mod == "env" && name == "uce_host_crypto_operation")
return(add([self](Caller, Span<const Val> args, Span<Val> results) -> Result<std::monostate, Trap> {
String encoded, out;
DValue request, response;
String error;
int32_t input_size = args[1].i32();
u32 cap = (u32)args[3].i32();
int32_t buf = args[2].i32();
if(input_size > 0 && input_size <= 64 * 1024 && self->hostcall_read(args[0].i32(), input_size, encoded) == "" && ucb_decode(encoded, request, &error))
{
String stage_key = "crypto_operation:" + encoded;
if(!self->hostcall_staged(stage_key, out))
{
response = crypto_operation_native(request);
out = ucb_encode(response);
if(buf == 0) self->hostcall_stage(stage_key, out);
}
}
else
{
response["error"] = "invalid_request";
out = ucb_encode(response);
}
if(buf && cap >= out.size()) self->hostcall_write(buf, out);
results[0] = Val((int32_t)out.size());
return(std::monostate());
}));
if(mod == "env" && name == "uce_host_log")
return(add([self](Caller, Span<const Val> args, Span<Val>) -> Result<std::monostate, Trap> {
String text;
@@ -3985,7 +4159,7 @@ private:
if(mod == "env" && name == "uce_host_file_truncate")
return(add([self](Caller, Span<const Val> args, Span<Val> results) -> Result<std::monostate, Trap> { String path,current; self->hostcall_read(args[0].i32(),args[1].i32(),path); self->hostcall_read(args[2].i32(),args[3].i32(),current); String r=self->resolve_guest_write(path,current); results[0]=Val((int32_t)(r!=""&&truncate(r.c_str(),(off_t)args[4].i64())==0)); return(std::monostate()); }));
if(mod == "env" && name == "uce_host_dir_remove")
return(add([self](Caller, Span<const Val> args, Span<Val> results) -> Result<std::monostate, Trap> { String path,current; self->hostcall_read(args[0].i32(),args[1].i32(),path); self->hostcall_read(args[2].i32(),args[3].i32(),current); String r=self->resolve_guest_write(path,current); bool rec=args[4].i32()!=0; bool ok=false; if(r!="") { if(rec) ok=std::filesystem::remove_all(r)>0; else ok=::rmdir(r.c_str())==0; } results[0]=Val((int32_t)ok); return(std::monostate()); }));
return(add([self](Caller, Span<const Val> args, Span<Val> results) -> Result<std::monostate, Trap> { String path,current; self->hostcall_read(args[0].i32(),args[1].i32(),path); self->hostcall_read(args[2].i32(),args[3].i32(),current); String r=self->resolve_guest_write(path,current); bool rec=args[4].i32()!=0; bool ok=false; if(r!="") { if(rec) { try { ok=std::filesystem::remove_all(r)>0; } catch(...) { ok=false; } } else ok=::rmdir(r.c_str())==0; } results[0]=Val((int32_t)ok); return(std::monostate()); }));
if(mod == "env" && name == "uce_host_file_temp")
return(add([self](Caller, Span<const Val> args, Span<Val> results) -> Result<std::monostate, Trap> { String prefix,current; self->hostcall_read(args[0].i32(),args[1].i32(),prefix); self->hostcall_read(args[2].i32(),args[3].i32(),current); u32 cap=(u32)args[5].i32(); int32_t buf=args[4].i32(); String out; String stage_key="file_temp:"+prefix+"\0"+current; if(!self->hostcall_staged(stage_key,out)) { if(prefix=="") prefix="/tmp/uce-temp"; String templ=self->resolve_guest_write(prefix+"XXXXXX",current); if(templ!="") { std::vector<char> t(templ.begin(), templ.end()); t.push_back(0); int fd=mkstemp(t.data()); if(fd>=0) { close(fd); out=t.data(); } } if(buf==0) self->hostcall_stage(stage_key,out); } if(buf&&cap>=out.size()) self->hostcall_write(buf,out); results[0]=Val((int32_t)out.size()); return(std::monostate()); }));
if(mod == "env" && name == "uce_host_file_chmod")
@@ -4195,9 +4369,15 @@ private:
}));
if(mod == "env" && name == "uce_host_memcache_command")
return(add([self](Caller caller, Span<const Val> args, Span<Val> results) -> Result<std::monostate, Trap> {
u64 handle = (u64)args[0].i64();
if(handle < 1 || handle > self->socket_fds.size() || self->socket_fds[(size_t)handle - 1] < 0)
{
results[0] = Val((int32_t)0);
return(std::monostate());
}
String command;
self->hostcall_read(args[1].i32(), args[2].i32(), command);
String key = "memcache:" + std::to_string((u64)args[0].i64()) + ":" + command;
String key = "memcache:" + std::to_string(handle) + ":" + command;
u32 cap = (u32)args[4].i32();
int32_t buf = args[3].i32();
String out;
@@ -4209,9 +4389,8 @@ private:
}
else
{
u64 socket_fd = (u64)args[0].i64();
wasm_socket_write_bounded(socket_fd, command + "\r\n", self->bounded_hostcall_timeout_ms(1000));
out = wasm_socket_read_bounded(socket_fd, 1024 * 128, self->bounded_hostcall_timeout_ms(1000));
u64 socket_fd = (u64)self->socket_fds[(size_t)handle - 1];
out = wasm_memcache_exchange(socket_fd, command, self->bounded_hostcall_timeout_ms(1000));
if(buf == 0)
{
self->staged_memcache_key = key;
@@ -4248,41 +4427,60 @@ private:
String password = request["password"].to_string();
String database = request["database"].to_string();
MySQL* db = 0;
for(auto* pooled : self->mysql_request_pool)
if(pooled && pooled->connection && pooled->request_host == host && pooled->request_username == username && pooled->request_password == password && pooled->request_database == database)
{
db = pooled;
connection_source = "request";
break;
}
bool ok = db != 0;
if(!db)
bool ok = false;
if(task_child_process)
{
bool reused = false;
bool persistent = false;
db = self->worker.mysql_checkout(host, username, password, database, reused, persistent);
connection_source = reused ? "worker" : "new";
ok = db && db->connection;
if(ok && db->connection)
// task() closes inherited descriptors after fork. Never reuse
// request/worker MySQL objects whose sockets were just closed.
db = new MySQL();
ok = db->connect(host, username, password, database);
connection_source = "task";
if(ok)
{
self->mysql_request_pool.push_back(db);
if(!persistent)
self->mysql_request_owned.push_back(db);
self->mysql_request_owned.push_back(db);
}
}
else
{
for(auto* pooled : self->mysql_request_pool)
if(pooled && pooled->connection && pooled->request_host == host && pooled->request_username == username && pooled->request_password == password && pooled->request_database == database)
{
db = pooled;
connection_source = "request";
break;
}
ok = db != 0;
if(!db)
{
bool reused = false;
bool persistent = false;
db = self->worker.mysql_checkout(host, username, password, database, reused, persistent);
connection_source = reused ? "worker" : "new";
ok = db && db->connection;
if(ok)
{
self->mysql_request_pool.push_back(db);
if(!persistent)
self->mysql_request_owned.push_back(db);
}
}
}
u64 handle = 0;
if(ok && db->connection)
if(ok && db->connection && self->mysql_next_handle != 0)
{
db->request_leases++;
self->mysql_handles.push_back(db);
handle = self->mysql_handles.size();
handle = self->mysql_next_handle;
self->mysql_next_handle = handle == UINT64_MAX ? 0 : handle + 1;
self->mysql_handles[handle] = db;
if(task_child_process) self->mysql_task_handles.insert(handle);
if(connection_source == "new") self->mysql_connection_open_count++;
else if(connection_source == "worker") self->mysql_connection_reuse_count++;
else if(connection_source == "request") self->mysql_request_pool_hit_count++;
}
response["handle"] = (f64)handle;
response["error_code"] = (f64)db->_preload_next_error_code;
response["statement_info"] = db->error();
response["error_code"] = (f64)(handle == 0 ? 2000 : db->_preload_next_error_code);
response["statement_info"] = handle == 0 ? String("mysql handle space exhausted") : db->error();
if(handle == 0 && db)
delete db;
}
@@ -4294,8 +4492,8 @@ private:
else
{
u64 handle = request["handle"].to_u64();
MySQL* db = (handle >= 1 && handle <= self->mysql_handles.size())
? self->mysql_handles[(size_t)handle - 1] : 0;
auto handle_it = self->mysql_handles.find(handle);
MySQL* db = handle_it == self->mysql_handles.end() || (task_child_process && self->mysql_task_handles.find(handle) == self->mysql_task_handles.end()) ? 0 : handle_it->second;
if(op == "query" && db)
{
response["result"] = db->query(request["query"].to_string());
@@ -4308,7 +4506,20 @@ private:
{
if(db->request_leases > 0)
db->request_leases--;
self->mysql_handles[(size_t)handle - 1] = 0;
self->mysql_handles.erase(handle);
self->mysql_task_handles.erase(handle);
if(task_child_process && db->request_leases == 0)
{
auto erase_db = [db](auto& pool) { pool.erase(std::remove(pool.begin(), pool.end(), db), pool.end()); };
erase_db(self->mysql_request_pool);
erase_db(self->mysql_request_owned);
delete db;
}
}
else if(op == "query" || op == "disconnect")
{
response["error_code"] = (f64)2000;
response["statement_info"] = "mysql handle is invalid or unavailable in this task";
}
}
}
@@ -4350,24 +4561,52 @@ private:
String host;
self->hostcall_read(args[0].i32(), args[1].i32(), host);
u64 fd = wasm_socket_connect_bounded(host, (u16)args[2].i32(), self->bounded_hostcall_timeout_ms(self->worker.cfg.invocation_timeout_ms));
results[0] = Val((int64_t)fd);
u64 handle = 0;
if(fd > 0)
{
self->socket_fds.push_back((int)fd);
handle = self->socket_fds.size();
}
results[0] = Val((int64_t)handle);
return(std::monostate());
}));
if(mod == "env" && name == "uce_host_socket_close")
return(add([](Caller, Span<const Val> args, Span<Val>) -> Result<std::monostate, Trap> {
::socket_close((u64)args[0].i64());
return(add([self](Caller, Span<const Val> args, Span<Val>) -> Result<std::monostate, Trap> {
u64 handle = (u64)args[0].i64();
if(handle >= 1 && handle <= self->socket_fds.size())
{
int& fd = self->socket_fds[(size_t)handle - 1];
if(fd >= 0)
{
::socket_close((u64)fd);
fd = -1;
}
}
return(std::monostate());
}));
if(mod == "env" && name == "uce_host_socket_write")
return(add([self](Caller, Span<const Val> args, Span<Val> results) -> Result<std::monostate, Trap> {
u64 handle = (u64)args[0].i64();
String data;
self->hostcall_read(args[1].i32(), args[2].i32(), data);
results[0] = Val(wasm_socket_write_bounded((u64)args[0].i64(), data, self->bounded_hostcall_timeout_ms(self->worker.cfg.invocation_timeout_ms)) ? (int32_t)1 : (int32_t)0);
bool ok = false;
if(handle >= 1 && handle <= self->socket_fds.size())
{
int fd = self->socket_fds[(size_t)handle - 1];
ok = fd >= 0 && wasm_socket_write_bounded((u64)fd, data, self->bounded_hostcall_timeout_ms(self->worker.cfg.invocation_timeout_ms));
}
results[0] = Val(ok ? (int32_t)1 : (int32_t)0);
return(std::monostate());
}));
if(mod == "env" && name == "uce_host_socket_read")
return(add([self](Caller caller, Span<const Val> args, Span<Val> results) -> Result<std::monostate, Trap> {
u64 sockfd = (u64)args[0].i64();
u64 handle = (u64)args[0].i64();
if(handle < 1 || handle > self->socket_fds.size() || self->socket_fds[(size_t)handle - 1] < 0)
{
results[0] = Val((int32_t)0);
return(std::monostate());
}
u64 sockfd = (u64)self->socket_fds[(size_t)handle - 1];
u32 max_length = (u32)args[1].i32();
u32 requested_timeout = (u32)args[2].i32();
u64 requested_ms = requested_timeout == 0 ? self->invocation_remaining_ms() : (u64)requested_timeout * 1000;
@@ -4375,7 +4614,7 @@ private:
int32_t buf = args[3].i32();
u32 cap = (u32)args[4].i32();
// The size and fetch calls share this key, while the remaining budget may change between them.
String key = std::to_string(sockfd) + ":" + std::to_string(max_length) + ":" + std::to_string(requested_timeout);
String key = std::to_string(handle) + ":" + std::to_string(max_length) + ":" + std::to_string(requested_timeout);
String out;
if(buf != 0 && self->staged_socket_read_key == key)
{
@@ -4442,7 +4681,7 @@ private:
// before the hostcall stack unwinds, so `self` points to the child's
// copy of this per-request workspace. The parent request can return and
// destroy its workspace without invalidating the child copy.
u64 task_timeout_ms = timeout > UINT64_MAX / 1000 ? UINT64_MAX : timeout * 1000;
u64 task_timeout_ms = timeout == 0 ? UINT64_MAX : (timeout > UINT64_MAX / 1000 ? UINT64_MAX : timeout * 1000);
auto run_callback = [self, callback_id, task_timeout_ms]() {
String error = self->run_task_callback(callback_id, task_timeout_ms);
if(error != "")
@@ -4597,6 +4836,9 @@ inline WasmResponse wasm_worker_serve(WasmWorker& worker, const Request& request
WasmResponse response;
f64 serve_started = time_precise();
auto workspace_start = std::chrono::steady_clock::now();
#ifdef UCE_WASM_HOST_CONNECTORS
worker.mysql_evict_idle(workspace_start);
#endif
f64 cpu_started = wasm_thread_cpu_time();
struct rusage thread_runtime_start = {};
bool thread_runtime_profiled = worker.cfg.profile_thread_runtime && getrusage(RUSAGE_THREAD, &thread_runtime_start) == 0;