Compare commits
12
Commits
88e2ad022d
..
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
09f743fdcf | ||
|
|
36d6207bf2 | ||
|
|
d0efab7db0 | ||
|
|
3d155203bd | ||
|
|
68b73343a2 | ||
|
|
10e6565037 | ||
|
|
0c75703186 | ||
|
|
91c745c44b | ||
|
|
e22cbf8882 | ||
|
|
80eaf0b69b | ||
|
|
4d68377704 | ||
|
|
a48f6af502 |
+12
-3
@@ -1,6 +1,11 @@
|
||||
# DOCS-TODO — `/doc` documentation overhaul
|
||||
# Completed `/doc` documentation overhaul (historical plan)
|
||||
|
||||
Briefing for the doc-system overhaul. Read this fully before touching anything.
|
||||
This June 2026 plan is complete and is retained only as implementation history.
|
||||
It is not the current task queue and its one-off execution rules do not govern
|
||||
new UCE work. Current documentation has 269 rendered pages and is covered by
|
||||
the sharded doc gate in `scripts/run_cli_tests.sh`.
|
||||
|
||||
The original briefing follows.
|
||||
The approved design and the two locked decisions are: **(1) examples are LIVE and
|
||||
SELF-VERIFYING** — real UCE code executed at render time, showing source + actual
|
||||
captured output, gated by the test suite; **(2) FULL SWEEP** of all 257 pages.
|
||||
@@ -163,7 +168,11 @@ and the suite stays green.
|
||||
```
|
||||
Expect the current 91 to grow by the new doc gate(s), 0 failed.
|
||||
|
||||
## Hard rules (non-negotiable)
|
||||
## Historical execution rules (no longer active)
|
||||
|
||||
These rules applied only while the overhaul was being carried out in June 2026.
|
||||
They are preserved to explain that work and must not override current repository
|
||||
or operator instructions.
|
||||
|
||||
- **NEVER `git commit`/push/tag.** Not pi, not its subagents, not anyone. Leave a clean
|
||||
working tree and report.
|
||||
|
||||
@@ -14,9 +14,9 @@ Implement the DOCS-TODO.md overhaul for the UCE `/doc` system on the sshfs sourc
|
||||
|
||||
## Current State
|
||||
|
||||
- Status: verifying; Phase 1 + gate complete, StringList conversion complete, Phase 2 example/filler sweep complete.
|
||||
- Last updated: 2026-06-16
|
||||
- Source of truth: `/root/mount_ssh/uce-dev-root-htdocs-uce`
|
||||
- Status: complete; retained as the implementation record for the June 2026 overhaul.
|
||||
- Last updated: 2026-07-20
|
||||
- Source of truth: `/root/mount_ssh/uce-upstream`
|
||||
- Runtime/live target: `root@10.4.2.110:/Code/uce.openfu.com/uce`
|
||||
|
||||
## Goal Tree
|
||||
@@ -38,10 +38,10 @@ Legend: `[ ]` not started, `[~]` in progress, `[x]` done, `[!]` blocked, `[-]` s
|
||||
- Done when: CLI runner checks every doc page and examples.
|
||||
- Verify: full host gate passes.
|
||||
|
||||
## Execution Queue
|
||||
## Completed Execution Queue
|
||||
|
||||
1. G2 area batching/delegation now that the mechanism is proven.
|
||||
2. Keep the host suite green after each area batch.
|
||||
1. Phase 2 area batching and delegation completed.
|
||||
2. The host suite and sharded 269-page documentation gate remain the ongoing regression boundary.
|
||||
|
||||
## Decisions
|
||||
|
||||
@@ -55,6 +55,7 @@ Legend: `[ ]` not started, `[~]` in progress, `[x]` done, `[!]` blocked, `[-]` s
|
||||
- 2026-06-16: Converted retiring `list_unique/list_sort/list_some/list_every/list_find` free functions to `StringList` methods, updated callers and method docs; full host gate passed: `Summary: 353 passed, 0 failed, 0 skipped`.
|
||||
- 2026-06-16: Added live examples across types, string, regex, time, markup, noise, uri, and session area pages; full host gate passed: `Summary: 353 passed, 0 failed, 0 skipped`.
|
||||
- 2026-06-16: Added `:example` blocks to all remaining doc pages, removed full PHP/JS Related Concepts filler, removed duplicate content code fences except resource-bound pages; full host gate passed: `Summary: 353 passed, 0 failed, 0 skipped`.
|
||||
- 2026-07-20: Marked the finished plan historical so it no longer presents completed work or one-off execution rules as current instructions.
|
||||
|
||||
## Change Log
|
||||
|
||||
|
||||
@@ -169,6 +169,13 @@ systemctl daemon-reload
|
||||
systemctl restart uce.service
|
||||
```
|
||||
|
||||
When `uce.socket` is active, restart only `uce.service` (or use
|
||||
`scripts/systemd/manage-uce-service.sh restart`). The socket unit must retain
|
||||
ownership of the FastCGI listener while workers change. Restarting
|
||||
`uce.socket` alongside the service removes that listener briefly and can expose
|
||||
HTTP 502 responses through nginx. Restart the socket unit only to repair the
|
||||
socket itself, with an explicit outage window.
|
||||
|
||||
## Runtime configuration
|
||||
|
||||
Create `/etc/uce/settings.cfg` from `etc/uce/settings.cfg` and adjust paths to your checkout root. Replace checkout-specific settings such as `WASM_CORE_PATH` with `<UCE_REPO_ROOT>/bin/wasm/core.wasm` or leave it relative to `COMPILER_SYS_PATH` as appropriate.
|
||||
@@ -185,6 +192,8 @@ FCGI_SOCKET_PATH=/run/uce/fastcgi.sock
|
||||
FCGI_SOCKET_MODE=0666
|
||||
CLI_SOCKET_PATH=/run/uce/cli.sock
|
||||
CLI_SOCKET_MODE=0600
|
||||
CLI_WORKER_COUNT=2
|
||||
CLI_WORKER_MAX_REQUESTS=8
|
||||
|
||||
SITE_DIRECTORY=/var/www/html
|
||||
HTTP_DOCUMENT_ROOT=/var/www/html
|
||||
@@ -202,7 +211,9 @@ WASM_MEMORY_LIMIT_BYTES=536870912
|
||||
WASM_EPOCH_DEADLINE_TICKS=200
|
||||
WASM_EPOCH_PERIOD_MS=50
|
||||
WASM_INVOCATION_TIMEOUT_MS=30000
|
||||
WASM_SERIALIZE_TIMEOUT_SECONDS=120
|
||||
MYSQL_PERSISTENT_POOL_SIZE=8
|
||||
MYSQL_PERSISTENT_POOL_IDLE_TIMEOUT_SECONDS=300
|
||||
|
||||
WORKER_COUNT=4
|
||||
MAX_MEMORY=16777216
|
||||
@@ -216,6 +227,8 @@ Important settings:
|
||||
|
||||
- `FCGI_SOCKET_PATH` is the Unix socket used for normal `.uce` requests. Set it explicitly and keep this value and the web-server `fastcgi_pass` path identical. The reference config uses `/run/uce/fastcgi.sock`; if you choose `/run/uce.sock`, use it in both places.
|
||||
- `CLI_SOCKET_PATH` is a local HTTP-over-Unix socket used by `scripts/uce-cli` and test/admin units. Keep it private (`CLI_SOCKET_MODE=0600`) unless you intentionally delegate admin/test execution to a trusted Unix group (`0660`).
|
||||
- `CLI_WORKER_COUNT` adds a transport-isolated CLI/test renderer pool. Use at least `2` on a live site so broad test/admin unit loads cannot evict or page out public FastCGI workers' hot modules and a CLI unit can make one nested CLI call without self-deadlocking. `0` preserves the legacy shared pool; dedicated CLI workers are additional to `WORKER_COUNT` and retain their own Wasmtime engine, module cache, and persistent connector pool.
|
||||
- `CLI_WORKER_MAX_REQUESTS` recycles each dedicated CLI worker after this many completed connections (default `8`, maximum `1024`); `0` disables recycling. Public FastCGI workers are not request-count recycled, preserving their warmed module caches. Recycling bounds the module set retained by broad test and administration runs while allowing one CLI invocation/test group to finish uninterrupted.
|
||||
- `FCGI_SOCKET_MODE` and `CLI_SOCKET_MODE` are octal permission modes applied after socket bind. Prefer tightening `FCGI_SOCKET_MODE` to `0660` when nginx/Apache can share a trusted group with the UCE worker.
|
||||
- `SITE_DIRECTORY` is the public site tree to scan for `.uce` files. Use `/var/www/html` when the web root is outside the runtime tree; relative paths are resolved from the runtime working directory. Installed regression gate scripts derive their temporary test root from this setting unless `UCE_TEST_SITE_DIRECTORY` is explicitly provided.
|
||||
- `HTTP_DOCUMENT_ROOT` is the root used by the built-in HTTP/WebSocket listener when it resolves upgrade requests. Set it to the same web root as nginx/Apache.
|
||||
@@ -225,6 +238,7 @@ Important settings:
|
||||
- `TMP_UPLOAD_PATH` and `SESSION_PATH` must be writable by the runtime.
|
||||
- `SESSION_COOKIE_SECURE=1` adds the `Secure` attribute to UCE-managed session cookies and should be used for HTTPS-only deployments. Leave it `0` only for local/plain-HTTP development.
|
||||
- `MYSQL_PERSISTENT_POOL_SIZE` caps credential-keyed connections retained by each Wasm worker. The default `8` is clamped to `64`; set it to `0` to restore request-lifetime connections. Cached sessions are reset before reuse.
|
||||
- `MYSQL_PERSISTENT_POOL_IDLE_TIMEOUT_SECONDS` retires a pooled connection at the first request boundary after it has been idle for this interval (default `300`, maximum `86400`). Set it to `0` to retain idle connections until capacity eviction, reset failure, worker exit, or database-side closure. Keep a positive value below the database server's idle timeout to avoid attempting stale sessions after traffic resumes. A completely idle UCE worker has no request boundary and therefore does not proactively close sockets.
|
||||
- `HTTP_PORT` is the built-in HTTP/WebSocket listener used for WebSocket upgrade traffic and direct local probes. Bind/firewall it for local access only; nginx/Apache should be the public entry point.
|
||||
- `WS_BROKER_OUTBOUND_TIMEOUT_SECONDS` controls how long a forwarded WS message can remain queued in the broker before being dropped (default `30`). Set to `0` to disable the timeout.
|
||||
- `WASM_COMPILE_SCRIPT` must point to `scripts/compile_wasm_unit` unless you provide an equivalent compiler. Relative paths are resolved from the runtime root/`COMPILER_SYS_PATH`. That script calls `scripts/check_unit_wasm.py` after linking each unit and uses the pinned WASI SDK on every deployment host.
|
||||
@@ -255,6 +269,15 @@ Important settings:
|
||||
4 KiB positional read. Cache-miss full-artifact reads use the same checks
|
||||
around 64 KiB positional chunks. A single in-progress regular-file syscall
|
||||
cannot be cooperatively interrupted; later reads and parsing cannot overrun the budget.
|
||||
Wasmtime's serialized-module deserialization call is also synchronous and
|
||||
cannot be interrupted; if it stalls, the timeout is reported after that call
|
||||
returns rather than at the nominal wall-clock boundary.
|
||||
Proactive and offline serialization instead runs in a short-lived child
|
||||
bounded by `WASM_SERIALIZE_TIMEOUT_SECONDS` (default `120`, clamped to
|
||||
`1`–`3600`), so each unit releases its pooling allocator arenas and threads
|
||||
on exit instead of accumulating them in the long-lived scanner. Serialization
|
||||
holds the unit compile lock and rechecks the wasm artifact identity before
|
||||
publication; timed-out/failed children leave no dead temporary artifact.
|
||||
Initial/final descriptor identity, unique selected metadata sections, and
|
||||
strict 64-bit LEB high-bit validation reject changed or ambiguous artifacts.
|
||||
|
||||
@@ -289,6 +312,7 @@ The server binary accepts only these process modes:
|
||||
```bash
|
||||
bin/uce_fastcgi.linux.bin # start the server
|
||||
bin/uce_fastcgi.linux.bin --precompile
|
||||
bin/uce_fastcgi.linux.bin --serialize-module /absolute/unit.uce.wasm
|
||||
bin/uce_fastcgi.linux.bin --help
|
||||
```
|
||||
|
||||
@@ -312,6 +336,12 @@ workers continue to use `/etc/uce/settings.cfg`. The parallel precompile
|
||||
regression uses private roots so a running proactive compiler cannot consume
|
||||
or publish its controlled race fixtures.
|
||||
|
||||
Direct `--precompile` is a listener-free whole-generation gate, not a
|
||||
single-unit readiness probe. It scans the configured source root, waits for
|
||||
unit publication locks, and has no internal wall-clock timeout. Automation
|
||||
must wrap it in a finite external timeout, as the managed restart does, before
|
||||
starting separately deadline-bounded CLI or HTTP invocations.
|
||||
|
||||
Prefer the managed restart when the service runs as an unprivileged user: it
|
||||
precompiles as that same user. Request-time publication still accepts readable
|
||||
artifacts produced by a trusted administrator. When Linux protected-hardlink
|
||||
@@ -715,6 +745,20 @@ if(valid && password_needs_rehash(encoded))
|
||||
|
||||
`password_hash()` returns a self-contained `$uce$scrypt$...` encoding with a random 16-byte salt and the bounded scrypt parameters `N=65536`, `r=8`, `p=1`. `password_verify()` accepts only structurally valid encodings with bounded cost parameters and compares the derived key in constant time. `password_needs_rehash()` reports malformed, legacy, or non-current parameters so applications can upgrade a credential after a successful legacy verification. Treat an empty hash as an operational failure and never store it. Application-level password length policy, rate limiting, and legacy-format verification remain the application's responsibility.
|
||||
|
||||
## Structured cryptographic operations
|
||||
|
||||
`crypto_operation(request)` is the bounded, algorithm-selected API for structured
|
||||
asymmetric operations. The initial allowlist supports `operation=key_generate`
|
||||
and `operation=jwt_sign` with `algorithm=ES256`. Key generation returns P-256
|
||||
public/private JWKs plus the RFC 7638 thumbprint (`kid`). JWT signing accepts only
|
||||
a consistent P-256 private JWK, forces `alg=ES256`, and emits a compact JWT with
|
||||
a 64-byte JOSE signature. It also supports `cbor_decode`, `cose_es256_parse`, and `es256_verify` with `algorithm=ES256`: inputs are canonical unpadded base64url fields; requests are capped at 32 KiB, while CBOR input is definite-length only and capped at 16 KiB decoded (21,846 base64url characters), 256 nodes, and depth 16; it rejects duplicate keys and trailing bytes, and returns an explicit typed tree (`unsigned`, `negative`, `bytes`, `text`, `array`, `map`) rather than DValue-coerced map keys. COSE accepts only EC2/-7/P-256 with exact 32-byte coordinates and an OpenSSL-valid point. Verification accepts only complete canonical DER ECDSA over `message_base64url` and returns `valid`; malformed material fails closed. V1 has no attestation, Ed25519, or RS256. Unknown operations and algorithms fail closed.
|
||||
|
||||
Existing typed digest, HMAC, password, randomness, and constant-time comparison
|
||||
functions remain separate. `crypto_operation()` exposes no raw signing, arbitrary
|
||||
digest/curve selection, encryption, network fetch, or token exchange. Store
|
||||
private JWKs as secrets.
|
||||
|
||||
## Operational footguns
|
||||
|
||||
- Keep the FastCGI socket path consistent: `FCGI_SOCKET_PATH` and the web-server `fastcgi_pass` must match exactly. The reference config uses `/run/uce/fastcgi.sock`; if you choose `/run/uce.sock`, use it in both places.
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
# UCE WASM Runtime Architecture
|
||||
|
||||
Status: current as of the W7e native-pipeline removal (June 2026). This document
|
||||
describes the **runtime architecture as built** — the process topology, the
|
||||
wasm membrane, the unified request dispatch, and the central WebSocket broker.
|
||||
Status: current as of the worker-role isolation and bounded serializer changes
|
||||
(July 2026). This document describes the **runtime architecture as built**. It
|
||||
covers the process topology, wasm membrane, request dispatch, and WebSocket broker.
|
||||
Native `.so` unit execution/dlopen fallback has been removed; the parser and
|
||||
preprocessor remain only as the front-end that emits C++ for wasm side-module
|
||||
compilation.
|
||||
@@ -26,8 +26,8 @@ gets invoked*.
|
||||
|
||||
```
|
||||
┌────────────────────────────┐
|
||||
nginx ──FastCGI──► worker pool (N processes) │ $FCGI_SOCKET_PATH (example `/run/uce/fastcgi.sock`)
|
||||
(port 80 etc.) │ uniform unit renderers │ (FastCGI + CLI)
|
||||
nginx ──FastCGI──► public workers (N processes)│ $FCGI_SOCKET_PATH (example `/run/uce/fastcgi.sock`)
|
||||
(port 80 etc.) │ public unit renderers │
|
||||
└─────────────▲──────────────┘
|
||||
│ forward render (FastCGI, FCGI_SOCKET_PATH)
|
||||
│
|
||||
@@ -42,13 +42,19 @@ gets invoked*.
|
||||
│ dispatcher(s) │ bind addr; forwards to pool
|
||||
└──────────────────┘
|
||||
|
||||
trusted CLI/test calls ──► ┌──────────────────┐
|
||||
(CLI socket) │ CLI workers │ isolated module and
|
||||
│ (M processes) │ connector caches
|
||||
└──────────────────┘
|
||||
|
||||
parent process: spawns/respawns all of the above + the proactive compiler.
|
||||
```
|
||||
|
||||
| Process | Owns | Renders units? | Source |
|
||||
|---|---|---|---|
|
||||
| **Parent** | nothing; supervises children | no | `main()`, `init_base_process()` |
|
||||
| **Worker** (×`WORKER_COUNT`) | `FCGI_SOCKET_PATH` (configured socket path; example `/run/uce/fastcgi.sock`) + `CLI_SOCKET_PATH` | **yes** — the only processes that run wasm | `listen_for_connections()` |
|
||||
| **Public worker** (×`WORKER_COUNT`) | `FCGI_SOCKET_PATH`; also `CLI_SOCKET_PATH` only when `CLI_WORKER_COUNT=0` | **yes** — runs public FastCGI wasm | `listen_for_connections()` |
|
||||
| **CLI worker** (×`CLI_WORKER_COUNT`) | `CLI_SOCKET_PATH` only | **yes** — isolates trusted CLI/test wasm and its module cache | `listen_for_connections()` |
|
||||
| **WS broker** (×1) | `HTTP_PORT` + every live WS connection + `WS_BROKER_SOCKET_PATH` | no — forwards to the pool | `run_ws_broker()` |
|
||||
| **serve_http dispatcher** (×bind) | one custom-server bind address | no — forwards to the pool | `custom_server_http_dispatcher_loop()` |
|
||||
| **Proactive compiler** | nothing; pre-compiles units | no | `run_proactive_compiler()` |
|
||||
@@ -192,6 +198,9 @@ the service and its post-fork workers restart. Direct launches without systemd
|
||||
activation retain the existing configured Unix/TCP listener behavior.
|
||||
The service preserves its runtime directory across service restarts because the
|
||||
socket unit, not the service unit, owns a listener path inside that directory.
|
||||
Normal deployments therefore restart only `uce.service`; restarting
|
||||
`uce.socket` at the same time removes the listener and forfeits the zero-gap
|
||||
handoff.
|
||||
On termination the parent asks render workers to close their listeners, finish
|
||||
accepted connections within the bounded worker drain interval, and only then
|
||||
exits. This prevents an accepted FastCGI request from being reset at handoff;
|
||||
@@ -211,6 +220,14 @@ hostcall loop therefore cannot renew an invocation indefinitely. Blocking host
|
||||
helpers retain operation-specific limits and cap them to the remaining
|
||||
invocation budget where the underlying operation is cancellable. Forked task
|
||||
callbacks receive a fresh invocation deadline capped by the task lifetime.
|
||||
The Memcached ASCII hostcall applies one deadline to the complete command write
|
||||
and protocol-framed response read. Retrieval payload lengths are parsed, so a
|
||||
payload containing `END` cannot terminate the response; multi-value and `stats`
|
||||
responses may span socket chunks up to the 8 MiB response bound. Timeout,
|
||||
malformed framing, or overflow shuts down the retained descriptor before
|
||||
returning an empty result, preventing unread response bytes from poisoning the
|
||||
next command on that connection. The native fallback uses the same parser and
|
||||
deadline contract.
|
||||
Synchronous compiler locks, transitive `#load` compilation, and compiler child
|
||||
processes consume that same deadline. Compiler children run in a dedicated
|
||||
process group; timeout kills the group, retains the previous generation, and
|
||||
@@ -266,6 +283,10 @@ Cold module compilation and deserialization are host work, so `load_unit()`
|
||||
refreshes the epoch deadline before its first guest call. Otherwise a component
|
||||
whose compilation outlasted the guest CPU budget would immediately trap in the
|
||||
following allocator/relocation call even though no guest loop consumed it.
|
||||
Wasmtime's serialized-module deserialization API is synchronous and has no
|
||||
cooperative cancellation point. The absolute invocation timeout is checked
|
||||
immediately after it returns, so a stalled deserialization may overrun the
|
||||
nominal wall boundary before UCE reports the timeout.
|
||||
|
||||
Unit artifacts live beneath an ABI-generation directory such as
|
||||
`BIN_DIRECTORY/units-c13-w7`: `c13` is the compiler/unit-metadata ABI and `w7`
|
||||
@@ -553,7 +574,15 @@ header free-functions are `inline`. The wasm backend exposes only declarations
|
||||
| `HTTP_PORT` | `8080` | Raw HTTP + WebSocket port — owned by the WS broker. |
|
||||
| `WS_BROKER_SOCKET_PATH` | `/run/uce/ws-broker.sock` | Broker command socket for `ws_*` flushes. |
|
||||
| `WS_BROKER_OUTBOUND_TIMEOUT_SECONDS` | `30` | Max lifetime in seconds for queued WS broker forwards before drop. |
|
||||
| `WORKER_COUNT` | `4` | Number of uniform worker processes. |
|
||||
| `WORKER_COUNT` | `4` | Number of public FastCGI worker processes. |
|
||||
| `CLI_WORKER_COUNT` | `0` built-in; `2` in the reference config | Additional CLI-only workers. Two permit one nested CLI invocation without self-deadlock; a positive count prevents test/admin module-cache churn from paging out public workers, while zero preserves the legacy shared pool. |
|
||||
| `CLI_WORKER_MAX_REQUESTS` | `8` | Completed CLI connections before a dedicated CLI worker is recycled (maximum `1024`; `0` disables). Public workers are not request-count recycled. |
|
||||
| `WASM_SERIALIZE_TIMEOUT_SECONDS` | `120` | Deadline for each short-lived serialized-module child; process exit reclaims pooling allocator arenas from proactive scanners. |
|
||||
|
||||
Proactive scanners never construct a Wasmtime serialization engine themselves.
|
||||
Each candidate is serialized by the executable's bounded `--serialize-module`
|
||||
child while holding the unit compile lock. Publication rechecks device, inode,
|
||||
size, mtime, and ctime, and the scanner removes dead child temporary artifacts.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -14,6 +14,12 @@ FCGI_PORT=9993
|
||||
CLI_SOCKET_PATH=/run/uce/cli.sock
|
||||
# Keep the CLI/admin socket private by default; set 0660 only for a trusted group.
|
||||
CLI_SOCKET_MODE=0600
|
||||
# Keep CLI/test module caches out of public FastCGI workers. Set to 0 only for
|
||||
# the legacy shared pool; dedicated workers are additional to WORKER_COUNT.
|
||||
CLI_WORKER_COUNT=2
|
||||
# Recycle dedicated CLI workers before broad test/admin runs retain every unit.
|
||||
# Zero disables recycling; public FastCGI workers are never request-count recycled.
|
||||
CLI_WORKER_MAX_REQUESTS=8
|
||||
|
||||
# Built-in HTTP/WebSocket listener used for WebSocket Upgrade requests.
|
||||
# Keep this behind nginx/Apache on localhost or firewall it from public access.
|
||||
@@ -59,7 +65,10 @@ WASM_MEMORY_LIMIT_BYTES=536870912
|
||||
WASM_EPOCH_DEADLINE_TICKS=200
|
||||
WASM_EPOCH_PERIOD_MS=50
|
||||
WASM_INVOCATION_TIMEOUT_MS=30000
|
||||
# Bound each short-lived proactive/precompile serialized-module child.
|
||||
WASM_SERIALIZE_TIMEOUT_SECONDS=120
|
||||
MYSQL_PERSISTENT_POOL_SIZE=8
|
||||
MYSQL_PERSISTENT_POOL_IDLE_TIMEOUT_SECONDS=300
|
||||
|
||||
# ENABLE THE BACKGROUND PROACTIVE COMPILER LOOP
|
||||
PROACTIVE_COMPILE_ENABLED=1
|
||||
|
||||
@@ -39,6 +39,7 @@ PUBLIC_APIS = [
|
||||
("hmac_sha256", True, "public"), ("hmac_sha256_hex", True, "public"), ("random_bytes", True, "public"),
|
||||
("crypto_equal", True, "public"), ("password_hash", True, "public"),
|
||||
("password_verify", True, "public"), ("password_needs_rehash", True, "public"),
|
||||
("crypto_operation", True, "public"),
|
||||
("gen_noise32", True, "public"), ("gen_noise64", True, "public"),
|
||||
("gen_noise01", True, "public"), ("gen_int", True, "public"), ("gen_float", True, "public"),
|
||||
("draw_int", True, "public"), ("draw_float", True, "public"),
|
||||
|
||||
@@ -67,6 +67,7 @@ if [[ "$action" == "run" ]]; then
|
||||
curl -sS --max-time "$curl_timeout" --fail-with-body --unix-socket "$socket_path" "${base_url}&group=${group}"
|
||||
done
|
||||
scripts/test_dependency_invalidation.sh
|
||||
scripts/test_compiler_lock_directory.sh
|
||||
scripts/test_abi_generation_rollout.sh
|
||||
scripts/test_parallel_precompile.sh
|
||||
timeout --signal=TERM --kill-after=5s 175s scripts/test_parallel_proactive_compile.sh
|
||||
@@ -77,8 +78,11 @@ if [[ "$action" == "run" ]]; then
|
||||
scripts/test_component_once_prefetch.sh
|
||||
scripts/test_relative_component_cache.sh
|
||||
scripts/test_password_hashing.sh
|
||||
scripts/test_crypto_operation_native.sh
|
||||
scripts/test_hardened_http_native.sh
|
||||
scripts/test_mysql_epoch_refresh.sh
|
||||
scripts/test_mysql_persistent_pool.sh
|
||||
scripts/test_mysql_persistent_pool_idle.sh
|
||||
scripts/test_log_timeliness.sh
|
||||
scripts/test_raw_http_request_log.sh
|
||||
scripts/test_component_resolution_ttl.sh
|
||||
@@ -89,5 +93,6 @@ if [[ "$action" == "run" ]]; then
|
||||
timeout --signal=TERM --kill-after=5s 240s scripts/test_dynamic_compile_failures.sh
|
||||
scripts/test_wasm_source_locations.sh
|
||||
scripts/test_server_arguments.sh
|
||||
scripts/test_cli_worker_isolation.sh
|
||||
scripts/test_socket_activation.sh
|
||||
fi
|
||||
|
||||
Executable
+153
@@ -0,0 +1,153 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
if [[ "${1:-}" != "--inside" ]]; then
|
||||
exec timeout --signal=TERM --kill-after=5s 120s unshare --mount --fork --kill-child=TERM "$0" --inside
|
||||
fi
|
||||
|
||||
root="/tmp/uce-cli-worker-isolation-$$"
|
||||
site="$root/site"
|
||||
work="$root/work"
|
||||
settings="$root/settings.cfg"
|
||||
log="$root/service.log"
|
||||
cli_socket="$root/run/cli.sock"
|
||||
fastcgi_socket="$root/run/fastcgi.sock"
|
||||
server_pid=""
|
||||
|
||||
cleanup() {
|
||||
status=$?
|
||||
if [[ -n "$server_pid" ]] && kill -0 "$server_pid" 2>/dev/null; then
|
||||
kill -TERM "$server_pid" 2>/dev/null || true
|
||||
deadline=$((SECONDS + 10))
|
||||
while kill -0 "$server_pid" 2>/dev/null && (( SECONDS < deadline )); do sleep 0.05; done
|
||||
if kill -0 "$server_pid" 2>/dev/null; then kill -KILL "$server_pid" 2>/dev/null || true; fi
|
||||
wait "$server_pid" 2>/dev/null || true
|
||||
fi
|
||||
if (( status != 0 )) && [[ -r "$log" ]]; then cat "$log" >&2; fi
|
||||
rm -rf "$root"
|
||||
return "$status"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
mkdir -p "$site" "$work" "$root/run" "$root/session" "$root/upload"
|
||||
cp /etc/uce/settings.cfg "$settings"
|
||||
cat >>"$settings" <<CFG
|
||||
BIN_DIRECTORY=$work
|
||||
PRECOMPILE_FILES_IN=$site
|
||||
SITE_DIRECTORY=$site
|
||||
FCGI_SOCKET_PATH=$fastcgi_socket
|
||||
FCGI_PORT=
|
||||
CLI_SOCKET_PATH=$cli_socket
|
||||
CLI_WORKER_COUNT=2
|
||||
CLI_WORKER_MAX_REQUESTS=3
|
||||
WS_BROKER_SOCKET_PATH=$root/run/ws.sock
|
||||
HTTP_PORT=
|
||||
HTTP_DOCUMENT_ROOT=$site
|
||||
SESSION_PATH=$root/session
|
||||
TMP_UPLOAD_PATH=$root/upload
|
||||
WASM_CORE_PATH=$(pwd)/bin/wasm/core.wasm
|
||||
WORKER_COUNT=2
|
||||
PROACTIVE_COMPILE_ENABLED=0
|
||||
CFG
|
||||
mount --bind "$settings" /etc/uce/settings.cfg
|
||||
|
||||
cat >"$site/isolation.uce" <<'UCE'
|
||||
RENDER(Request& context)
|
||||
{
|
||||
print(request_perf()["worker_pid"].to_string());
|
||||
}
|
||||
|
||||
CLI(Request& context)
|
||||
{
|
||||
print(request_perf()["worker_pid"].to_string());
|
||||
}
|
||||
UCE
|
||||
|
||||
cat >"$site/nested.uce" <<UCE
|
||||
CLI(Request& context)
|
||||
{
|
||||
print(shell_exec("cd $(pwd) && timeout 10s scripts/uce-cli --socket $cli_socket /isolation.uce"));
|
||||
}
|
||||
UCE
|
||||
|
||||
bin/uce_fastcgi.linux.bin >"$log" 2>&1 &
|
||||
server_pid=$!
|
||||
deadline=$((SECONDS + 20))
|
||||
while [[ ! -S "$cli_socket" ]] && (( SECONDS < deadline )); do sleep 0.05; done
|
||||
[[ -S "$cli_socket" ]] || { echo "private UCE CLI socket was not ready" >&2; exit 1; }
|
||||
timeout --signal=TERM --kill-after=1s 30s scripts/uce-cli --socket "$cli_socket" /isolation.uce >/dev/null
|
||||
timeout --signal=TERM --kill-after=1s 30s scripts/uce-cli --socket "$cli_socket" /nested.uce | grep -Eq '^[0-9]+([.]0+)?$'
|
||||
|
||||
fastcgi_request() {
|
||||
SCRIPT_FILENAME="$site/isolation.uce" SCRIPT_NAME=/isolation.uce REQUEST_URI=/isolation.uce REQUEST_METHOD=GET DOCUMENT_ROOT="$site" \
|
||||
timeout --signal=TERM --kill-after=1s 10s cgi-fcgi -bind -connect "$fastcgi_socket" | tr -d '\r' | awk 'NF{last=$0} END{print last}'
|
||||
}
|
||||
fastcgi_request >/dev/null
|
||||
|
||||
cli_request() {
|
||||
local output
|
||||
local deadline=$((SECONDS + 20))
|
||||
while (( SECONDS < deadline )); do
|
||||
if output=$(timeout --signal=TERM --kill-after=1s 10s scripts/uce-cli --socket "$cli_socket" /isolation.uce 2>/dev/null); then
|
||||
printf '%s\n' "$output"
|
||||
return 0
|
||||
fi
|
||||
sleep 0.05
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
cli_pids=()
|
||||
http_pids=()
|
||||
|
||||
# Concurrent completions must count individually and retire both initial CLI
|
||||
# workers despite neither connection pool necessarily becoming idle first.
|
||||
concurrent_jobs=()
|
||||
for n in $(seq 1 12); do
|
||||
cli_request >"$root/concurrent-$n" & concurrent_jobs+=("$!")
|
||||
done
|
||||
for job in "${concurrent_jobs[@]}"; do wait "$job"; done
|
||||
mapfile -t concurrent_cli < <(cat "$root"/concurrent-* | sort -u)
|
||||
cli_pids+=("${concurrent_cli[@]}")
|
||||
|
||||
for _ in $(seq 1 12); do
|
||||
cli_pids+=("$(cli_request)")
|
||||
http_pids+=("$(fastcgi_request)")
|
||||
done
|
||||
mapfile -t unique_cli < <(printf '%s\n' "${cli_pids[@]}" | sort -u)
|
||||
mapfile -t unique_http < <(printf '%s\n' "${http_pids[@]}" | sort -u)
|
||||
[[ ${#unique_cli[@]} -ge 4 ]] || { echo "CLI workers did not recycle after three requests: ${unique_cli[*]}" >&2; exit 1; }
|
||||
[[ ${#unique_http[@]} -ge 1 ]] || { echo "HTTP requests reached no public worker" >&2; exit 1; }
|
||||
for cli_pid in "${unique_cli[@]}"; do
|
||||
if printf '%s\n' "${unique_http[@]}" | grep -Fxq "$cli_pid"; then
|
||||
echo "CLI worker also served public HTTP: $cli_pid" >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
[[ $(grep -c 'wasm FastCGI worker ready' "$log") -eq 2 ]]
|
||||
[[ $(grep -c 'wasm CLI worker ready' "$log") -ge 4 ]]
|
||||
|
||||
echo "CLI worker isolation and recycling passed: CLI ${unique_cli[*]}, public ${unique_http[*]}"
|
||||
|
||||
# A CLI-only runtime must not allocate a listener-less public Wasmtime worker.
|
||||
kill -TERM "$server_pid"
|
||||
wait "$server_pid"
|
||||
server_pid=""
|
||||
rm -f "$cli_socket" "$fastcgi_socket"
|
||||
cat >>"$settings" <<CFG
|
||||
FCGI_SOCKET_PATH=
|
||||
CLI_WORKER_COUNT=1
|
||||
CLI_WORKER_MAX_REQUESTS=0
|
||||
CFG
|
||||
: >"$log"
|
||||
bin/uce_fastcgi.linux.bin >"$log" 2>&1 &
|
||||
server_pid=$!
|
||||
deadline=$((SECONDS + 20))
|
||||
while [[ ! -S "$cli_socket" ]] && (( SECONDS < deadline )); do sleep 0.05; done
|
||||
[[ -S "$cli_socket" ]] || { echo "CLI-only socket was not ready" >&2; exit 1; }
|
||||
cli_request >/dev/null
|
||||
[[ $(grep -c 'wasm CLI worker ready' "$log") -eq 1 ]]
|
||||
[[ $(grep -c 'wasm FastCGI worker ready' "$log") -eq 0 ]]
|
||||
|
||||
echo "CLI-only worker allocation passed"
|
||||
Executable
+55
@@ -0,0 +1,55 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
if [[ "${1:-}" != "--inside" ]]; then
|
||||
exec timeout --signal=TERM --kill-after=5s 90s unshare --mount --fork --kill-child=TERM "$0" --inside
|
||||
fi
|
||||
|
||||
root="/tmp/uce-compiler-lock-directory-$$"
|
||||
site="$root/site"
|
||||
work="$root/missing/parents/work"
|
||||
settings="$root/settings.cfg"
|
||||
log="$root/service.log"
|
||||
server_pid=""
|
||||
|
||||
cleanup() {
|
||||
if [[ -n "$server_pid" ]] && kill -0 "$server_pid" 2>/dev/null; then
|
||||
kill -TERM "$server_pid" 2>/dev/null || true
|
||||
wait "$server_pid" 2>/dev/null || true
|
||||
fi
|
||||
rm -rf "$root"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
mkdir -p "$site" "$root/run" "$root/session" "$root/upload"
|
||||
printf '%s\n' 'CLI(Request& context) { print("recursive-lock-directory-ok"); }' >"$site/test.uce"
|
||||
cp /etc/uce/settings.cfg "$settings"
|
||||
cat >>"$settings" <<CFG
|
||||
BIN_DIRECTORY=$work
|
||||
PRECOMPILE_FILES_IN=$site
|
||||
SITE_DIRECTORY=$site
|
||||
FCGI_SOCKET_PATH=$root/run/fastcgi.sock
|
||||
FCGI_PORT=
|
||||
CLI_SOCKET_PATH=$root/run/cli.sock
|
||||
WS_BROKER_SOCKET_PATH=$root/run/ws.sock
|
||||
HTTP_PORT=
|
||||
HTTP_DOCUMENT_ROOT=$site
|
||||
SESSION_PATH=$root/session
|
||||
TMP_UPLOAD_PATH=$root/upload
|
||||
WASM_CORE_PATH=$(pwd)/bin/wasm/core.wasm
|
||||
WORKER_COUNT=1
|
||||
PROACTIVE_COMPILE_ENABLED=0
|
||||
CFG
|
||||
mount --bind "$settings" /etc/uce/settings.cfg
|
||||
|
||||
bin/uce_fastcgi.linux.bin >"$log" 2>&1 &
|
||||
server_pid=$!
|
||||
deadline=$((SECONDS + 20))
|
||||
while [[ ! -S "$root/run/cli.sock" ]] && (( SECONDS < deadline )); do sleep 0.05; done
|
||||
[[ -S "$root/run/cli.sock" ]] || { cat "$log" >&2; exit 1; }
|
||||
|
||||
rm -rf "$work"
|
||||
response=$(curl -sS --max-time 45 --fail-with-body --unix-socket "$root/run/cli.sock" http://localhost/test.uce) || { cat "$log" >&2; exit 1; }
|
||||
[[ "$response" == *"recursive-lock-directory-ok"* ]] || { printf '%s\n' "$response" >&2; cat "$log" >&2; exit 1; }
|
||||
find "$work" -type f -name '*.lock' -print -quit | grep -q . || { echo "compiler did not recreate a nested lock path" >&2; cat "$log" >&2; exit 1; }
|
||||
echo "compiler recursive lock directory passed"
|
||||
Executable
+176
@@ -0,0 +1,176 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
test_source="/tmp/uce-oauth-es256-native-$$.cpp"
|
||||
test_binary="/tmp/uce-oauth-es256-native-$$"
|
||||
cleanup() { rm -f "$test_source" "$test_binary"; }
|
||||
trap cleanup EXIT
|
||||
|
||||
cat >"$test_source" <<'EOF'
|
||||
#include "src/lib/types.cpp"
|
||||
#include "src/lib/dvalue.cpp"
|
||||
#include "src/lib/functionlib.cpp"
|
||||
#include <openssl/bn.h>
|
||||
#include <openssl/core_names.h>
|
||||
#include <openssl/ecdsa.h>
|
||||
#include <openssl/evp.h>
|
||||
#include <openssl/params.h>
|
||||
#include <memory>
|
||||
|
||||
String base64_encode(String raw)
|
||||
{
|
||||
if(raw.empty()) return("");
|
||||
String out(4 * ((raw.size() + 2) / 3), 0);
|
||||
int size = EVP_EncodeBlock((unsigned char*)out.data(), (const unsigned char*)raw.data(), (int)raw.size());
|
||||
out.resize(size > 0 ? (size_t)size : 0);
|
||||
return(out);
|
||||
}
|
||||
|
||||
String base64_decode(String raw, bool& ok)
|
||||
{
|
||||
ok = false;
|
||||
if(raw.empty() || raw.size() % 4) return("");
|
||||
String out(3 * raw.size() / 4, 0);
|
||||
int size = EVP_DecodeBlock((unsigned char*)out.data(), (const unsigned char*)raw.data(), (int)raw.size());
|
||||
if(size < 0) return("");
|
||||
while(!raw.empty() && raw.back() == '=') { size--; raw.pop_back(); }
|
||||
out.resize((size_t)size); ok = true; return(out);
|
||||
}
|
||||
|
||||
#include "src/lib/hash.cpp"
|
||||
|
||||
static String b64url_encode(String text)
|
||||
{
|
||||
String out=replace(replace(base64_encode(text),"+","-"),"/","_"); while(!out.empty()&&out.back()=='=') out.pop_back(); return out;
|
||||
}
|
||||
|
||||
static String b64url_decode(String text)
|
||||
{
|
||||
text = replace(replace(text, "-", "+"), "_", "/");
|
||||
while(text.size() % 4) text += "=";
|
||||
bool ok = false;
|
||||
String result = base64_decode(text, ok);
|
||||
return(ok ? result : String(""));
|
||||
}
|
||||
|
||||
static bool verify(DValue public_jwk, String jwt)
|
||||
{
|
||||
StringList parts = split(jwt, ".");
|
||||
if(parts.size() != 3) return(false);
|
||||
String x = b64url_decode(public_jwk["x"].to_string());
|
||||
String y = b64url_decode(public_jwk["y"].to_string());
|
||||
String raw = b64url_decode(parts[2]);
|
||||
if(x.size() != 32 || y.size() != 32 || raw.size() != 64) return(false);
|
||||
unsigned char point[65] = {4}; memcpy(point + 1, x.data(), 32); memcpy(point + 33, y.data(), 32);
|
||||
OSSL_PARAM params[] = { OSSL_PARAM_construct_utf8_string(OSSL_PKEY_PARAM_GROUP_NAME, (char*)"prime256v1", 0), OSSL_PARAM_construct_octet_string(OSSL_PKEY_PARAM_PUB_KEY, point, sizeof(point)), OSSL_PARAM_construct_end() };
|
||||
EVP_PKEY_CTX* build = EVP_PKEY_CTX_new_from_name(0, "EC", 0); EVP_PKEY* key = 0;
|
||||
if(!build || EVP_PKEY_fromdata_init(build) <= 0 || EVP_PKEY_fromdata(build, &key, EVP_PKEY_PUBLIC_KEY, params) <= 0) { EVP_PKEY_CTX_free(build); return(false); }
|
||||
EVP_PKEY_CTX_free(build);
|
||||
BIGNUM* r = BN_bin2bn((const unsigned char*)raw.data(), 32, 0); BIGNUM* s = BN_bin2bn((const unsigned char*)raw.data() + 32, 32, 0);
|
||||
ECDSA_SIG* sig = ECDSA_SIG_new(); int der_size = r && s && sig && ECDSA_SIG_set0(sig, r, s) ? i2d_ECDSA_SIG(sig, 0) : 0; r = s = 0;
|
||||
String der(der_size > 0 ? (size_t)der_size : 0, 0); unsigned char* out = (unsigned char*)der.data();
|
||||
bool ok = der_size > 0 && i2d_ECDSA_SIG(sig, &out) == der_size;
|
||||
EVP_MD_CTX* verify_ctx = EVP_MD_CTX_new();
|
||||
String signing_input = parts[0] + "." + parts[1];
|
||||
ok = ok && verify_ctx && EVP_DigestVerifyInit(verify_ctx, 0, EVP_sha256(), 0, key) > 0 && EVP_DigestVerify(verify_ctx, (const unsigned char*)der.data(), der.size(), (const unsigned char*)signing_input.data(), signing_input.size()) == 1;
|
||||
EVP_MD_CTX_free(verify_ctx); ECDSA_SIG_free(sig); EVP_PKEY_free(key); return(ok);
|
||||
}
|
||||
|
||||
int main()
|
||||
{
|
||||
DValue key_request; key_request["operation"] = "key_generate"; key_request["algorithm"] = "ES256";
|
||||
DValue key = crypto_operation_native(key_request);
|
||||
DValue header; header["alg"] = "none"; header["kid"] = key["kid"]; DValue claims; claims["iss"] = "https://client.example";
|
||||
auto sign = [&](DValue private_jwk) { DValue request; request["operation"] = "jwt_sign"; request["algorithm"] = "ES256"; request["private_jwk"] = private_jwk; request["protected_header"] = header; request["claims"] = claims; return(crypto_operation_native(request)); };
|
||||
DValue signed_result = sign(key["private_jwk"]); String jwt = signed_result["jwt"].to_string();
|
||||
DValue wrong_curve = key["private_jwk"]; wrong_curve["crv"] = "P-384";
|
||||
DValue malformed = key["private_jwk"]; malformed["x"] = "bad=";
|
||||
DValue mismatch = key["private_jwk"]; String d = mismatch["d"].to_string(); d[0] = d[0] == 'A' ? 'B' : 'A'; mismatch["d"] = d;
|
||||
const String cose="pQECAyYgASFYIGsX0fLhLEJH-Lzm5WOkQPJ3A32BLeszoPShOUXYmMKWIlggT-NC4v4af5uO5-tKfA-eFivOM1drMV7Oy7ZAaDe_UfU", message="d2ViYXV0aG4gZml4ZWQgbWVzc2FnZQ", signature="MEUCIQCkatZK1VVsjk17uvyzyhjdAkMNWXPjxSOMqWcjmM_8XAIgDaSk3Qufyd0_6r9Dm9A8RQbFco-FdTBulq7bvRGoBC4";
|
||||
auto op = [&](String operation) { DValue r; r["operation"]=operation; r["algorithm"]="ES256"; return r; };
|
||||
auto parse = [&](String key) { DValue r=op("cose_es256_parse"); r["cose_key_base64url"]=key; return crypto_operation_native(r); };
|
||||
auto verify_cose = [&](String key,String msg,String sig) { DValue r=op("es256_verify"); r["cose_key_base64url"]=key; r["message_base64url"]=msg; r["signature_der_base64url"]=sig; return crypto_operation_native(r); };
|
||||
auto decode_cbor = [&](String bytes) { DValue r=op("cbor_decode"); r["cbor_base64url"]=b64url_encode(bytes); return crypto_operation_native(r); };
|
||||
DValue unsupported; unsupported["operation"] = "encrypt"; unsupported["algorithm"] = "ES256";
|
||||
DValue unknown_algorithm; unknown_algorithm["operation"] = "key_generate"; unknown_algorithm["algorithm"] = "none";
|
||||
DValue untyped_algorithm = key_request; untyped_algorithm["algorithm"] = (f64)256;
|
||||
DValue list_header; list_header.set_array(); DValue list_item; list_item = "not-an-object"; list_header.push(list_item); DValue list_request; list_request["operation"] = "jwt_sign"; list_request["algorithm"] = "ES256"; list_request["private_jwk"] = key["private_jwk"]; list_request["protected_header"] = list_header; list_request["claims"] = claims;
|
||||
DValue control_request = list_request; control_request["protected_header"] = header; control_request["claims"] = claims; control_request["claims"]["bad"] = String("control\nbyte");
|
||||
DValue oversized = key_request; oversized["ignored"] = String(33000, 'x');
|
||||
DValue nonfinite = key_request; nonfinite["ignored"] = std::numeric_limits<f64>::quiet_NaN();
|
||||
String tampered = jwt; if(tampered.size() > 2) tampered[tampered.size() - 2] = tampered[tampered.size() - 2] == 'A' ? 'B' : 'A';
|
||||
bool kid_ok = key["ok"].to_bool() && key["kid"].to_string() == key["thumbprint"].to_string();
|
||||
bool signed_ok = signed_result["ok"].to_bool() && jwt != "" && verify(key["public_jwk"], jwt);
|
||||
bool tamper_ok = !verify(key["public_jwk"], tampered);
|
||||
auto cbor_error = [&](String bytes) { return(decode_cbor(bytes)["error"].to_string() == "invalid_cbor"); };
|
||||
auto cose_error = [&](String encoded) { return(parse(encoded)["error"].to_string() == "invalid_cose_key"); };
|
||||
String cose_raw = b64url_decode(cose);
|
||||
String wrong_kty = cose_raw; wrong_kty[2] = 1;
|
||||
String wrong_alg = cose_raw; wrong_alg[4] = 0x27;
|
||||
String wrong_cose_curve = cose_raw; wrong_cose_curve[6] = 2;
|
||||
String missing_label = cose_raw; missing_label[0] = 0xa4; missing_label.resize(missing_label.size() - 35);
|
||||
String short_x = String("\xa5\x01\x02\x03\x26\x20\x01\x21\x58\x1f",10) + String(31, 'x') + String("\x22\x58\x20", 3) + String(32, 'y');
|
||||
String duplicate_label = cose_raw; duplicate_label[0] = 0xa6; duplicate_label += String("\x01\x02", 2);
|
||||
String invalid_point = String("\xa5\x01\x02\x03\x26\x20\x01\x21\x58\x20", 10) + String(32, 'x') + String("\x22\x58\x20", 3) + String(32, 'y');
|
||||
String der = b64url_decode(signature);
|
||||
String tampered_der = der; tampered_der[10] ^= 1;
|
||||
String trailing_der = der + String("\x00", 1);
|
||||
String noncanonical_der = der; noncanonical_der[1]++; noncanonical_der[3]++; noncanonical_der.insert(4, 1, '\0');
|
||||
DValue valid_verify = verify_cose(cose, message, signature);
|
||||
DValue tampered_signature = verify_cose(cose, message, b64url_encode(tampered_der));
|
||||
DValue tampered_message = verify_cose(cose, b64url_encode("tampered"), signature);
|
||||
DValue malformed_der = verify_cose(cose, message, b64url_encode(String("\x30\x00", 2)));
|
||||
DValue noncanonical_der_result = verify_cose(cose, message, b64url_encode(noncanonical_der));
|
||||
DValue trailing_der_result = verify_cose(cose, message, b64url_encode(trailing_der));
|
||||
DValue invalid_point_result = verify_cose(b64url_encode(invalid_point), message, signature);
|
||||
String large_cbor = String("\x58\x81", 2) + String(129, 'x');
|
||||
String oversized_cbor = String("\x5a\x00\x00\x40\x00", 5) + String(16384, 'x');
|
||||
String duplicate_compound = String("\xa2\x82\x41[\x41]\x01\x82\x41[\x41]\x02", 13);
|
||||
String distinct_compound = String("\xa2\x82\x41[\x41]\x01\x82\x41[\x41[\x02", 13);
|
||||
String node_overflow = String("\x99\x01\x00", 3) + String(256, 0);
|
||||
String decoded;
|
||||
bool cbor_valid = decode_cbor(String("\x82\x01\x62ok", 5))["ok"].to_bool();
|
||||
bool cbor_large = decode_cbor(large_cbor)["ok"].to_bool();
|
||||
bool cbor_control_text = decode_cbor(String("\x61\x01", 2))["ok"].to_bool();
|
||||
bool cbor_duplicate = cbor_error(String("\xa2\x01\x02\x01\x03", 5));
|
||||
bool cbor_compound_duplicate = cbor_error(duplicate_compound);
|
||||
bool cbor_compound_distinct = decode_cbor(distinct_compound)["ok"].to_bool();
|
||||
bool cbor_invalid_utf8 = cbor_error(String("\x61\x80", 2));
|
||||
bool cbor_truncated = cbor_error(String("\xa1", 1));
|
||||
bool cbor_trailing = cbor_error(String("\x01\x02", 2));
|
||||
bool cbor_depth = cbor_error(String(17, '\x81') + "\x00");
|
||||
bool cbor_nodes = cbor_error(node_overflow);
|
||||
bool cbor_size = cbor_error(oversized_cbor);
|
||||
bool cbor_indefinite = cbor_error(String("\x9f\x01\xff", 3));
|
||||
bool cbor_integer_overflow = cbor_error(String("\x5b\xff\xff\xff\xff\xff\xff\xff\xff", 9));
|
||||
bool cbor_nonminimal = cbor_error(String("\x18\x17", 2));
|
||||
bool b64_padding = !uce_base64url_decode(cose + "=", decoded, UCE_CBOR_MAX_BASE64URL);
|
||||
bool b64_truncated = !uce_base64url_decode("A", decoded, UCE_CBOR_MAX_BASE64URL);
|
||||
bool b64_trailing_bits = !uce_base64url_decode("AB", decoded, UCE_CBOR_MAX_BASE64URL);
|
||||
bool cose_valid = parse(cose)["ok"].to_bool();
|
||||
bool cose_wrong_kty = cose_error(b64url_encode(wrong_kty));
|
||||
bool cose_wrong_alg = cose_error(b64url_encode(wrong_alg));
|
||||
bool cose_wrong_curve = cose_error(b64url_encode(wrong_cose_curve));
|
||||
bool cose_short_coordinate = cose_error(b64url_encode(short_x));
|
||||
bool cose_missing_label = cose_error(b64url_encode(missing_label));
|
||||
bool cose_duplicate_label = cose_error(b64url_encode(duplicate_label));
|
||||
bool cose_invalid_point = invalid_point_result["error"].to_string() == "invalid_key_or_payload";
|
||||
bool verify_valid = valid_verify["ok"].to_bool() && valid_verify["valid"].to_bool();
|
||||
bool verify_tampered_signature = tampered_signature["ok"].to_bool() && !tampered_signature["valid"].to_bool();
|
||||
bool verify_tampered_message = tampered_message["ok"].to_bool() && !tampered_message["valid"].to_bool();
|
||||
bool verify_malformed_der = malformed_der["error"].to_string() == "invalid_signature";
|
||||
bool verify_noncanonical_der = noncanonical_der_result["error"].to_string() == "invalid_signature";
|
||||
bool verify_trailing_der = trailing_der_result["error"].to_string() == "invalid_signature";
|
||||
bool cbor_ok = cbor_valid && cbor_large && cbor_control_text && cbor_duplicate && cbor_compound_duplicate && cbor_compound_distinct && cbor_invalid_utf8 && cbor_truncated && cbor_trailing && cbor_depth && cbor_nodes && cbor_size && cbor_indefinite && cbor_integer_overflow && cbor_nonminimal;
|
||||
bool cose_ok = cose_valid && cose_wrong_kty && cose_wrong_alg && cose_wrong_curve && cose_short_coordinate && cose_missing_label && cose_duplicate_label && cose_invalid_point;
|
||||
bool cose_negative = verify_valid && verify_tampered_signature && verify_tampered_message && verify_malformed_der && verify_noncanonical_der && verify_trailing_der;
|
||||
bool b64_ok = b64_padding && b64_truncated && b64_trailing_bits;
|
||||
bool negatives_ok = !sign(wrong_curve)["ok"].to_bool() && !sign(malformed)["ok"].to_bool() && !sign(mismatch)["ok"].to_bool() && crypto_operation_native(unsupported)["error"].to_string() == "unsupported_operation" && crypto_operation_native(unknown_algorithm)["error"].to_string() == "unsupported_algorithm" && crypto_operation_native(untyped_algorithm)["error"].to_string() == "invalid_request" && crypto_operation_native(list_request)["error"].to_string() == "invalid_key_or_payload" && crypto_operation_native(control_request)["error"].to_string() == "invalid_request" && crypto_operation_native(oversized)["error"].to_string() == "invalid_request" && crypto_operation_native(nonfinite)["error"].to_string() == "invalid_request";
|
||||
if(!(kid_ok && signed_ok && tamper_ok && negatives_ok && cose_ok && cbor_ok && cose_negative && b64_ok)) std::cerr << "kid=" << kid_ok << " signed=" << signed_ok << " tamper=" << tamper_ok << " negatives=" << negatives_ok << " cbor=" << cbor_ok << " cose=" << cose_ok << " verify=" << cose_negative << " b64=" << b64_ok << " large=" << cbor_large << " compound=" << cbor_compound_duplicate << "/" << cbor_compound_distinct << " der=" << verify_malformed_der << "/" << verify_noncanonical_der << "/" << verify_trailing_der << " cborparts=" << cbor_valid << cbor_control_text << cbor_duplicate << cbor_invalid_utf8 << cbor_truncated << cbor_trailing << cbor_depth << cbor_nodes << cbor_size << cbor_indefinite << cbor_integer_overflow << cbor_nonminimal << "\\n";
|
||||
return(kid_ok && signed_ok && tamper_ok && negatives_ok && cose_ok && cbor_ok && cose_negative && b64_ok ? 0 : 1);
|
||||
}
|
||||
EOF
|
||||
"${CXX:-c++}" -std=c++20 -fpermissive -I. "$test_source" -lpcre2-8 -lcrypto -o "$test_binary"
|
||||
"$test_binary"
|
||||
echo "native structured crypto operation passed"
|
||||
@@ -450,14 +450,23 @@ fi
|
||||
|
||||
worker_count=$(awk -F= '/^[[:space:]]*WORKER_COUNT[[:space:]]*=/ {gsub(/^[[:space:]]+|[[:space:]]+$/, "", $2); print $2; exit}' /etc/uce/settings.cfg 2>/dev/null || true)
|
||||
worker_count="${worker_count:-4}"
|
||||
cli_worker_count=$(awk -F= '/^[[:space:]]*CLI_WORKER_COUNT[[:space:]]*=/ {gsub(/^[[:space:]]+|[[:space:]]+$/, "", $2); value=$2} END{print value}' /etc/uce/settings.cfg 2>/dev/null || true)
|
||||
cli_worker_count="${cli_worker_count:-0}"
|
||||
cli_worker_max_requests=$(awk -F= '/^[[:space:]]*CLI_WORKER_MAX_REQUESTS[[:space:]]*=/ {gsub(/^[[:space:]]+|[[:space:]]+$/, "", $2); value=$2} END{print value}' /etc/uce/settings.cfg 2>/dev/null || true)
|
||||
cli_worker_max_requests="${cli_worker_max_requests:-8}"
|
||||
worker_pids=""
|
||||
for _ in {1..48}; do
|
||||
output=$(scripts/uce-cli "/$test_name/parent.uce")
|
||||
worker_pids+="${output##*:}"$'\n'
|
||||
done
|
||||
unique_workers=$(printf '%s' "$worker_pids" | sed '/^$/d' | sort -u | wc -l)
|
||||
if (( unique_workers > worker_count )); then
|
||||
echo "worker pool recycled during 48 requests: $unique_workers PIDs for $worker_count workers" >&2
|
||||
if (( cli_worker_count > 0 && cli_worker_max_requests > 0 )); then
|
||||
if (( unique_workers <= cli_worker_count )); then
|
||||
echo "dedicated CLI pool did not recycle during 48 requests: $unique_workers PIDs for $cli_worker_count workers" >&2
|
||||
exit 1
|
||||
fi
|
||||
elif (( unique_workers > worker_count )); then
|
||||
echo "shared worker pool recycled during 48 requests: $unique_workers PIDs for $worker_count workers" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
#include "src/lib/types.cpp"
|
||||
#include "src/lib/dvalue.cpp"
|
||||
#include "src/lib/functionlib.cpp"
|
||||
#include "src/wasm/hardened_http_internal.h"
|
||||
#include <iostream>
|
||||
struct Fake { std::vector<String> answers, argv, env; String input; HardenedHttpExecResult result; };
|
||||
static DValue request() { DValue r; r["method"]="POST"; r["url"]="https://api.example.test/token"; r["headers"]["Accept"]="application/json"; r["body"]="client_secret=SECRET"; DValue& s=r["security"]; s["https_only"].set_bool(true); s["public_dns_only"].set_bool(true); s["pin_dns"].set_bool(true); s["isolated_curl"].set_bool(true); s["no_redirects"].set_bool(true); return r; }
|
||||
static DValue run(Fake& f,DValue r) { HardenedHttpHooks h; h.resolve=[&](String){return f.answers;}; h.execute=[&](std::vector<String> a,String i,std::vector<String> e,u64,size_t){f.argv=a;f.input=i;f.env=e;return f.result;}; return hardened_http_request_internal(r,9000,h); }
|
||||
static String headers(int status=200) { return "HTTP/1.1 "+std::to_string(status)+" OK\r\nContent-Type: application/json\r\n\r\n"; }
|
||||
static bool exists(String path) { return access(path.c_str(),F_OK)==0; }
|
||||
static void wait_for(String path) { for(int n=0;n<500&&!exists(path);n++) usleep(1000); }
|
||||
static bool process_live(pid_t pid) { char state=0; String path="/proc/"+std::to_string((long long)pid)+"/stat"; int fd=open(path.c_str(),O_RDONLY); if(fd<0) return false; char text[256]{}; ssize_t n=read(fd,text,sizeof(text)-1); close(fd); if(n<=0) return false; char* close_paren=strrchr(text,')'); return close_paren&&close_paren[2]!='Z'; }
|
||||
int main(int argc,char** argv) {
|
||||
if(argc==3&&String(argv[1])=="--child-timeout") { pid_t child=fork(); if(child==0) { usleep(100000); int fd=open(argv[2],O_WRONLY|O_CREAT|O_TRUNC,0600); if(fd>=0) { write(fd,"leaked",6); close(fd); } _exit(0); } for(;;) pause(); }
|
||||
if(argc==2&&String(argv[1])=="--child-output") { String x(4096,'x'); for(int n=0;n<64;n++) write(1,x.data(),x.size()); return 0; }
|
||||
if(argc==3&&String(argv[1])=="--child-fds") { int inherited=atoi(argv[2]); bool closed=fcntl(inherited,F_GETFD)==-1&&errno==EBADF; bool header=fcntl(3,F_GETFD)!=-1; write(1,closed&&header?"closed":"open",closed&&header?6:4); return closed&&header?0:1; }
|
||||
if(argc==5&&String(argv[1])=="--child-job") { int ready=open(argv[3],O_WRONLY|O_CREAT|O_TRUNC,0600); if(ready>=0) { write(ready,"ready",5); close(ready); } pid_t child=fork(); if(child==0) { int pidfile=open(argv[4],O_WRONLY|O_CREAT|O_TRUNC,0600); if(pidfile>=0) { String pid=std::to_string((long long)getpid()); write(pidfile,pid.data(),pid.size()); close(pidfile); } usleep(150000); int marker=open(argv[2],O_WRONLY|O_CREAT|O_TRUNC,0600); if(marker>=0) { write(marker,"leaked",6); close(marker); } _exit(0); } for(;;) pause(); }
|
||||
if(argc==3&&String(argv[1])=="--child-async-timeout") { setsid(); HardenedHttpExecResult result=hardened_http_exec_argv_capture({"/bin/sleep","1"},"",20,4096,false,false); if(result.timed_out&&result.exit_code==137) { int fd=open(argv[2],O_WRONLY|O_CREAT|O_TRUNC,0600); if(fd>=0) { write(fd,"typed",5); close(fd); } } return 0; }
|
||||
if(argc==3&&String(argv[1])=="--child-async-output") { setsid(); HardenedHttpExecResult result=hardened_http_exec_argv_capture({"/proc/self/exe","--child-output"},"",1000,1024,false,false); if(result.output_limited&&result.exit_code==137) { int fd=open(argv[2],O_WRONLY|O_CREAT|O_TRUNC,0600); if(fd>=0) { write(fd,"typed",5); close(fd); } } return 0; }
|
||||
bool ok=true; auto need=[&](bool x,const char* n){if(!x){std::cerr<<n<<"\n";ok=false;}};
|
||||
Fake f; f.answers={"8.8.8.8"}; f.result.exit_code=0; f.result.stderr_text="SECRET"; f.result.headers_text=headers(); f.result.body_text="one\r\n\r\ntwo"; DValue out=run(f,request());
|
||||
need(out["error"].to_string()==""&&out["error"].to_string().find("SECRET")==String::npos&&out["body"].to_string()=="one\r\n\r\ntwo","body framing / non-secret errors"); need(f.argv.size()>2&&f.argv[0]=="/usr/bin/curl"&&f.argv[1]=="--disable","absolute curl disable"); need(f.env.size()==1&&f.env[0]=="PATH=/usr/bin:/bin","clean env"); need(std::find(f.argv.begin(),f.argv.end(),"SECRET")==f.argv.end()&&f.input.find("SECRET")!=String::npos,"secret stdin only"); need(std::find(f.argv.begin(),f.argv.end(),"--resolve")!=f.argv.end(),"pinned dns");
|
||||
Fake interim; interim.answers={"8.8.8.8"}; interim.result.exit_code=0; interim.result.headers_text="HTTP/1.1 100 Continue\r\n\r\n"+headers(201); interim.result.body_text="{}"; need(run(interim,request())["status"].to_u64()==201,"interim headers");
|
||||
for(String bad:{"127.0.0.1","10.0.0.1","169.254.1.1","192.168.1.1","::1","2001:db8::1"}) { Fake x; x.answers={"8.8.8.8",bad}; need(run(x,request())["error"].to_string()=="unsafe_dns","dns matrix"); }
|
||||
for(auto pair:std::vector<std::pair<int,String>>{{302,headers(302)},{500,headers(500)},{200,"bad\r\n\r\n"}}) { Fake x; x.answers={"8.8.8.8"};x.result.exit_code=0;x.result.headers_text=pair.second;need(run(x,request())["error"].to_string()==(pair.first==302?"redirect_not_allowed":pair.first==500?"http_status":"malformed_output"),"status errors"); }
|
||||
DValue bad=request(); bad["method"]="TRACE"; Fake x; need(run(x,bad)["error"].to_string()=="invalid_request"&&x.argv.empty(),"method validation"); bad=request();bad["headers"]["Host"]="evil";need(run(x,bad)["error"].to_string()=="invalid_request","header validation"); bad=request();bad["headers"]["Accept"]="ok\r\nInjected: x";need(run(x,bad)["error"].to_string()=="invalid_request","header CRLF validation");bad=request();bad["url"]="https://127.0.0.1/";need(run(x,bad)["error"].to_string()=="invalid_request","url validation");bad=request();bad["follow_redirects"].set_bool(true);need(run(x,bad)["error"].to_string()=="invalid_request","redirect composition");
|
||||
for(String key:{"https_only","public_dns_only","pin_dns","isolated_curl","no_redirects"}) { bad=request(); bad["security"].remove(key); need(run(x,bad)["error"].to_string()=="invalid_request","partial hardening fails closed"); bad=request(); bad["security"][key].set_bool(false); need(run(x,bad)["error"].to_string()=="invalid_request","false hardening fails closed"); bad=request(); bad["security"][key]="true"; need(run(x,bad)["error"].to_string()=="invalid_request","non-boolean hardening fails closed"); }
|
||||
DValue legacy=request(); legacy["security"].clear(); legacy["security"]["unrelated"]="value"; need(!hardened_http_security_requested(legacy.key("security")),"unknown security object remains legacy");
|
||||
Fake large;large.answers={"8.8.8.8"};large.result.exit_code=0;large.result.headers_text=headers();large.result.body_text=String(65537,'x');need(run(large,request())["error"].to_string()=="response_too_large","body cap");
|
||||
String base="/tmp/hardened-http-"+std::to_string((long long)getpid()), marker=base+"-marker"; unlink(marker.c_str()); HardenedHttpExecResult timeout=hardened_http_exec_argv_capture({"/proc/self/exe","--child-timeout",marker},"",30,4096,false); usleep(150000); need(timeout.timed_out&&timeout.exit_code==137&&!exists(marker),"timeout kills descendants"); unlink(marker.c_str());
|
||||
HardenedHttpExecResult overflow=hardened_http_exec_argv_capture({"/proc/self/exe","--child-output"},"",1000,1024,false);need(overflow.output_limited&&overflow.body_text.size()<=1024,"output cap/reap");
|
||||
int inherited=open("/dev/null",O_RDONLY), high_inherited=fcntl(inherited,F_DUPFD,10); close(inherited); HardenedHttpExecResult fds=hardened_http_exec_argv_capture({"/proc/self/exe","--child-fds",std::to_string(high_inherited)},"",1000,4096,false); close(high_inherited); need(fds.exit_code==0&&fds.body_text=="closed","only stdio and header fd inherited");
|
||||
String ready=base+"-ready", descendant=base+"-descendant", sentinel=base+"-sentinel"; unlink(marker.c_str()); unlink(ready.c_str()); unlink(descendant.c_str()); unlink(sentinel.c_str()); pid_t worker=fork(); if(worker==0) { setsid(); hardened_http_exec_argv_capture({"/proc/self/exe","--child-job",marker,ready,descendant},"",5000,4096,false,false); _exit(0); } wait_for(ready); wait_for(descendant); pid_t unrelated=fork(); if(unrelated==0) { setsid(); usleep(150000); int fd=open(sentinel.c_str(),O_WRONLY|O_CREAT|O_TRUNC,0600); if(fd>=0) { write(fd,"alive",5); close(fd); } _exit(0); } pid_t descendant_pid=exists(descendant)?(pid_t)strtol([](String path){ int fd=open(path.c_str(),O_RDONLY); char text[32]{}; ssize_t n=fd<0?-1:read(fd,text,sizeof(text)-1); if(fd>=0) close(fd); return String(text,n>0?(size_t)n:0); }(descendant).c_str(),0,10):0; need(exists(ready)&&descendant_pid>0,"async worker and descendant started"); kill(-worker,SIGKILL); waitpid(worker,0,0); usleep(250000); waitpid(unrelated,0,0); need(!process_live(descendant_pid)&&!exists(marker)&&exists(sentinel),"async cancellation leaves no live descendant and kills only its worker group"); unlink(marker.c_str()); unlink(ready.c_str()); unlink(descendant.c_str()); unlink(sentinel.c_str());
|
||||
String async_timeout=base+"-async-timeout"; unlink(async_timeout.c_str()); pid_t timeout_worker=fork(); if(timeout_worker==0) { execl("/proc/self/exe","test_hardened_http_native","--child-async-timeout",async_timeout.c_str(),(char*)0); _exit(127); } int timeout_status=0; waitpid(timeout_worker,&timeout_status,0); need(WIFEXITED(timeout_status)&&WEXITSTATUS(timeout_status)==0&&exists(async_timeout),"async timeout records a typed terminal result before worker exit"); unlink(async_timeout.c_str());
|
||||
String async_output=base+"-async-output"; unlink(async_output.c_str()); pid_t output_worker=fork(); if(output_worker==0) { execl("/proc/self/exe","test_hardened_http_native","--child-async-output",async_output.c_str(),(char*)0); _exit(127); } int output_status=0; waitpid(output_worker,&output_status,0); need(WIFEXITED(output_status)&&WEXITSTATUS(output_status)==0&&exists(async_output),"async output cap records a typed terminal result before worker exit"); unlink(async_output.c_str());
|
||||
return ok?0:1;
|
||||
}
|
||||
Executable
+8
@@ -0,0 +1,8 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
repo=$(cd "$(dirname "$0")/.." && pwd)
|
||||
bin="${TMPDIR:-/tmp}/test_hardened_http_native.$$"
|
||||
trap 'rm -f "$bin"' EXIT
|
||||
compiler=$(command -v clang++ || command -v g++)
|
||||
"$compiler" -std=c++20 -fpermissive -I"$repo" "$repo/scripts/test_hardened_http_native.cpp" -lpcre2-8 -o "$bin"
|
||||
timeout --signal=TERM --kill-after=2s 15s "$bin"
|
||||
Executable
+145
@@ -0,0 +1,145 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
if [[ "${1:-}" != "--inside" ]]; then
|
||||
exec timeout --signal=TERM --kill-after=5s 150s unshare --mount --fork --kill-child=TERM "$0" --inside
|
||||
fi
|
||||
|
||||
name="mysql-pool-idle-test-$$"
|
||||
root="/tmp/$name"
|
||||
site="$root/site"
|
||||
work="$root/work"
|
||||
settings="$root/settings.cfg"
|
||||
log="$root/service.log"
|
||||
socket="$root/run/cli.sock"
|
||||
test_user="uce_pool_idle_$$"
|
||||
test_database="uce_pool_idle_$$"
|
||||
test_password=$(printf '%s' "$name-$(date +%s%N)" | sha256sum | cut -c1-32)
|
||||
server_pid=""
|
||||
|
||||
cleanup() {
|
||||
status=$?
|
||||
if [[ -n "$server_pid" ]] && kill -0 "$server_pid" 2>/dev/null; then
|
||||
kill -TERM "$server_pid" 2>/dev/null || true
|
||||
deadline=$((SECONDS + 10))
|
||||
while kill -0 "$server_pid" 2>/dev/null && (( SECONDS < deadline )); do sleep 0.05; done
|
||||
if kill -0 "$server_pid" 2>/dev/null; then kill -KILL "$server_pid" 2>/dev/null || true; fi
|
||||
wait "$server_pid" 2>/dev/null || true
|
||||
fi
|
||||
mariadb -e "DROP DATABASE IF EXISTS \`$test_database\`; DROP USER IF EXISTS '$test_user'@'127.0.0.1'" >/dev/null 2>&1 || true
|
||||
if (( status != 0 )) && [[ -r "$log" ]]; then cat "$log" >&2; fi
|
||||
rm -rf "$root"
|
||||
return "$status"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
mkdir -p "$site" "$work" "$root/run" "$root/session" "$root/upload"
|
||||
cp /etc/uce/settings.cfg "$settings"
|
||||
cat >>"$settings" <<CFG
|
||||
BIN_DIRECTORY=$work
|
||||
PRECOMPILE_FILES_IN=$site
|
||||
SITE_DIRECTORY=$site
|
||||
FCGI_SOCKET_PATH=$root/run/fastcgi.sock
|
||||
FCGI_PORT=
|
||||
CLI_SOCKET_PATH=$socket
|
||||
CLI_WORKER_COUNT=1
|
||||
CLI_WORKER_MAX_REQUESTS=0
|
||||
WS_BROKER_SOCKET_PATH=$root/run/ws.sock
|
||||
HTTP_PORT=
|
||||
HTTP_DOCUMENT_ROOT=$site
|
||||
SESSION_PATH=$root/session
|
||||
TMP_UPLOAD_PATH=$root/upload
|
||||
WASM_CORE_PATH=$(pwd)/bin/wasm/core.wasm
|
||||
WORKER_COUNT=1
|
||||
PROACTIVE_COMPILE_ENABLED=0
|
||||
MYSQL_PERSISTENT_POOL_SIZE=8
|
||||
MYSQL_PERSISTENT_POOL_IDLE_TIMEOUT_SECONDS=2
|
||||
CFG
|
||||
mount --bind "$settings" /etc/uce/settings.cfg
|
||||
|
||||
mariadb -e "DROP DATABASE IF EXISTS \`$test_database\`; DROP USER IF EXISTS '$test_user'@'127.0.0.1'; CREATE DATABASE \`$test_database\`; CREATE USER '$test_user'@'127.0.0.1' IDENTIFIED BY '$test_password'; GRANT ALL ON \`$test_database\`.* TO '$test_user'@'127.0.0.1'"
|
||||
|
||||
cat >"$site/database.uce" <<UCE
|
||||
CLI(Request& context)
|
||||
{
|
||||
MySQL* db = mysql_connect("127.0.0.1", "$test_user", "$test_password", "$test_database");
|
||||
if(!mysql_connected(db)) { print("connect-failed"); return; }
|
||||
String connection_id;
|
||||
mysql_query(db, "SELECT CONNECTION_ID() AS id").each([&](DValue row, String key) { connection_id = row["id"].to_string(); });
|
||||
String source;
|
||||
DValue perf = request_perf();
|
||||
perf["mysql_operations"].each([&](DValue operation, String key) {
|
||||
if(operation["op"].to_string() == "connect") source = operation["source"].to_string();
|
||||
});
|
||||
print(perf["worker_pid"].to_string(), "|", connection_id, "|", source);
|
||||
mysql_disconnect(db);
|
||||
}
|
||||
UCE
|
||||
printf '%s\n' 'CLI(Request& context) { print(request_perf()["worker_pid"].to_string(), "|plain"); }' >"$site/plain.uce"
|
||||
|
||||
timeout --signal=TERM --kill-after=5s 120s bin/uce_fastcgi.linux.bin >"$log" 2>&1 &
|
||||
server_pid=$!
|
||||
deadline=$((SECONDS + 20))
|
||||
while [[ ! -S "$socket" ]] && (( SECONDS < deadline )); do sleep 0.05; done
|
||||
[[ -S "$socket" ]] || { echo "private UCE CLI socket was not ready" >&2; exit 1; }
|
||||
|
||||
request() {
|
||||
timeout --signal=TERM --kill-after=1s 30s scripts/uce-cli --socket "$socket" "$1"
|
||||
}
|
||||
|
||||
first=$(request /database.uce)
|
||||
second=$(request /database.uce)
|
||||
IFS='|' read -r first_pid first_id first_source <<<"$first"
|
||||
IFS='|' read -r second_pid second_id second_source <<<"$second"
|
||||
[[ "$first_source" == "new" && "$second_source" == "worker" ]]
|
||||
[[ "$first_pid" == "$second_pid" && "$first_id" == "$second_id" ]]
|
||||
|
||||
sleep 3
|
||||
plain=$(request /plain.uce)
|
||||
[[ "$plain" == "$first_pid|plain" ]]
|
||||
deadline=$((SECONDS + 5))
|
||||
while (( SECONDS < deadline )); do
|
||||
connections=$(mariadb --batch --skip-column-names -e "SELECT COUNT(*) FROM information_schema.PROCESSLIST WHERE USER='$test_user'")
|
||||
[[ "$connections" == "0" ]] && break
|
||||
sleep 0.05
|
||||
done
|
||||
[[ "$connections" == "0" ]] || { echo "expired pooled connection remained after a request boundary" >&2; exit 1; }
|
||||
|
||||
third=$(request /database.uce)
|
||||
IFS='|' read -r third_pid third_id third_source <<<"$third"
|
||||
[[ "$third_pid" == "$first_pid" && "$third_source" == "new" && "$third_id" != "$first_id" ]]
|
||||
|
||||
sleep 3
|
||||
plain=$(request /plain.uce)
|
||||
[[ "$plain" == "$first_pid|plain" ]]
|
||||
deadline=$((SECONDS + 5))
|
||||
while (( SECONDS < deadline )); do
|
||||
connections=$(mariadb --batch --skip-column-names -e "SELECT COUNT(*) FROM information_schema.PROCESSLIST WHERE USER='$test_user'")
|
||||
[[ "$connections" == "0" ]] && break
|
||||
sleep 0.05
|
||||
done
|
||||
[[ "$connections" == "0" ]] || { echo "final pooled fixture connection remained after idle eviction" >&2; exit 1; }
|
||||
|
||||
retained=$(request /database.uce)
|
||||
IFS='|' read -r retained_pid retained_id retained_source <<<"$retained"
|
||||
[[ "$retained_pid" == "$first_pid" && "$retained_source" == "new" ]]
|
||||
connections=$(mariadb --batch --skip-column-names -e "SELECT COUNT(*) FROM information_schema.PROCESSLIST WHERE USER='$test_user'")
|
||||
[[ "$connections" == "1" ]] || { echo "shutdown fixture connection was not retained in the worker pool" >&2; exit 1; }
|
||||
|
||||
journal_cursor=$(journalctl -u mariadb.service -n 0 --show-cursor --no-pager | sed -n 's/^-- cursor: //p')
|
||||
[[ -n "$journal_cursor" ]] || { echo "could not capture MariaDB journal cursor" >&2; exit 1; }
|
||||
kill -TERM "$server_pid"
|
||||
deadline=$((SECONDS + 10))
|
||||
while kill -0 "$server_pid" 2>/dev/null && (( SECONDS < deadline )); do sleep 0.05; done
|
||||
[[ ! -e "/proc/$server_pid" ]] || { echo "private UCE server did not stop cleanly" >&2; exit 1; }
|
||||
wait "$server_pid" 2>/dev/null || true
|
||||
server_pid=""
|
||||
sleep 1
|
||||
shutdown_log=$(journalctl -u mariadb.service --after-cursor "$journal_cursor" --no-pager)
|
||||
if grep -F "$test_user" <<<"$shutdown_log" | grep -Fq "Aborted connection"; then
|
||||
echo "clean UCE shutdown aborted a retained MySQL connection" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Persistent MySQL idle eviction preserved hot reuse, retired expired connections, and closed a retained connection on clean shutdown"
|
||||
@@ -134,6 +134,15 @@ printf '0\n' >"$root/maximum"
|
||||
flock -u 7
|
||||
mapfile -t scanner_pids < <(awk -F '\t' -v site="$site/" '$3 ~ ("^" site "unit-[0-9]+[.]uce$") { count[$4]++ } END { for(pid in count) if(count[pid] >= 2) print pid }' "$shim_log" | sort -n)
|
||||
[[ "${#scanner_pids[@]}" -eq 2 ]] || { echo "controlled units were not split between both scanners" >&2; cat "$shim_log" >&2; exit 1; }
|
||||
assert_scanner_memory_bounded() {
|
||||
for scanner_pid in "${scanner_pids[@]}"; do
|
||||
retained_kb=$(awk '/^(VmRSS|VmSwap):/{total += $2} END{print total + 0}' "/proc/$scanner_pid/status")
|
||||
# Scanner bookkeeping is small; 64 MiB leaves ample headroom while still
|
||||
# catching the measured old 12-serialization retention (~73 MiB).
|
||||
(( retained_kb < 65536 )) || { echo "proactive scanner retained serialized-module arenas: pid=$scanner_pid retained_kb=$retained_kb" >&2; exit 1; }
|
||||
done
|
||||
}
|
||||
assert_scanner_memory_bounded
|
||||
blockers=()
|
||||
victims=()
|
||||
for scanner_pid in "${scanner_pids[@]}"; do
|
||||
@@ -202,4 +211,8 @@ while [[ ! -s "$artifacts/priority.uce.cwasm" ]] && (( SECONDS < deadline )); do
|
||||
priority_nice=$(awk -F '\t' -v path="$site/priority.uce" '$3 == path { print $2; exit }' "$shim_log")
|
||||
[[ "$priority_nice" == "5" ]] || { echo "priority queue was not owned by the nice-5 compiler: $priority_nice" >&2; cat "$shim_log" >&2; exit 1; }
|
||||
|
||||
# Recheck after dependency fanout, failure/retry, recovery, and priority work;
|
||||
# the historical leak accumulated in the long-lived scanners across phases.
|
||||
assert_scanner_memory_bounded
|
||||
|
||||
printf '%s\n' 'parallel proactive compile passed'
|
||||
|
||||
@@ -38,7 +38,8 @@ socket_inode=$(stat -c %i "$socket_path")
|
||||
invoke() {
|
||||
local output="$1"
|
||||
shift
|
||||
timeout --signal=TERM --kill-after=1s 2s unshare --mount --fork \
|
||||
local timeout_seconds="${INVOKE_TIMEOUT_SECONDS:-2}"
|
||||
timeout --signal=TERM --kill-after=1s "$timeout_seconds" unshare --mount --fork \
|
||||
bash -c 'mount --bind "$1" /etc/uce/settings.cfg; exec "$2" "${@:3}"' \
|
||||
_ "$cfg" "$binary" "$@" >"$output.stdout" 2>"$output.stderr"
|
||||
}
|
||||
@@ -47,10 +48,11 @@ for option in --help -h; do
|
||||
invoke "$root/help" "$option"
|
||||
grep -q '^Usage: uce_fastcgi' "$root/help.stdout"
|
||||
grep -q -- '--precompile' "$root/help.stdout"
|
||||
grep -q -- '--serialize-module' "$root/help.stdout"
|
||||
[[ ! -s "$root/help.stderr" ]]
|
||||
done
|
||||
|
||||
for arguments in '--unknown' '--precompile extra' '--help extra'; do
|
||||
for arguments in '--unknown' '--precompile extra' '--serialize-module' '--help extra'; do
|
||||
read -r -a argv <<<"$arguments"
|
||||
set +e
|
||||
invoke "$root/invalid" "${argv[@]}"
|
||||
@@ -62,6 +64,36 @@ for arguments in '--unknown' '--precompile extra' '--help extra'; do
|
||||
[[ ! -s "$root/invalid.stdout" ]]
|
||||
done
|
||||
|
||||
cp bin/wasm/core.wasm "$root/serialize.wasm"
|
||||
INVOKE_TIMEOUT_SECONDS=30 invoke "$root/serialize" --serialize-module "$root/serialize.wasm"
|
||||
[[ -s "$root/serialize.cwasm" ]]
|
||||
[[ ! -s "$root/serialize.stderr" ]]
|
||||
|
||||
# Serialization shares the unit compile lock. If the wasm path is replaced
|
||||
# while the serializer is waiting, it must read the replacement and must not
|
||||
# publish a stale native artifact for the old inode.
|
||||
cp bin/wasm/core.wasm "$root/race.wasm"
|
||||
exec 8>"$root/race.wasm.lock"
|
||||
flock 8
|
||||
INVOKE_TIMEOUT_SECONDS=30 invoke "$root/race" --serialize-module "$root/race.wasm" &
|
||||
race_pid=$!
|
||||
sleep 0.1
|
||||
kill -0 "$race_pid"
|
||||
printf 'not wasm\n' >"$root/race.wasm.next"
|
||||
mv "$root/race.wasm.next" "$root/race.wasm"
|
||||
flock -u 8
|
||||
set +e
|
||||
wait "$race_pid"
|
||||
race_rc=$?
|
||||
set -e
|
||||
[[ $race_rc -eq 1 ]]
|
||||
grep -Eqi 'wasm|magic|WebAssembly|compile' "$root/race.stderr"
|
||||
[[ ! -e "$root/race.cwasm" ]]
|
||||
if compgen -G "$root/race.cwasm.*.tmp" >/dev/null; then
|
||||
echo "failed serialization left a temporary artifact" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
[[ -S "$socket_path" ]]
|
||||
[[ "$(stat -c %i "$socket_path")" == "$socket_inode" ]]
|
||||
kill -0 "$listener_pid"
|
||||
|
||||
@@ -24,6 +24,9 @@ inode_before=$(stat -c %i "$socket_path")
|
||||
load_pid=$!
|
||||
sleep 0.05
|
||||
systemctl restart "$service_name"
|
||||
systemctl is-active --quiet "$socket_name"
|
||||
inode_during=$(stat -c %i "$socket_path")
|
||||
[[ "$inode_during" = "$inode_before" ]]
|
||||
wait "$load_pid"
|
||||
|
||||
inode_after=$(stat -c %i "$socket_path")
|
||||
|
||||
@@ -31,7 +31,7 @@ cleanup() {
|
||||
}
|
||||
trap cleanup EXIT
|
||||
mkdir -p "$site/components" "$work" "$root/run" "$root/session" "$root/upload"
|
||||
sed -E '/^[[:space:]]*(BIN_DIRECTORY|PRECOMPILE_FILES_IN|SITE_DIRECTORY|FCGI_SOCKET_PATH|FCGI_PORT|CLI_SOCKET_PATH|WS_BROKER_SOCKET_PATH|HTTP_PORT|HTTP_DOCUMENT_ROOT|SESSION_PATH|TMP_UPLOAD_PATH|WASM_CORE_PATH|WASM_COMPILE_SCRIPT|WASM_INVOCATION_TIMEOUT_MS|WASM_EPOCH_PERIOD_MS|PROACTIVE_COMPILE_ENABLED|WORKER_COUNT|COMPILE_FAILURE_RETRY_SECONDS)[[:space:]]*=/d' \
|
||||
sed -E '/^[[:space:]]*(BIN_DIRECTORY|PRECOMPILE_FILES_IN|SITE_DIRECTORY|FCGI_SOCKET_PATH|FCGI_PORT|CLI_SOCKET_PATH|CLI_WORKER_COUNT|CLI_WORKER_MAX_REQUESTS|WS_BROKER_SOCKET_PATH|HTTP_PORT|HTTP_DOCUMENT_ROOT|SESSION_PATH|TMP_UPLOAD_PATH|WASM_CORE_PATH|WASM_COMPILE_SCRIPT|WASM_INVOCATION_TIMEOUT_MS|WASM_EPOCH_PERIOD_MS|PROACTIVE_COMPILE_ENABLED|WORKER_COUNT|COMPILE_FAILURE_RETRY_SECONDS)[[:space:]]*=/d' \
|
||||
/etc/uce/settings.cfg >"$settings"
|
||||
cat >>"$settings" <<CFG
|
||||
BIN_DIRECTORY=$work
|
||||
@@ -40,12 +40,14 @@ SITE_DIRECTORY=$site
|
||||
FCGI_SOCKET_PATH=$root/run/fastcgi.sock
|
||||
FCGI_PORT=
|
||||
CLI_SOCKET_PATH=$root/run/cli.sock
|
||||
CLI_WORKER_COUNT=1
|
||||
CLI_WORKER_MAX_REQUESTS=0
|
||||
WS_BROKER_SOCKET_PATH=$root/run/ws.sock
|
||||
HTTP_PORT=
|
||||
HTTP_DOCUMENT_ROOT=$site
|
||||
SESSION_PATH=$root/session
|
||||
TMP_UPLOAD_PATH=$root/upload
|
||||
WASM_CORE_PATH=$(pwd)/bin/wasm/core.wasm
|
||||
WASM_CORE_PATH=$(pwd -P)/bin/wasm/core.wasm
|
||||
WASM_COMPILE_SCRIPT=$root/compile
|
||||
WASM_INVOCATION_TIMEOUT_MS=2000
|
||||
WASM_EPOCH_PERIOD_MS=20
|
||||
|
||||
@@ -21,11 +21,20 @@ if [[ -r "$settings_file" ]]; then
|
||||
[[ -n "${UCE_CLI_SOCKET:-}" ]] || socket_path=$(awk -F= '/^[[:space:]]*CLI_SOCKET_PATH[[:space:]]*=/ {gsub(/^[[:space:]]+|[[:space:]]+$/, "", $2); print $2; exit}' "$settings_file")
|
||||
[[ -n "${BIN_DIRECTORY:-}" ]] || bin_directory=$(awk -F= '/^[[:space:]]*BIN_DIRECTORY[[:space:]]*=/ {gsub(/^[[:space:]]+|[[:space:]]+$/, "", $2); print $2; exit}' "$settings_file")
|
||||
invocation_ms=$(awk -F= '/^[[:space:]]*WASM_INVOCATION_TIMEOUT_MS[[:space:]]*=/ {gsub(/^[[:space:]]+|[[:space:]]+$/, "", $2); print $2; exit}' "$settings_file")
|
||||
cli_worker_count=$(awk -F= '/^[[:space:]]*CLI_WORKER_COUNT[[:space:]]*=/ {gsub(/^[[:space:]]+|[[:space:]]+$/, "", $2); value=$2} END{print value}' "$settings_file")
|
||||
fi
|
||||
site_directory="${site_directory:-site}"
|
||||
socket_path="${socket_path:-/run/uce/cli.sock}"
|
||||
bin_directory="${bin_directory:-/tmp/uce/work}"
|
||||
invocation_ms="${invocation_ms:-30000}"
|
||||
cli_worker_count="${cli_worker_count:-0}"
|
||||
curl_transport=(--unix-socket "$socket_path")
|
||||
test_base_url="http://localhost"
|
||||
if (( cli_worker_count > 0 )); then
|
||||
test_http_host="${UCE_TEST_HTTP_HOST:-uce.openfu.com}"
|
||||
curl_transport=(--resolve "$test_http_host:80:127.0.0.1")
|
||||
test_base_url="http://$test_http_host"
|
||||
fi
|
||||
test_name="invocation-timeout-test-$$"
|
||||
source_dir="$site_directory/$test_name"
|
||||
pid_file="/tmp/uce-$test_name-worker"
|
||||
@@ -42,13 +51,15 @@ mkdir -p "$source_dir"
|
||||
cache_dir="$(scripts/unit_cache_directory "$bin_directory")$(realpath "$source_dir")"
|
||||
|
||||
printf '%s\n' \
|
||||
'CLI(Request& context) {' \
|
||||
'void timeout_test_run(Request& context) {' \
|
||||
' if(context.get["warm"] == "1") { print("warm"); return; }' \
|
||||
" file_put_contents(\"$pid_file\", std::to_string(request_perf()[\"worker_pid\"].to_u64()));" \
|
||||
' while(true) time_precise();' \
|
||||
'}' >"$source_dir/hostcall-loop.uce"
|
||||
'}' \
|
||||
'CLI(Request& context) { timeout_test_run(context); }' \
|
||||
'RENDER(Request& context) { timeout_test_run(context); }' >"$source_dir/hostcall-loop.uce"
|
||||
printf '%s\n' \
|
||||
'CLI(Request& context) {' \
|
||||
'void timeout_test_run(Request& context) {' \
|
||||
' if(context.get["warm"] == "1") { print("warm"); return; }' \
|
||||
' if(context.get["quick"] == "1") { print(shell_exec("printf quick")); return; }' \
|
||||
' if(context.get["status"] == "1") { DValue spec; spec["cmd"] = "exit 7"; spec["timeout_ms"] = (f64)500; print(shell_exec(spec)["exit_code"].to_u64()); return; }' \
|
||||
@@ -56,46 +67,50 @@ printf '%s\n' \
|
||||
' if(context.get["job"] == "1") { DValue spec; spec["cmd"] = "sleep 2"; spec["timeout_ms"] = (f64)5000; u64 job = shell_spawn(spec); f64 started = time_precise(); job_await(job, 300); u64 elapsed = (u64)((time_precise() - started) * 1000); job_cancel(job); print(elapsed); return; }' \
|
||||
" file_put_contents(\"$pid_file\", std::to_string(request_perf()[\"worker_pid\"].to_u64()));" \
|
||||
' print(shell_exec("printf shell-start; sleep 60 & printf shell-end"));' \
|
||||
'}' >"$source_dir/legacy-shell.uce"
|
||||
'}' \
|
||||
'CLI(Request& context) { timeout_test_run(context); }' \
|
||||
'RENDER(Request& context) { timeout_test_run(context); }' >"$source_dir/legacy-shell.uce"
|
||||
printf '%s\n' \
|
||||
'CLI(Request& context) {' \
|
||||
'void timeout_test_run(Request& context) {' \
|
||||
' if(context.get["warm"] == "1") { print("warm"); return; }' \
|
||||
" file_put_contents(\"$pid_file\", std::to_string(request_perf()[\"worker_pid\"].to_u64()));" \
|
||||
' while(true) sleep(60);' \
|
||||
' print("sleep-end");' \
|
||||
'}' >"$source_dir/sleep.uce"
|
||||
'}' \
|
||||
'CLI(Request& context) { timeout_test_run(context); }' \
|
||||
'RENDER(Request& context) { timeout_test_run(context); }' >"$source_dir/sleep.uce"
|
||||
printf '%s\n' \
|
||||
'CLI(Request& context) { print(request_perf()["worker_pid"].to_u64(), "|health"); }' >"$source_dir/health.uce"
|
||||
'void timeout_test_run(Request& context) { print(request_perf()["worker_pid"].to_u64(), "|health"); }' \
|
||||
'CLI(Request& context) { timeout_test_run(context); }' \
|
||||
'RENDER(Request& context) { timeout_test_run(context); }' >"$source_dir/health.uce"
|
||||
|
||||
max_seconds=$(( (invocation_ms + 15000) / 1000 ))
|
||||
(( max_seconds >= 10 )) || max_seconds=10
|
||||
|
||||
request() {
|
||||
local unit="$1"
|
||||
curl -sS --max-time "$max_seconds" -o "$body_file" -w '%{http_code}' --unix-socket "$socket_path" "http://localhost/$test_name/$unit"
|
||||
curl -sS --max-time "$max_seconds" -o "$body_file" -w '%{http_code}' "${curl_transport[@]}" "$test_base_url/$test_name/$unit"
|
||||
}
|
||||
|
||||
same_worker_health() {
|
||||
local expected_pid="$1"
|
||||
for _ in $(seq 1 32); do
|
||||
local health
|
||||
health=$(curl -sS --max-time 5 --fail-with-body --unix-socket "$socket_path" "http://localhost/$test_name/health.uce")
|
||||
[[ "$health" == "$expected_pid|health" ]] && return 0
|
||||
done
|
||||
return 1
|
||||
kill -0 "$expected_pid" 2>/dev/null || return 1
|
||||
local health
|
||||
health=$(curl -sS --max-time 5 --fail-with-body "${curl_transport[@]}" "$test_base_url/$test_name/health.uce")
|
||||
[[ "$health" =~ ^[0-9]+\|health$ ]]
|
||||
}
|
||||
|
||||
curl -sS --max-time "$max_seconds" --fail-with-body --unix-socket "$socket_path" "http://localhost/$test_name/hostcall-loop.uce?warm=1" >/dev/null
|
||||
curl -sS --max-time "$max_seconds" --fail-with-body --unix-socket "$socket_path" "http://localhost/$test_name/legacy-shell.uce?warm=1" >/dev/null
|
||||
curl -sS --max-time "$max_seconds" --fail-with-body --unix-socket "$socket_path" "http://localhost/$test_name/sleep.uce?warm=1" >/dev/null
|
||||
curl -sS --max-time "$max_seconds" --fail-with-body --unix-socket "$socket_path" "http://localhost/$test_name/health.uce" >/dev/null
|
||||
quick=$(curl -sS --max-time 5 --fail-with-body --unix-socket "$socket_path" "http://localhost/$test_name/legacy-shell.uce?quick=1")
|
||||
curl -sS --max-time "$max_seconds" --fail-with-body "${curl_transport[@]}" "$test_base_url/$test_name/hostcall-loop.uce?warm=1" >/dev/null
|
||||
curl -sS --max-time "$max_seconds" --fail-with-body "${curl_transport[@]}" "$test_base_url/$test_name/legacy-shell.uce?warm=1" >/dev/null
|
||||
curl -sS --max-time "$max_seconds" --fail-with-body "${curl_transport[@]}" "$test_base_url/$test_name/sleep.uce?warm=1" >/dev/null
|
||||
curl -sS --max-time "$max_seconds" --fail-with-body "${curl_transport[@]}" "$test_base_url/$test_name/health.uce" >/dev/null
|
||||
quick=$(curl -sS --max-time 5 --fail-with-body "${curl_transport[@]}" "$test_base_url/$test_name/legacy-shell.uce?quick=1")
|
||||
[[ "$quick" == quick ]] || { echo "quick legacy shell returned: $quick" >&2; exit 1; }
|
||||
exit_status=$(curl -sS --max-time 5 --fail-with-body --unix-socket "$socket_path" "http://localhost/$test_name/legacy-shell.uce?status=1")
|
||||
exit_status=$(curl -sS --max-time 5 --fail-with-body "${curl_transport[@]}" "$test_base_url/$test_name/legacy-shell.uce?status=1")
|
||||
[[ "$exit_status" == 7 ]] || { echo "structured shell lost exit status: $exit_status" >&2; exit 1; }
|
||||
zero_timeout=$(curl -sS --max-time 5 --fail-with-body --unix-socket "$socket_path" "http://localhost/$test_name/legacy-shell.uce?zero=1")
|
||||
zero_timeout=$(curl -sS --max-time 5 --fail-with-body "${curl_transport[@]}" "$test_base_url/$test_name/legacy-shell.uce?zero=1")
|
||||
[[ "$zero_timeout" == zero ]] || { echo "structured shell zero-timeout default changed: $zero_timeout" >&2; exit 1; }
|
||||
job_elapsed=$(curl -sS --max-time 5 --fail-with-body --unix-socket "$socket_path" "http://localhost/$test_name/legacy-shell.uce?job=1")
|
||||
job_elapsed=$(curl -sS --max-time 5 --fail-with-body "${curl_transport[@]}" "$test_base_url/$test_name/legacy-shell.uce?job=1")
|
||||
[[ "$job_elapsed" =~ ^[0-9]+$ && "$job_elapsed" -ge 200 && "$job_elapsed" -lt 500 ]] || { echo "job_await two-call duration was ${job_elapsed}ms" >&2; exit 1; }
|
||||
|
||||
for unit in hostcall-loop.uce legacy-shell.uce sleep.uce; do
|
||||
|
||||
@@ -29,7 +29,7 @@ cleanup() {
|
||||
}
|
||||
trap cleanup EXIT
|
||||
mkdir -p "$site/components" "$work" "$root/run" "$root/session" "$root/upload"
|
||||
sed -E '/^[[:space:]]*(BIN_DIRECTORY|PRECOMPILE_FILES_IN|SITE_DIRECTORY|FCGI_SOCKET_PATH|FCGI_PORT|CLI_SOCKET_PATH|WS_BROKER_SOCKET_PATH|HTTP_PORT|HTTP_DOCUMENT_ROOT|SESSION_PATH|TMP_UPLOAD_PATH|WASM_CORE_PATH|WASM_INVOCATION_TIMEOUT_MS|WASM_EPOCH_PERIOD_MS|PROACTIVE_COMPILE_ENABLED|WORKER_COUNT)[[:space:]]*=/d' \
|
||||
sed -E '/^[[:space:]]*(BIN_DIRECTORY|PRECOMPILE_FILES_IN|SITE_DIRECTORY|FCGI_SOCKET_PATH|FCGI_PORT|CLI_SOCKET_PATH|CLI_WORKER_COUNT|CLI_WORKER_MAX_REQUESTS|WS_BROKER_SOCKET_PATH|HTTP_PORT|HTTP_DOCUMENT_ROOT|SESSION_PATH|TMP_UPLOAD_PATH|WASM_CORE_PATH|WASM_INVOCATION_TIMEOUT_MS|WASM_EPOCH_PERIOD_MS|PROACTIVE_COMPILE_ENABLED|SERVE_LAST_KNOWN_GOOD|SHOW_DYNAMIC_COMPILE_ERRORS|WORKER_COUNT)[[:space:]]*=/d' \
|
||||
/etc/uce/settings.cfg >"$settings"
|
||||
cat >>"$settings" <<CFG
|
||||
BIN_DIRECTORY=$work
|
||||
@@ -38,15 +38,19 @@ SITE_DIRECTORY=$site
|
||||
FCGI_SOCKET_PATH=$root/run/fastcgi.sock
|
||||
FCGI_PORT=
|
||||
CLI_SOCKET_PATH=$socket
|
||||
CLI_WORKER_COUNT=1
|
||||
CLI_WORKER_MAX_REQUESTS=0
|
||||
WS_BROKER_SOCKET_PATH=$root/run/ws.sock
|
||||
HTTP_PORT=
|
||||
HTTP_DOCUMENT_ROOT=$site
|
||||
SESSION_PATH=$root/session
|
||||
TMP_UPLOAD_PATH=$root/upload
|
||||
WASM_CORE_PATH=$(pwd)/bin/wasm/core.wasm
|
||||
WASM_INVOCATION_TIMEOUT_MS=100
|
||||
WASM_INVOCATION_TIMEOUT_MS=1000
|
||||
WASM_EPOCH_PERIOD_MS=10
|
||||
PROACTIVE_COMPILE_ENABLED=0
|
||||
SERVE_LAST_KNOWN_GOOD=0
|
||||
SHOW_DYNAMIC_COMPILE_ERRORS=1
|
||||
WORKER_COUNT=1
|
||||
CFG
|
||||
mount --bind "$settings" /etc/uce/settings.cfg
|
||||
@@ -259,14 +263,14 @@ wait_ready() {
|
||||
prepare_deadline
|
||||
deadline_wasm="$cache/components/deadline.uce.wasm"
|
||||
deadline_counter="$root/deadline.counter"
|
||||
printf '%s delay 150 - - %s\n' "$deadline_wasm" "$deadline_counter" >"$control"
|
||||
printf '%s delay 1100 - - %s\n' "$deadline_wasm" "$deadline_counter" >"$control"
|
||||
started=$(date +%s%N)
|
||||
set +e
|
||||
deadline_output=$(request target=components/deadline.uce 2>&1)
|
||||
set -e
|
||||
elapsed_ms=$(( ($(date +%s%N) - started) / 1000000 ))
|
||||
[[ "$deadline_output" == *UCE_INVOCATION_TIMEOUT:* ]] || { echo "metadata scan lacked canonical timeout: $deadline_output" >&2; exit 1; }
|
||||
(( elapsed_ms >= 100 && elapsed_ms < 2000 )) || { echo "metadata timeout took ${elapsed_ms}ms" >&2; exit 1; }
|
||||
(( elapsed_ms >= 1000 && elapsed_ms < 3000 )) || { echo "metadata timeout took ${elapsed_ms}ms" >&2; exit 1; }
|
||||
[[ $(wc -l <"$deadline_counter") -eq 1 ]] || { echo "metadata scanner read past the first delayed refill" >&2; exit 1; }
|
||||
rm -f "$control"
|
||||
[[ "$(request health=1)" == "$worker_pid|health" ]]
|
||||
@@ -275,14 +279,14 @@ restore deadline
|
||||
|
||||
rm -f "$cache/components/deadline.uce.cwasm"
|
||||
touch "$deadline_wasm"
|
||||
printf '%s delay 150 - - %s\n' "$deadline_wasm" "$root/full-deadline.counter" >"$control"
|
||||
printf '%s delay 1100 - - %s\n' "$deadline_wasm" "$root/full-deadline.counter" >"$control"
|
||||
started=$(date +%s%N)
|
||||
set +e
|
||||
full_deadline_output=$(request target=components/deadline.uce 2>&1)
|
||||
set -e
|
||||
elapsed_ms=$(( ($(date +%s%N) - started) / 1000000 ))
|
||||
[[ "$full_deadline_output" == *UCE_INVOCATION_TIMEOUT:* ]] || { echo "full artifact read lacked canonical timeout: $full_deadline_output" >&2; exit 1; }
|
||||
(( elapsed_ms >= 100 && elapsed_ms < 2000 )) || { echo "full artifact timeout took ${elapsed_ms}ms" >&2; exit 1; }
|
||||
(( elapsed_ms >= 1000 && elapsed_ms < 3000 )) || { echo "full artifact timeout took ${elapsed_ms}ms" >&2; exit 1; }
|
||||
[[ $(wc -l <"$root/full-deadline.counter") -eq 1 ]] || { echo "full artifact reader continued after the first delayed chunk" >&2; exit 1; }
|
||||
rm -f "$control"
|
||||
[[ "$(request health=1)" == "$worker_pid|health" ]]
|
||||
|
||||
@@ -14,6 +14,7 @@ hmac_sha256
|
||||
hmac_sha256_hex
|
||||
random_bytes
|
||||
crypto_equal
|
||||
crypto_operation
|
||||
password_hash
|
||||
password_verify
|
||||
password_needs_rehash
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
:sig
|
||||
DValue crypto_operation(DValue request)
|
||||
|
||||
:params
|
||||
request : structured operation, algorithm, and operation-specific fields
|
||||
return value : map with ok, bounded error code, and operation-specific output
|
||||
|
||||
:content
|
||||
Runs one explicitly supported structured asymmetric cryptographic operation. The initial allowlist is `key_generate` with `ES256` and `jwt_sign` with `ES256`. Unknown operations and algorithms fail closed.
|
||||
|
||||
ES256 signing validates that `x`, `y`, and `d` form one P-256 key, forces the protected `alg` to `ES256`, and emits a compact JWT with a 64-byte JOSE signature. Requests are capped at 32 KiB; `cbor_decode` accepts at most 16 KiB decoded CBOR (21,846 canonical base64url characters), 256 nodes, and depth 16. CBOR is definite-length only, validates UTF-8 text, preserves typed map keys, rejects structurally duplicate keys and trailing bytes, and reports malformed CBOR as `invalid_cbor`. Header and claims roots must be JSON objects containing valid UTF-8 without raw control bytes.
|
||||
|
||||
This function does not replace typed digest, HMAC, password, randomness, or constant-time comparison APIs. It exposes no raw signing, arbitrary curve/digest selection, encryption, or generic OpenSSL access. Keep returned private JWKs secret.
|
||||
|
||||
:example
|
||||
DValue key_request;
|
||||
key_request["operation"] = "key_generate";
|
||||
key_request["algorithm"] = "ES256";
|
||||
DValue key = crypto_operation(key_request);
|
||||
|
||||
DValue sign_request;
|
||||
sign_request["operation"] = "jwt_sign";
|
||||
sign_request["algorithm"] = "ES256";
|
||||
sign_request["private_jwk"] = key["private_jwk"];
|
||||
sign_request["protected_header"]["typ"] = "JWT";
|
||||
sign_request["claims"]["iss"] = "https://client.example";
|
||||
DValue signed_jwt = crypto_operation(sign_request);
|
||||
print(signed_jwt["ok"].to_bool() ? "signed" : "failed", "\n");
|
||||
@@ -1,8 +1,18 @@
|
||||
# http_request
|
||||
|
||||
Performs a bounded outbound HTTP(S) request using the runtime `curl` binary. Request fields: `method`, `url`, `headers`, `body`, `timeout_ms`, `follow_redirects`.
|
||||
Performs an outbound HTTP(S) request using the runtime `curl` binary. Existing request fields remain backward-compatible: `method`, `url`, `headers`, `body`, `timeout_ms`, and `follow_redirects`.
|
||||
|
||||
Returns `{ status, headers, body, error }`. `headers` is a name/value map. A missing `curl` binary returns a clear `error` string.
|
||||
An absent `security` map, or a map containing none of these recognized keys, keeps the legacy request behavior. If a `security` map contains any recognized key, it selects hardening and **every** recognized field below must be explicitly boolean `true`; missing, partial, non-true, or `false` fields fail closed with `invalid_request`.
|
||||
|
||||
- `https_only` rejects non-HTTPS URLs, IP-literal hosts, and URL userinfo.
|
||||
- `public_dns_only` validates **every** DNS answer against the public IPv4 policy. IPv6 answers currently fail closed.
|
||||
- `pin_dns` pins curl to one validated answer with `--resolve`, retaining the URL hostname for TLS/SNI.
|
||||
- `isolated_curl` uses absolute `/usr/bin/curl`, `--disable` as argv[1], cleared environment, no proxy/config/netrc/HSTS/Alt-Svc inheritance, no redirects, a three-second connect bound, and a ten-second total bound.
|
||||
- `no_redirects` makes redirect following and `follow_redirects=true` invalid composition.
|
||||
|
||||
Hardened requests bound body input/output to 64 KiB and response headers to 8 KiB. Async hardened requests keep curl and its descendants in the job worker process group, so cancelling that job kills that group only. Methods are limited to `GET`, `POST`, `PUT`, `PATCH`, `DELETE`, `HEAD`, and `OPTIONS`; header names/values are validated and caller-controlled `Host`, framing, connection, and expectation headers are rejected. Sensitive request bodies go to curl stdin, never argv. Errors are typed non-secret values such as `invalid_request`, `unsafe_dns`, `timeout`, `response_too_large`, `redirect_not_allowed`, `http_status`, and `network_failure`.
|
||||
|
||||
Returns `{ status, headers, body, error }`.
|
||||
|
||||
:see
|
||||
>socket
|
||||
@@ -12,3 +22,10 @@ DValue req; req["method"] = "GET"; req["url"] = "http://127.0.0.1/doc/index.uce"
|
||||
req["headers"]["Host"] = "uce.openfu.com"; req["timeout_ms"] = (f64)2000;
|
||||
DValue resp = http_request(req);
|
||||
print("HTTP ", resp["status"].to_u64(), ", ", resp["body"].to_string().length(), " bytes returned\n");
|
||||
|
||||
:example
|
||||
// Hardened requests reject IP-literal hosts before they start curl.
|
||||
DValue hardened; hardened["method"]="GET"; hardened["url"]="https://127.0.0.1/";
|
||||
for(String key:{"https_only","public_dns_only","pin_dns","isolated_curl","no_redirects"}) hardened["security"][key].set_bool(true);
|
||||
DValue rejected=http_request(hardened);
|
||||
print(rejected["error"].to_string(), "\n");
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# http_request_async
|
||||
|
||||
Starts the same bounded curl-backed request as `http_request()` in the file-backed async job registry and returns a job id. Use `job_await()` or `job_result()` to retrieve the HTTP result.
|
||||
Starts the same request shape as `http_request()`, including its opt-in `security` hardening fields, in the file-backed async job registry and returns a job id. An unknown `security` object remains legacy; any recognized hardening key requires all five keys to be explicit `true` or the request fails closed. Hardened curl remains in this job's worker process group, so `job_cancel()` kills curl and its descendants without signalling other jobs. Use `job_await()` or `job_result()` to retrieve the HTTP result.
|
||||
|
||||
:see
|
||||
>socket
|
||||
|
||||
@@ -13,6 +13,7 @@ return value : string containing the Memcache server's response
|
||||
Executes a raw command on an open Memcache connection and returns the server response as a string.
|
||||
|
||||
This is the low-level escape hatch for Memcache operations that are not covered by the dedicated helpers.
|
||||
UCE reads the complete framed ASCII response, including multi-value retrievals and `stats`, rather than returning a socket chunk. The complete write and read share a one-second deadline and responses are limited to 8 MiB. A timeout, malformed frame, or oversized response returns an empty string and makes that connection unusable so unread bytes cannot affect a later command. Commands ending in `noreply` and `quit` return immediately with an empty string.
|
||||
|
||||
:example
|
||||
u64 conn = memcache_connect();
|
||||
|
||||
@@ -16,7 +16,7 @@ Establishes a connection to a MySQL server and returns a request-owned pointer t
|
||||
|
||||
This connection handle is then used with helpers such as `mysql_query()`, `mysql_error()`, and `mysql_disconnect()`.
|
||||
|
||||
MySQL handles are request-scoped framework resources. Repeated `mysql_connect()` calls with the same host, credentials, and database reuse one server connection within the current request. Database identity is part of both request-local and worker-persistent pool keys. Each call creates a lease and `mysql_disconnect()` releases that lease. At request cleanup UCE returns the server connection to the current worker's persistent pool, then resets it before cross-request reuse. The pool holds up to `MYSQL_PERSISTENT_POOL_SIZE` connections per worker (default 8); set the size to 0 to disable cross-request reuse. Never store a `MySQL*` in globals, sessions, or other state that can outlive the current request.
|
||||
MySQL handles are request-scoped framework resources. Repeated `mysql_connect()` calls with the same host, credentials, and database reuse one server connection within the current request. Database identity is part of both request-local and worker-persistent pool keys. Each call creates a lease and `mysql_disconnect()` releases that lease. At request cleanup UCE returns the server connection to the current worker's persistent pool, then resets it before cross-request reuse. The pool holds up to `MYSQL_PERSISTENT_POOL_SIZE` connections per worker (default 8); set the size to 0 to disable cross-request reuse. `MYSQL_PERSISTENT_POOL_IDLE_TIMEOUT_SECONDS` retires idle entries at the first later request boundary (default 300, 0 keeps unlimited idle retention). A completely idle worker has no boundary and does not proactively close sockets. Never store a `MySQL*` in globals, sessions, or other state that can outlive the current request.
|
||||
|
||||
:example
|
||||
MySQL* db = mysql_connect();
|
||||
|
||||
@@ -16,6 +16,8 @@ Executes a MySQL query and returns the resulting data, if any.
|
||||
|
||||
`params` provides the query parameter values used by the statement. Use named `:name` placeholders only; positional `?` placeholders are rejected.
|
||||
|
||||
Ordinary placeholders are escaped and quoted as SQL string values. For grammar positions that require an unquoted non-negative integer, such as `LIMIT` and `OFFSET`, append `!` to the placeholder (`:limit!`). Unsigned placeholders fail before query execution unless their value is a non-empty sequence of decimal digits. They never accept signs, whitespace, expressions, identifiers, or other SQL fragments.
|
||||
|
||||
The result is returned as a `DValue`, which makes it easy to iterate through rows and read fields with the usual `DValue` accessors.
|
||||
|
||||
After an insert, update, or delete, use `mysql_affected_rows()` to inspect how many rows changed.
|
||||
@@ -24,7 +26,8 @@ After an insert, update, or delete, use `mysql_affected_rows()` to inspect how m
|
||||
MySQL* db = mysql_connect();
|
||||
if(db != 0)
|
||||
{
|
||||
DValue rows = mysql_query(db, "select 'ada@example.test' as email, 1 + 1 as total");
|
||||
StringMap params; params["limit"] = "1";
|
||||
DValue rows = mysql_query(db, "select 'ada@example.test' as email, 1 + 1 as total limit :limit!", params);
|
||||
String email = "none"; String total = "?";
|
||||
rows.each([&](DValue r, String key) { email = r["email"].to_string(); total = r["total"].to_string(); });
|
||||
print(email, " / total=", total, "\n");
|
||||
|
||||
@@ -27,7 +27,7 @@ The request log's `wasm-ready`, `wasm`, `workspace`, `invoke`, `collect`, and `p
|
||||
|
||||
Workspace birth is subdivided into `birth_policy_us`, `birth_import_us`, `birth_instantiate_us`, `birth_exports_us`, and `birth_initialize_us`. Request-context transfer reports `context_bytes`, `context_encode_us`, `context_allocate_us`, `context_write_us`, `context_guest_apply_us`, and `context_free_us`. These bounded aggregate fields expose sizes and timing only, never request values.
|
||||
|
||||
Wasm FastCGI workers retain up to `MYSQL_PERSISTENT_POOL_SIZE` credential-keyed MySQL connections (default `8`; set `0` to disable). UCE calls the client library's connection-reset operation before another request receives a cached connection, clearing transactions, temporary tables, session variables, and selected databases while avoiding a new authentication handshake. Same-request leases continue to share state until request cleanup.
|
||||
Wasm FastCGI workers retain up to `MYSQL_PERSISTENT_POOL_SIZE` credential-keyed MySQL connections (default `8`; set `0` to disable). `MYSQL_PERSISTENT_POOL_IDLE_TIMEOUT_SECONDS` retires an idle entry at the first later request boundary (default `300`; `0` disables age eviction). UCE calls the client library's connection-reset operation before another request receives a cached connection, clearing transactions, temporary tables, session variables, and selected databases while avoiding a new authentication handshake. Same-request leases continue to share state until request cleanup.
|
||||
|
||||
:example
|
||||
DValue perf = request_perf();
|
||||
|
||||
@@ -21,7 +21,7 @@ Starts a repeating background worker process.
|
||||
|
||||
If a process with the same `key` is already running anywhere in the runtime instance, `task_repeat()` does not start a second worker and instead returns the PID of the existing one. Coordination is through the same shared task state used by `task()`.
|
||||
|
||||
`timeout` bounds the lifetime of the repeating worker. The default is ten minutes. Pass `0` only for workers that have another shutdown path.
|
||||
`timeout` bounds the lifetime of the repeating worker. The default is ten minutes. Pass `0` only for workers that have another shutdown path. In Wasm, each individual callback invocation remains bounded by the runtime invocation timeout even when the repeating process lifetime is unbounded.
|
||||
|
||||
:example
|
||||
task_repeat("doc-demo-repeat", 60.0, []() { usleep(10000); });
|
||||
|
||||
@@ -33,3 +33,9 @@ The example uses query-string routing in the same style as the PHP starter, but
|
||||
Direct requests to `/examples/uce-starter/index.uce` still work, but self-links are canonicalized back to `/examples/uce-starter/`.
|
||||
|
||||
The demo account pages use a small file-backed user store under `/tmp/uce-starter-data/` with session-based login state.
|
||||
|
||||
|
||||
## Optional Datastar Assets
|
||||
|
||||
This starter vendors Datastar v1.0.2 as `js/datastar.js` for future server-rendered interactive islands. It is not loaded by the default page shell. `lib/datastar.uce` contains small Datastar SSE formatting helpers, and `views/datastar-example.uce` is an unlinked opt-in example route.
|
||||
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,38 @@
|
||||
String datastar_data_lines(String field, String value)
|
||||
{
|
||||
String out = "";
|
||||
for(String line : split(value, "\n"))
|
||||
out += "data: " + field + " " + line + "\n";
|
||||
return(out);
|
||||
}
|
||||
|
||||
void datastar_sse_headers(Request& context)
|
||||
{
|
||||
context.call["app"]["page_type"] = "blank";
|
||||
context.header["Content-Type"] = "text/event-stream; charset=utf-8";
|
||||
context.header["Cache-Control"] = "no-cache, no-store, must-revalidate";
|
||||
context.header["X-Accel-Buffering"] = "no";
|
||||
}
|
||||
|
||||
String datastar_event(String event_name, StringMap fields)
|
||||
{
|
||||
String out = "event: " + event_name + "\n";
|
||||
for(const auto& field : fields)
|
||||
out += datastar_data_lines(field.first, field.second);
|
||||
out += "\n";
|
||||
return(out);
|
||||
}
|
||||
|
||||
String datastar_patch_elements(String html)
|
||||
{
|
||||
StringMap fields;
|
||||
fields["elements"] = html;
|
||||
return(datastar_event("datastar-patch-elements", fields));
|
||||
}
|
||||
|
||||
String datastar_patch_signals(String signals_json)
|
||||
{
|
||||
StringMap fields;
|
||||
fields["signals"] = signals_json;
|
||||
return(datastar_event("datastar-patch-signals", fields));
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
#load "../lib/app.uce"
|
||||
#load "../lib/datastar.uce"
|
||||
|
||||
COMPONENT(Request& context)
|
||||
{
|
||||
if(context.get["stream"] == "1")
|
||||
{
|
||||
datastar_sse_headers(context);
|
||||
print(datastar_patch_elements("<div id=\"datastar-example-result\">Server-rendered UCE Datastar fragment</div>"));
|
||||
return;
|
||||
}
|
||||
|
||||
<>
|
||||
<h1>Datastar optional example</h1>
|
||||
<p>Datastar is vendored beside the starter client libraries for future server-rendered islands. It is not loaded globally.</p>
|
||||
<script type="module" src="<?= app_asset_url("js/datastar.js", context) ?>"></script>
|
||||
<button class="btn" data-on:click="@get('<?= app_link("datastar-example", context) ?>&stream=1')">Patch a fragment</button>
|
||||
<div id="datastar-example-result">Waiting for a server patch.</div>
|
||||
</>;
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
@@ -20,6 +20,37 @@ CLI(Request& context)
|
||||
print(cli_arg(context, "missing", "fallback"), "\n");
|
||||
return;
|
||||
}
|
||||
if(action == "task_repeat_unbounded")
|
||||
{
|
||||
String marker = "/tmp/uce-task-repeat-unbounded.txt";
|
||||
file_unlink(marker);
|
||||
pid_t pid = task_repeat("uce-task-repeat-unbounded", 0.05, [marker]() { file_put_contents(marker, "ran"); }, 0);
|
||||
usleep(200000);
|
||||
bool ran = file_get_contents(marker) == "ran";
|
||||
if(pid > 0) task_kill(pid, 15);
|
||||
file_unlink(marker);
|
||||
print(pid > 0 && ran ? "task repeat unbounded ok\n" : "task repeat unbounded failed\n");
|
||||
return;
|
||||
}
|
||||
if(action == "mysql_unsigned_params")
|
||||
{
|
||||
MySQL db;
|
||||
StringMap params; params["limit"] = "1";
|
||||
String parsed = db.parse_query_parameters("SELECT 7 LIMIT :limit!", params);
|
||||
StringMap comparison; comparison["left"] = "7"; comparison["right"] = "8";
|
||||
String inequality = db.parse_query_parameters("SELECT :left!=:right", comparison);
|
||||
StringMap empty_name; empty_name[""] = "1";
|
||||
db.parameter_error = false; db.statement_info = "";
|
||||
db.parse_query_parameters("SELECT 1 LIMIT :!", empty_name);
|
||||
bool empty_rejected = db.parameter_error;
|
||||
StringMap invalid; invalid["limit"] = "1;SELECT 9";
|
||||
db.parameter_error = false;
|
||||
db.statement_info = "";
|
||||
db.parse_query_parameters("SELECT 1 LIMIT :limit!", invalid);
|
||||
bool ok = parsed.find("LIMIT 1") != String::npos && inequality.find("'7'!='8'") != String::npos && empty_rejected && db.parameter_error && db.statement_info.find("must contain only decimal digits") != String::npos;
|
||||
print(ok ? "mysql unsigned params ok\n" : "mysql unsigned params failed: parsed=" + parsed + ", parameter_error=" + (db.parameter_error ? "1" : "0") + ", statement=" + db.statement_info + "\n");
|
||||
return;
|
||||
}
|
||||
context.set_status(400, "CLI Error");
|
||||
print("unknown cli action: ", action, "\n");
|
||||
}
|
||||
|
||||
+11
-3
@@ -31,7 +31,7 @@ RENDER(Request& context)
|
||||
StringMap colon_uri_headers = split_http_headers("GET /clock.uce?t=12:30 HTTP/1.1\r\nHost: colon.example\r\n");
|
||||
check("split_http_headers() request line with colon in URI", colon_uri_headers["REQUEST_METHOD"] == "GET" && colon_uri_headers["DOCUMENT_URI"] == "/clock.uce" && colon_uri_headers["QUERY_STRING"] == "t=12:30" && colon_uri_headers["HTTP_HOST"] == "colon.example", var_dump(colon_uri_headers));
|
||||
check("trim() / split_kv() / split_http_headers()", trim(" padded value ") == "padded value" && kv["alpha"] == "one" && kv["empty"] == "" && http_headers["REQUEST_METHOD"] == "GET" && http_headers["DOCUMENT_URI"] == "/demo.uce" && http_headers["QUERY_STRING"] == "x=1" && http_headers["HTTP_X_EMPTY"] == "" && leading_crlf_headers["REQUEST_METHOD"] == "GET" && leading_crlf_headers["DOCUMENT_URI"] == "/lead.uce" && leading_crlf_headers["HTTP_HOST"] == "lead.example" && header_only["REQUEST_METHOD"] == "" && header_only["HTTP_HOST"] == "example.test" && header_only["HTTP_X_TOKEN"] == "abc", trim(" padded value ") + " / " + var_dump(kv) + " / " + var_dump(http_headers) + " / " + var_dump(leading_crlf_headers) + " / " + var_dump(header_only));
|
||||
check("replace()", replace("alpha-beta-beta", "beta", "done") == "alpha-done-done", replace("alpha-beta-beta", "beta", "done"));
|
||||
check("replace()", replace("alpha-beta-beta", "beta", "done") == "alpha-done-done" && replace("hello", "", "X") == "hello", replace("alpha-beta-beta", "beta", "done"));
|
||||
check("html_escape() attribute-safe quotes", html_escape("<&>\"Don't") == "<&>"Don't", html_escape("<&>\"Don't"));
|
||||
check("regex_match()", regex_match("[A-Z][a-z]+", "Alice") && !regex_match("[A-Z][a-z]+", "Alice!"), "full-string validation");
|
||||
|
||||
@@ -212,11 +212,19 @@ RENDER(Request& context)
|
||||
String decoded_base64 = base64_decode(encoded_base64, base64_ok);
|
||||
bool invalid_base64_ok = true;
|
||||
base64_decode("AA=A", invalid_base64_ok);
|
||||
check("base64_encode() / base64_decode() binary-safe", base64_ok && decoded_base64 == binary_payload && decoded_base64.size() == binary_payload.size() && !invalid_base64_ok, encoded_base64 + " bytes=" + std::to_string((u64)decoded_base64.size()));
|
||||
bool early_padding_base64_ok = true;
|
||||
base64_decode("A=AA", early_padding_base64_ok);
|
||||
check("base64_encode() / base64_decode() binary-safe", base64_ok && decoded_base64 == binary_payload && decoded_base64.size() == binary_payload.size() && !invalid_base64_ok && !early_padding_base64_ok, encoded_base64 + " bytes=" + std::to_string((u64)decoded_base64.size()));
|
||||
|
||||
String utf8_sample = "A\xC3\xA9";
|
||||
auto utf8_parts = split_utf8(utf8_sample);
|
||||
check("split_utf8()", utf8_parts.size() == 2, "count=" + std::to_string(utf8_parts.size()));
|
||||
String truncated_utf8 = "abc";
|
||||
truncated_utf8.push_back((char)0xE2);
|
||||
truncated_utf8.push_back((char)0x82);
|
||||
auto truncated_utf8_parts = split_utf8(truncated_utf8);
|
||||
String lone_utf8_lead(1, (char)0xF0);
|
||||
auto compound_lone_utf8_parts = split_utf8(lone_utf8_lead, true);
|
||||
check("split_utf8()", utf8_parts.size() == 2 && truncated_utf8_parts.size() == 4 && truncated_utf8_parts[3].size() == 2 && compound_lone_utf8_parts.size() == 1 && compound_lone_utf8_parts[0] == lone_utf8_lead, "count=" + std::to_string(utf8_parts.size()) + " truncated=" + std::to_string(truncated_utf8_parts.size()));
|
||||
|
||||
DValue payload;
|
||||
payload["name"] = "uce";
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
#include "testlib.h"
|
||||
|
||||
RENDER(Request& context)
|
||||
{
|
||||
u64 passed = 0;
|
||||
u64 failed = 0;
|
||||
u64 skipped = 0;
|
||||
auto check = [&](String name, bool ok, String detail)
|
||||
{
|
||||
site_tests_case(name, ok ? "pass" : "fail", detail);
|
||||
if(ok) passed++; else failed++;
|
||||
};
|
||||
site_tests_page_start("Structured crypto operations", "Algorithm-selected P-256 JWK creation and ES256 JWT signing.");
|
||||
DValue key_request;
|
||||
key_request["operation"] = "key_generate";
|
||||
key_request["algorithm"] = "ES256";
|
||||
DValue key = crypto_operation(key_request);
|
||||
DValue header;
|
||||
header["alg"] = "none";
|
||||
header["kid"] = key["kid"];
|
||||
DValue claims;
|
||||
claims["iss"] = "https://client.example";
|
||||
claims["sub"] = "client.example";
|
||||
DValue sign_request;
|
||||
sign_request["operation"] = "jwt_sign";
|
||||
sign_request["algorithm"] = "ES256";
|
||||
sign_request["private_jwk"] = key["private_jwk"];
|
||||
sign_request["protected_header"] = header;
|
||||
sign_request["claims"] = claims;
|
||||
DValue signed_result = crypto_operation(sign_request);
|
||||
String jwt = signed_result["jwt"].to_string();
|
||||
StringList parts = split(jwt, ".");
|
||||
String header_b64 = parts.size() == 3 ? parts[0] : "";
|
||||
while(header_b64.size() % 4) header_b64 += "=";
|
||||
String decoded_header = base64_decode(header_b64);
|
||||
DValue wrong_curve = key["private_jwk"];
|
||||
wrong_curve["crv"] = "P-384";
|
||||
DValue malformed = key["private_jwk"];
|
||||
malformed["x"] = "not_base64url=";
|
||||
DValue wrong_request = sign_request;
|
||||
wrong_request["private_jwk"] = wrong_curve;
|
||||
DValue malformed_request = sign_request;
|
||||
malformed_request["private_jwk"] = malformed;
|
||||
DValue cose_request;
|
||||
cose_request["operation"] = "cose_es256_parse";
|
||||
cose_request["algorithm"] = "ES256";
|
||||
cose_request["cose_key_base64url"] = "pQECAyYgASFYIGsX0fLhLEJH-Lzm5WOkQPJ3A32BLeszoPShOUXYmMKWIlggT-NC4v4af5uO5-tKfA-eFivOM1drMV7Oy7ZAaDe_UfU";
|
||||
DValue cose = crypto_operation(cose_request);
|
||||
DValue verify_request;
|
||||
verify_request["operation"] = "es256_verify";
|
||||
verify_request["algorithm"] = "ES256";
|
||||
verify_request["cose_key_base64url"] = cose_request["cose_key_base64url"];
|
||||
verify_request["message_base64url"] = "d2ViYXV0aG4gZml4ZWQgbWVzc2FnZQ";
|
||||
verify_request["signature_der_base64url"] = "MEUCIQCkatZK1VVsjk17uvyzyhjdAkMNWXPjxSOMqWcjmM_8XAIgDaSk3Qufyd0_6r9Dm9A8RQbFco-FdTBulq7bvRGoBC4";
|
||||
DValue unsupported;
|
||||
unsupported["operation"] = "encrypt";
|
||||
unsupported["algorithm"] = "ES256";
|
||||
DValue list_header;
|
||||
list_header.set_array();
|
||||
DValue list_item;
|
||||
list_item = "not-an-object";
|
||||
list_header.push(list_item);
|
||||
DValue list_request = sign_request;
|
||||
list_request["protected_header"] = list_header;
|
||||
DValue control_request = sign_request;
|
||||
control_request["claims"]["bad"] = String("control\nbyte");
|
||||
check("crypto_operation() generates an ES256 key", key["ok"].to_bool() && key["private_jwk"]["kty"].to_string() == "EC" && key["private_jwk"]["crv"].to_string() == "P-256" && key["public_jwk"]["d"].to_string() == "" && key["kid"].to_string() == key["thumbprint"].to_string(), key["kid"].to_string());
|
||||
check("crypto_operation() signs ES256 JWTs", signed_result["ok"].to_bool() && parts.size() == 3 && parts[2].size() == 86 && decoded_header.find("ES256") != String::npos, signed_result["error"].to_string());
|
||||
check("crypto_operation() rejects invalid signing requests", !crypto_operation(wrong_request)["ok"].to_bool() && !crypto_operation(malformed_request)["ok"].to_bool() && !crypto_operation(list_request)["ok"].to_bool() && crypto_operation(control_request)["error"].to_string() == "invalid_request", "signing validation");
|
||||
check("crypto_operation() parses an ES256 COSE key", cose["ok"].to_bool(), cose["error"].to_string());
|
||||
check("crypto_operation() verifies ES256 DER signatures", crypto_operation(verify_request)["ok"].to_bool() && crypto_operation(verify_request)["valid"].to_bool(), crypto_operation(verify_request)["error"].to_string());
|
||||
check("crypto_operation() rejects unsupported operations", crypto_operation(unsupported)["error"].to_string() == "unsupported_operation", crypto_operation(unsupported)["error"].to_string());
|
||||
site_tests_summary(passed, failed, skipped, "Structured crypto tests generate ephemeral P-256 keys and retain no key material.");
|
||||
site_tests_page_end();
|
||||
}
|
||||
@@ -8,6 +8,7 @@ markdown.uce|Markdown|Markdown parsing, rendering, and component hooks.|http sui
|
||||
units.uce|Units|unit_call(), lifecycle hooks, and unit metadata.|http suite uce public|Units|1|1
|
||||
websockets.ws.uce|WebSockets|Browser-driven WebSocket helper checks.|http suite uce public websocket|WebSockets|1|1
|
||||
io.uce|Filesystem|Filesystem helpers that are restricted outside trusted networks.|http suite uce internal|Filesystem|1|1
|
||||
crypto_operation.uce|Structured Crypto|Algorithm-selected key generation and JWT signing coverage.|http suite uce public crypto|Structured crypto operations|1|1
|
||||
sqlite.uce|SQLite|SQLite connector with prepared named parameters and DValue rows.|http suite uce internal sqlite|SQLite|1|1
|
||||
zip.uce|ZIP|Archive helpers that create and extract temporary server-side files.|http suite uce internal|ZIP|1|1
|
||||
services.uce|Sockets And Services|Network/service helpers that are restricted outside trusted networks.|http suite uce internal|Sockets And Services|1|1
|
||||
|
||||
@@ -31,6 +31,8 @@ RENDER(Request& context)
|
||||
};
|
||||
|
||||
String nested = preprocessor_nested_literal();
|
||||
String trailing_backslash = "C:\\Users\\";
|
||||
String escaped_quote = "before\"after";
|
||||
|
||||
site_tests_page_start("Preprocessor", "Regression coverage for literal output rewriting and parser edge cases.");
|
||||
?>
|
||||
@@ -43,6 +45,8 @@ RENDER(Request& context)
|
||||
check("raw string terminator in nested literal", contains(nested, "nested )\" marker"), nested);
|
||||
check("entrypoint @fragment attribute captures output", context.call["fragments"]["preprocessor-test"].to_string() == "fragment attr once", context.call["fragments"]["preprocessor-test"].to_string());
|
||||
check("inline code island after dangerous literal", true, "parser returned to C++ after rendering literal content containing )\"");
|
||||
check("quote scanner handles trailing escaped backslash", trailing_backslash == "C:\\Users\\", trailing_backslash);
|
||||
check("quote scanner retains escaped quote content", escaped_quote == "before\"after", escaped_quote);
|
||||
|
||||
site_tests_summary(passed, failed, skipped, "Literal content containing the C++ raw-string terminator sequence must compile and render unchanged.");
|
||||
site_tests_page_end();
|
||||
|
||||
+43
-5
@@ -28,14 +28,20 @@ RENDER(Request& context)
|
||||
u64 sockfd = socket_connect("127.0.0.1", 80);
|
||||
if(sockfd != 0)
|
||||
{
|
||||
u64 closed_handle = socket_connect("127.0.0.1", 80);
|
||||
bool opaque_handles = sockfd == 1 && closed_handle == 2;
|
||||
socket_close(closed_handle);
|
||||
socket_close(closed_handle);
|
||||
socket_close(999999);
|
||||
bool closed_write_rejected = !socket_write(closed_handle, "invalid");
|
||||
bool write_ok = socket_write(sockfd, "GET /tests/index.uce HTTP/1.0\r\nHost: uce.openfu.com\r\n\r\n");
|
||||
String response = socket_read(sockfd, 4096, 2);
|
||||
socket_close(sockfd);
|
||||
bool has_nul = response.find(String("\0", 1)) != String::npos;
|
||||
mark(
|
||||
"socket_connect() / socket_write() / socket_read()",
|
||||
(write_ok && response.find("200 OK") != String::npos && !has_nul) ? "pass" : "fail",
|
||||
response.substr(0, response.length() > 220 ? 220 : response.length()) + (has_nul ? " [unexpected NUL]" : "")
|
||||
"socket handles reject invalid and closed indices",
|
||||
(opaque_handles && closed_write_rejected && write_ok && response.find("200 OK") != String::npos && !has_nul) ? "pass" : "fail",
|
||||
"handles=" + std::to_string(sockfd) + "/" + std::to_string(closed_handle) + " " + response.substr(0, response.length() > 180 ? 180 : response.length()) + (has_nul ? " [unexpected NUL]" : "")
|
||||
);
|
||||
}
|
||||
else
|
||||
@@ -90,10 +96,25 @@ RENDER(Request& context)
|
||||
StringMap mem_multi = memcache_get_multiple(memfd, {"site-tests-key", "site-tests-key2"});
|
||||
memcache_delete(memfd, "site-tests-key");
|
||||
memcache_delete(memfd, "site-tests-key2");
|
||||
String large_mem_value(200 * 1024, 'x');
|
||||
large_mem_value.replace(1024, 7, "\r\nEND\r\n");
|
||||
bool large_mem_stored = memcache_set(memfd, "site-tests-large", large_mem_value);
|
||||
String large_mem_loaded = memcache_get(memfd, "site-tests-large");
|
||||
String second_large_mem_value(160 * 1024, 'y');
|
||||
bool second_large_mem_stored = memcache_set(memfd, "site-tests-large-2", second_large_mem_value);
|
||||
StringMap large_mem_multi = memcache_get_multiple(memfd, {"site-tests-large", "site-tests-large-2"});
|
||||
String no_reply = memcache_command(memfd, "set site-tests-noreply 0 60 5 noreply\r\nvalue");
|
||||
String no_reply_value = memcache_get(memfd, "site-tests-noreply");
|
||||
String stats_mem = memcache_command(memfd, "stats");
|
||||
String stats_reset_mem = memcache_command(memfd, "stats reset");
|
||||
String after_stats_reset = memcache_get(memfd, "site-tests-noreply");
|
||||
bool large_mem_deleted = memcache_delete(memfd, "site-tests-large");
|
||||
bool second_large_mem_deleted = memcache_delete(memfd, "site-tests-large-2");
|
||||
bool no_reply_deleted = memcache_delete(memfd, "site-tests-noreply");
|
||||
mark(
|
||||
"memcache_connect() / memcache_command() / memcache_get_multiple()",
|
||||
mem_value == "value-1" && contains(raw_mem, "VALUE site-tests-key") && mem_multi["site-tests-key"] == "value-1" && mem_multi["site-tests-key2"] == "value-2" ? "pass" : "fail",
|
||||
"memcache value=" + mem_value + " raw=" + raw_mem + " multi=" + var_dump(mem_multi)
|
||||
mem_value == "value-1" && contains(raw_mem, "VALUE site-tests-key") && mem_multi["site-tests-key"] == "value-1" && mem_multi["site-tests-key2"] == "value-2" && large_mem_stored && large_mem_loaded == large_mem_value && second_large_mem_stored && large_mem_multi["site-tests-large"] == large_mem_value && large_mem_multi["site-tests-large-2"] == second_large_mem_value && no_reply == "" && no_reply_value == "value" && contains(stats_mem, "STAT ") && trim(stats_reset_mem) == "RESET" && after_stats_reset == "value" && large_mem_deleted && second_large_mem_deleted && no_reply_deleted ? "pass" : "fail",
|
||||
"memcache value=" + mem_value + " raw=" + raw_mem + " multi=" + var_dump(mem_multi) + " large=" + std::to_string(large_mem_loaded.size()) + " multi-large=" + std::to_string(large_mem_multi["site-tests-large"].size() + large_mem_multi["site-tests-large-2"].size()) + " noreply=" + no_reply_value + " stats=" + std::to_string(stats_mem.size()) + "/" + trim(stats_reset_mem) + " after-reset=" + after_stats_reset + " deleted=" + (large_mem_deleted && second_large_mem_deleted && no_reply_deleted ? "true" : "false")
|
||||
);
|
||||
}
|
||||
|
||||
@@ -115,6 +136,15 @@ RENDER(Request& context)
|
||||
parsed_underscore.find("'Ada'") != String::npos && parsed_underscore.find("_name") == String::npos ? "pass" : "fail",
|
||||
parsed_underscore
|
||||
);
|
||||
StringMap quoted_params;
|
||||
quoted_params["name_looking_text"] = "SUBSTITUTED";
|
||||
String escaped_quote_query = "SELECT 'a\\':name_looking_text'";
|
||||
String parsed_escaped_quote = placeholder_guard.parse_query_parameters(escaped_quote_query, quoted_params);
|
||||
mark(
|
||||
"mysql named placeholders stay opaque after escaped quotes",
|
||||
parsed_escaped_quote.find(":name_looking_text") != String::npos && parsed_escaped_quote.find("SUBSTITUTED") == String::npos ? "pass" : "fail",
|
||||
parsed_escaped_quote
|
||||
);
|
||||
MySQL unavailable_mysql;
|
||||
unavailable_mysql.connect("127.0.0.1", "__uce_intentionally_missing__", "not-a-real-password");
|
||||
mark(
|
||||
@@ -158,6 +188,14 @@ RENDER(Request& context)
|
||||
inserted == 2 && updated == 1 && selected == 0 ? "pass" : "fail",
|
||||
"inserted=" + std::to_string((u64)inserted) + " updated=" + std::to_string((u64)updated) + " selected=" + std::to_string((u64)selected)
|
||||
);
|
||||
mysql.query("INSERT INTO uce_affected_rows_test VALUES (1,30)");
|
||||
u32 duplicate_error_code = mysql._preload_next_error_code;
|
||||
String duplicate_error = mysql.error();
|
||||
mark(
|
||||
"mysql query failures retain server diagnostics",
|
||||
duplicate_error_code > 1 && duplicate_error != "" && duplicate_error != "Unknown server error" ? "pass" : "fail",
|
||||
"code=" + std::to_string((u64)duplicate_error_code) + " error=" + duplicate_error
|
||||
);
|
||||
}
|
||||
DValue mysql_perf = request_perf();
|
||||
String mysql_operations = json_encode(mysql_perf["mysql_operations"]);
|
||||
|
||||
@@ -18,6 +18,16 @@ RENDER(Request& context)
|
||||
auto unit_paths = units_list();
|
||||
DValue info = unit_info("call_helpers.uce");
|
||||
DValue relative_info = unit_info("components/../relative-child.uce");
|
||||
String outside_unit = "/tmp/uce-site-tests-outside-unit.uce";
|
||||
file_put_contents(outside_unit, "RENDER(Request& context) {}");
|
||||
DValue outside_info = unit_info(outside_unit);
|
||||
file_unlink(outside_unit);
|
||||
String generated_dir = path_join(context.params["UCE_BIN_DIRECTORY"], "site-tests-generated");
|
||||
mkdir(generated_dir);
|
||||
String generated_unit = path_join(generated_dir, "generated.uce");
|
||||
file_put_contents(generated_unit, "RENDER(Request& context) {}");
|
||||
DValue generated_info = unit_info(generated_unit);
|
||||
file_unlink(generated_unit);
|
||||
|
||||
ob_start();
|
||||
unit_call("call_helpers.uce", "emit_marker");
|
||||
@@ -32,6 +42,8 @@ RENDER(Request& context)
|
||||
check("units_list()", unit_paths.size() > 0, "count=" + std::to_string(unit_paths.size()));
|
||||
check("unit_info()", info["path"].to_string() != "", json_encode(info));
|
||||
check("compiler canonicalizes relative unit paths", relative_info["path"].to_string().find("/../") == String::npos && str_ends_with(relative_info["path"].to_string(), "/site/tests/relative-child.uce"), json_encode(relative_info));
|
||||
check("compiler rejects units outside document root", outside_info["path"].to_string() == "", json_encode(outside_info));
|
||||
check("compiler accepts generated units in BIN_DIRECTORY", generated_info["path"].to_string() == generated_unit, json_encode(generated_info));
|
||||
check("unit_compile()", unit_compile("call_helpers.uce"), "call_helpers.uce");
|
||||
check("unit_call()", call_output.find("UNIT_CALL_EXPORT_OK") != String::npos, call_output);
|
||||
check("unit_render()", render_output.find("data-unit-render=\"ok\"") != String::npos, render_output);
|
||||
|
||||
@@ -362,6 +362,7 @@ FastCGIServer::listen(const std::string& local_path)
|
||||
close(server_socket);
|
||||
throw;
|
||||
}
|
||||
server_socket_types[server_socket] = 'F';
|
||||
return server_socket;
|
||||
}
|
||||
|
||||
@@ -400,6 +401,23 @@ FastCGIServer::close_http_listeners()
|
||||
}
|
||||
}
|
||||
|
||||
void
|
||||
FastCGIServer::close_listeners_except(char type)
|
||||
{
|
||||
for(std::vector<int>::iterator it = server_sockets.begin(); it != server_sockets.end();)
|
||||
{
|
||||
int socket_handle = *it;
|
||||
if(server_socket_types[socket_handle] != type)
|
||||
{
|
||||
close(socket_handle);
|
||||
server_socket_types.erase(socket_handle);
|
||||
it = server_sockets.erase(it);
|
||||
continue;
|
||||
}
|
||||
++it;
|
||||
}
|
||||
}
|
||||
|
||||
bool
|
||||
FastCGIServer::is_http_like_type(char type)
|
||||
{
|
||||
@@ -672,10 +690,12 @@ FastCGIServer::process(int timeout_ms)
|
||||
Connection* doomed_connection = it->second;
|
||||
client_sockets.erase(it++);
|
||||
delete doomed_connection;
|
||||
if(calls_until_termination != -1 && client_sockets.size() == 0)
|
||||
{
|
||||
if(calls_until_termination > 0)
|
||||
calls_until_termination -= 1;
|
||||
if(calls_until_termination <= 0)
|
||||
if(calls_until_termination == 0)
|
||||
{
|
||||
close_listeners_except(0);
|
||||
if(client_sockets.empty())
|
||||
exit(0);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -96,6 +96,7 @@ public:
|
||||
std::map<int, Connection*> client_sockets;
|
||||
|
||||
void close_http_listeners();
|
||||
void close_listeners_except(char type);
|
||||
void read_fgci(Connection&);
|
||||
static bool is_http_like_type(char type);
|
||||
Connection* open_client_connection(int server_socket, int client_socket);
|
||||
|
||||
@@ -74,8 +74,14 @@ void compiler_code_state_consume(CompilerCodeState& state, String& buffer, const
|
||||
|
||||
if(state.inside_quote)
|
||||
{
|
||||
if(state.quote_char == c && (i == 0 || content[i-1] != '\\'))
|
||||
state.inside_quote = false;
|
||||
if(state.quote_char == c)
|
||||
{
|
||||
u32 backslashes = 0;
|
||||
for(u32 j = i; j > 0 && content[j - 1] == '\\'; j -= 1)
|
||||
backslashes += 1;
|
||||
if(backslashes % 2 == 0)
|
||||
state.inside_quote = false;
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
|
||||
+39
-6
@@ -646,7 +646,10 @@ int compiler_open_lock_file(String file_name, String purpose, bool nonblocking =
|
||||
(void)purpose;
|
||||
auto lock_dir = dirname(file_name);
|
||||
if(lock_dir != "")
|
||||
mkdir(lock_dir);
|
||||
{
|
||||
std::error_code error;
|
||||
std::filesystem::create_directories(lock_dir, error);
|
||||
}
|
||||
int fdlock = open(file_name.c_str(), O_RDWR | O_CREAT, 0666);
|
||||
if(fdlock == -1 && (errno == EACCES || errno == EPERM))
|
||||
fdlock = open(file_name.c_str(), O_RDONLY | O_CLOEXEC);
|
||||
@@ -676,7 +679,10 @@ int compiler_open_lock_file_bounded(String file_name, String purpose, CompilerDe
|
||||
return(compiler_open_lock_file(file_name, purpose));
|
||||
auto lock_dir = dirname(file_name);
|
||||
if(lock_dir != "")
|
||||
mkdir(lock_dir);
|
||||
{
|
||||
std::error_code error;
|
||||
std::filesystem::create_directories(lock_dir, error);
|
||||
}
|
||||
int fdlock = open(file_name.c_str(), O_RDWR | O_CREAT, 0666);
|
||||
if(fdlock == -1 && (errno == EACCES || errno == EPERM))
|
||||
fdlock = open(file_name.c_str(), O_RDONLY | O_CLOEXEC);
|
||||
@@ -734,14 +740,41 @@ static void compiler_mark_source_generation_nonblocking(Request* context)
|
||||
String compiler_normalize_unit_path(Request* context, String file_name)
|
||||
{
|
||||
file_name = trim(file_name);
|
||||
if(file_name == "")
|
||||
if(file_name == "" || !context || !context->server)
|
||||
return("");
|
||||
if(file_name[0] != '/')
|
||||
file_name = expand_path(file_name, context->server->config["COMPILER_SYS_PATH"]);
|
||||
String canonical = path_real(file_name);
|
||||
if(canonical != "")
|
||||
return(canonical);
|
||||
return(file_name);
|
||||
if(canonical == "")
|
||||
{
|
||||
std::error_code error;
|
||||
canonical = std::filesystem::weakly_canonical(file_name, error).string();
|
||||
if(error || canonical == "")
|
||||
return("");
|
||||
}
|
||||
StringList allowed_roots = {
|
||||
context->params["DOCUMENT_ROOT"],
|
||||
context->server->config["PRECOMPILE_FILES_IN"],
|
||||
context->server->config["HTTP_DOCUMENT_ROOT"],
|
||||
path_join(context->server->config["COMPILER_SYS_PATH"], context->server->config["SITE_DIRECTORY"]),
|
||||
context->server->config["BIN_DIRECTORY"]
|
||||
};
|
||||
for(String allowed_root : allowed_roots)
|
||||
{
|
||||
allowed_root = trim(allowed_root);
|
||||
if(allowed_root == "")
|
||||
continue;
|
||||
if(allowed_root[0] != '/')
|
||||
allowed_root = expand_path(allowed_root, context->server->config["COMPILER_SYS_PATH"]);
|
||||
allowed_root = path_real(allowed_root);
|
||||
if(allowed_root == "")
|
||||
continue;
|
||||
if(allowed_root[allowed_root.length() - 1] != '/')
|
||||
allowed_root += "/";
|
||||
if(canonical + "/" == allowed_root || str_starts_with(canonical, allowed_root))
|
||||
return(canonical);
|
||||
}
|
||||
return("");
|
||||
}
|
||||
|
||||
bool compiler_is_known_unit_file(String file_name)
|
||||
|
||||
@@ -140,6 +140,8 @@ bool contains(String haystack, String needle)
|
||||
|
||||
String replace(String s, String search, String replace_with)
|
||||
{
|
||||
if(search == "")
|
||||
return(s);
|
||||
s64 last_spos = 0;
|
||||
auto spos = s.find(search);
|
||||
if(spos == std::string::npos)
|
||||
@@ -791,7 +793,7 @@ DValue array_merge(DValue a, DValue b)
|
||||
StringList split_utf8(String s, bool compound_characters)
|
||||
{
|
||||
StringList result;
|
||||
auto len = s.size();
|
||||
s64 len = (s64)s.size();
|
||||
String codepoint = "";
|
||||
for(s64 i = 0; i < len; i++)
|
||||
{
|
||||
@@ -800,13 +802,13 @@ StringList split_utf8(String s, bool compound_characters)
|
||||
{
|
||||
codepoint = "";
|
||||
codepoint.append(1, c);
|
||||
if(is_bit_set(c, 6))
|
||||
if(is_bit_set(c, 6) && i + 1 < len)
|
||||
{
|
||||
codepoint.append(1, s[++i]);
|
||||
if(is_bit_set(c, 5))
|
||||
if(is_bit_set(c, 5) && i + 1 < len)
|
||||
{
|
||||
codepoint.append(1, s[++i]);
|
||||
if(is_bit_set(c, 4))
|
||||
if(is_bit_set(c, 4) && i + 1 < len)
|
||||
{
|
||||
codepoint.append(1, s[++i]);
|
||||
}
|
||||
@@ -837,7 +839,7 @@ StringList split_utf8(String s, bool compound_characters)
|
||||
join_next = true;
|
||||
last_was_regional = false;
|
||||
}
|
||||
else if(s[0] == '\xF0' && s[1] == '\x9F' && s[2] == '\x87' && s[3] >= '\xA6' && s[3] <= '\xBF') // Regional indicator letters
|
||||
else if(s.size() == 4 && s[0] == '\xF0' && s[1] == '\x9F' && s[2] == '\x87' && s[3] >= '\xA6' && s[3] <= '\xBF') // Regional indicator letters
|
||||
{
|
||||
if(last_was_regional)
|
||||
{
|
||||
@@ -850,7 +852,7 @@ StringList split_utf8(String s, bool compound_characters)
|
||||
last_was_regional = true;
|
||||
}
|
||||
}
|
||||
else if(s[0] == '\xEF' && s[1] == '\xB8' && s[2] >= '\x80' && s[2] <= '\x8F') // Variation selector
|
||||
else if(s.size() == 3 && s[0] == '\xEF' && s[1] == '\xB8' && s[2] >= '\x80' && s[2] <= '\x8F') // Variation selector
|
||||
{
|
||||
compound_result[compound_result.size()-1] += s;
|
||||
last_was_regional = false;
|
||||
|
||||
@@ -41,6 +41,7 @@ void SHA1Final(unsigned char digest[20], SHA1_CTX* context);
|
||||
#include <string.h>
|
||||
#include <sys/types.h> /* for u_int*_t */
|
||||
#include "hash.h"
|
||||
#include "uri.h"
|
||||
|
||||
#ifndef BYTE_ORDER
|
||||
#if (BSD >= 199103)
|
||||
@@ -422,8 +423,69 @@ bool crypto_equal_native(String a, String b)
|
||||
return(diff == 0);
|
||||
}
|
||||
|
||||
static bool uce_crypto_utf8_string(String value)
|
||||
{
|
||||
for(size_t i = 0; i < value.size();)
|
||||
{
|
||||
u8 c = (u8)value[i];
|
||||
if(c < 0x80) { i++; continue; }
|
||||
size_t need = c >= 0xC2 && c <= 0xDF ? 1 : (c >= 0xE0 && c <= 0xEF ? 2 : (c >= 0xF0 && c <= 0xF4 ? 3 : 0));
|
||||
if(need == 0 || i + need >= value.size()) return(false);
|
||||
for(size_t j = 1; j <= need; j++) if(((u8)value[i + j] & 0xC0) != 0x80) return(false);
|
||||
u8 second = (u8)value[i + 1];
|
||||
if((c == 0xE0 && second < 0xA0) || (c == 0xED && second >= 0xA0) || (c == 0xF0 && second < 0x90) || (c == 0xF4 && second >= 0x90)) return(false);
|
||||
i += need + 1;
|
||||
}
|
||||
return(true);
|
||||
}
|
||||
|
||||
static bool uce_crypto_utf8_json_string(String value)
|
||||
{
|
||||
if(!uce_crypto_utf8_string(value)) return(false);
|
||||
for(unsigned char c : value) if(c < 0x20) return(false);
|
||||
return(true);
|
||||
}
|
||||
|
||||
static bool uce_crypto_value_valid(const DValue& value, size_t depth, size_t& nodes, size_t& bytes)
|
||||
{
|
||||
if(depth > 16 || ++nodes > 256) return(false);
|
||||
const DValue& item = value.deref();
|
||||
if(item.type == 'S')
|
||||
{
|
||||
bytes += item._String.size();
|
||||
return(bytes <= 32768 && uce_crypto_utf8_json_string(item._String));
|
||||
}
|
||||
if(item.type == 'F') return(std::isfinite(item._float));
|
||||
if(item.type == 'B') return(true);
|
||||
if(item.type != 'M') return(false);
|
||||
bool list = item.is_list();
|
||||
for(const auto& child : item._map)
|
||||
{
|
||||
if(!list)
|
||||
{
|
||||
bytes += child.first.size();
|
||||
if(bytes > 32768 || !uce_crypto_utf8_json_string(child.first)) return(false);
|
||||
}
|
||||
if(!uce_crypto_value_valid(child.second, depth + 1, nodes, bytes)) return(false);
|
||||
}
|
||||
return(true);
|
||||
}
|
||||
|
||||
bool crypto_operation_request_valid(DValue request)
|
||||
{
|
||||
const DValue& root = request.deref();
|
||||
if(root.type != 'M' || root.is_list()) return(false);
|
||||
size_t nodes = 0, bytes = 0;
|
||||
return(uce_crypto_value_valid(root, 0, nodes, bytes));
|
||||
}
|
||||
|
||||
#ifndef __UCE_WASM_CORE__
|
||||
#include <openssl/bn.h>
|
||||
#include <openssl/core_names.h>
|
||||
#include <openssl/ecdsa.h>
|
||||
#include <openssl/evp.h>
|
||||
#include <openssl/params.h>
|
||||
#include <openssl/param_build.h>
|
||||
#include <openssl/rand.h>
|
||||
|
||||
namespace {
|
||||
@@ -525,4 +587,385 @@ bool password_needs_rehash_native(String encoded)
|
||||
String salt, digest;
|
||||
return(!uce_password_parts(encoded, n, r, p, salt, digest) || n != UCE_PASSWORD_SCRYPT_N || r != UCE_PASSWORD_SCRYPT_R || p != UCE_PASSWORD_SCRYPT_P);
|
||||
}
|
||||
|
||||
namespace {
|
||||
static constexpr size_t UCE_ES256_COORDINATE_BYTES = 32;
|
||||
static constexpr size_t UCE_ES256_SIGNATURE_BYTES = 64;
|
||||
static constexpr size_t UCE_ES256_JSON_MAX = 16 * 1024;
|
||||
static constexpr size_t UCE_ES256_VALUE_MAX = 256;
|
||||
static constexpr size_t UCE_ES256_DEPTH_MAX = 16;
|
||||
static constexpr size_t UCE_ES256_COORDINATE_BASE64URL_MAX = 43;
|
||||
static constexpr size_t UCE_CBOR_MAX_BYTES = 16 * 1024;
|
||||
static constexpr size_t UCE_CBOR_MAX_BASE64URL = 21846;
|
||||
static constexpr size_t UCE_ES256_DER_MAX_BYTES = 144;
|
||||
static constexpr size_t UCE_ES256_DER_BASE64URL_MAX = 192;
|
||||
|
||||
struct UcePkeyDeleter { void operator()(EVP_PKEY* value) const { EVP_PKEY_free(value); } };
|
||||
struct UcePkeyCtxDeleter { void operator()(EVP_PKEY_CTX* value) const { EVP_PKEY_CTX_free(value); } };
|
||||
struct UceEcdsaSigDeleter { void operator()(ECDSA_SIG* value) const { ECDSA_SIG_free(value); } };
|
||||
struct UceBnDeleter { void operator()(BIGNUM* value) const { BN_clear_free(value); } };
|
||||
struct UceParamBldDeleter { void operator()(OSSL_PARAM_BLD* value) const { OSSL_PARAM_BLD_free(value); } };
|
||||
struct UceParamsDeleter { void operator()(OSSL_PARAM* value) const { OSSL_PARAM_free(value); } };
|
||||
|
||||
static String uce_base64url_encode(const unsigned char* bytes, size_t size)
|
||||
{
|
||||
String encoded = base64_encode(String((const char*)bytes, size));
|
||||
encoded = replace(replace(encoded, "+", "-"), "/", "_");
|
||||
while(!encoded.empty() && encoded.back() == '=') encoded.pop_back();
|
||||
return(encoded);
|
||||
}
|
||||
|
||||
static bool uce_base64url_decode(String encoded, String& decoded, size_t max_encoded)
|
||||
{
|
||||
if(encoded.empty() || encoded.size() > max_encoded || encoded.find('=') != String::npos || encoded.size() % 4 == 1)
|
||||
return(false);
|
||||
for(char c : encoded)
|
||||
if(!(c >= 'A' && c <= 'Z') && !(c >= 'a' && c <= 'z') && !(c >= '0' && c <= '9') && c != '-' && c != '_')
|
||||
return(false);
|
||||
String padded = replace(replace(encoded, "-", "+"), "_", "/");
|
||||
while(padded.size() % 4) padded += "=";
|
||||
bool ok = false;
|
||||
decoded = base64_decode(padded, ok);
|
||||
return(ok && uce_base64url_encode((const unsigned char*)decoded.data(), decoded.size()) == encoded);
|
||||
}
|
||||
|
||||
static bool uce_es256_json_value(const DValue& value, size_t depth, size_t& values, size_t& bytes)
|
||||
{
|
||||
if(depth > UCE_ES256_DEPTH_MAX || ++values > UCE_ES256_VALUE_MAX)
|
||||
return(false);
|
||||
const DValue& item = value.deref();
|
||||
if(item.type == 'S')
|
||||
return((bytes += item._String.size()) <= UCE_ES256_JSON_MAX);
|
||||
if(item.type == 'F' || item.type == 'B')
|
||||
return(true);
|
||||
if(item.type != 'M')
|
||||
return(false);
|
||||
for(const auto& child : item._map)
|
||||
{
|
||||
if((bytes += child.first.size()) > UCE_ES256_JSON_MAX || !uce_es256_json_value(child.second, depth + 1, values, bytes))
|
||||
return(false);
|
||||
}
|
||||
return(true);
|
||||
}
|
||||
|
||||
static bool uce_es256_json_map(const DValue& value)
|
||||
{
|
||||
if(value.deref().type != 'M' || value.deref().is_list())
|
||||
return(false);
|
||||
size_t values = 0, bytes = 0;
|
||||
return(uce_es256_json_value(value, 0, values, bytes));
|
||||
}
|
||||
|
||||
static bool uce_es256_jwk_string(const DValue& jwk, const String& field, String& value)
|
||||
{
|
||||
const DValue* found = jwk.key(field);
|
||||
if(!found)
|
||||
return(false);
|
||||
const DValue& item = found->deref();
|
||||
if(item.type != 'S' || item._String.empty() || item._String.size() > 32768)
|
||||
return(false);
|
||||
value = item._String;
|
||||
return(true);
|
||||
}
|
||||
|
||||
static std::unique_ptr<EVP_PKEY, UcePkeyDeleter> uce_es256_key_from_jwk(const DValue& jwk)
|
||||
{
|
||||
if(jwk.deref().type != 'M')
|
||||
return(nullptr);
|
||||
String kty, crv, x64, y64, d64, x, y, d;
|
||||
if(!uce_es256_jwk_string(jwk, "kty", kty) || !uce_es256_jwk_string(jwk, "crv", crv) || !uce_es256_jwk_string(jwk, "x", x64) || !uce_es256_jwk_string(jwk, "y", y64) || !uce_es256_jwk_string(jwk, "d", d64) ||
|
||||
kty != "EC" || crv != "P-256" || !uce_base64url_decode(x64, x, UCE_ES256_COORDINATE_BASE64URL_MAX) || !uce_base64url_decode(y64, y, UCE_ES256_COORDINATE_BASE64URL_MAX) || !uce_base64url_decode(d64, d, UCE_ES256_COORDINATE_BASE64URL_MAX) ||
|
||||
x.size() != UCE_ES256_COORDINATE_BYTES || y.size() != UCE_ES256_COORDINATE_BYTES || d.size() != UCE_ES256_COORDINATE_BYTES)
|
||||
return(nullptr);
|
||||
unsigned char public_key[65];
|
||||
public_key[0] = 4;
|
||||
memcpy(public_key + 1, x.data(), x.size());
|
||||
memcpy(public_key + 33, y.data(), y.size());
|
||||
std::unique_ptr<BIGNUM, UceBnDeleter> private_bn(BN_bin2bn((const unsigned char*)d.data(), d.size(), 0));
|
||||
std::unique_ptr<OSSL_PARAM_BLD, UceParamBldDeleter> builder(OSSL_PARAM_BLD_new());
|
||||
if(!private_bn || !builder || OSSL_PARAM_BLD_push_utf8_string(builder.get(), OSSL_PKEY_PARAM_GROUP_NAME, "prime256v1", 0) <= 0 ||
|
||||
OSSL_PARAM_BLD_push_octet_string(builder.get(), OSSL_PKEY_PARAM_PUB_KEY, public_key, sizeof(public_key)) <= 0 ||
|
||||
OSSL_PARAM_BLD_push_BN(builder.get(), OSSL_PKEY_PARAM_PRIV_KEY, private_bn.get()) <= 0)
|
||||
return(nullptr);
|
||||
std::unique_ptr<OSSL_PARAM, UceParamsDeleter> params(OSSL_PARAM_BLD_to_param(builder.get()));
|
||||
std::unique_ptr<EVP_PKEY_CTX, UcePkeyCtxDeleter> ctx(EVP_PKEY_CTX_new_from_name(0, "EC", 0));
|
||||
EVP_PKEY* raw = 0;
|
||||
if(!params || !ctx || EVP_PKEY_fromdata_init(ctx.get()) <= 0 || EVP_PKEY_fromdata(ctx.get(), &raw, EVP_PKEY_KEYPAIR, params.get()) <= 0)
|
||||
return(nullptr);
|
||||
std::unique_ptr<EVP_PKEY, UcePkeyDeleter> key(raw);
|
||||
std::unique_ptr<EVP_PKEY_CTX, UcePkeyCtxDeleter> check(EVP_PKEY_CTX_new(key.get(), 0));
|
||||
return(check && EVP_PKEY_public_check(check.get()) > 0 && EVP_PKEY_private_check(check.get()) > 0 && EVP_PKEY_pairwise_check(check.get()) > 0 ? std::move(key) : nullptr);
|
||||
}
|
||||
|
||||
static bool uce_es256_key_coordinates(EVP_PKEY* key, String& x, String& y, String& d)
|
||||
{
|
||||
unsigned char public_key[65]; size_t public_key_size = sizeof(public_key);
|
||||
BIGNUM* private_bn = 0;
|
||||
if(EVP_PKEY_get_octet_string_param(key, OSSL_PKEY_PARAM_PUB_KEY, public_key, sizeof(public_key), &public_key_size) <= 0 || public_key_size != sizeof(public_key) || public_key[0] != 4 ||
|
||||
EVP_PKEY_get_bn_param(key, OSSL_PKEY_PARAM_PRIV_KEY, &private_bn) <= 0)
|
||||
return(false);
|
||||
std::unique_ptr<BIGNUM, UceBnDeleter> private_key(private_bn);
|
||||
if(BN_bn2binpad(private_key.get(), (unsigned char*)d.data(), UCE_ES256_COORDINATE_BYTES) != UCE_ES256_COORDINATE_BYTES)
|
||||
return(false);
|
||||
x.assign((const char*)public_key + 1, UCE_ES256_COORDINATE_BYTES);
|
||||
y.assign((const char*)public_key + 33, UCE_ES256_COORDINATE_BYTES);
|
||||
return(true);
|
||||
}
|
||||
|
||||
enum class UceCborKind { Unsigned, Negative, Bytes, Text, Array, Map };
|
||||
struct UceCbor { UceCborKind kind; u64 number = 0; String bytes; std::vector<UceCbor> items; };
|
||||
static constexpr size_t UCE_CBOR_MAX_NODES = 256, UCE_CBOR_MAX_DEPTH = 16;
|
||||
|
||||
static bool uce_cbor_uint(const String& in, size_t& p, u8 ai, u64& n)
|
||||
{
|
||||
if(ai < 24) { n = ai; return(true); }
|
||||
size_t count = ai == 24 ? 1 : ai == 25 ? 2 : ai == 26 ? 4 : ai == 27 ? 8 : 0;
|
||||
if(!count || p > in.size() || count > in.size() - p) return(false);
|
||||
n = 0;
|
||||
for(size_t i = 0; i < count; i++)
|
||||
{
|
||||
if(n > (UINT64_MAX - (u8)in[p + i]) / 256) return(false);
|
||||
n = n * 256 + (u8)in[p + i];
|
||||
}
|
||||
p += count;
|
||||
return((count == 1 && n >= 24) || (count == 2 && n > 0xff) || (count == 4 && n > 0xffff) || (count == 8 && n > 0xffffffff));
|
||||
}
|
||||
|
||||
static bool uce_cbor_equal(const UceCbor& left, const UceCbor& right)
|
||||
{
|
||||
if(left.kind != right.kind || left.number != right.number || left.bytes != right.bytes || left.items.size() != right.items.size()) return(false);
|
||||
for(size_t i = 0; i < left.items.size(); i++) if(!uce_cbor_equal(left.items[i], right.items[i])) return(false);
|
||||
return(true);
|
||||
}
|
||||
|
||||
static bool uce_cbor_read(const String& in, size_t& p, UceCbor& out, size_t depth, size_t& nodes)
|
||||
{
|
||||
if(p >= in.size() || depth > UCE_CBOR_MAX_DEPTH || ++nodes > UCE_CBOR_MAX_NODES) return(false);
|
||||
u8 header = (u8)in[p++], major = header >> 5, ai = header & 31;
|
||||
u64 n = 0;
|
||||
if(ai == 31 || !uce_cbor_uint(in, p, ai, n)) return(false);
|
||||
if(major == 0 || major == 1)
|
||||
{
|
||||
out.kind = major ? UceCborKind::Negative : UceCborKind::Unsigned;
|
||||
out.number = n;
|
||||
return(true);
|
||||
}
|
||||
if(major == 2 || major == 3)
|
||||
{
|
||||
if(n > UCE_CBOR_MAX_BYTES || n > in.size() - p || (major == 3 && !uce_crypto_utf8_string(String(in.data() + p, (size_t)n)))) return(false);
|
||||
out.kind = major == 2 ? UceCborKind::Bytes : UceCborKind::Text;
|
||||
out.bytes.assign(in.data() + p, (size_t)n);
|
||||
p += (size_t)n;
|
||||
return(true);
|
||||
}
|
||||
if((major != 4 && major != 5) || n > UCE_CBOR_MAX_NODES) return(false);
|
||||
size_t item_count = (size_t)n * (major == 5 ? 2 : 1);
|
||||
if(item_count > UCE_CBOR_MAX_NODES - nodes) return(false);
|
||||
out.kind = major == 4 ? UceCborKind::Array : UceCborKind::Map;
|
||||
out.items.reserve(item_count);
|
||||
for(size_t i = 0; i < item_count; i++)
|
||||
{
|
||||
UceCbor child;
|
||||
if(!uce_cbor_read(in, p, child, depth + 1, nodes)) return(false);
|
||||
if(major == 5 && !(i & 1))
|
||||
for(size_t previous = 0; previous < out.items.size(); previous += 2)
|
||||
if(uce_cbor_equal(out.items[previous], child)) return(false);
|
||||
out.items.push_back(std::move(child));
|
||||
}
|
||||
return(true);
|
||||
}
|
||||
static DValue uce_cbor_value(const UceCbor& value)
|
||||
{
|
||||
DValue out; switch(value.kind) { case UceCborKind::Unsigned: out["type"]="unsigned"; out["value"]=std::to_string(value.number); break; case UceCborKind::Negative: out["type"]="negative"; out["value"]=value.number==UINT64_MAX?"-18446744073709551616":"-"+std::to_string(value.number+1); break; case UceCborKind::Bytes: out["type"]="bytes"; out["base64url"]=uce_base64url_encode((const unsigned char*)value.bytes.data(),value.bytes.size()); break; case UceCborKind::Text: out["type"]="text"; out["value"]=value.bytes; break; case UceCborKind::Array: out["type"]="array"; out["items"].set_array(); for(const auto& x:value.items) out["items"].push(uce_cbor_value(x)); break; case UceCborKind::Map: out["type"]="map"; out["entries"].set_array(); for(size_t i=0;i<value.items.size();i+=2) { DValue entry; entry["key"]=uce_cbor_value(value.items[i]); entry["value"]=uce_cbor_value(value.items[i+1]); out["entries"].push(entry); } } return out;
|
||||
}
|
||||
static bool uce_cbor_es256(const UceCbor& cose,String& x,String& y)
|
||||
{
|
||||
if(cose.kind!=UceCborKind::Map) return false; const UceCbor *kty=0,*alg=0,*crv=0,*xx=0,*yy=0;
|
||||
for(size_t i=0;i<cose.items.size();i+=2) { const UceCbor& k=cose.items[i]; const UceCbor& v=cose.items[i+1]; if(k.kind!=UceCborKind::Unsigned&&k.kind!=UceCborKind::Negative) continue; bool neg=k.kind==UceCborKind::Negative; if(!neg&&k.number==1) kty=&v; else if(!neg&&k.number==3) alg=&v; else if(neg&&k.number==0) crv=&v; else if(neg&&k.number==1) xx=&v; else if(neg&&k.number==2) yy=&v; }
|
||||
return(kty&&alg&&crv&&xx&&yy&&kty->kind==UceCborKind::Unsigned&&kty->number==2&&alg->kind==UceCborKind::Negative&&alg->number==6&&crv->kind==UceCborKind::Unsigned&&crv->number==1&&xx->kind==UceCborKind::Bytes&&yy->kind==UceCborKind::Bytes&&xx->bytes.size()==32&&yy->bytes.size()==32&&(x=xx->bytes,true)&&(y=yy->bytes,true));
|
||||
}
|
||||
static std::unique_ptr<EVP_PKEY,UcePkeyDeleter> uce_es256_public_key(String x,String y)
|
||||
{
|
||||
if(x.size()!=32||y.size()!=32) return nullptr; unsigned char point[65]={4}; memcpy(point+1,x.data(),32); memcpy(point+33,y.data(),32); OSSL_PARAM p[]={OSSL_PARAM_construct_utf8_string(OSSL_PKEY_PARAM_GROUP_NAME,(char*)"prime256v1",0),OSSL_PARAM_construct_octet_string(OSSL_PKEY_PARAM_PUB_KEY,point,sizeof(point)),OSSL_PARAM_construct_end()}; std::unique_ptr<EVP_PKEY_CTX,UcePkeyCtxDeleter> ctx(EVP_PKEY_CTX_new_from_name(0,"EC",0)); EVP_PKEY* raw=0; if(!ctx||EVP_PKEY_fromdata_init(ctx.get())<=0||EVP_PKEY_fromdata(ctx.get(),&raw,EVP_PKEY_PUBLIC_KEY,p)<=0) return nullptr; std::unique_ptr<EVP_PKEY,UcePkeyDeleter> key(raw); std::unique_ptr<EVP_PKEY_CTX,UcePkeyCtxDeleter> check(EVP_PKEY_CTX_new(key.get(),0)); return check&&EVP_PKEY_public_check(check.get())>0?std::move(key):nullptr;
|
||||
}
|
||||
static bool uce_es256_cose(String encoded,String& x,String& y)
|
||||
{
|
||||
String raw; UceCbor cose; size_t p=0,nodes=0; return uce_base64url_decode(encoded, raw, UCE_CBOR_MAX_BASE64URL) && raw.size() <= UCE_CBOR_MAX_BYTES && uce_cbor_read(raw, p, cose, 0, nodes) && p == raw.size() && uce_cbor_es256(cose, x, y) && uce_es256_public_key(x, y);
|
||||
}
|
||||
|
||||
static DValue uce_es256_jwk(String x, String y, String d = "")
|
||||
{
|
||||
DValue jwk;
|
||||
jwk["kty"] = "EC";
|
||||
jwk["crv"] = "P-256";
|
||||
jwk["x"] = uce_base64url_encode((const unsigned char*)x.data(), x.size());
|
||||
jwk["y"] = uce_base64url_encode((const unsigned char*)y.data(), y.size());
|
||||
if(d != "") jwk["d"] = uce_base64url_encode((const unsigned char*)d.data(), d.size());
|
||||
return(jwk);
|
||||
}
|
||||
|
||||
static String uce_es256_thumbprint(const DValue& public_jwk)
|
||||
{
|
||||
const DValue* x = public_jwk.key("x");
|
||||
const DValue* y = public_jwk.key("y");
|
||||
if(!x || !y)
|
||||
return("");
|
||||
String canonical = "{\"crv\":\"P-256\",\"kty\":\"EC\",\"x\":\"" + x->to_string() + "\",\"y\":\"" + y->to_string() + "\"}";
|
||||
String digest = sha256_native(canonical);
|
||||
return(uce_base64url_encode((const unsigned char*)digest.data(), digest.size()));
|
||||
}
|
||||
}
|
||||
|
||||
static DValue uce_es256_key_create()
|
||||
{
|
||||
std::unique_ptr<EVP_PKEY_CTX, UcePkeyCtxDeleter> ctx(EVP_PKEY_CTX_new_from_name(0, "EC", 0));
|
||||
EVP_PKEY* raw = 0;
|
||||
OSSL_PARAM params[] = { OSSL_PARAM_construct_utf8_string(OSSL_PKEY_PARAM_GROUP_NAME, (char*)"prime256v1", 0), OSSL_PARAM_construct_end() };
|
||||
if(!ctx || EVP_PKEY_keygen_init(ctx.get()) <= 0 || EVP_PKEY_CTX_set_params(ctx.get(), params) <= 0 || EVP_PKEY_generate(ctx.get(), &raw) <= 0)
|
||||
return(DValue());
|
||||
std::unique_ptr<EVP_PKEY, UcePkeyDeleter> key(raw);
|
||||
String x(UCE_ES256_COORDINATE_BYTES, 0), y(UCE_ES256_COORDINATE_BYTES, 0), d(UCE_ES256_COORDINATE_BYTES, 0);
|
||||
if(!uce_es256_key_coordinates(key.get(), x, y, d))
|
||||
return(DValue());
|
||||
DValue result;
|
||||
result["public_jwk"] = uce_es256_jwk(x, y);
|
||||
String kid = uce_es256_thumbprint(result["public_jwk"]);
|
||||
result["public_jwk"]["kid"] = kid;
|
||||
result["private_jwk"] = uce_es256_jwk(x, y, d);
|
||||
result["private_jwk"]["kid"] = kid;
|
||||
result["kid"] = kid;
|
||||
result["thumbprint"] = kid;
|
||||
return(result);
|
||||
}
|
||||
|
||||
static String uce_es256_jwt(DValue private_jwk, DValue protected_header, DValue claims)
|
||||
{
|
||||
if(!uce_es256_json_map(protected_header) || !uce_es256_json_map(claims))
|
||||
return("");
|
||||
std::unique_ptr<EVP_PKEY, UcePkeyDeleter> key = uce_es256_key_from_jwk(private_jwk);
|
||||
if(!key)
|
||||
return("");
|
||||
protected_header["alg"] = "ES256";
|
||||
String header_json = json_encode(protected_header);
|
||||
String claims_json = json_encode(claims);
|
||||
if(header_json.size() > UCE_ES256_JSON_MAX || claims_json.size() > UCE_ES256_JSON_MAX)
|
||||
return("");
|
||||
String signing_input = uce_base64url_encode((const unsigned char*)header_json.data(), header_json.size()) + "." + uce_base64url_encode((const unsigned char*)claims_json.data(), claims_json.size());
|
||||
std::unique_ptr<EVP_MD_CTX, decltype(&EVP_MD_CTX_free)> ctx(EVP_MD_CTX_new(), EVP_MD_CTX_free);
|
||||
size_t der_size = 0;
|
||||
if(!ctx || EVP_DigestSignInit(ctx.get(), 0, EVP_sha256(), 0, key.get()) <= 0 || EVP_DigestSign(ctx.get(), 0, &der_size, (const unsigned char*)signing_input.data(), signing_input.size()) <= 0 || der_size == 0 || der_size > 256)
|
||||
return("");
|
||||
String der(der_size, 0);
|
||||
if(EVP_DigestSign(ctx.get(), (unsigned char*)der.data(), &der_size, (const unsigned char*)signing_input.data(), signing_input.size()) <= 0)
|
||||
return("");
|
||||
const unsigned char* cursor = (const unsigned char*)der.data();
|
||||
std::unique_ptr<ECDSA_SIG, UceEcdsaSigDeleter> signature(d2i_ECDSA_SIG(0, &cursor, der_size));
|
||||
const BIGNUM *r = 0, *s = 0;
|
||||
if(!signature || cursor != (const unsigned char*)der.data() + der_size)
|
||||
return("");
|
||||
ECDSA_SIG_get0(signature.get(), &r, &s);
|
||||
String jose(UCE_ES256_SIGNATURE_BYTES, 0);
|
||||
if(!r || !s || BN_bn2binpad(r, (unsigned char*)jose.data(), UCE_ES256_COORDINATE_BYTES) != UCE_ES256_COORDINATE_BYTES || BN_bn2binpad(s, (unsigned char*)jose.data() + UCE_ES256_COORDINATE_BYTES, UCE_ES256_COORDINATE_BYTES) != UCE_ES256_COORDINATE_BYTES)
|
||||
return("");
|
||||
return(signing_input + "." + uce_base64url_encode((const unsigned char*)jose.data(), jose.size()));
|
||||
}
|
||||
|
||||
DValue crypto_operation_native(DValue request)
|
||||
{
|
||||
DValue result;
|
||||
result["ok"].set_bool(false);
|
||||
if(!crypto_operation_request_valid(request))
|
||||
{
|
||||
result["error"] = "invalid_request";
|
||||
return(result);
|
||||
}
|
||||
String operation, algorithm;
|
||||
if(!uce_es256_jwk_string(request, "operation", operation) || !uce_es256_jwk_string(request, "algorithm", algorithm))
|
||||
{
|
||||
result["error"] = "invalid_request";
|
||||
return(result);
|
||||
}
|
||||
if(algorithm != "ES256")
|
||||
{
|
||||
result["error"] = "unsupported_algorithm";
|
||||
return(result);
|
||||
}
|
||||
if(operation == "key_generate")
|
||||
{
|
||||
DValue key = uce_es256_key_create();
|
||||
if(key["private_jwk"]["d"].to_string() == "")
|
||||
{
|
||||
result["error"] = "operation_failed";
|
||||
return(result);
|
||||
}
|
||||
key["ok"].set_bool(true);
|
||||
key["operation"] = operation;
|
||||
key["algorithm"] = algorithm;
|
||||
return(key);
|
||||
}
|
||||
if(operation == "cbor_decode")
|
||||
{
|
||||
String encoded, raw; UceCbor value; size_t p=0,nodes=0;
|
||||
if(!uce_es256_jwk_string(request, "cbor_base64url", encoded) || !uce_base64url_decode(encoded, raw, UCE_CBOR_MAX_BASE64URL) || raw.size() > UCE_CBOR_MAX_BYTES || !uce_cbor_read(raw, p, value, 0, nodes) || p != raw.size()) { result["error"] = "invalid_cbor"; return(result); }
|
||||
result["ok"].set_bool(true); result["operation"]=operation; result["value"]=uce_cbor_value(value); return result;
|
||||
}
|
||||
if(operation == "cose_es256_parse")
|
||||
{
|
||||
String encoded,x,y; if(!uce_es256_jwk_string(request,"cose_key_base64url",encoded)||!uce_es256_cose(encoded,x,y)) { result["error"]="invalid_cose_key"; return result; }
|
||||
result["ok"].set_bool(true); result["operation"]=operation; result["algorithm"]=algorithm; result["x_base64url"]=uce_base64url_encode((const unsigned char*)x.data(),x.size()); result["y_base64url"]=uce_base64url_encode((const unsigned char*)y.data(),y.size()); return result;
|
||||
}
|
||||
if(operation == "es256_verify")
|
||||
{
|
||||
String encoded, message64, signature64, x, y, message, der;
|
||||
if(!uce_es256_jwk_string(request, "cose_key_base64url", encoded) || !uce_es256_jwk_string(request, "message_base64url", message64) || !uce_es256_jwk_string(request, "signature_der_base64url", signature64) || !uce_es256_cose(encoded, x, y) || !uce_base64url_decode(message64, message, UCE_CBOR_MAX_BASE64URL) || !uce_base64url_decode(signature64, der, UCE_ES256_DER_BASE64URL_MAX) || message.size() > UCE_CBOR_MAX_BYTES || der.empty() || der.size() > UCE_ES256_DER_MAX_BYTES)
|
||||
{
|
||||
result["error"] = "invalid_key_or_payload";
|
||||
return(result);
|
||||
}
|
||||
const unsigned char* cursor = (const unsigned char*)der.data();
|
||||
std::unique_ptr<ECDSA_SIG, UceEcdsaSigDeleter> signature(d2i_ECDSA_SIG(0, &cursor, der.size()));
|
||||
int canonical = signature ? i2d_ECDSA_SIG(signature.get(), 0) : 0;
|
||||
String reencoded(canonical > 0 ? (size_t)canonical : 0, 0);
|
||||
unsigned char* dest = (unsigned char*)reencoded.data();
|
||||
if(!signature || cursor != (const unsigned char*)der.data() + der.size() || canonical <= 0 || i2d_ECDSA_SIG(signature.get(), &dest) != canonical || reencoded != der)
|
||||
{
|
||||
result["error"] = "invalid_signature";
|
||||
return(result);
|
||||
}
|
||||
std::unique_ptr<EVP_PKEY, UcePkeyDeleter> key = uce_es256_public_key(x, y);
|
||||
std::unique_ptr<EVP_MD_CTX, decltype(&EVP_MD_CTX_free)> ctx(EVP_MD_CTX_new(), EVP_MD_CTX_free);
|
||||
if(!key || !ctx || EVP_DigestVerifyInit(ctx.get(), 0, EVP_sha256(), 0, key.get()) <= 0)
|
||||
{
|
||||
result["error"] = "operation_failed";
|
||||
return(result);
|
||||
}
|
||||
int verified = EVP_DigestVerify(ctx.get(), (const unsigned char*)der.data(), der.size(), (const unsigned char*)message.data(), message.size());
|
||||
if(verified < 0)
|
||||
{
|
||||
result["error"] = "operation_failed";
|
||||
return(result);
|
||||
}
|
||||
result["ok"].set_bool(true);
|
||||
result["operation"] = operation;
|
||||
result["algorithm"] = algorithm;
|
||||
result["valid"].set_bool(verified == 1);
|
||||
return(result);
|
||||
}
|
||||
if(operation == "jwt_sign")
|
||||
{
|
||||
String jwt = uce_es256_jwt(request["private_jwk"], request["protected_header"], request["claims"]);
|
||||
if(jwt == "")
|
||||
{
|
||||
result["error"] = "invalid_key_or_payload";
|
||||
return(result);
|
||||
}
|
||||
result["ok"].set_bool(true);
|
||||
result["operation"] = operation;
|
||||
result["algorithm"] = algorithm;
|
||||
result["jwt"] = jwt;
|
||||
return(result);
|
||||
}
|
||||
result["error"] = "unsupported_operation";
|
||||
return(result);
|
||||
}
|
||||
#endif
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
#pragma once
|
||||
|
||||
struct DValue;
|
||||
|
||||
/* ================ sha1.h ================ */
|
||||
/*
|
||||
SHA-1 in C
|
||||
@@ -27,6 +29,8 @@ bool crypto_equal_native(String a, String b);
|
||||
String password_hash_native(String password);
|
||||
bool password_verify_native(String password, String encoded);
|
||||
bool password_needs_rehash_native(String encoded);
|
||||
bool crypto_operation_request_valid(DValue request);
|
||||
DValue crypto_operation_native(DValue request);
|
||||
String sha256(String data);
|
||||
String sha256_hex(String data);
|
||||
String hmac_sha256(String key, String data);
|
||||
|
||||
+48
-10
@@ -246,9 +246,10 @@ DValue MySQL::query(String q)
|
||||
statement_info = "mysql connection is not open";
|
||||
return(DValue());
|
||||
}
|
||||
_preload_next_error_code = mysql_query((MYSQL*)connection, q.c_str());
|
||||
int query_status = mysql_query((MYSQL*)connection, q.c_str());
|
||||
_preload_next_error_code = query_status == 0 ? 0 : mysql_errno((MYSQL*)connection);
|
||||
DValue result;
|
||||
if(_preload_next_error_code == 0)
|
||||
if(query_status == 0)
|
||||
result = get_pending_result();
|
||||
return(result);
|
||||
}
|
||||
@@ -291,11 +292,16 @@ static bool mysql_has_unquoted_positional_placeholder(String query)
|
||||
|
||||
DValue MySQL::query(String q, StringMap params)
|
||||
{
|
||||
// Positional ? placeholders survive named substitution (values are always
|
||||
// quoted by escape()), so the check in query(String) covers this path too.
|
||||
return(query(
|
||||
parse_query_parameters(q, params).c_str()
|
||||
));
|
||||
// Positional ? placeholders survive named substitution (ordinary values
|
||||
// are quoted by escape()), so the check in query(String) covers this path.
|
||||
parameter_error = false;
|
||||
String parsed = parse_query_parameters(q, params);
|
||||
if(parameter_error)
|
||||
{
|
||||
_preload_next_error_code = CR_UNKNOWN_ERROR;
|
||||
return(DValue());
|
||||
}
|
||||
return(query(parsed));
|
||||
}
|
||||
|
||||
String MySQL::parse_query_parameters(String query, StringMap map)
|
||||
@@ -305,6 +311,7 @@ String MySQL::parse_query_parameters(String query, StringMap map)
|
||||
|
||||
u8 mode = 0;
|
||||
char quote;
|
||||
bool escaped = false;
|
||||
String identifier;
|
||||
for(u32 i = 0; i < query.length(); i++)
|
||||
{
|
||||
@@ -321,6 +328,7 @@ String MySQL::parse_query_parameters(String query, StringMap map)
|
||||
result.append(1, c);
|
||||
mode = 2;
|
||||
quote = c;
|
||||
escaped = false;
|
||||
}
|
||||
else
|
||||
{
|
||||
@@ -329,10 +337,25 @@ String MySQL::parse_query_parameters(String query, StringMap map)
|
||||
}
|
||||
else if(mode == 1) // identifier mode
|
||||
{
|
||||
if(isalnum(c) || c == '_')
|
||||
if(isalnum((unsigned char)c) || c == '_')
|
||||
{
|
||||
identifier.append(1, c);
|
||||
}
|
||||
else if(c == '!' && query[i + 1] != '=')
|
||||
{
|
||||
String value = map[identifier];
|
||||
bool valid = identifier != "" && value != "";
|
||||
for(char digit : value)
|
||||
if(!isdigit((unsigned char)digit)) valid = false;
|
||||
if(!valid)
|
||||
{
|
||||
parameter_error = true;
|
||||
statement_info = "mysql unsigned parameter :" + identifier + "! must contain only decimal digits";
|
||||
return("");
|
||||
}
|
||||
result.append(value);
|
||||
mode = 0;
|
||||
}
|
||||
else
|
||||
{
|
||||
result.append(escape(map[identifier]));
|
||||
@@ -342,9 +365,19 @@ String MySQL::parse_query_parameters(String query, StringMap map)
|
||||
}
|
||||
else if(mode == 2) // quoted mode
|
||||
{
|
||||
result.append(1, c);
|
||||
if(escaped)
|
||||
{
|
||||
escaped = false;
|
||||
continue;
|
||||
}
|
||||
if(c == '\\')
|
||||
{
|
||||
escaped = true;
|
||||
continue;
|
||||
}
|
||||
if(c == quote)
|
||||
mode = 0;
|
||||
result.append(1, c);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -390,10 +423,15 @@ String MySQL::error()
|
||||
case(CR_OUT_OF_MEMORY):
|
||||
p = "Out of memory";
|
||||
break;
|
||||
default:
|
||||
case(CR_UNKNOWN_ERROR):
|
||||
p = "Unknown server error";
|
||||
break;
|
||||
default:
|
||||
if(connection && mysql_error((MYSQL*)connection)[0] != '\0')
|
||||
p = mysql_error((MYSQL*)connection);
|
||||
else
|
||||
p = "Unknown server error";
|
||||
break;
|
||||
}
|
||||
_preload_next_error_code = 0;
|
||||
return(p);
|
||||
|
||||
@@ -20,6 +20,7 @@ struct MySQL {
|
||||
u32 row_count = 0;
|
||||
u64 insert_id = 0;
|
||||
String statement_info = ""; //
|
||||
bool parameter_error = false;
|
||||
bool request_cleanup_delete = false;
|
||||
bool request_pooled = false;
|
||||
bool worker_persistent = false;
|
||||
|
||||
+240
-21
@@ -64,6 +64,7 @@ int uce_host_crypto_equal(const char* a, size_t a_len, const char* b, size_t b_l
|
||||
size_t uce_host_password_hash(const char* password, size_t password_len, char* out, size_t cap);
|
||||
int uce_host_password_verify(const char* password, size_t password_len, const char* encoded, size_t encoded_len);
|
||||
int uce_host_password_needs_rehash(const char* encoded, size_t encoded_len);
|
||||
size_t uce_host_crypto_operation(const char* in, size_t in_len, char* out, size_t cap);
|
||||
size_t uce_host_http_request(const char* in, size_t in_len, char* out, size_t cap);
|
||||
uint64_t uce_host_http_request_async(const char* in, size_t in_len);
|
||||
size_t uce_host_shell_exec_dv(const char* in, size_t in_len, char* out, size_t cap);
|
||||
@@ -284,6 +285,25 @@ String password_hash(String password)
|
||||
}
|
||||
bool password_verify(String password, String encoded) { return(uce_host_password_verify(password.data(), password.size(), encoded.data(), encoded.size()) != 0); }
|
||||
bool password_needs_rehash(String encoded) { return(uce_host_password_needs_rehash(encoded.data(), encoded.size()) != 0); }
|
||||
DValue crypto_operation(DValue request)
|
||||
{
|
||||
if(!crypto_operation_request_valid(request))
|
||||
{
|
||||
DValue result;
|
||||
result["ok"].set_bool(false);
|
||||
result["error"] = "invalid_request";
|
||||
return(result);
|
||||
}
|
||||
String encoded = ucb_encode(request);
|
||||
size_t required = uce_host_crypto_operation(encoded.data(), encoded.size(), 0, 0);
|
||||
if(required == 0 || required > 64 * 1024)
|
||||
return(DValue());
|
||||
String out(required, 0);
|
||||
size_t got = uce_host_crypto_operation(encoded.data(), encoded.size(), &out[0], required);
|
||||
DValue result;
|
||||
String error;
|
||||
return(got <= required && ucb_decode(String(out.data(), got), result, &error) ? result : DValue());
|
||||
}
|
||||
String base64_decode(String raw) { return(wasm_string_hostcall_1(uce_host_base64_decode, raw)); }
|
||||
String random_bytes(u64 n)
|
||||
{
|
||||
@@ -305,6 +325,7 @@ u64 http_request_async(DValue req)
|
||||
return((u64)uce_host_http_request_async(encoded.data(), encoded.size()));
|
||||
}
|
||||
|
||||
|
||||
DValue shell_exec(DValue spec)
|
||||
{
|
||||
String encoded = ucb_encode(spec);
|
||||
@@ -737,6 +758,7 @@ StringMap default_config()
|
||||
#include <fcntl.h>
|
||||
#include <limits.h>
|
||||
#include <sys/file.h>
|
||||
#include <poll.h>
|
||||
#include <errno.h>
|
||||
#include "sys.h"
|
||||
#include "hash.h"
|
||||
@@ -752,10 +774,12 @@ bool crypto_equal(String a, String b) { return(crypto_equal_native(a, b)); }
|
||||
String password_hash(String password) { return(password_hash_native(password)); }
|
||||
bool password_verify(String password, String encoded) { return(password_verify_native(password, encoded)); }
|
||||
bool password_needs_rehash(String encoded) { return(password_needs_rehash_native(encoded)); }
|
||||
DValue crypto_operation(DValue request) { return(crypto_operation_native(request)); }
|
||||
|
||||
// Single definitions for the native split build (declared extern in sys.h).
|
||||
pid_t parent_pid = 0;
|
||||
pid_t my_pid = 0;
|
||||
bool task_child_process = false;
|
||||
|
||||
namespace {
|
||||
|
||||
@@ -1291,6 +1315,185 @@ String socket_read(u64 sockfd, u32 max_length, u32 timeout)
|
||||
return("");
|
||||
}
|
||||
|
||||
static bool memcache_decimal(String value, u64& result)
|
||||
{
|
||||
if(value == "")
|
||||
return(false);
|
||||
result = 0;
|
||||
for(char c : value)
|
||||
{
|
||||
if(c < '0' || c > '9' || result > (UINT64_MAX - (u64)(c - '0')) / 10)
|
||||
return(false);
|
||||
result = result * 10 + (u64)(c - '0');
|
||||
}
|
||||
return(true);
|
||||
}
|
||||
|
||||
static String memcache_command_line(const String& command)
|
||||
{
|
||||
size_t end = command.find("\r\n");
|
||||
return(trim(command.substr(0, end)));
|
||||
}
|
||||
|
||||
bool uce_memcache_command_has_no_reply(const String& command)
|
||||
{
|
||||
StringList words = split(memcache_command_line(command), " ");
|
||||
if(words.size() == 0)
|
||||
return(false);
|
||||
String verb = to_lower(words[0]);
|
||||
return(verb == "quit" || to_lower(words.back()) == "noreply");
|
||||
}
|
||||
|
||||
MemcacheResponseState uce_memcache_response_advance(const String& command, const String& response, MemcacheResponseParser& parser)
|
||||
{
|
||||
while(true)
|
||||
{
|
||||
if(parser.mode == MemcacheResponseParser::Mode::FirstLine)
|
||||
{
|
||||
size_t line_end = response.find("\r\n");
|
||||
if(line_end == String::npos)
|
||||
return(MemcacheResponseState::Incomplete);
|
||||
String line = response.substr(0, line_end);
|
||||
if(line == "ERROR" || line.rfind("CLIENT_ERROR", 0) == 0 || line.rfind("SERVER_ERROR", 0) == 0 || line == "RESET" || line == "OK")
|
||||
return(MemcacheResponseState::Complete);
|
||||
if(line.rfind("VALUE ", 0) == 0)
|
||||
parser.mode = MemcacheResponseParser::Mode::RetrievalHeader;
|
||||
else if(line.rfind("VA ", 0) == 0)
|
||||
{
|
||||
StringList fields = split(line, " ");
|
||||
u64 length = 0;
|
||||
if(fields.size() < 2 || !memcache_decimal(fields[1], length) || length > SIZE_MAX - line_end - 4)
|
||||
return(MemcacheResponseState::Malformed);
|
||||
parser.data_end = line_end + 2 + (size_t)length;
|
||||
parser.mode = MemcacheResponseParser::Mode::MetaData;
|
||||
}
|
||||
else
|
||||
{
|
||||
StringList request_words = split(memcache_command_line(command), " ");
|
||||
String request_verb = request_words.size() ? to_lower(request_words[0]) : String("");
|
||||
bool end_terminated = line.rfind("STAT ", 0) == 0 || line.rfind("ITEM ", 0) == 0 ||
|
||||
request_verb == "stats" || (request_verb == "lru_crawler" && request_words.size() > 1 && to_lower(request_words[1]) == "metadump");
|
||||
if(!end_terminated)
|
||||
return(MemcacheResponseState::Complete);
|
||||
parser.mode = MemcacheResponseParser::Mode::Lines;
|
||||
}
|
||||
}
|
||||
|
||||
if(parser.mode == MemcacheResponseParser::Mode::RetrievalHeader)
|
||||
{
|
||||
size_t line_end = response.find("\r\n", parser.offset);
|
||||
if(line_end == String::npos)
|
||||
return(MemcacheResponseState::Incomplete);
|
||||
String line = response.substr(parser.offset, line_end - parser.offset);
|
||||
if(line == "END")
|
||||
return(MemcacheResponseState::Complete);
|
||||
StringList fields = split(line, " ");
|
||||
u64 length = 0;
|
||||
if(fields.size() < 4 || fields[0] != "VALUE" || !memcache_decimal(fields[3], length) || length > SIZE_MAX - line_end - 4)
|
||||
return(MemcacheResponseState::Malformed);
|
||||
parser.data_end = line_end + 2 + (size_t)length;
|
||||
parser.mode = MemcacheResponseParser::Mode::RetrievalData;
|
||||
}
|
||||
|
||||
if(parser.mode == MemcacheResponseParser::Mode::RetrievalData)
|
||||
{
|
||||
if(response.size() < parser.data_end + 2)
|
||||
return(MemcacheResponseState::Incomplete);
|
||||
if(response.compare(parser.data_end, 2, "\r\n") != 0)
|
||||
return(MemcacheResponseState::Malformed);
|
||||
parser.offset = parser.data_end + 2;
|
||||
parser.mode = MemcacheResponseParser::Mode::RetrievalHeader;
|
||||
}
|
||||
|
||||
if(parser.mode == MemcacheResponseParser::Mode::MetaData)
|
||||
{
|
||||
if(response.size() < parser.data_end + 2)
|
||||
return(MemcacheResponseState::Incomplete);
|
||||
return(response.compare(parser.data_end, 2, "\r\n") == 0 ? MemcacheResponseState::Complete : MemcacheResponseState::Malformed);
|
||||
}
|
||||
|
||||
if(parser.mode == MemcacheResponseParser::Mode::Lines)
|
||||
{
|
||||
size_t line_end = response.find("\r\n", parser.offset);
|
||||
if(line_end == String::npos)
|
||||
return(MemcacheResponseState::Incomplete);
|
||||
if(response.compare(parser.offset, line_end - parser.offset, "END") == 0)
|
||||
return(MemcacheResponseState::Complete);
|
||||
parser.offset = line_end + 2;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
static bool native_socket_wait(int fd, short events, u64 deadline)
|
||||
{
|
||||
while(true)
|
||||
{
|
||||
u64 now = monotonic_ms();
|
||||
if(now >= deadline)
|
||||
return(false);
|
||||
pollfd item = {fd, events, 0};
|
||||
int timeout = (int)std::min<u64>(deadline - now, INT_MAX);
|
||||
int ready = poll(&item, 1, timeout);
|
||||
if(ready > 0)
|
||||
return((item.revents & events) != 0);
|
||||
if(ready == 0 || errno != EINTR)
|
||||
return(false);
|
||||
}
|
||||
}
|
||||
|
||||
static String native_memcache_exchange(u64 connection, String command)
|
||||
{
|
||||
static constexpr size_t max_response = 8 * 1024 * 1024;
|
||||
int fd = (int)connection;
|
||||
u64 deadline = monotonic_ms() + 1000;
|
||||
String request = command + "\r\n";
|
||||
size_t sent = 0;
|
||||
while(sent < request.size())
|
||||
{
|
||||
if(!native_socket_wait(fd, POLLOUT, deadline))
|
||||
break;
|
||||
ssize_t count = send(fd, request.data() + sent, request.size() - sent, MSG_DONTWAIT | MSG_NOSIGNAL);
|
||||
if(count > 0)
|
||||
sent += (size_t)count;
|
||||
else if(count < 0 && (errno == EINTR || errno == EAGAIN || errno == EWOULDBLOCK))
|
||||
continue;
|
||||
else
|
||||
break;
|
||||
}
|
||||
if(sent != request.size())
|
||||
{
|
||||
shutdown(fd, SHUT_RDWR);
|
||||
return("");
|
||||
}
|
||||
if(uce_memcache_command_has_no_reply(command))
|
||||
return("");
|
||||
|
||||
String response;
|
||||
response.reserve(16 * 1024);
|
||||
MemcacheResponseParser parser;
|
||||
while(response.size() < max_response)
|
||||
{
|
||||
MemcacheResponseState state = uce_memcache_response_advance(command, response, parser);
|
||||
if(state == MemcacheResponseState::Complete)
|
||||
return(response);
|
||||
if(state == MemcacheResponseState::Malformed || !native_socket_wait(fd, POLLIN, deadline))
|
||||
break;
|
||||
char buffer[64 * 1024];
|
||||
size_t remaining = max_response - response.size();
|
||||
ssize_t count = recv(fd, buffer, std::min(remaining, sizeof(buffer)), MSG_DONTWAIT);
|
||||
if(count > 0)
|
||||
response.append(buffer, (size_t)count);
|
||||
else if(count < 0 && (errno == EINTR || errno == EAGAIN || errno == EWOULDBLOCK))
|
||||
continue;
|
||||
else
|
||||
break;
|
||||
}
|
||||
if(uce_memcache_response_advance(command, response, parser) == MemcacheResponseState::Complete)
|
||||
return(response);
|
||||
shutdown(fd, SHUT_RDWR);
|
||||
return("");
|
||||
}
|
||||
|
||||
String memcache_escape_key(String key)
|
||||
{
|
||||
String result;
|
||||
@@ -1320,26 +1523,18 @@ u64 memcache_connect(String host, u16 port)
|
||||
|
||||
String memcache_command(u64 connection, String command)
|
||||
{
|
||||
socket_write(connection, command+"\r\n");
|
||||
return(socket_read(connection)); // FIXME: do multi-chunk until END line is received!
|
||||
return(native_memcache_exchange(connection, command));
|
||||
}
|
||||
|
||||
bool memcache_set(u64 connection, String key, String value, u64 expires_in)
|
||||
{
|
||||
socket_write(connection,
|
||||
// set KEY META_DATA EXPIRY_TIME LENGTH_IN_BYTES
|
||||
String("set ") + memcache_escape_key(key) + " 0 " + std::to_string(expires_in) + " " + std::to_string(value.length()) + "\r\n" +
|
||||
value + "\r\n");
|
||||
return("STORED" == trim(socket_read(connection)));
|
||||
return("STORED" == trim(memcache_command(connection,
|
||||
String("set ") + memcache_escape_key(key) + " 0 " + std::to_string(expires_in) + " " + std::to_string(value.length()) + "\r\n" + value)));
|
||||
}
|
||||
|
||||
bool memcache_delete(u64 connection, String key)
|
||||
{
|
||||
socket_write(connection,
|
||||
// set KEY META_DATA EXPIRY_TIME LENGTH_IN_BYTES
|
||||
String("delete ") + memcache_escape_key(key) + "\r\n"
|
||||
);
|
||||
return("DELETED" == trim(socket_read(connection)));
|
||||
return("DELETED" == trim(memcache_command(connection, String("delete ") + memcache_escape_key(key))));
|
||||
}
|
||||
|
||||
String memcache_get(u64 connection, String key, String default_value)
|
||||
@@ -1389,6 +1584,7 @@ void on_segfault(int sig)
|
||||
|
||||
struct Worker {
|
||||
pid_t pid;
|
||||
char listener_type = 0;
|
||||
};
|
||||
|
||||
std::map<pid_t, Worker> workers;
|
||||
@@ -1617,27 +1813,41 @@ DValue process_exec(String cmd, String input, StringMap env, u64 timeout_ms, u64
|
||||
return(result);
|
||||
}
|
||||
|
||||
pid_t spawn_subprocess(std::function<void()> exec_after_spawn)
|
||||
pid_t spawn_subprocess(std::function<void()> exec_after_spawn, char listener_type = 0)
|
||||
{
|
||||
parent_pid = getpid();
|
||||
pid_t p;
|
||||
p = fork();
|
||||
sigset_t blocked;
|
||||
sigset_t previous;
|
||||
sigemptyset(&blocked);
|
||||
sigaddset(&blocked, SIGCHLD);
|
||||
if(sigprocmask(SIG_BLOCK, &blocked, &previous) != 0)
|
||||
{
|
||||
perror("sigprocmask");
|
||||
return(0);
|
||||
}
|
||||
pid_t p = fork();
|
||||
if(p == 0)
|
||||
{
|
||||
sigprocmask(SIG_SETMASK, &previous, 0);
|
||||
my_pid = getpid();
|
||||
//printf("(C) child procress started, PID:%i\n", my_pid);
|
||||
prctl(PR_SET_PDEATHSIG, SIGHUP);
|
||||
exec_after_spawn();
|
||||
return(0);
|
||||
}
|
||||
else
|
||||
if(p < 0)
|
||||
{
|
||||
Worker w;
|
||||
w.pid = p;
|
||||
workers[w.pid] = w;
|
||||
printf("(P) child procress spawned: PID %i\n", p);
|
||||
return(p);
|
||||
perror("fork worker");
|
||||
sigprocmask(SIG_SETMASK, &previous, 0);
|
||||
return(0);
|
||||
}
|
||||
Worker w;
|
||||
w.pid = p;
|
||||
w.listener_type = listener_type;
|
||||
workers[w.pid] = w;
|
||||
sigprocmask(SIG_SETMASK, &previous, 0);
|
||||
printf("(P) child procress spawned: PID %i\n", p);
|
||||
return(p);
|
||||
}
|
||||
|
||||
String runtime_safe_key(String key, String label)
|
||||
@@ -1788,6 +1998,7 @@ pid_t task(String key, std::function<void()> exec_after_spawn, u64 timeout)
|
||||
{
|
||||
close_locked_file(lock_fd);
|
||||
my_pid = getpid();
|
||||
task_child_process = true;
|
||||
// The FastCGI worker handles termination to drain accepted requests.
|
||||
// Generic task children do not run that drain loop, so inheriting those
|
||||
// handlers would turn task_kill(SIGTERM) into a no-op.
|
||||
@@ -1904,6 +2115,9 @@ StringMap make_server_settings()
|
||||
cfg["WASM_MEMORY_LIMIT_BYTES"] = std::to_string(512ull * 1024 * 1024);
|
||||
cfg["WASM_EPOCH_DEADLINE_TICKS"] = "200";
|
||||
cfg["WASM_EPOCH_PERIOD_MS"] = "50";
|
||||
cfg["WASM_SERIALIZE_TIMEOUT_SECONDS"] = "120";
|
||||
cfg["MYSQL_PERSISTENT_POOL_SIZE"] = "8";
|
||||
cfg["MYSQL_PERSISTENT_POOL_IDLE_TIMEOUT_SECONDS"] = "300";
|
||||
cfg["SETUP_TEMPLATE"] = "scripts/setup.h.template";
|
||||
cfg["LIT_ESC"] = "3d5b5_1";
|
||||
cfg["CONTENT_TYPE"] = "text/html; charset=utf-8";
|
||||
@@ -1911,6 +2125,11 @@ StringMap make_server_settings()
|
||||
cfg["FCGI_SOCKET_MODE"] = "0666";
|
||||
cfg["CLI_SOCKET_PATH"] = "/run/uce/cli.sock";
|
||||
cfg["CLI_SOCKET_MODE"] = "0600";
|
||||
// Zero preserves the legacy shared renderer pool. A positive count keeps
|
||||
// trusted CLI/test module caches out of public FastCGI workers.
|
||||
cfg["CLI_WORKER_COUNT"] = "0";
|
||||
// Bound retained CLI/test module state. Zero disables recycling.
|
||||
cfg["CLI_WORKER_MAX_REQUESTS"] = "8";
|
||||
// Command socket the WS broker listens on; workers flush ws_* dispatch
|
||||
// command batches here at workspace teardown.
|
||||
cfg["WS_BROKER_SOCKET_PATH"] = "/run/uce/ws-broker.sock";
|
||||
|
||||
@@ -51,6 +51,7 @@ bool crypto_equal(String a, String b);
|
||||
String password_hash(String password);
|
||||
bool password_verify(String password, String encoded);
|
||||
bool password_needs_rehash(String encoded);
|
||||
DValue crypto_operation(DValue request);
|
||||
String shell_escape(String raw);
|
||||
String basename(String fn);
|
||||
String dirname(String fn);
|
||||
@@ -128,6 +129,15 @@ String signal_name(s32 sig);
|
||||
String memcache_escape_key(String key);
|
||||
StringList memcache_escape_keys(StringList keys);
|
||||
u64 memcache_connect(String host = "127.0.0.1", u16 port = 11211);
|
||||
enum class MemcacheResponseState { Incomplete, Complete, Malformed };
|
||||
struct MemcacheResponseParser
|
||||
{
|
||||
enum class Mode { FirstLine, RetrievalHeader, RetrievalData, Lines, MetaData } mode = Mode::FirstLine;
|
||||
size_t offset = 0;
|
||||
size_t data_end = 0;
|
||||
};
|
||||
__attribute__((visibility("hidden"))) bool uce_memcache_command_has_no_reply(const String& command);
|
||||
__attribute__((visibility("hidden"))) MemcacheResponseState uce_memcache_response_advance(const String& command, const String& response, MemcacheResponseParser& parser);
|
||||
String memcache_command(u64 connection, String command);
|
||||
bool memcache_set(u64 connection, String key, String value, u64 expires_in = 60*60);
|
||||
bool memcache_delete(u64 connection, String key);
|
||||
@@ -139,9 +149,11 @@ StringMap memcache_get_multiple(u64 connection, StringList keys);
|
||||
#if defined(__UCE_WASM_CORE__) || defined(__UCE_WASM_UNIT__)
|
||||
pid_t parent_pid = 0;
|
||||
pid_t my_pid = 0;
|
||||
bool task_child_process = false;
|
||||
#else
|
||||
extern pid_t parent_pid;
|
||||
extern pid_t my_pid;
|
||||
extern bool task_child_process;
|
||||
#endif
|
||||
|
||||
void on_segfault(int sig);
|
||||
|
||||
+5
-1
@@ -97,7 +97,11 @@ String base64_decode(String raw, bool& ok)
|
||||
return("");
|
||||
if(padding > 0 && i + 4 != cleaned.length())
|
||||
return("");
|
||||
if(cleaned[i + 2] == '=' && cleaned[i + 3] != '=')
|
||||
if(cleaned[i] == '=' || cleaned[i + 1] == '=')
|
||||
return("");
|
||||
if(padding == 1 && (cleaned[i + 2] == '=' || cleaned[i + 3] != '='))
|
||||
return("");
|
||||
if(padding == 2 && (cleaned[i + 2] != '=' || cleaned[i + 3] != '='))
|
||||
return("");
|
||||
|
||||
result.append(1, (char)((values[0] << 2) | (values[1] >> 4)));
|
||||
|
||||
+112
-16
@@ -1132,7 +1132,8 @@ void run_ws_broker()
|
||||
if(server_state.config["WS_BROKER_SOCKET_PATH"] != "")
|
||||
{
|
||||
ws_broker.listen(server_state.config["WS_BROKER_SOCKET_PATH"]);
|
||||
chmod(server_state.config["WS_BROKER_SOCKET_PATH"].c_str(), S_IRWXU | S_IRGRP | S_IWGRP);
|
||||
if(chmod(server_state.config["WS_BROKER_SOCKET_PATH"].c_str(), S_IRWXU | S_IRGRP | S_IWGRP) != 0)
|
||||
fprintf(stderr, "(!) Could not chmod socket %s to %04o: %s\n", server_state.config["WS_BROKER_SOCKET_PATH"].c_str(), 0760, strerror(errno));
|
||||
}
|
||||
while(!termination_signal_received)
|
||||
{
|
||||
@@ -1265,6 +1266,58 @@ bool proactive_compile_queue_has(StringList& queue, String file_name)
|
||||
return(std::find(queue.begin(), queue.end(), file_name) != queue.end());
|
||||
}
|
||||
|
||||
String serialized_module_path(String wasm_path)
|
||||
{
|
||||
if(wasm_path.size() >= 5 && wasm_path.rfind(".wasm") == wasm_path.size() - 5)
|
||||
return(wasm_path.substr(0, wasm_path.size() - 5) + ".cwasm");
|
||||
return(wasm_path + ".cwasm");
|
||||
}
|
||||
|
||||
void cleanup_dead_serialization_temps(String wasm_path)
|
||||
{
|
||||
String cached_path = serialized_module_path(wasm_path);
|
||||
String directory = dirname(cached_path);
|
||||
String prefix = basename(cached_path) + ".";
|
||||
std::error_code ec;
|
||||
for(auto const& entry : std::filesystem::directory_iterator(directory, ec))
|
||||
{
|
||||
if(ec)
|
||||
break;
|
||||
String name = entry.path().filename().string();
|
||||
if(!str_starts_with(name, prefix) || name.size() <= prefix.size() + 4 || name.rfind(".tmp") != name.size() - 4)
|
||||
continue;
|
||||
String pid_text = name.substr(prefix.size(), name.size() - prefix.size() - 4);
|
||||
char* end = 0;
|
||||
errno = 0;
|
||||
long pid = strtol(pid_text.c_str(), &end, 10);
|
||||
if(pid <= 0 || errno == ERANGE || !end || *end != '\0')
|
||||
continue;
|
||||
if(kill((pid_t)pid, 0) != 0 && errno == ESRCH)
|
||||
file_unlink(entry.path().string());
|
||||
}
|
||||
}
|
||||
|
||||
String proactive_serialize_module(String wasm_path)
|
||||
{
|
||||
String executable = compiler_source_path_real("/proc/self/exe");
|
||||
if(executable == "")
|
||||
return("cannot resolve UCE executable for isolated serialization");
|
||||
u64 timeout_seconds = std::max<u64>(1, std::min<u64>(3600,
|
||||
to_u64(server_state.config["WASM_SERIALIZE_TIMEOUT_SECONDS"], 120)));
|
||||
DValue execution = process_exec(
|
||||
"exec " + shell_escape(executable) + " --serialize-module " + shell_escape(wasm_path),
|
||||
"", StringMap(), timeout_seconds * 1000, 64 * 1024);
|
||||
cleanup_dead_serialization_temps(wasm_path);
|
||||
if(execution["timed_out"].to_bool())
|
||||
return("serialized-module child timed out after " + std::to_string(timeout_seconds) + " seconds");
|
||||
if(execution["exit_code"].to_s64() != 0)
|
||||
{
|
||||
String diagnostic = trim(first(execution["stderr"].to_string(), execution["stdout"].to_string()));
|
||||
return(diagnostic == "" ? "serialized-module child failed" : diagnostic);
|
||||
}
|
||||
return("");
|
||||
}
|
||||
|
||||
u64 bounded_compile_jobs(String value, u64 fallback = 2)
|
||||
{
|
||||
value = trim(value);
|
||||
@@ -1318,7 +1371,7 @@ bool proactive_compile_unit(Request& context, String file_name, bool& source_mis
|
||||
if(!source_missing && !failed && wasm_serialized_module_needs_refresh(wasm_path))
|
||||
{
|
||||
printf("(i) proactive serialize %s\n", file_name.c_str());
|
||||
String serialize_error = wasm_serialize_module_artifact(wasm_path);
|
||||
String serialize_error = proactive_serialize_module(wasm_path);
|
||||
if(serialize_error != "")
|
||||
{
|
||||
printf("(!) proactive serialize failed for %s: %s\n", file_name.c_str(), serialize_error.c_str());
|
||||
@@ -1558,16 +1611,26 @@ void ensure_proactive_compiler()
|
||||
priority_compiler_pid = spawn_compiler("priority compiler", run_priority_compiler);
|
||||
}
|
||||
|
||||
void listen_for_connections()
|
||||
void listen_for_connections(char listener_type = 0)
|
||||
{
|
||||
install_process_fault_handlers();
|
||||
// Workers are uniform FastCGI/CLI renderers; the WS broker owns the HTTP/WS
|
||||
// port and every connection, so workers never accept raw HTTP themselves.
|
||||
server.close_http_listeners();
|
||||
// The transport's legacy eight-connection recycle predates persistent
|
||||
// Wasmtime engines. Recycling makes every ninth request pay engine/module
|
||||
// startup; request-scoped workspaces already isolate and release user state.
|
||||
// The WS broker owns the HTTP/WS port. A configured dedicated CLI pool keeps
|
||||
// test/admin module-cache churn out of public FastCGI renderers; zero keeps
|
||||
// the legacy shared listener set.
|
||||
if(listener_type == 'F' || listener_type == 'C')
|
||||
server.close_listeners_except(listener_type);
|
||||
else
|
||||
server.close_http_listeners();
|
||||
// Public workers keep their hot engines and modules. Dedicated CLI workers
|
||||
// recycle so broad test/admin runs cannot retain an unbounded module set.
|
||||
server.calls_until_termination = -1;
|
||||
if(listener_type == 'C')
|
||||
{
|
||||
u64 max_requests = std::min<u64>(1024,
|
||||
to_u64(server_state.config["CLI_WORKER_MAX_REQUESTS"], 8));
|
||||
if(max_requests > 0)
|
||||
server.calls_until_termination = (int)max_requests;
|
||||
}
|
||||
server.on_request = &handle_request;
|
||||
server.on_data = &handle_data;
|
||||
server.on_complete = &handle_complete;
|
||||
@@ -1578,7 +1641,8 @@ void listen_for_connections()
|
||||
String wasm_error = wasm_backend_start(startup_context);
|
||||
f64 wasm_ms = (time_precise() - wasm_start) * 1000.0;
|
||||
if(wasm_error == "")
|
||||
printf("(P) wasm worker ready: PID %i in %.3f ms\n", getpid(), wasm_ms);
|
||||
printf("(P) wasm %s worker ready: PID %i in %.3f ms\n",
|
||||
listener_type == 'F' ? "FastCGI" : listener_type == 'C' ? "CLI" : "shared", getpid(), wasm_ms);
|
||||
else
|
||||
fprintf(stderr, "(!) wasm worker initialization failed: PID %i in %.3f ms: %s\n", getpid(), wasm_ms, wasm_error.c_str());
|
||||
while(!termination_signal_received)
|
||||
@@ -1870,9 +1934,11 @@ void print_fastcgi_usage(FILE* stream)
|
||||
{
|
||||
fprintf(stream,
|
||||
"Usage: uce_fastcgi [--precompile]\n"
|
||||
" uce_fastcgi --serialize-module PATH\n"
|
||||
" uce_fastcgi --help\n\n"
|
||||
"Without options, start the FastCGI server.\n"
|
||||
" --precompile Compile the current source generation without starting listeners.\n"
|
||||
" --serialize-module PATH Serialize one Wasm artifact in an isolated process.\n"
|
||||
" -h, --help Show this help and exit.\n");
|
||||
}
|
||||
|
||||
@@ -1888,7 +1954,8 @@ int main(int argc, char** argv)
|
||||
return(0);
|
||||
}
|
||||
bool precompile = argc == 2 && String(argv[1]) == "--precompile";
|
||||
if(argc != 1 && !precompile)
|
||||
bool serialize_module = argc == 3 && String(argv[1]) == "--serialize-module";
|
||||
if(argc != 1 && !precompile && !serialize_module)
|
||||
{
|
||||
fprintf(stderr, "invalid arguments\n");
|
||||
print_fastcgi_usage(stderr);
|
||||
@@ -1898,6 +1965,16 @@ int main(int argc, char** argv)
|
||||
// after a fault does not allocate.
|
||||
backtrace(request_fault_frames, 4);
|
||||
process_start_directory();
|
||||
if(serialize_module)
|
||||
{
|
||||
String error = wasm_serialize_module_artifact(argv[2]);
|
||||
if(error != "")
|
||||
{
|
||||
fprintf(stderr, "%s\n", error.c_str());
|
||||
return(1);
|
||||
}
|
||||
return(0);
|
||||
}
|
||||
if(precompile)
|
||||
return(precompile_unit_generation());
|
||||
|
||||
@@ -1923,11 +2000,30 @@ int main(int argc, char** argv)
|
||||
if(!termination_signal_received)
|
||||
ensure_ws_broker();
|
||||
|
||||
while(workers.size() < int_val(server_state.config["WORKER_COUNT"]))
|
||||
{
|
||||
if(!termination_signal_received)
|
||||
spawn_subprocess(listen_for_connections);
|
||||
}
|
||||
u64 cli_worker_count = server_state.config["CLI_SOCKET_PATH"] == "" ? 0 :
|
||||
std::min<u64>(16, to_u64(server_state.config["CLI_WORKER_COUNT"], 0));
|
||||
bool has_public_listener = false;
|
||||
for(auto& listener : server.server_socket_types)
|
||||
if(listener.second == 'F')
|
||||
has_public_listener = true;
|
||||
u64 configured_worker_count = std::max<s64>(1, int_val(server_state.config["WORKER_COUNT"]));
|
||||
u64 public_worker_count = cli_worker_count > 0 && !has_public_listener ? 0 :
|
||||
(server.server_sockets.empty() ? 0 : configured_worker_count);
|
||||
auto role_count = [&](char listener_type) {
|
||||
u64 count = 0;
|
||||
for(auto& worker : workers)
|
||||
if(worker.second.listener_type == listener_type)
|
||||
count++;
|
||||
return(count);
|
||||
};
|
||||
auto spawn_listener_worker = [&](char listener_type) {
|
||||
return(spawn_subprocess([listener_type]() { listen_for_connections(listener_type); }, listener_type));
|
||||
};
|
||||
char public_listener_type = cli_worker_count > 0 ? 'F' : 0;
|
||||
while(!termination_signal_received && role_count(public_listener_type) < public_worker_count)
|
||||
if(spawn_listener_worker(public_listener_type) <= 0) { sleep(1); break; }
|
||||
while(!termination_signal_received && role_count('C') < cli_worker_count)
|
||||
if(spawn_listener_worker('C') <= 0) { sleep(1); break; }
|
||||
sleep(1);
|
||||
}
|
||||
|
||||
|
||||
+11
-2
@@ -121,6 +121,13 @@ static String wasm_backend_ensure_started(Request* context)
|
||||
return(g_wasm_init_error);
|
||||
}
|
||||
wc.mysql_persistent_pool_size = std::min<u64>(to_u64(cfg["MYSQL_PERSISTENT_POOL_SIZE"], 8), 64);
|
||||
u64 mysql_idle_timeout = to_u64(first(cfg["MYSQL_PERSISTENT_POOL_IDLE_TIMEOUT_SECONDS"], "300"), UINT64_MAX);
|
||||
if(mysql_idle_timeout == UINT64_MAX || mysql_idle_timeout > 86400)
|
||||
{
|
||||
g_wasm_init_error = "MYSQL_PERSISTENT_POOL_IDLE_TIMEOUT_SECONDS must be an integer no greater than 86400";
|
||||
return(g_wasm_init_error);
|
||||
}
|
||||
wc.mysql_persistent_pool_idle_timeout_seconds = mysql_idle_timeout;
|
||||
wc.profile_hostcall_cpu = to_bool(cfg["WASM_PROFILE_HOSTCALL_CPU"], false);
|
||||
wc.profile_thread_runtime = to_bool(cfg["WASM_PROFILE_THREAD_RUNTIME"], false);
|
||||
wc.verbose = to_bool(cfg["WASM_BACKEND_VERBOSE"], false);
|
||||
@@ -422,8 +429,8 @@ String wasm_backend_serve(Request& request, const String& entry_unit, const Stri
|
||||
return("");
|
||||
}
|
||||
|
||||
// Stop the ticker before the worker process exits (best-effort; forked workers
|
||||
// are usually killed, but a clean ager-out path should join the thread).
|
||||
// Stop the ticker before destroying the worker state it references. Deleting
|
||||
// the worker also closes retained connector resources on a clean process exit.
|
||||
void wasm_backend_shutdown()
|
||||
{
|
||||
if(g_wasm_epoch_ticker)
|
||||
@@ -434,4 +441,6 @@ void wasm_backend_shutdown()
|
||||
delete g_wasm_epoch_ticker;
|
||||
g_wasm_epoch_ticker = 0;
|
||||
}
|
||||
delete g_wasm_worker;
|
||||
g_wasm_worker = 0;
|
||||
}
|
||||
|
||||
+42
-3
@@ -207,6 +207,7 @@ String MySQL::parse_query_parameters(String query, StringMap map)
|
||||
|
||||
u8 mode = 0;
|
||||
char quote = 0;
|
||||
bool escaped = false;
|
||||
String identifier;
|
||||
for(u32 i = 0; i < query.length(); i++)
|
||||
{
|
||||
@@ -223,14 +224,30 @@ String MySQL::parse_query_parameters(String query, StringMap map)
|
||||
result.append(1, c);
|
||||
mode = 2;
|
||||
quote = c;
|
||||
escaped = false;
|
||||
}
|
||||
else
|
||||
result.append(1, c);
|
||||
}
|
||||
else if(mode == 1)
|
||||
{
|
||||
if(isalnum(c) || c == '_')
|
||||
if(isalnum((unsigned char)c) || c == '_')
|
||||
identifier.append(1, c);
|
||||
else if(c == '!' && query[i + 1] != '=')
|
||||
{
|
||||
String value = map[identifier];
|
||||
bool valid = identifier != "" && value != "";
|
||||
for(char digit : value)
|
||||
if(!isdigit((unsigned char)digit)) valid = false;
|
||||
if(!valid)
|
||||
{
|
||||
parameter_error = true;
|
||||
statement_info = "mysql unsigned parameter :" + identifier + "! must contain only decimal digits";
|
||||
return("");
|
||||
}
|
||||
result.append(value);
|
||||
mode = 0;
|
||||
}
|
||||
else
|
||||
{
|
||||
result.append(escape(map[identifier]));
|
||||
@@ -240,9 +257,19 @@ String MySQL::parse_query_parameters(String query, StringMap map)
|
||||
}
|
||||
else if(mode == 2)
|
||||
{
|
||||
result.append(1, c);
|
||||
if(escaped)
|
||||
{
|
||||
escaped = false;
|
||||
continue;
|
||||
}
|
||||
if(c == '\\')
|
||||
{
|
||||
escaped = true;
|
||||
continue;
|
||||
}
|
||||
if(c == quote)
|
||||
mode = 0;
|
||||
result.append(1, c);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -315,7 +342,17 @@ DValue MySQL::query(String q)
|
||||
return(result ? *result : DValue());
|
||||
}
|
||||
|
||||
DValue MySQL::query(String q, StringMap params) { return(query(parse_query_parameters(q, params))); }
|
||||
DValue MySQL::query(String q, StringMap params)
|
||||
{
|
||||
parameter_error = false;
|
||||
String parsed = parse_query_parameters(q, params);
|
||||
if(parameter_error)
|
||||
{
|
||||
_preload_next_error_code = 2000;
|
||||
return(DValue());
|
||||
}
|
||||
return(query(parsed));
|
||||
}
|
||||
DValue MySQL::get_pending_result() { return(DValue()); }
|
||||
|
||||
// sqlite runs host-side (the host links libsqlite and owns the connections in
|
||||
@@ -461,6 +498,8 @@ extern "C" void uce_wasm_link_anchors()
|
||||
(void*)(double (*)(double))&cos,
|
||||
(void*)(double (*)(double))&sin,
|
||||
(void*)(double (*)(double))&round,
|
||||
// structured crypto dispatcher is hostcall-backed and otherwise unreferenced by core
|
||||
(void*)&crypto_operation,
|
||||
};
|
||||
(void)libc_anchors;
|
||||
}
|
||||
|
||||
@@ -13,6 +13,7 @@ uce_host_crypto_equal
|
||||
uce_host_password_hash
|
||||
uce_host_password_verify
|
||||
uce_host_password_needs_rehash
|
||||
uce_host_crypto_operation
|
||||
uce_host_task_spawn
|
||||
uce_host_task_pid
|
||||
uce_host_task_kill
|
||||
|
||||
@@ -0,0 +1,146 @@
|
||||
// Internal worker-local implementation for opt-in generic hardened HTTP.
|
||||
#pragma once
|
||||
#include <arpa/inet.h>
|
||||
#include <cctype>
|
||||
#include <cerrno>
|
||||
#include <fcntl.h>
|
||||
#include <functional>
|
||||
#include <signal.h>
|
||||
#include <sys/wait.h>
|
||||
#include <sys/syscall.h>
|
||||
#include <unistd.h>
|
||||
|
||||
static u64 hardened_http_monotonic_ms() { timespec ts{}; clock_gettime(CLOCK_MONOTONIC,&ts); return (u64)ts.tv_sec*1000ull+(u64)ts.tv_nsec/1000000ull; }
|
||||
static void hardened_http_close_inherited_fds()
|
||||
{
|
||||
#ifdef SYS_close_range
|
||||
if(syscall(SYS_close_range,4u,~0u,0u)==0) return;
|
||||
#endif
|
||||
long max_fd=sysconf(_SC_OPEN_MAX); if(max_fd<4) max_fd=4;
|
||||
for(int fd=4;fd<max_fd;fd++) close(fd);
|
||||
}
|
||||
|
||||
struct HardenedHttpExecResult { int exit_code=-1; bool timed_out=false, output_limited=false; String headers_text, body_text, stderr_text; };
|
||||
struct HardenedHttpHooks {
|
||||
std::function<std::vector<String>(String)> resolve;
|
||||
std::function<HardenedHttpExecResult(std::vector<String>, String, std::vector<String>, u64, size_t)> execute;
|
||||
};
|
||||
|
||||
// Hardened requests deliberately support public IPv4 answers only. Every IPv6
|
||||
// answer fails closed until this generic transport has a reviewed IPv6 policy.
|
||||
// IPv4 must be globally unicast: reject IANA special-purpose 0/8, 10/8,
|
||||
// 100.64/10, 127/8, 169.254/16, 172.16/12, 192.0.0/24, 192.0.2/24,
|
||||
// 192.31.196/24, 192.52.193/24, 192.88.99/24, 192.175.48/24, 192.168/16,
|
||||
// 198.18/15, 198.51.100/24, 203.0.113/24, and all 224/4 multicast/reserved.
|
||||
static bool hardened_http_public_address(String text)
|
||||
{
|
||||
in_addr v4{};
|
||||
if(inet_pton(AF_INET,text.c_str(),&v4)!=1) return false; // Includes every AAAA candidate.
|
||||
u32 x=ntohl(v4.s_addr); u8 a=x>>24,b=x>>16,c=x>>8;
|
||||
if(a==0||a==10||a==127||a>=224||(a==100&&b>=64&&b<=127)||(a==169&&b==254)||(a==172&&b>=16&&b<=31)||(a==192&&(b==0||b==2||b==168))||(a==192&&b==31&&c==196)||(a==192&&b==52&&c==193)||(a==192&&b==88&&c==99)||(a==192&&b==175&&c==48)||(a==198&&(b==18||b==19||b==51))||(a==203&&b==0&&c==113)) return false;
|
||||
return true;
|
||||
}
|
||||
// Three private pipes keep curl's -D headers and -o body distinct. The child
|
||||
// inherits only stdin/stdout/stderr and the header pipe on fd 3. Async workers
|
||||
// retain their own process group so job_cancel can kill curl and its descendants.
|
||||
static HardenedHttpExecResult hardened_http_exec_argv_capture(std::vector<String> argv, String input, u64 timeout_ms, size_t output_limit, bool clean_env=true, bool own_process_group=true)
|
||||
{
|
||||
HardenedHttpExecResult r; if(argv.empty()) return r;
|
||||
int in[2], body[2], headers[2], err[2];
|
||||
if(pipe(in)||pipe(body)||pipe(headers)||pipe(err)) return r;
|
||||
pid_t pid=fork();
|
||||
if(pid==0) {
|
||||
if(own_process_group) setpgid(0,0);
|
||||
dup2(in[0],0); dup2(body[1],1); dup2(err[1],2); dup2(headers[1],3);
|
||||
hardened_http_close_inherited_fds();
|
||||
std::vector<char*> args; for(String& a:argv) args.push_back((char*)a.c_str()); args.push_back(0);
|
||||
if(clean_env) { clearenv(); setenv("PATH","/usr/bin:/bin",1); execv(args[0],args.data()); } else execvp(args[0],args.data());
|
||||
_exit(127);
|
||||
}
|
||||
auto close_all=[&](){ for(int fd: {in[0],in[1],body[0],body[1],headers[0],headers[1],err[0],err[1]}) close(fd); };
|
||||
if(pid<0) { close_all(); return r; }
|
||||
if(own_process_group) setpgid(pid,pid);
|
||||
close(in[0]); close(body[1]); close(headers[1]); close(err[1]);
|
||||
for(int fd: {in[1],body[0],headers[0],err[0]}) fcntl(fd,F_SETFL,fcntl(fd,F_GETFL,0)|O_NONBLOCK);
|
||||
size_t input_off=0, total=0; bool in_open=true,body_open=true,headers_open=true,err_open=true,exited=false,killed=false; int status=0; u64 deadline=hardened_http_monotonic_ms()+timeout_ms;
|
||||
auto terminate=[&](bool timeout){ if(timeout) r.timed_out=true; if(!killed) { if(own_process_group) kill(-pid,SIGKILL); kill(pid,SIGKILL); killed=true; } };
|
||||
auto drain=[&](int fd,bool& open,String& dst) { char buf[4096]; ssize_t n; while((n=read(fd,buf,sizeof(buf)))>0) { if(total+(size_t)n>output_limit) { r.output_limited=true; terminate(false); } else { dst+=String(buf,n); total+=(size_t)n; } } if(n==0) { close(fd); open=false; } };
|
||||
while(in_open||body_open||headers_open||err_open||!exited) {
|
||||
if(!exited) { pid_t w=waitpid(pid,&status,WNOHANG); if(w==pid) exited=true; }
|
||||
if(in_open) { if(input_off<input.size()) { ssize_t n=write(in[1],input.data()+input_off,input.size()-input_off); if(n>0) input_off+=(size_t)n; else if(n<0&&errno!=EINTR&&errno!=EAGAIN&&errno!=EWOULDBLOCK) { close(in[1]); in_open=false; } } else { close(in[1]); in_open=false; } }
|
||||
drain(body[0],body_open,r.body_text); drain(headers[0],headers_open,r.headers_text); drain(err[0],err_open,r.stderr_text);
|
||||
if(!killed&&hardened_http_monotonic_ms()>=deadline) terminate(true);
|
||||
if(killed&&!exited) { while(waitpid(pid,&status,0)<0&&errno==EINTR) {} exited=true; }
|
||||
if(in_open||body_open||headers_open||err_open||!exited) usleep(1000);
|
||||
}
|
||||
if(WIFEXITED(status)) r.exit_code=WEXITSTATUS(status); else if(WIFSIGNALED(status)) r.exit_code=128+WTERMSIG(status);
|
||||
return r;
|
||||
}
|
||||
|
||||
static bool hardened_http_parse_headers(String text, u64& status, DValue& filtered)
|
||||
{
|
||||
status=0; filtered.set_array(); size_t pos=0; bool found=false;
|
||||
while(pos<text.size()) {
|
||||
size_t end=text.find("\r\n\r\n",pos); size_t sep=4; if(end==String::npos) { end=text.find("\n\n",pos); sep=2; } if(end==String::npos) return false;
|
||||
String block=replace(text.substr(pos,end-pos),"\r",""); size_t nl=block.find('\n'); String first=nl==String::npos?block:block.substr(0,nl);
|
||||
if(first.size()<12||first.rfind("HTTP/",0)!=0||first[8]!=' '||!isdigit((unsigned char)first[9])||!isdigit((unsigned char)first[10])||!isdigit((unsigned char)first[11])) return false;
|
||||
status=strtoull(first.substr(9,3).c_str(),0,10); DValue current; current.set_array();
|
||||
if(nl!=String::npos) for(String line:split(block.substr(nl+1),"\n")) { size_t c=line.find(':'); if(c==String::npos) return false; String k=to_lower(trim(line.substr(0,c))); if(k=="content-type"||k=="cache-control") current[k]=trim(line.substr(c+1)); }
|
||||
filtered=current; found=true; pos=end+sep;
|
||||
}
|
||||
return found;
|
||||
}
|
||||
|
||||
static bool hardened_http_token(String text)
|
||||
{
|
||||
if(text=="") return false;
|
||||
for(unsigned char c:text) if(!(isalnum(c)||c=='-'||c=='_')) return false;
|
||||
return true;
|
||||
}
|
||||
static bool hardened_http_header_value(String text)
|
||||
{
|
||||
if(text.size()>8192||text.find('\0')!=String::npos||text.find('\r')!=String::npos||text.find('\n')!=String::npos) return false;
|
||||
return true;
|
||||
}
|
||||
static bool hardened_http_ip_literal(String text)
|
||||
{
|
||||
in_addr v4{}; in6_addr v6{};
|
||||
return inet_pton(AF_INET,text.c_str(),&v4)==1 || inet_pton(AF_INET6,text.c_str(),&v6)==1;
|
||||
}
|
||||
static bool hardened_http_url(String url, String& host, String& port)
|
||||
{
|
||||
if(url.find('\0')!=String::npos||url.find_first_of("\\ \r\n\t")!=String::npos||url.rfind("https://",0)!=0) return false;
|
||||
String rest=url.substr(8); size_t slash=rest.find('/'); String authority=slash==String::npos?rest:rest.substr(0,slash);
|
||||
if(authority==""||authority.find('@')!=String::npos||authority.find('[')!=String::npos||authority.find(']')!=String::npos) return false;
|
||||
size_t colon=authority.rfind(':'); host=colon==String::npos?authority:authority.substr(0,colon); port=colon==String::npos?"443":authority.substr(colon+1);
|
||||
if(host==""||hardened_http_ip_literal(host)||!hardened_http_token(port)) return false;
|
||||
for(unsigned char c:host) if(!(isalnum(c)||c=='.'||c=='-')) return false;
|
||||
char* end=0; unsigned long n=strtoul(port.c_str(),&end,10); return end!=port.c_str()&&*end==0&&n>0&&n<=65535;
|
||||
}
|
||||
static bool hardened_http_security_requested(const DValue* security)
|
||||
{
|
||||
return security && (security->key("https_only") || security->key("public_dns_only") || security->key("pin_dns") || security->key("isolated_curl") || security->key("no_redirects"));
|
||||
}
|
||||
static bool hardened_http_true(const DValue* value)
|
||||
{
|
||||
return value && value->get_type_name()=="bool" && value->to_bool();
|
||||
}
|
||||
static DValue hardened_http_request_internal(const DValue& req, u64 timeout_ms, const HardenedHttpHooks& hooks)
|
||||
{
|
||||
DValue r; r["status"]=(f64)0; r["headers"].set_array(); r["body"]=""; r["error"]="";
|
||||
const DValue* sec=req.key("security");
|
||||
if(req.key("follow_redirects") && req.key("follow_redirects")->to_bool()) { r["error"]="invalid_request"; return r; }
|
||||
timeout_ms=std::min<u64>(std::max<u64>(1,timeout_ms),10000);
|
||||
if(!sec || !hardened_http_true(sec->key("https_only")) || !hardened_http_true(sec->key("public_dns_only")) || !hardened_http_true(sec->key("pin_dns")) || !hardened_http_true(sec->key("isolated_curl")) || !hardened_http_true(sec->key("no_redirects"))) { r["error"]="invalid_request"; return r; }
|
||||
String method=req.key("method")?req.key("method")->to_string():"GET"; String url=req.key("url")?req.key("url")->to_string():""; String host,port;
|
||||
if((method!="GET"&&method!="POST"&&method!="PUT"&&method!="PATCH"&&method!="DELETE"&&method!="HEAD"&&method!="OPTIONS")||!hardened_http_url(url,host,port)) { r["error"]="invalid_request"; return r; }
|
||||
String body=req.key("body")?req.key("body")->to_string():""; if(body.size()>65536||body.find('\0')!=String::npos) { r["error"]="invalid_request"; return r; }
|
||||
std::vector<String> argv={"/usr/bin/curl","--disable","-sS","--http1.1","--proto","=https","--proto-redir","=https","--noproxy","*","--proxy","","--alt-svc","","--hsts","","--cacert","/etc/ssl/certs/ca-certificates.crt","--connect-timeout","3","--max-time",std::to_string(std::max<u64>(1,timeout_ms/1000)),"--max-filesize","65536","-X",method,"-D","/proc/self/fd/3","-o","/proc/self/fd/1"};
|
||||
const DValue* hs=req.key("headers"); if(hs) { bool valid=true; hs->each([&](const DValue& v,String k) { String value=v.to_string(), lower=to_lower(k); if(!hardened_http_token(k)||!hardened_http_header_value(value)||lower=="host"||lower=="content-length"||lower=="transfer-encoding"||lower=="connection"||lower=="proxy-connection"||lower=="expect") valid=false; else { argv.push_back("-H"); argv.push_back(k+": "+value); } }); if(!valid) { r["error"]="invalid_request"; return r; } }
|
||||
std::vector<String> answers=hooks.resolve(host); String address; for(String a:answers) { if(!hardened_http_public_address(a)) { r["error"]="unsafe_dns"; return r; } if(address=="") address=a; } if(address=="") { r["error"]="unsafe_dns"; return r; }
|
||||
argv.push_back("--resolve"); argv.push_back(host+":"+port+":"+address); if(req.key("body")) { argv.push_back("--data-binary"); argv.push_back("@-"); } argv.push_back(url);
|
||||
HardenedHttpExecResult pr=hooks.execute(argv,body,{"PATH=/usr/bin:/bin"},timeout_ms,72*1024); if(pr.output_limited||pr.body_text.size()>65536||pr.headers_text.size()>8192) { r["error"]="response_too_large"; return r; }
|
||||
u64 status; DValue headers; if(!hardened_http_parse_headers(pr.headers_text,status,headers)) { r["error"]=pr.timed_out?"timeout":"malformed_output"; return r; }
|
||||
r["status"]=(f64)status; r["headers"]=headers; r["body"]=pr.body_text;
|
||||
if(status/100==3) r["error"]="redirect_not_allowed"; else if(pr.exit_code!=0) r["error"]=pr.timed_out?"timeout":"network_failure"; else if(status/100!=2) r["error"]="http_status"; return r;
|
||||
}
|
||||
+302
-60
@@ -39,6 +39,7 @@
|
||||
#include <string>
|
||||
#include <arpa/inet.h>
|
||||
#include <netinet/in.h>
|
||||
#include <netdb.h>
|
||||
#include <sys/socket.h>
|
||||
#include <sys/stat.h>
|
||||
#include <sys/file.h>
|
||||
@@ -55,6 +56,7 @@
|
||||
#include <sys/wait.h>
|
||||
#include <signal.h>
|
||||
#include <poll.h>
|
||||
#include "hardened_http_internal.h"
|
||||
|
||||
struct WasmDylinkInfo
|
||||
{
|
||||
@@ -153,6 +155,7 @@ struct WasmWorkerConfig
|
||||
u64 epoch_period_ms = 50;
|
||||
u64 invocation_timeout_ms = 30000;
|
||||
u64 mysql_persistent_pool_size = 8;
|
||||
u64 mysql_persistent_pool_idle_timeout_seconds = 300;
|
||||
bool profile_hostcall_cpu = false;
|
||||
bool profile_thread_runtime = false;
|
||||
bool verbose = false;
|
||||
@@ -389,15 +392,12 @@ static u64 wasm_socket_connect_bounded(const String& host, u16 port, u64 timeout
|
||||
}
|
||||
if(fd <= 0)
|
||||
return(0);
|
||||
if(context)
|
||||
context->resources.sockets.push_back(fd);
|
||||
return((u64)fd);
|
||||
}
|
||||
|
||||
static bool wasm_socket_write_bounded(u64 socket_fd, const String& data, u64 timeout_ms)
|
||||
static bool wasm_socket_write_until(u64 socket_fd, const String& data, u64 deadline)
|
||||
{
|
||||
int fd = (int)socket_fd;
|
||||
u64 deadline = wasm_deadline_after_ms(timeout_ms);
|
||||
size_t offset = 0;
|
||||
while(offset < data.size())
|
||||
{
|
||||
@@ -412,6 +412,11 @@ static bool wasm_socket_write_bounded(u64 socket_fd, const String& data, u64 tim
|
||||
return(true);
|
||||
}
|
||||
|
||||
static bool wasm_socket_write_bounded(u64 socket_fd, const String& data, u64 timeout_ms)
|
||||
{
|
||||
return(wasm_socket_write_until(socket_fd, data, wasm_deadline_after_ms(timeout_ms)));
|
||||
}
|
||||
|
||||
static String wasm_socket_read_bounded(u64 socket_fd, u32 max_length, u64 timeout_ms)
|
||||
{
|
||||
if(max_length == 0 || !wasm_socket_wait((int)socket_fd, POLLIN, wasm_deadline_after_ms(timeout_ms)))
|
||||
@@ -421,6 +426,42 @@ static String wasm_socket_read_bounded(u64 socket_fd, u32 max_length, u64 timeou
|
||||
return(count > 0 ? String(buffer.data(), (size_t)count) : String(""));
|
||||
}
|
||||
|
||||
static String wasm_memcache_exchange(u64 socket_fd, const String& command, u64 timeout_ms)
|
||||
{
|
||||
static constexpr size_t max_response = 8 * 1024 * 1024;
|
||||
int fd = (int)socket_fd;
|
||||
u64 deadline = wasm_deadline_after_ms(timeout_ms);
|
||||
if(!wasm_socket_write_until(socket_fd, command + "\r\n", deadline))
|
||||
{
|
||||
shutdown(fd, SHUT_RDWR);
|
||||
return("");
|
||||
}
|
||||
if(uce_memcache_command_has_no_reply(command))
|
||||
return("");
|
||||
String response;
|
||||
response.reserve(16 * 1024);
|
||||
MemcacheResponseParser parser;
|
||||
while(response.size() < max_response)
|
||||
{
|
||||
MemcacheResponseState state = uce_memcache_response_advance(command, response, parser);
|
||||
if(state == MemcacheResponseState::Complete)
|
||||
return(response);
|
||||
if(state == MemcacheResponseState::Malformed || !wasm_socket_wait(fd, POLLIN, deadline))
|
||||
break;
|
||||
char buffer[64 * 1024];
|
||||
size_t remaining = max_response - response.size();
|
||||
ssize_t count = recv(fd, buffer, std::min(remaining, sizeof(buffer)), MSG_DONTWAIT);
|
||||
if(count > 0)
|
||||
response.append(buffer, (size_t)count);
|
||||
else if(count == 0 || (errno != EINTR && errno != EAGAIN && errno != EWOULDBLOCK))
|
||||
break;
|
||||
}
|
||||
if(uce_memcache_response_advance(command, response, parser) == MemcacheResponseState::Complete)
|
||||
return(response);
|
||||
shutdown(fd, SHUT_RDWR);
|
||||
return("");
|
||||
}
|
||||
|
||||
static f64 wasm_thread_cpu_time()
|
||||
{
|
||||
struct timespec ts;
|
||||
@@ -581,7 +622,7 @@ static u64 uce_shell_spawn_spec(const DValue& spec)
|
||||
}
|
||||
|
||||
|
||||
static DValue uce_exec_argv_capture(std::vector<String> argv, String input, u64 timeout_ms)
|
||||
static DValue uce_exec_argv_capture(std::vector<String> argv, String input, u64 timeout_ms, size_t output_limit=0, bool clean_env=false)
|
||||
{
|
||||
DValue r; r["exit_code"]=(f64)-1; r["stdout"]=""; r["stderr"]=""; r["timed_out"].set_bool(false);
|
||||
if(argv.empty()) { r["stderr"]="empty argv"; return(r); }
|
||||
@@ -598,7 +639,9 @@ static DValue uce_exec_argv_capture(std::vector<String> argv, String input, u64
|
||||
dup2(inpipe[0],0); dup2(outpipe[1],1); dup2(errpipe[1],2);
|
||||
close(inpipe[0]); close(inpipe[1]); close(outpipe[0]); close(outpipe[1]); close(errpipe[0]); close(errpipe[1]);
|
||||
std::vector<char*> args; for(auto& a: argv) args.push_back((char*)a.c_str()); args.push_back(0);
|
||||
execvp(args[0], args.data()); _exit(127);
|
||||
if(clean_env) { clearenv(); setenv("PATH", "/usr/bin:/bin", 1); execv(args[0], args.data()); }
|
||||
else execvp(args[0], args.data());
|
||||
_exit(127);
|
||||
}
|
||||
if(pid < 0)
|
||||
{
|
||||
@@ -614,8 +657,8 @@ static DValue uce_exec_argv_capture(std::vector<String> argv, String input, u64
|
||||
{
|
||||
if(!exited) { pid_t w=waitpid(pid,&status,WNOHANG); if(w==pid) { exited=true; status_valid=true; } else if(w<0&&errno==ECHILD) { u64 transfer_deadline=wasm_deadline_after_ms(50); do { status_valid=child_exit_status_take(pid,status,child_status_snapshot); if(!status_valid)sched_yield(); } while(!status_valid&&wasm_monotonic_ms()<transfer_deadline); exited=true; if(!status_valid)r["stderr"]=r["stderr"].to_string()+"lost child exit status"; } }
|
||||
if(in_open) { if(input_off<input.size()) { ssize_t n=write(inpipe[1], input.data()+input_off, input.size()-input_off); if(n>0) input_off+=(size_t)n; else if(n<0 && errno!=EINTR && errno!=EAGAIN && errno!=EWOULDBLOCK) { close(inpipe[1]); in_open=false; } } else { close(inpipe[1]); in_open=false; } }
|
||||
char buf[4096]; ssize_t n; while((n=read(outpipe[0],buf,sizeof(buf)))>0) r["stdout"] = r["stdout"].to_string()+String(buf,n); if(n==0&&out_open){close(outpipe[0]);out_open=false;}
|
||||
while((n=read(errpipe[0],buf,sizeof(buf)))>0) r["stderr"] = r["stderr"].to_string()+String(buf,n); if(n==0&&err_open){close(errpipe[0]);err_open=false;}
|
||||
char buf[4096]; ssize_t n; while((n=read(outpipe[0],buf,sizeof(buf)))>0) { if(output_limit && r["stdout"].to_string().size()+(size_t)n>output_limit) { r["output_limited"].set_bool(true); kill(-pid,SIGKILL); kill(pid,SIGKILL); } else r["stdout"] = r["stdout"].to_string()+String(buf,n); } if(n==0&&out_open){close(outpipe[0]);out_open=false;}
|
||||
while((n=read(errpipe[0],buf,sizeof(buf)))>0) { if(output_limit && r["stderr"].to_string().size()+(size_t)n>output_limit) { r["output_limited"].set_bool(true); kill(-pid,SIGKILL); kill(pid,SIGKILL); } else r["stderr"] = r["stderr"].to_string()+String(buf,n); } if(n==0&&err_open){close(errpipe[0]);err_open=false;}
|
||||
if((out_open || err_open || !exited) && wasm_monotonic_ms() >= deadline) { r["timed_out"].set_bool(true); kill(-pid,SIGKILL); kill(pid,SIGKILL); if(!exited) status_valid=waitpid(pid,&status,0)==pid; exited=true; if(in_open){close(inpipe[1]);in_open=false;} if(out_open){close(outpipe[0]);out_open=false;} if(err_open){close(errpipe[0]);err_open=false;} }
|
||||
if(out_open || err_open || !exited) usleep(10000);
|
||||
}
|
||||
@@ -630,8 +673,11 @@ static bool uce_header_name_safe(String name)
|
||||
return(true);
|
||||
}
|
||||
|
||||
static DValue uce_hardened_http_request_value(const DValue& req, u64 timeout_ms, bool keep_worker_process_group=false);
|
||||
|
||||
static DValue uce_http_request_value(const DValue& req)
|
||||
{
|
||||
const DValue* security=req.key("security"); if(hardened_http_security_requested(security)) { u64 requested=req.key("timeout_ms")?req.key("timeout_ms")->to_u64(5000):5000; return(uce_hardened_http_request_value(req,std::max<u64>(1,requested))); }
|
||||
DValue r; r["status"]=(f64)0; r["headers"].set_array(); r["body"]=""; r["error"]="";
|
||||
const DValue* method_value = req.key("method");
|
||||
const DValue* url_value = req.key("url");
|
||||
@@ -665,19 +711,35 @@ static DValue uce_http_request_value(const DValue& req)
|
||||
return(r);
|
||||
}
|
||||
|
||||
static DValue uce_hardened_http_request_value(const DValue& req, u64 timeout_ms, bool keep_worker_process_group)
|
||||
{
|
||||
HardenedHttpHooks hooks;
|
||||
hooks.resolve=[](String host) { std::vector<String> answers; addrinfo hints{}; hints.ai_socktype=SOCK_STREAM; hints.ai_family=AF_UNSPEC; addrinfo* result=0; if(getaddrinfo(host.c_str(), "443", &hints, &result)!=0) return answers; for(addrinfo* p=result;p;p=p->ai_next) { char text[INET6_ADDRSTRLEN]; if(p->ai_family==AF_INET && inet_ntop(AF_INET,&((sockaddr_in*)p->ai_addr)->sin_addr,text,sizeof(text))) answers.push_back(text); else if(p->ai_family==AF_INET6 && inet_ntop(AF_INET6,&((sockaddr_in6*)p->ai_addr)->sin6_addr,text,sizeof(text))) answers.push_back(text); else answers.push_back(""); } freeaddrinfo(result); return answers; };
|
||||
hooks.execute=[keep_worker_process_group](std::vector<String> argv, String input, std::vector<String>, u64 deadline, size_t limit) { return hardened_http_exec_argv_capture(argv,input,deadline,limit,true,!keep_worker_process_group); };
|
||||
return hardened_http_request_internal(req,timeout_ms,hooks);
|
||||
}
|
||||
|
||||
static u64 uce_http_spawn_spec(const DValue& req)
|
||||
{
|
||||
uce_job_reap(); u64 id=uce_job_new("http"); if(!id) return(0);
|
||||
int ready[2]; if(pipe(ready)) { DValue r; r["error"]="pipe failed"; uce_job_finish(id,r,"failed"); return(id); }
|
||||
pid_t pid=fork();
|
||||
if(pid==0) { setsid(); uce_write_text(uce_job_path(id)+"/worker_pid", std::to_string((long long)getpid())); uce_write_text(uce_job_path(id)+"/state", "running"); DValue result=uce_http_request_value(req); uce_job_finish(id,result,result["error"].to_string()==""?"done":"failed"); _exit(0); }
|
||||
if(pid<0) { DValue r; r["error"]="fork failed"; uce_job_finish(id,r,"failed"); return(id); }
|
||||
if(pid==0) { close(ready[0]); if(setsid()<0) _exit(127); char ok='1'; if(write(ready[1],&ok,1)!=1) _exit(127); close(ready[1]); uce_write_text(uce_job_path(id)+"/worker_pid", std::to_string((long long)getpid())); uce_write_text(uce_job_path(id)+"/state", "running"); const DValue* security=req.key("security"); bool hardened=hardened_http_security_requested(security); DValue result=hardened ? uce_hardened_http_request_value(req,req.key("timeout_ms")?std::max<u64>(1,req.key("timeout_ms")->to_u64(5000)):5000,true) : uce_http_request_value(req); uce_job_finish(id,result,result["error"].to_string()==""?"done":"failed"); _exit(0); }
|
||||
close(ready[1]); char ok=0; ssize_t started; do { started=read(ready[0],&ok,1); } while(started<0&&errno==EINTR); close(ready[0]);
|
||||
if(pid<0 || started!=1 || ok!='1') { DValue r; r["error"]="async worker start failed"; uce_job_finish(id,r,"failed"); return(id); }
|
||||
uce_write_text(uce_job_path(id)+"/worker_pid", std::to_string((long long)pid)); uce_write_text(uce_job_path(id)+"/state", "running"); return(id);
|
||||
}
|
||||
|
||||
static DValue uce_job_status_value(u64 id)
|
||||
{
|
||||
DValue r; String dir=uce_job_path(id); r["job_id"]=(f64)id;
|
||||
if(id==0 || !std::filesystem::is_directory(dir)) { r["state"]="missing"; return(r); }
|
||||
bool is_directory = false;
|
||||
if(id != 0)
|
||||
{
|
||||
try { is_directory = std::filesystem::is_directory(dir); }
|
||||
catch(...) { is_directory = false; }
|
||||
}
|
||||
if(!is_directory) { r["state"]="missing"; return(r); }
|
||||
String state=trim(uce_read_text(dir+"/state")); if(state=="") state="pending"; r["state"]=state;
|
||||
r["kind"]=trim(uce_read_text(dir+"/kind")); r["pid"]=(f64)strtoull(uce_read_text(dir+"/worker_pid").c_str(),0,10);
|
||||
r["done"].set_bool(state=="done"||state=="failed"||state=="cancelled");
|
||||
@@ -1271,6 +1333,14 @@ static bool wasm_read_metadata_file(const String& path, std::vector<u8>& metadat
|
||||
|
||||
class WasmWorkspace;
|
||||
|
||||
#ifdef UCE_WASM_HOST_CONNECTORS
|
||||
struct WasmMySQLPersistentConnection
|
||||
{
|
||||
MySQL* db = 0;
|
||||
std::chrono::steady_clock::time_point idle_since;
|
||||
};
|
||||
#endif
|
||||
|
||||
class WasmWorker
|
||||
{
|
||||
public:
|
||||
@@ -1283,25 +1353,59 @@ public:
|
||||
#ifdef UCE_WASM_HOST_CONNECTORS
|
||||
~WasmWorker()
|
||||
{
|
||||
for(auto* db : mysql_persistent_pool)
|
||||
delete db;
|
||||
for(auto& entry : mysql_persistent_pool)
|
||||
delete entry.db;
|
||||
}
|
||||
|
||||
void mysql_evict_idle(std::chrono::steady_clock::time_point now = std::chrono::steady_clock::now())
|
||||
{
|
||||
if(cfg.mysql_persistent_pool_idle_timeout_seconds == 0)
|
||||
return;
|
||||
auto timeout = std::chrono::seconds(cfg.mysql_persistent_pool_idle_timeout_seconds);
|
||||
for(size_t i = 0; i < mysql_persistent_pool.size();)
|
||||
{
|
||||
auto& entry = mysql_persistent_pool[i];
|
||||
if(entry.idle_since != std::chrono::steady_clock::time_point() && now - entry.idle_since >= timeout)
|
||||
{
|
||||
delete entry.db;
|
||||
mysql_persistent_pool.erase(mysql_persistent_pool.begin() + i);
|
||||
continue;
|
||||
}
|
||||
i++;
|
||||
}
|
||||
}
|
||||
|
||||
void mysql_release(MySQL* db)
|
||||
{
|
||||
if(!db)
|
||||
return;
|
||||
db->request_leases = 0;
|
||||
for(auto& entry : mysql_persistent_pool)
|
||||
if(entry.db == db)
|
||||
{
|
||||
entry.idle_since = std::chrono::steady_clock::now();
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
MySQL* mysql_checkout(const String& host, const String& username, const String& password, const String& database, bool& reused, bool& persistent)
|
||||
{
|
||||
mysql_evict_idle();
|
||||
reused = false;
|
||||
persistent = false;
|
||||
for(size_t i = 0; i < mysql_persistent_pool.size(); i++)
|
||||
{
|
||||
MySQL* db = mysql_persistent_pool[i];
|
||||
MySQL* db = mysql_persistent_pool[i].db;
|
||||
if(!db || !db->connection || db->request_host != host || db->request_username != username || db->request_password != password || db->request_database != database)
|
||||
continue;
|
||||
if(db->reset_connection())
|
||||
{
|
||||
mysql_persistent_pool[i].idle_since = std::chrono::steady_clock::time_point();
|
||||
if(i + 1 < mysql_persistent_pool.size())
|
||||
{
|
||||
auto entry = mysql_persistent_pool[i];
|
||||
mysql_persistent_pool.erase(mysql_persistent_pool.begin() + i);
|
||||
mysql_persistent_pool.push_back(db);
|
||||
mysql_persistent_pool.push_back(entry);
|
||||
}
|
||||
reused = true;
|
||||
persistent = true;
|
||||
@@ -1326,15 +1430,17 @@ public:
|
||||
{
|
||||
while(mysql_persistent_pool.size() >= cfg.mysql_persistent_pool_size)
|
||||
{
|
||||
delete mysql_persistent_pool.front();
|
||||
delete mysql_persistent_pool.front().db;
|
||||
mysql_persistent_pool.erase(mysql_persistent_pool.begin());
|
||||
}
|
||||
mysql_persistent_pool.push_back(db);
|
||||
WasmMySQLPersistentConnection entry;
|
||||
entry.db = db;
|
||||
mysql_persistent_pool.push_back(entry);
|
||||
}
|
||||
return(db);
|
||||
}
|
||||
|
||||
std::vector<MySQL*> mysql_persistent_pool;
|
||||
std::vector<WasmMySQLPersistentConnection> mysql_persistent_pool;
|
||||
#endif
|
||||
|
||||
String init()
|
||||
@@ -1557,14 +1663,30 @@ public:
|
||||
|
||||
static String serialize_module_artifact(const String& wasm_path)
|
||||
{
|
||||
int lock_fd = open((wasm_path + ".lock").c_str(), O_RDWR | O_CREAT | O_CLOEXEC, 0666);
|
||||
if(lock_fd < 0 || flock(lock_fd, LOCK_EX) != 0)
|
||||
{
|
||||
if(lock_fd >= 0)
|
||||
close(lock_fd);
|
||||
return("cannot lock " + wasm_path);
|
||||
}
|
||||
struct SerializationLock
|
||||
{
|
||||
int fd;
|
||||
~SerializationLock() { flock(fd, LOCK_UN); close(fd); }
|
||||
} lock{lock_fd};
|
||||
if(!serialized_module_needs_refresh(wasm_path))
|
||||
return("");
|
||||
struct stat initial_stat;
|
||||
if(stat(wasm_path.c_str(), &initial_stat) != 0 || !S_ISREG(initial_stat.st_mode))
|
||||
return("cannot stat " + wasm_path);
|
||||
std::vector<u8> bytes;
|
||||
if(!wasm_read_file(wasm_path, bytes))
|
||||
return("cannot read " + wasm_path);
|
||||
wasmtime::Engine engine = make_engine();
|
||||
String error;
|
||||
auto module = compile_and_cache_module(engine, cached_wasm_path(wasm_path), bytes, error);
|
||||
auto module = compile_and_cache_module(engine, cached_wasm_path(wasm_path), bytes, error,
|
||||
wasm_path, &initial_stat);
|
||||
if(!module)
|
||||
return(error);
|
||||
if(serialized_module_needs_refresh(wasm_path))
|
||||
@@ -1612,8 +1734,15 @@ private:
|
||||
return(std::nullopt);
|
||||
}
|
||||
|
||||
static bool same_artifact(const struct stat& left, const struct stat& right)
|
||||
{
|
||||
return(left.st_dev == right.st_dev && left.st_ino == right.st_ino && left.st_size == right.st_size &&
|
||||
left.st_mtim.tv_sec == right.st_mtim.tv_sec && left.st_mtim.tv_nsec == right.st_mtim.tv_nsec &&
|
||||
left.st_ctim.tv_sec == right.st_ctim.tv_sec && left.st_ctim.tv_nsec == right.st_ctim.tv_nsec);
|
||||
}
|
||||
|
||||
static std::optional<wasmtime::Module> compile_and_cache_module(wasmtime::Engine& engine, const String& cached_path,
|
||||
std::vector<u8>& bytes, String& compile_error)
|
||||
std::vector<u8>& bytes, String& compile_error, const String& source_path = "", const struct stat* expected_source = 0)
|
||||
{
|
||||
auto compiled = wasmtime::Module::compile(engine, bytes);
|
||||
if(!compiled)
|
||||
@@ -1636,6 +1765,14 @@ private:
|
||||
{
|
||||
out.flush();
|
||||
out.close();
|
||||
struct stat current_source;
|
||||
if(expected_source && (stat(source_path.c_str(), ¤t_source) != 0 ||
|
||||
!same_artifact(*expected_source, current_source)))
|
||||
{
|
||||
(void)std::remove(tmp.c_str());
|
||||
compile_error = "wasm artifact changed during serialization: " + source_path;
|
||||
return(std::nullopt);
|
||||
}
|
||||
if(std::rename(tmp.c_str(), cached_path.c_str()) != 0)
|
||||
(void)std::remove(tmp.c_str());
|
||||
}
|
||||
@@ -1752,6 +1889,7 @@ public:
|
||||
bool writable = false;
|
||||
};
|
||||
std::vector<FileHandle> file_handles;
|
||||
std::vector<int> socket_fds;
|
||||
|
||||
struct RequestPerfSnapshot
|
||||
{
|
||||
@@ -1909,7 +2047,9 @@ public:
|
||||
// the wasm-side enforcement of request-scoped DB lifecycle; app code should
|
||||
// never cache these opaque handles across requests.
|
||||
std::vector<SQLite*> sqlite_handles;
|
||||
std::vector<MySQL*> mysql_handles;
|
||||
std::map<u64, MySQL*> mysql_handles;
|
||||
std::set<u64> mysql_task_handles;
|
||||
u64 mysql_next_handle = 1;
|
||||
std::vector<MySQL*> mysql_request_pool;
|
||||
std::vector<MySQL*> mysql_request_owned;
|
||||
#endif
|
||||
@@ -1926,13 +2066,20 @@ public:
|
||||
h.fd = -1;
|
||||
}
|
||||
}
|
||||
for(auto& fd : socket_fds)
|
||||
{
|
||||
if(fd >= 0)
|
||||
{
|
||||
::socket_close((u64)fd);
|
||||
fd = -1;
|
||||
}
|
||||
}
|
||||
#ifdef UCE_WASM_HOST_CONNECTORS
|
||||
for(auto* db : sqlite_handles)
|
||||
if(db)
|
||||
delete db; // ~SQLite disconnects
|
||||
for(auto* db : mysql_request_pool)
|
||||
if(db)
|
||||
db->request_leases = 0;
|
||||
worker.mysql_release(db);
|
||||
for(auto* db : mysql_request_owned)
|
||||
if(db)
|
||||
delete db; // ~MySQL disconnects
|
||||
@@ -3559,6 +3706,33 @@ private:
|
||||
return(add([self](Caller, Span<const Val> args, Span<Val> results) -> Result<std::monostate, Trap> { String password,encoded; self->hostcall_read(args[0].i32(), args[1].i32(), password); self->hostcall_read(args[2].i32(), args[3].i32(), encoded); bool valid=password_verify_native(password,encoded); results[0]=Val((int32_t)(valid?1:0)); return(std::monostate()); }));
|
||||
if(mod == "env" && name == "uce_host_password_needs_rehash")
|
||||
return(add([self](Caller, Span<const Val> args, Span<Val> results) -> Result<std::monostate, Trap> { String encoded; self->hostcall_read(args[0].i32(), args[1].i32(), encoded); results[0]=Val((int32_t)(password_needs_rehash_native(encoded)?1:0)); return(std::monostate()); }));
|
||||
if(mod == "env" && name == "uce_host_crypto_operation")
|
||||
return(add([self](Caller, Span<const Val> args, Span<Val> results) -> Result<std::monostate, Trap> {
|
||||
String encoded, out;
|
||||
DValue request, response;
|
||||
String error;
|
||||
int32_t input_size = args[1].i32();
|
||||
u32 cap = (u32)args[3].i32();
|
||||
int32_t buf = args[2].i32();
|
||||
if(input_size > 0 && input_size <= 64 * 1024 && self->hostcall_read(args[0].i32(), input_size, encoded) == "" && ucb_decode(encoded, request, &error))
|
||||
{
|
||||
String stage_key = "crypto_operation:" + encoded;
|
||||
if(!self->hostcall_staged(stage_key, out))
|
||||
{
|
||||
response = crypto_operation_native(request);
|
||||
out = ucb_encode(response);
|
||||
if(buf == 0) self->hostcall_stage(stage_key, out);
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
response["error"] = "invalid_request";
|
||||
out = ucb_encode(response);
|
||||
}
|
||||
if(buf && cap >= out.size()) self->hostcall_write(buf, out);
|
||||
results[0] = Val((int32_t)out.size());
|
||||
return(std::monostate());
|
||||
}));
|
||||
if(mod == "env" && name == "uce_host_log")
|
||||
return(add([self](Caller, Span<const Val> args, Span<Val>) -> Result<std::monostate, Trap> {
|
||||
String text;
|
||||
@@ -3985,7 +4159,7 @@ private:
|
||||
if(mod == "env" && name == "uce_host_file_truncate")
|
||||
return(add([self](Caller, Span<const Val> args, Span<Val> results) -> Result<std::monostate, Trap> { String path,current; self->hostcall_read(args[0].i32(),args[1].i32(),path); self->hostcall_read(args[2].i32(),args[3].i32(),current); String r=self->resolve_guest_write(path,current); results[0]=Val((int32_t)(r!=""&&truncate(r.c_str(),(off_t)args[4].i64())==0)); return(std::monostate()); }));
|
||||
if(mod == "env" && name == "uce_host_dir_remove")
|
||||
return(add([self](Caller, Span<const Val> args, Span<Val> results) -> Result<std::monostate, Trap> { String path,current; self->hostcall_read(args[0].i32(),args[1].i32(),path); self->hostcall_read(args[2].i32(),args[3].i32(),current); String r=self->resolve_guest_write(path,current); bool rec=args[4].i32()!=0; bool ok=false; if(r!="") { if(rec) ok=std::filesystem::remove_all(r)>0; else ok=::rmdir(r.c_str())==0; } results[0]=Val((int32_t)ok); return(std::monostate()); }));
|
||||
return(add([self](Caller, Span<const Val> args, Span<Val> results) -> Result<std::monostate, Trap> { String path,current; self->hostcall_read(args[0].i32(),args[1].i32(),path); self->hostcall_read(args[2].i32(),args[3].i32(),current); String r=self->resolve_guest_write(path,current); bool rec=args[4].i32()!=0; bool ok=false; if(r!="") { if(rec) { try { ok=std::filesystem::remove_all(r)>0; } catch(...) { ok=false; } } else ok=::rmdir(r.c_str())==0; } results[0]=Val((int32_t)ok); return(std::monostate()); }));
|
||||
if(mod == "env" && name == "uce_host_file_temp")
|
||||
return(add([self](Caller, Span<const Val> args, Span<Val> results) -> Result<std::monostate, Trap> { String prefix,current; self->hostcall_read(args[0].i32(),args[1].i32(),prefix); self->hostcall_read(args[2].i32(),args[3].i32(),current); u32 cap=(u32)args[5].i32(); int32_t buf=args[4].i32(); String out; String stage_key="file_temp:"+prefix+"\0"+current; if(!self->hostcall_staged(stage_key,out)) { if(prefix=="") prefix="/tmp/uce-temp"; String templ=self->resolve_guest_write(prefix+"XXXXXX",current); if(templ!="") { std::vector<char> t(templ.begin(), templ.end()); t.push_back(0); int fd=mkstemp(t.data()); if(fd>=0) { close(fd); out=t.data(); } } if(buf==0) self->hostcall_stage(stage_key,out); } if(buf&&cap>=out.size()) self->hostcall_write(buf,out); results[0]=Val((int32_t)out.size()); return(std::monostate()); }));
|
||||
if(mod == "env" && name == "uce_host_file_chmod")
|
||||
@@ -4195,9 +4369,15 @@ private:
|
||||
}));
|
||||
if(mod == "env" && name == "uce_host_memcache_command")
|
||||
return(add([self](Caller caller, Span<const Val> args, Span<Val> results) -> Result<std::monostate, Trap> {
|
||||
u64 handle = (u64)args[0].i64();
|
||||
if(handle < 1 || handle > self->socket_fds.size() || self->socket_fds[(size_t)handle - 1] < 0)
|
||||
{
|
||||
results[0] = Val((int32_t)0);
|
||||
return(std::monostate());
|
||||
}
|
||||
String command;
|
||||
self->hostcall_read(args[1].i32(), args[2].i32(), command);
|
||||
String key = "memcache:" + std::to_string((u64)args[0].i64()) + ":" + command;
|
||||
String key = "memcache:" + std::to_string(handle) + ":" + command;
|
||||
u32 cap = (u32)args[4].i32();
|
||||
int32_t buf = args[3].i32();
|
||||
String out;
|
||||
@@ -4209,9 +4389,8 @@ private:
|
||||
}
|
||||
else
|
||||
{
|
||||
u64 socket_fd = (u64)args[0].i64();
|
||||
wasm_socket_write_bounded(socket_fd, command + "\r\n", self->bounded_hostcall_timeout_ms(1000));
|
||||
out = wasm_socket_read_bounded(socket_fd, 1024 * 128, self->bounded_hostcall_timeout_ms(1000));
|
||||
u64 socket_fd = (u64)self->socket_fds[(size_t)handle - 1];
|
||||
out = wasm_memcache_exchange(socket_fd, command, self->bounded_hostcall_timeout_ms(1000));
|
||||
if(buf == 0)
|
||||
{
|
||||
self->staged_memcache_key = key;
|
||||
@@ -4248,41 +4427,60 @@ private:
|
||||
String password = request["password"].to_string();
|
||||
String database = request["database"].to_string();
|
||||
MySQL* db = 0;
|
||||
for(auto* pooled : self->mysql_request_pool)
|
||||
if(pooled && pooled->connection && pooled->request_host == host && pooled->request_username == username && pooled->request_password == password && pooled->request_database == database)
|
||||
{
|
||||
db = pooled;
|
||||
connection_source = "request";
|
||||
break;
|
||||
}
|
||||
bool ok = db != 0;
|
||||
if(!db)
|
||||
bool ok = false;
|
||||
if(task_child_process)
|
||||
{
|
||||
bool reused = false;
|
||||
bool persistent = false;
|
||||
db = self->worker.mysql_checkout(host, username, password, database, reused, persistent);
|
||||
connection_source = reused ? "worker" : "new";
|
||||
ok = db && db->connection;
|
||||
if(ok && db->connection)
|
||||
// task() closes inherited descriptors after fork. Never reuse
|
||||
// request/worker MySQL objects whose sockets were just closed.
|
||||
db = new MySQL();
|
||||
ok = db->connect(host, username, password, database);
|
||||
connection_source = "task";
|
||||
if(ok)
|
||||
{
|
||||
self->mysql_request_pool.push_back(db);
|
||||
if(!persistent)
|
||||
self->mysql_request_owned.push_back(db);
|
||||
self->mysql_request_owned.push_back(db);
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
for(auto* pooled : self->mysql_request_pool)
|
||||
if(pooled && pooled->connection && pooled->request_host == host && pooled->request_username == username && pooled->request_password == password && pooled->request_database == database)
|
||||
{
|
||||
db = pooled;
|
||||
connection_source = "request";
|
||||
break;
|
||||
}
|
||||
ok = db != 0;
|
||||
if(!db)
|
||||
{
|
||||
bool reused = false;
|
||||
bool persistent = false;
|
||||
db = self->worker.mysql_checkout(host, username, password, database, reused, persistent);
|
||||
connection_source = reused ? "worker" : "new";
|
||||
ok = db && db->connection;
|
||||
if(ok)
|
||||
{
|
||||
self->mysql_request_pool.push_back(db);
|
||||
if(!persistent)
|
||||
self->mysql_request_owned.push_back(db);
|
||||
}
|
||||
}
|
||||
}
|
||||
u64 handle = 0;
|
||||
if(ok && db->connection)
|
||||
if(ok && db->connection && self->mysql_next_handle != 0)
|
||||
{
|
||||
db->request_leases++;
|
||||
self->mysql_handles.push_back(db);
|
||||
handle = self->mysql_handles.size();
|
||||
handle = self->mysql_next_handle;
|
||||
self->mysql_next_handle = handle == UINT64_MAX ? 0 : handle + 1;
|
||||
self->mysql_handles[handle] = db;
|
||||
if(task_child_process) self->mysql_task_handles.insert(handle);
|
||||
if(connection_source == "new") self->mysql_connection_open_count++;
|
||||
else if(connection_source == "worker") self->mysql_connection_reuse_count++;
|
||||
else if(connection_source == "request") self->mysql_request_pool_hit_count++;
|
||||
}
|
||||
response["handle"] = (f64)handle;
|
||||
response["error_code"] = (f64)db->_preload_next_error_code;
|
||||
response["statement_info"] = db->error();
|
||||
response["error_code"] = (f64)(handle == 0 ? 2000 : db->_preload_next_error_code);
|
||||
response["statement_info"] = handle == 0 ? String("mysql handle space exhausted") : db->error();
|
||||
if(handle == 0 && db)
|
||||
delete db;
|
||||
}
|
||||
@@ -4294,8 +4492,8 @@ private:
|
||||
else
|
||||
{
|
||||
u64 handle = request["handle"].to_u64();
|
||||
MySQL* db = (handle >= 1 && handle <= self->mysql_handles.size())
|
||||
? self->mysql_handles[(size_t)handle - 1] : 0;
|
||||
auto handle_it = self->mysql_handles.find(handle);
|
||||
MySQL* db = handle_it == self->mysql_handles.end() || (task_child_process && self->mysql_task_handles.find(handle) == self->mysql_task_handles.end()) ? 0 : handle_it->second;
|
||||
if(op == "query" && db)
|
||||
{
|
||||
response["result"] = db->query(request["query"].to_string());
|
||||
@@ -4308,7 +4506,20 @@ private:
|
||||
{
|
||||
if(db->request_leases > 0)
|
||||
db->request_leases--;
|
||||
self->mysql_handles[(size_t)handle - 1] = 0;
|
||||
self->mysql_handles.erase(handle);
|
||||
self->mysql_task_handles.erase(handle);
|
||||
if(task_child_process && db->request_leases == 0)
|
||||
{
|
||||
auto erase_db = [db](auto& pool) { pool.erase(std::remove(pool.begin(), pool.end(), db), pool.end()); };
|
||||
erase_db(self->mysql_request_pool);
|
||||
erase_db(self->mysql_request_owned);
|
||||
delete db;
|
||||
}
|
||||
}
|
||||
else if(op == "query" || op == "disconnect")
|
||||
{
|
||||
response["error_code"] = (f64)2000;
|
||||
response["statement_info"] = "mysql handle is invalid or unavailable in this task";
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -4350,24 +4561,52 @@ private:
|
||||
String host;
|
||||
self->hostcall_read(args[0].i32(), args[1].i32(), host);
|
||||
u64 fd = wasm_socket_connect_bounded(host, (u16)args[2].i32(), self->bounded_hostcall_timeout_ms(self->worker.cfg.invocation_timeout_ms));
|
||||
results[0] = Val((int64_t)fd);
|
||||
u64 handle = 0;
|
||||
if(fd > 0)
|
||||
{
|
||||
self->socket_fds.push_back((int)fd);
|
||||
handle = self->socket_fds.size();
|
||||
}
|
||||
results[0] = Val((int64_t)handle);
|
||||
return(std::monostate());
|
||||
}));
|
||||
if(mod == "env" && name == "uce_host_socket_close")
|
||||
return(add([](Caller, Span<const Val> args, Span<Val>) -> Result<std::monostate, Trap> {
|
||||
::socket_close((u64)args[0].i64());
|
||||
return(add([self](Caller, Span<const Val> args, Span<Val>) -> Result<std::monostate, Trap> {
|
||||
u64 handle = (u64)args[0].i64();
|
||||
if(handle >= 1 && handle <= self->socket_fds.size())
|
||||
{
|
||||
int& fd = self->socket_fds[(size_t)handle - 1];
|
||||
if(fd >= 0)
|
||||
{
|
||||
::socket_close((u64)fd);
|
||||
fd = -1;
|
||||
}
|
||||
}
|
||||
return(std::monostate());
|
||||
}));
|
||||
if(mod == "env" && name == "uce_host_socket_write")
|
||||
return(add([self](Caller, Span<const Val> args, Span<Val> results) -> Result<std::monostate, Trap> {
|
||||
u64 handle = (u64)args[0].i64();
|
||||
String data;
|
||||
self->hostcall_read(args[1].i32(), args[2].i32(), data);
|
||||
results[0] = Val(wasm_socket_write_bounded((u64)args[0].i64(), data, self->bounded_hostcall_timeout_ms(self->worker.cfg.invocation_timeout_ms)) ? (int32_t)1 : (int32_t)0);
|
||||
bool ok = false;
|
||||
if(handle >= 1 && handle <= self->socket_fds.size())
|
||||
{
|
||||
int fd = self->socket_fds[(size_t)handle - 1];
|
||||
ok = fd >= 0 && wasm_socket_write_bounded((u64)fd, data, self->bounded_hostcall_timeout_ms(self->worker.cfg.invocation_timeout_ms));
|
||||
}
|
||||
results[0] = Val(ok ? (int32_t)1 : (int32_t)0);
|
||||
return(std::monostate());
|
||||
}));
|
||||
if(mod == "env" && name == "uce_host_socket_read")
|
||||
return(add([self](Caller caller, Span<const Val> args, Span<Val> results) -> Result<std::monostate, Trap> {
|
||||
u64 sockfd = (u64)args[0].i64();
|
||||
u64 handle = (u64)args[0].i64();
|
||||
if(handle < 1 || handle > self->socket_fds.size() || self->socket_fds[(size_t)handle - 1] < 0)
|
||||
{
|
||||
results[0] = Val((int32_t)0);
|
||||
return(std::monostate());
|
||||
}
|
||||
u64 sockfd = (u64)self->socket_fds[(size_t)handle - 1];
|
||||
u32 max_length = (u32)args[1].i32();
|
||||
u32 requested_timeout = (u32)args[2].i32();
|
||||
u64 requested_ms = requested_timeout == 0 ? self->invocation_remaining_ms() : (u64)requested_timeout * 1000;
|
||||
@@ -4375,7 +4614,7 @@ private:
|
||||
int32_t buf = args[3].i32();
|
||||
u32 cap = (u32)args[4].i32();
|
||||
// The size and fetch calls share this key, while the remaining budget may change between them.
|
||||
String key = std::to_string(sockfd) + ":" + std::to_string(max_length) + ":" + std::to_string(requested_timeout);
|
||||
String key = std::to_string(handle) + ":" + std::to_string(max_length) + ":" + std::to_string(requested_timeout);
|
||||
String out;
|
||||
if(buf != 0 && self->staged_socket_read_key == key)
|
||||
{
|
||||
@@ -4442,7 +4681,7 @@ private:
|
||||
// before the hostcall stack unwinds, so `self` points to the child's
|
||||
// copy of this per-request workspace. The parent request can return and
|
||||
// destroy its workspace without invalidating the child copy.
|
||||
u64 task_timeout_ms = timeout > UINT64_MAX / 1000 ? UINT64_MAX : timeout * 1000;
|
||||
u64 task_timeout_ms = timeout == 0 ? UINT64_MAX : (timeout > UINT64_MAX / 1000 ? UINT64_MAX : timeout * 1000);
|
||||
auto run_callback = [self, callback_id, task_timeout_ms]() {
|
||||
String error = self->run_task_callback(callback_id, task_timeout_ms);
|
||||
if(error != "")
|
||||
@@ -4597,6 +4836,9 @@ inline WasmResponse wasm_worker_serve(WasmWorker& worker, const Request& request
|
||||
WasmResponse response;
|
||||
f64 serve_started = time_precise();
|
||||
auto workspace_start = std::chrono::steady_clock::now();
|
||||
#ifdef UCE_WASM_HOST_CONNECTORS
|
||||
worker.mysql_evict_idle(workspace_start);
|
||||
#endif
|
||||
f64 cpu_started = wasm_thread_cpu_time();
|
||||
struct rusage thread_runtime_start = {};
|
||||
bool thread_runtime_profiled = worker.cfg.profile_thread_runtime && getrusage(RUSAGE_THREAD, &thread_runtime_start) == 0;
|
||||
|
||||
Reference in New Issue
Block a user