Compare commits
12
Commits
88e2ad022d
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
09f743fdcf | ||
|
|
36d6207bf2 | ||
|
|
d0efab7db0 | ||
|
|
3d155203bd | ||
|
|
68b73343a2 | ||
|
|
10e6565037 | ||
|
|
0c75703186 | ||
|
|
91c745c44b | ||
|
|
e22cbf8882 | ||
|
|
80eaf0b69b | ||
|
|
4d68377704 | ||
|
|
a48f6af502 |
+12
-3
@@ -1,6 +1,11 @@
|
|||||||
# DOCS-TODO — `/doc` documentation overhaul
|
# Completed `/doc` documentation overhaul (historical plan)
|
||||||
|
|
||||||
Briefing for the doc-system overhaul. Read this fully before touching anything.
|
This June 2026 plan is complete and is retained only as implementation history.
|
||||||
|
It is not the current task queue and its one-off execution rules do not govern
|
||||||
|
new UCE work. Current documentation has 269 rendered pages and is covered by
|
||||||
|
the sharded doc gate in `scripts/run_cli_tests.sh`.
|
||||||
|
|
||||||
|
The original briefing follows.
|
||||||
The approved design and the two locked decisions are: **(1) examples are LIVE and
|
The approved design and the two locked decisions are: **(1) examples are LIVE and
|
||||||
SELF-VERIFYING** — real UCE code executed at render time, showing source + actual
|
SELF-VERIFYING** — real UCE code executed at render time, showing source + actual
|
||||||
captured output, gated by the test suite; **(2) FULL SWEEP** of all 257 pages.
|
captured output, gated by the test suite; **(2) FULL SWEEP** of all 257 pages.
|
||||||
@@ -163,7 +168,11 @@ and the suite stays green.
|
|||||||
```
|
```
|
||||||
Expect the current 91 to grow by the new doc gate(s), 0 failed.
|
Expect the current 91 to grow by the new doc gate(s), 0 failed.
|
||||||
|
|
||||||
## Hard rules (non-negotiable)
|
## Historical execution rules (no longer active)
|
||||||
|
|
||||||
|
These rules applied only while the overhaul was being carried out in June 2026.
|
||||||
|
They are preserved to explain that work and must not override current repository
|
||||||
|
or operator instructions.
|
||||||
|
|
||||||
- **NEVER `git commit`/push/tag.** Not pi, not its subagents, not anyone. Leave a clean
|
- **NEVER `git commit`/push/tag.** Not pi, not its subagents, not anyone. Leave a clean
|
||||||
working tree and report.
|
working tree and report.
|
||||||
|
|||||||
@@ -14,9 +14,9 @@ Implement the DOCS-TODO.md overhaul for the UCE `/doc` system on the sshfs sourc
|
|||||||
|
|
||||||
## Current State
|
## Current State
|
||||||
|
|
||||||
- Status: verifying; Phase 1 + gate complete, StringList conversion complete, Phase 2 example/filler sweep complete.
|
- Status: complete; retained as the implementation record for the June 2026 overhaul.
|
||||||
- Last updated: 2026-06-16
|
- Last updated: 2026-07-20
|
||||||
- Source of truth: `/root/mount_ssh/uce-dev-root-htdocs-uce`
|
- Source of truth: `/root/mount_ssh/uce-upstream`
|
||||||
- Runtime/live target: `root@10.4.2.110:/Code/uce.openfu.com/uce`
|
- Runtime/live target: `root@10.4.2.110:/Code/uce.openfu.com/uce`
|
||||||
|
|
||||||
## Goal Tree
|
## Goal Tree
|
||||||
@@ -38,10 +38,10 @@ Legend: `[ ]` not started, `[~]` in progress, `[x]` done, `[!]` blocked, `[-]` s
|
|||||||
- Done when: CLI runner checks every doc page and examples.
|
- Done when: CLI runner checks every doc page and examples.
|
||||||
- Verify: full host gate passes.
|
- Verify: full host gate passes.
|
||||||
|
|
||||||
## Execution Queue
|
## Completed Execution Queue
|
||||||
|
|
||||||
1. G2 area batching/delegation now that the mechanism is proven.
|
1. Phase 2 area batching and delegation completed.
|
||||||
2. Keep the host suite green after each area batch.
|
2. The host suite and sharded 269-page documentation gate remain the ongoing regression boundary.
|
||||||
|
|
||||||
## Decisions
|
## Decisions
|
||||||
|
|
||||||
@@ -55,6 +55,7 @@ Legend: `[ ]` not started, `[~]` in progress, `[x]` done, `[!]` blocked, `[-]` s
|
|||||||
- 2026-06-16: Converted retiring `list_unique/list_sort/list_some/list_every/list_find` free functions to `StringList` methods, updated callers and method docs; full host gate passed: `Summary: 353 passed, 0 failed, 0 skipped`.
|
- 2026-06-16: Converted retiring `list_unique/list_sort/list_some/list_every/list_find` free functions to `StringList` methods, updated callers and method docs; full host gate passed: `Summary: 353 passed, 0 failed, 0 skipped`.
|
||||||
- 2026-06-16: Added live examples across types, string, regex, time, markup, noise, uri, and session area pages; full host gate passed: `Summary: 353 passed, 0 failed, 0 skipped`.
|
- 2026-06-16: Added live examples across types, string, regex, time, markup, noise, uri, and session area pages; full host gate passed: `Summary: 353 passed, 0 failed, 0 skipped`.
|
||||||
- 2026-06-16: Added `:example` blocks to all remaining doc pages, removed full PHP/JS Related Concepts filler, removed duplicate content code fences except resource-bound pages; full host gate passed: `Summary: 353 passed, 0 failed, 0 skipped`.
|
- 2026-06-16: Added `:example` blocks to all remaining doc pages, removed full PHP/JS Related Concepts filler, removed duplicate content code fences except resource-bound pages; full host gate passed: `Summary: 353 passed, 0 failed, 0 skipped`.
|
||||||
|
- 2026-07-20: Marked the finished plan historical so it no longer presents completed work or one-off execution rules as current instructions.
|
||||||
|
|
||||||
## Change Log
|
## Change Log
|
||||||
|
|
||||||
|
|||||||
@@ -169,6 +169,13 @@ systemctl daemon-reload
|
|||||||
systemctl restart uce.service
|
systemctl restart uce.service
|
||||||
```
|
```
|
||||||
|
|
||||||
|
When `uce.socket` is active, restart only `uce.service` (or use
|
||||||
|
`scripts/systemd/manage-uce-service.sh restart`). The socket unit must retain
|
||||||
|
ownership of the FastCGI listener while workers change. Restarting
|
||||||
|
`uce.socket` alongside the service removes that listener briefly and can expose
|
||||||
|
HTTP 502 responses through nginx. Restart the socket unit only to repair the
|
||||||
|
socket itself, with an explicit outage window.
|
||||||
|
|
||||||
## Runtime configuration
|
## Runtime configuration
|
||||||
|
|
||||||
Create `/etc/uce/settings.cfg` from `etc/uce/settings.cfg` and adjust paths to your checkout root. Replace checkout-specific settings such as `WASM_CORE_PATH` with `<UCE_REPO_ROOT>/bin/wasm/core.wasm` or leave it relative to `COMPILER_SYS_PATH` as appropriate.
|
Create `/etc/uce/settings.cfg` from `etc/uce/settings.cfg` and adjust paths to your checkout root. Replace checkout-specific settings such as `WASM_CORE_PATH` with `<UCE_REPO_ROOT>/bin/wasm/core.wasm` or leave it relative to `COMPILER_SYS_PATH` as appropriate.
|
||||||
@@ -185,6 +192,8 @@ FCGI_SOCKET_PATH=/run/uce/fastcgi.sock
|
|||||||
FCGI_SOCKET_MODE=0666
|
FCGI_SOCKET_MODE=0666
|
||||||
CLI_SOCKET_PATH=/run/uce/cli.sock
|
CLI_SOCKET_PATH=/run/uce/cli.sock
|
||||||
CLI_SOCKET_MODE=0600
|
CLI_SOCKET_MODE=0600
|
||||||
|
CLI_WORKER_COUNT=2
|
||||||
|
CLI_WORKER_MAX_REQUESTS=8
|
||||||
|
|
||||||
SITE_DIRECTORY=/var/www/html
|
SITE_DIRECTORY=/var/www/html
|
||||||
HTTP_DOCUMENT_ROOT=/var/www/html
|
HTTP_DOCUMENT_ROOT=/var/www/html
|
||||||
@@ -202,7 +211,9 @@ WASM_MEMORY_LIMIT_BYTES=536870912
|
|||||||
WASM_EPOCH_DEADLINE_TICKS=200
|
WASM_EPOCH_DEADLINE_TICKS=200
|
||||||
WASM_EPOCH_PERIOD_MS=50
|
WASM_EPOCH_PERIOD_MS=50
|
||||||
WASM_INVOCATION_TIMEOUT_MS=30000
|
WASM_INVOCATION_TIMEOUT_MS=30000
|
||||||
|
WASM_SERIALIZE_TIMEOUT_SECONDS=120
|
||||||
MYSQL_PERSISTENT_POOL_SIZE=8
|
MYSQL_PERSISTENT_POOL_SIZE=8
|
||||||
|
MYSQL_PERSISTENT_POOL_IDLE_TIMEOUT_SECONDS=300
|
||||||
|
|
||||||
WORKER_COUNT=4
|
WORKER_COUNT=4
|
||||||
MAX_MEMORY=16777216
|
MAX_MEMORY=16777216
|
||||||
@@ -216,6 +227,8 @@ Important settings:
|
|||||||
|
|
||||||
- `FCGI_SOCKET_PATH` is the Unix socket used for normal `.uce` requests. Set it explicitly and keep this value and the web-server `fastcgi_pass` path identical. The reference config uses `/run/uce/fastcgi.sock`; if you choose `/run/uce.sock`, use it in both places.
|
- `FCGI_SOCKET_PATH` is the Unix socket used for normal `.uce` requests. Set it explicitly and keep this value and the web-server `fastcgi_pass` path identical. The reference config uses `/run/uce/fastcgi.sock`; if you choose `/run/uce.sock`, use it in both places.
|
||||||
- `CLI_SOCKET_PATH` is a local HTTP-over-Unix socket used by `scripts/uce-cli` and test/admin units. Keep it private (`CLI_SOCKET_MODE=0600`) unless you intentionally delegate admin/test execution to a trusted Unix group (`0660`).
|
- `CLI_SOCKET_PATH` is a local HTTP-over-Unix socket used by `scripts/uce-cli` and test/admin units. Keep it private (`CLI_SOCKET_MODE=0600`) unless you intentionally delegate admin/test execution to a trusted Unix group (`0660`).
|
||||||
|
- `CLI_WORKER_COUNT` adds a transport-isolated CLI/test renderer pool. Use at least `2` on a live site so broad test/admin unit loads cannot evict or page out public FastCGI workers' hot modules and a CLI unit can make one nested CLI call without self-deadlocking. `0` preserves the legacy shared pool; dedicated CLI workers are additional to `WORKER_COUNT` and retain their own Wasmtime engine, module cache, and persistent connector pool.
|
||||||
|
- `CLI_WORKER_MAX_REQUESTS` recycles each dedicated CLI worker after this many completed connections (default `8`, maximum `1024`); `0` disables recycling. Public FastCGI workers are not request-count recycled, preserving their warmed module caches. Recycling bounds the module set retained by broad test and administration runs while allowing one CLI invocation/test group to finish uninterrupted.
|
||||||
- `FCGI_SOCKET_MODE` and `CLI_SOCKET_MODE` are octal permission modes applied after socket bind. Prefer tightening `FCGI_SOCKET_MODE` to `0660` when nginx/Apache can share a trusted group with the UCE worker.
|
- `FCGI_SOCKET_MODE` and `CLI_SOCKET_MODE` are octal permission modes applied after socket bind. Prefer tightening `FCGI_SOCKET_MODE` to `0660` when nginx/Apache can share a trusted group with the UCE worker.
|
||||||
- `SITE_DIRECTORY` is the public site tree to scan for `.uce` files. Use `/var/www/html` when the web root is outside the runtime tree; relative paths are resolved from the runtime working directory. Installed regression gate scripts derive their temporary test root from this setting unless `UCE_TEST_SITE_DIRECTORY` is explicitly provided.
|
- `SITE_DIRECTORY` is the public site tree to scan for `.uce` files. Use `/var/www/html` when the web root is outside the runtime tree; relative paths are resolved from the runtime working directory. Installed regression gate scripts derive their temporary test root from this setting unless `UCE_TEST_SITE_DIRECTORY` is explicitly provided.
|
||||||
- `HTTP_DOCUMENT_ROOT` is the root used by the built-in HTTP/WebSocket listener when it resolves upgrade requests. Set it to the same web root as nginx/Apache.
|
- `HTTP_DOCUMENT_ROOT` is the root used by the built-in HTTP/WebSocket listener when it resolves upgrade requests. Set it to the same web root as nginx/Apache.
|
||||||
@@ -225,6 +238,7 @@ Important settings:
|
|||||||
- `TMP_UPLOAD_PATH` and `SESSION_PATH` must be writable by the runtime.
|
- `TMP_UPLOAD_PATH` and `SESSION_PATH` must be writable by the runtime.
|
||||||
- `SESSION_COOKIE_SECURE=1` adds the `Secure` attribute to UCE-managed session cookies and should be used for HTTPS-only deployments. Leave it `0` only for local/plain-HTTP development.
|
- `SESSION_COOKIE_SECURE=1` adds the `Secure` attribute to UCE-managed session cookies and should be used for HTTPS-only deployments. Leave it `0` only for local/plain-HTTP development.
|
||||||
- `MYSQL_PERSISTENT_POOL_SIZE` caps credential-keyed connections retained by each Wasm worker. The default `8` is clamped to `64`; set it to `0` to restore request-lifetime connections. Cached sessions are reset before reuse.
|
- `MYSQL_PERSISTENT_POOL_SIZE` caps credential-keyed connections retained by each Wasm worker. The default `8` is clamped to `64`; set it to `0` to restore request-lifetime connections. Cached sessions are reset before reuse.
|
||||||
|
- `MYSQL_PERSISTENT_POOL_IDLE_TIMEOUT_SECONDS` retires a pooled connection at the first request boundary after it has been idle for this interval (default `300`, maximum `86400`). Set it to `0` to retain idle connections until capacity eviction, reset failure, worker exit, or database-side closure. Keep a positive value below the database server's idle timeout to avoid attempting stale sessions after traffic resumes. A completely idle UCE worker has no request boundary and therefore does not proactively close sockets.
|
||||||
- `HTTP_PORT` is the built-in HTTP/WebSocket listener used for WebSocket upgrade traffic and direct local probes. Bind/firewall it for local access only; nginx/Apache should be the public entry point.
|
- `HTTP_PORT` is the built-in HTTP/WebSocket listener used for WebSocket upgrade traffic and direct local probes. Bind/firewall it for local access only; nginx/Apache should be the public entry point.
|
||||||
- `WS_BROKER_OUTBOUND_TIMEOUT_SECONDS` controls how long a forwarded WS message can remain queued in the broker before being dropped (default `30`). Set to `0` to disable the timeout.
|
- `WS_BROKER_OUTBOUND_TIMEOUT_SECONDS` controls how long a forwarded WS message can remain queued in the broker before being dropped (default `30`). Set to `0` to disable the timeout.
|
||||||
- `WASM_COMPILE_SCRIPT` must point to `scripts/compile_wasm_unit` unless you provide an equivalent compiler. Relative paths are resolved from the runtime root/`COMPILER_SYS_PATH`. That script calls `scripts/check_unit_wasm.py` after linking each unit and uses the pinned WASI SDK on every deployment host.
|
- `WASM_COMPILE_SCRIPT` must point to `scripts/compile_wasm_unit` unless you provide an equivalent compiler. Relative paths are resolved from the runtime root/`COMPILER_SYS_PATH`. That script calls `scripts/check_unit_wasm.py` after linking each unit and uses the pinned WASI SDK on every deployment host.
|
||||||
@@ -255,6 +269,15 @@ Important settings:
|
|||||||
4 KiB positional read. Cache-miss full-artifact reads use the same checks
|
4 KiB positional read. Cache-miss full-artifact reads use the same checks
|
||||||
around 64 KiB positional chunks. A single in-progress regular-file syscall
|
around 64 KiB positional chunks. A single in-progress regular-file syscall
|
||||||
cannot be cooperatively interrupted; later reads and parsing cannot overrun the budget.
|
cannot be cooperatively interrupted; later reads and parsing cannot overrun the budget.
|
||||||
|
Wasmtime's serialized-module deserialization call is also synchronous and
|
||||||
|
cannot be interrupted; if it stalls, the timeout is reported after that call
|
||||||
|
returns rather than at the nominal wall-clock boundary.
|
||||||
|
Proactive and offline serialization instead runs in a short-lived child
|
||||||
|
bounded by `WASM_SERIALIZE_TIMEOUT_SECONDS` (default `120`, clamped to
|
||||||
|
`1`–`3600`), so each unit releases its pooling allocator arenas and threads
|
||||||
|
on exit instead of accumulating them in the long-lived scanner. Serialization
|
||||||
|
holds the unit compile lock and rechecks the wasm artifact identity before
|
||||||
|
publication; timed-out/failed children leave no dead temporary artifact.
|
||||||
Initial/final descriptor identity, unique selected metadata sections, and
|
Initial/final descriptor identity, unique selected metadata sections, and
|
||||||
strict 64-bit LEB high-bit validation reject changed or ambiguous artifacts.
|
strict 64-bit LEB high-bit validation reject changed or ambiguous artifacts.
|
||||||
|
|
||||||
@@ -289,6 +312,7 @@ The server binary accepts only these process modes:
|
|||||||
```bash
|
```bash
|
||||||
bin/uce_fastcgi.linux.bin # start the server
|
bin/uce_fastcgi.linux.bin # start the server
|
||||||
bin/uce_fastcgi.linux.bin --precompile
|
bin/uce_fastcgi.linux.bin --precompile
|
||||||
|
bin/uce_fastcgi.linux.bin --serialize-module /absolute/unit.uce.wasm
|
||||||
bin/uce_fastcgi.linux.bin --help
|
bin/uce_fastcgi.linux.bin --help
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -312,6 +336,12 @@ workers continue to use `/etc/uce/settings.cfg`. The parallel precompile
|
|||||||
regression uses private roots so a running proactive compiler cannot consume
|
regression uses private roots so a running proactive compiler cannot consume
|
||||||
or publish its controlled race fixtures.
|
or publish its controlled race fixtures.
|
||||||
|
|
||||||
|
Direct `--precompile` is a listener-free whole-generation gate, not a
|
||||||
|
single-unit readiness probe. It scans the configured source root, waits for
|
||||||
|
unit publication locks, and has no internal wall-clock timeout. Automation
|
||||||
|
must wrap it in a finite external timeout, as the managed restart does, before
|
||||||
|
starting separately deadline-bounded CLI or HTTP invocations.
|
||||||
|
|
||||||
Prefer the managed restart when the service runs as an unprivileged user: it
|
Prefer the managed restart when the service runs as an unprivileged user: it
|
||||||
precompiles as that same user. Request-time publication still accepts readable
|
precompiles as that same user. Request-time publication still accepts readable
|
||||||
artifacts produced by a trusted administrator. When Linux protected-hardlink
|
artifacts produced by a trusted administrator. When Linux protected-hardlink
|
||||||
@@ -715,6 +745,20 @@ if(valid && password_needs_rehash(encoded))
|
|||||||
|
|
||||||
`password_hash()` returns a self-contained `$uce$scrypt$...` encoding with a random 16-byte salt and the bounded scrypt parameters `N=65536`, `r=8`, `p=1`. `password_verify()` accepts only structurally valid encodings with bounded cost parameters and compares the derived key in constant time. `password_needs_rehash()` reports malformed, legacy, or non-current parameters so applications can upgrade a credential after a successful legacy verification. Treat an empty hash as an operational failure and never store it. Application-level password length policy, rate limiting, and legacy-format verification remain the application's responsibility.
|
`password_hash()` returns a self-contained `$uce$scrypt$...` encoding with a random 16-byte salt and the bounded scrypt parameters `N=65536`, `r=8`, `p=1`. `password_verify()` accepts only structurally valid encodings with bounded cost parameters and compares the derived key in constant time. `password_needs_rehash()` reports malformed, legacy, or non-current parameters so applications can upgrade a credential after a successful legacy verification. Treat an empty hash as an operational failure and never store it. Application-level password length policy, rate limiting, and legacy-format verification remain the application's responsibility.
|
||||||
|
|
||||||
|
## Structured cryptographic operations
|
||||||
|
|
||||||
|
`crypto_operation(request)` is the bounded, algorithm-selected API for structured
|
||||||
|
asymmetric operations. The initial allowlist supports `operation=key_generate`
|
||||||
|
and `operation=jwt_sign` with `algorithm=ES256`. Key generation returns P-256
|
||||||
|
public/private JWKs plus the RFC 7638 thumbprint (`kid`). JWT signing accepts only
|
||||||
|
a consistent P-256 private JWK, forces `alg=ES256`, and emits a compact JWT with
|
||||||
|
a 64-byte JOSE signature. It also supports `cbor_decode`, `cose_es256_parse`, and `es256_verify` with `algorithm=ES256`: inputs are canonical unpadded base64url fields; requests are capped at 32 KiB, while CBOR input is definite-length only and capped at 16 KiB decoded (21,846 base64url characters), 256 nodes, and depth 16; it rejects duplicate keys and trailing bytes, and returns an explicit typed tree (`unsigned`, `negative`, `bytes`, `text`, `array`, `map`) rather than DValue-coerced map keys. COSE accepts only EC2/-7/P-256 with exact 32-byte coordinates and an OpenSSL-valid point. Verification accepts only complete canonical DER ECDSA over `message_base64url` and returns `valid`; malformed material fails closed. V1 has no attestation, Ed25519, or RS256. Unknown operations and algorithms fail closed.
|
||||||
|
|
||||||
|
Existing typed digest, HMAC, password, randomness, and constant-time comparison
|
||||||
|
functions remain separate. `crypto_operation()` exposes no raw signing, arbitrary
|
||||||
|
digest/curve selection, encryption, network fetch, or token exchange. Store
|
||||||
|
private JWKs as secrets.
|
||||||
|
|
||||||
## Operational footguns
|
## Operational footguns
|
||||||
|
|
||||||
- Keep the FastCGI socket path consistent: `FCGI_SOCKET_PATH` and the web-server `fastcgi_pass` must match exactly. The reference config uses `/run/uce/fastcgi.sock`; if you choose `/run/uce.sock`, use it in both places.
|
- Keep the FastCGI socket path consistent: `FCGI_SOCKET_PATH` and the web-server `fastcgi_pass` must match exactly. The reference config uses `/run/uce/fastcgi.sock`; if you choose `/run/uce.sock`, use it in both places.
|
||||||
|
|||||||
@@ -1,8 +1,8 @@
|
|||||||
# UCE WASM Runtime Architecture
|
# UCE WASM Runtime Architecture
|
||||||
|
|
||||||
Status: current as of the W7e native-pipeline removal (June 2026). This document
|
Status: current as of the worker-role isolation and bounded serializer changes
|
||||||
describes the **runtime architecture as built** — the process topology, the
|
(July 2026). This document describes the **runtime architecture as built**. It
|
||||||
wasm membrane, the unified request dispatch, and the central WebSocket broker.
|
covers the process topology, wasm membrane, request dispatch, and WebSocket broker.
|
||||||
Native `.so` unit execution/dlopen fallback has been removed; the parser and
|
Native `.so` unit execution/dlopen fallback has been removed; the parser and
|
||||||
preprocessor remain only as the front-end that emits C++ for wasm side-module
|
preprocessor remain only as the front-end that emits C++ for wasm side-module
|
||||||
compilation.
|
compilation.
|
||||||
@@ -26,8 +26,8 @@ gets invoked*.
|
|||||||
|
|
||||||
```
|
```
|
||||||
┌────────────────────────────┐
|
┌────────────────────────────┐
|
||||||
nginx ──FastCGI──► worker pool (N processes) │ $FCGI_SOCKET_PATH (example `/run/uce/fastcgi.sock`)
|
nginx ──FastCGI──► public workers (N processes)│ $FCGI_SOCKET_PATH (example `/run/uce/fastcgi.sock`)
|
||||||
(port 80 etc.) │ uniform unit renderers │ (FastCGI + CLI)
|
(port 80 etc.) │ public unit renderers │
|
||||||
└─────────────▲──────────────┘
|
└─────────────▲──────────────┘
|
||||||
│ forward render (FastCGI, FCGI_SOCKET_PATH)
|
│ forward render (FastCGI, FCGI_SOCKET_PATH)
|
||||||
│
|
│
|
||||||
@@ -42,13 +42,19 @@ gets invoked*.
|
|||||||
│ dispatcher(s) │ bind addr; forwards to pool
|
│ dispatcher(s) │ bind addr; forwards to pool
|
||||||
└──────────────────┘
|
└──────────────────┘
|
||||||
|
|
||||||
|
trusted CLI/test calls ──► ┌──────────────────┐
|
||||||
|
(CLI socket) │ CLI workers │ isolated module and
|
||||||
|
│ (M processes) │ connector caches
|
||||||
|
└──────────────────┘
|
||||||
|
|
||||||
parent process: spawns/respawns all of the above + the proactive compiler.
|
parent process: spawns/respawns all of the above + the proactive compiler.
|
||||||
```
|
```
|
||||||
|
|
||||||
| Process | Owns | Renders units? | Source |
|
| Process | Owns | Renders units? | Source |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| **Parent** | nothing; supervises children | no | `main()`, `init_base_process()` |
|
| **Parent** | nothing; supervises children | no | `main()`, `init_base_process()` |
|
||||||
| **Worker** (×`WORKER_COUNT`) | `FCGI_SOCKET_PATH` (configured socket path; example `/run/uce/fastcgi.sock`) + `CLI_SOCKET_PATH` | **yes** — the only processes that run wasm | `listen_for_connections()` |
|
| **Public worker** (×`WORKER_COUNT`) | `FCGI_SOCKET_PATH`; also `CLI_SOCKET_PATH` only when `CLI_WORKER_COUNT=0` | **yes** — runs public FastCGI wasm | `listen_for_connections()` |
|
||||||
|
| **CLI worker** (×`CLI_WORKER_COUNT`) | `CLI_SOCKET_PATH` only | **yes** — isolates trusted CLI/test wasm and its module cache | `listen_for_connections()` |
|
||||||
| **WS broker** (×1) | `HTTP_PORT` + every live WS connection + `WS_BROKER_SOCKET_PATH` | no — forwards to the pool | `run_ws_broker()` |
|
| **WS broker** (×1) | `HTTP_PORT` + every live WS connection + `WS_BROKER_SOCKET_PATH` | no — forwards to the pool | `run_ws_broker()` |
|
||||||
| **serve_http dispatcher** (×bind) | one custom-server bind address | no — forwards to the pool | `custom_server_http_dispatcher_loop()` |
|
| **serve_http dispatcher** (×bind) | one custom-server bind address | no — forwards to the pool | `custom_server_http_dispatcher_loop()` |
|
||||||
| **Proactive compiler** | nothing; pre-compiles units | no | `run_proactive_compiler()` |
|
| **Proactive compiler** | nothing; pre-compiles units | no | `run_proactive_compiler()` |
|
||||||
@@ -192,6 +198,9 @@ the service and its post-fork workers restart. Direct launches without systemd
|
|||||||
activation retain the existing configured Unix/TCP listener behavior.
|
activation retain the existing configured Unix/TCP listener behavior.
|
||||||
The service preserves its runtime directory across service restarts because the
|
The service preserves its runtime directory across service restarts because the
|
||||||
socket unit, not the service unit, owns a listener path inside that directory.
|
socket unit, not the service unit, owns a listener path inside that directory.
|
||||||
|
Normal deployments therefore restart only `uce.service`; restarting
|
||||||
|
`uce.socket` at the same time removes the listener and forfeits the zero-gap
|
||||||
|
handoff.
|
||||||
On termination the parent asks render workers to close their listeners, finish
|
On termination the parent asks render workers to close their listeners, finish
|
||||||
accepted connections within the bounded worker drain interval, and only then
|
accepted connections within the bounded worker drain interval, and only then
|
||||||
exits. This prevents an accepted FastCGI request from being reset at handoff;
|
exits. This prevents an accepted FastCGI request from being reset at handoff;
|
||||||
@@ -211,6 +220,14 @@ hostcall loop therefore cannot renew an invocation indefinitely. Blocking host
|
|||||||
helpers retain operation-specific limits and cap them to the remaining
|
helpers retain operation-specific limits and cap them to the remaining
|
||||||
invocation budget where the underlying operation is cancellable. Forked task
|
invocation budget where the underlying operation is cancellable. Forked task
|
||||||
callbacks receive a fresh invocation deadline capped by the task lifetime.
|
callbacks receive a fresh invocation deadline capped by the task lifetime.
|
||||||
|
The Memcached ASCII hostcall applies one deadline to the complete command write
|
||||||
|
and protocol-framed response read. Retrieval payload lengths are parsed, so a
|
||||||
|
payload containing `END` cannot terminate the response; multi-value and `stats`
|
||||||
|
responses may span socket chunks up to the 8 MiB response bound. Timeout,
|
||||||
|
malformed framing, or overflow shuts down the retained descriptor before
|
||||||
|
returning an empty result, preventing unread response bytes from poisoning the
|
||||||
|
next command on that connection. The native fallback uses the same parser and
|
||||||
|
deadline contract.
|
||||||
Synchronous compiler locks, transitive `#load` compilation, and compiler child
|
Synchronous compiler locks, transitive `#load` compilation, and compiler child
|
||||||
processes consume that same deadline. Compiler children run in a dedicated
|
processes consume that same deadline. Compiler children run in a dedicated
|
||||||
process group; timeout kills the group, retains the previous generation, and
|
process group; timeout kills the group, retains the previous generation, and
|
||||||
@@ -266,6 +283,10 @@ Cold module compilation and deserialization are host work, so `load_unit()`
|
|||||||
refreshes the epoch deadline before its first guest call. Otherwise a component
|
refreshes the epoch deadline before its first guest call. Otherwise a component
|
||||||
whose compilation outlasted the guest CPU budget would immediately trap in the
|
whose compilation outlasted the guest CPU budget would immediately trap in the
|
||||||
following allocator/relocation call even though no guest loop consumed it.
|
following allocator/relocation call even though no guest loop consumed it.
|
||||||
|
Wasmtime's serialized-module deserialization API is synchronous and has no
|
||||||
|
cooperative cancellation point. The absolute invocation timeout is checked
|
||||||
|
immediately after it returns, so a stalled deserialization may overrun the
|
||||||
|
nominal wall boundary before UCE reports the timeout.
|
||||||
|
|
||||||
Unit artifacts live beneath an ABI-generation directory such as
|
Unit artifacts live beneath an ABI-generation directory such as
|
||||||
`BIN_DIRECTORY/units-c13-w7`: `c13` is the compiler/unit-metadata ABI and `w7`
|
`BIN_DIRECTORY/units-c13-w7`: `c13` is the compiler/unit-metadata ABI and `w7`
|
||||||
@@ -553,7 +574,15 @@ header free-functions are `inline`. The wasm backend exposes only declarations
|
|||||||
| `HTTP_PORT` | `8080` | Raw HTTP + WebSocket port — owned by the WS broker. |
|
| `HTTP_PORT` | `8080` | Raw HTTP + WebSocket port — owned by the WS broker. |
|
||||||
| `WS_BROKER_SOCKET_PATH` | `/run/uce/ws-broker.sock` | Broker command socket for `ws_*` flushes. |
|
| `WS_BROKER_SOCKET_PATH` | `/run/uce/ws-broker.sock` | Broker command socket for `ws_*` flushes. |
|
||||||
| `WS_BROKER_OUTBOUND_TIMEOUT_SECONDS` | `30` | Max lifetime in seconds for queued WS broker forwards before drop. |
|
| `WS_BROKER_OUTBOUND_TIMEOUT_SECONDS` | `30` | Max lifetime in seconds for queued WS broker forwards before drop. |
|
||||||
| `WORKER_COUNT` | `4` | Number of uniform worker processes. |
|
| `WORKER_COUNT` | `4` | Number of public FastCGI worker processes. |
|
||||||
|
| `CLI_WORKER_COUNT` | `0` built-in; `2` in the reference config | Additional CLI-only workers. Two permit one nested CLI invocation without self-deadlock; a positive count prevents test/admin module-cache churn from paging out public workers, while zero preserves the legacy shared pool. |
|
||||||
|
| `CLI_WORKER_MAX_REQUESTS` | `8` | Completed CLI connections before a dedicated CLI worker is recycled (maximum `1024`; `0` disables). Public workers are not request-count recycled. |
|
||||||
|
| `WASM_SERIALIZE_TIMEOUT_SECONDS` | `120` | Deadline for each short-lived serialized-module child; process exit reclaims pooling allocator arenas from proactive scanners. |
|
||||||
|
|
||||||
|
Proactive scanners never construct a Wasmtime serialization engine themselves.
|
||||||
|
Each candidate is serialized by the executable's bounded `--serialize-module`
|
||||||
|
child while holding the unit compile lock. Publication rechecks device, inode,
|
||||||
|
size, mtime, and ctime, and the scanner removes dead child temporary artifacts.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@@ -14,6 +14,12 @@ FCGI_PORT=9993
|
|||||||
CLI_SOCKET_PATH=/run/uce/cli.sock
|
CLI_SOCKET_PATH=/run/uce/cli.sock
|
||||||
# Keep the CLI/admin socket private by default; set 0660 only for a trusted group.
|
# Keep the CLI/admin socket private by default; set 0660 only for a trusted group.
|
||||||
CLI_SOCKET_MODE=0600
|
CLI_SOCKET_MODE=0600
|
||||||
|
# Keep CLI/test module caches out of public FastCGI workers. Set to 0 only for
|
||||||
|
# the legacy shared pool; dedicated workers are additional to WORKER_COUNT.
|
||||||
|
CLI_WORKER_COUNT=2
|
||||||
|
# Recycle dedicated CLI workers before broad test/admin runs retain every unit.
|
||||||
|
# Zero disables recycling; public FastCGI workers are never request-count recycled.
|
||||||
|
CLI_WORKER_MAX_REQUESTS=8
|
||||||
|
|
||||||
# Built-in HTTP/WebSocket listener used for WebSocket Upgrade requests.
|
# Built-in HTTP/WebSocket listener used for WebSocket Upgrade requests.
|
||||||
# Keep this behind nginx/Apache on localhost or firewall it from public access.
|
# Keep this behind nginx/Apache on localhost or firewall it from public access.
|
||||||
@@ -59,7 +65,10 @@ WASM_MEMORY_LIMIT_BYTES=536870912
|
|||||||
WASM_EPOCH_DEADLINE_TICKS=200
|
WASM_EPOCH_DEADLINE_TICKS=200
|
||||||
WASM_EPOCH_PERIOD_MS=50
|
WASM_EPOCH_PERIOD_MS=50
|
||||||
WASM_INVOCATION_TIMEOUT_MS=30000
|
WASM_INVOCATION_TIMEOUT_MS=30000
|
||||||
|
# Bound each short-lived proactive/precompile serialized-module child.
|
||||||
|
WASM_SERIALIZE_TIMEOUT_SECONDS=120
|
||||||
MYSQL_PERSISTENT_POOL_SIZE=8
|
MYSQL_PERSISTENT_POOL_SIZE=8
|
||||||
|
MYSQL_PERSISTENT_POOL_IDLE_TIMEOUT_SECONDS=300
|
||||||
|
|
||||||
# ENABLE THE BACKGROUND PROACTIVE COMPILER LOOP
|
# ENABLE THE BACKGROUND PROACTIVE COMPILER LOOP
|
||||||
PROACTIVE_COMPILE_ENABLED=1
|
PROACTIVE_COMPILE_ENABLED=1
|
||||||
|
|||||||
@@ -39,6 +39,7 @@ PUBLIC_APIS = [
|
|||||||
("hmac_sha256", True, "public"), ("hmac_sha256_hex", True, "public"), ("random_bytes", True, "public"),
|
("hmac_sha256", True, "public"), ("hmac_sha256_hex", True, "public"), ("random_bytes", True, "public"),
|
||||||
("crypto_equal", True, "public"), ("password_hash", True, "public"),
|
("crypto_equal", True, "public"), ("password_hash", True, "public"),
|
||||||
("password_verify", True, "public"), ("password_needs_rehash", True, "public"),
|
("password_verify", True, "public"), ("password_needs_rehash", True, "public"),
|
||||||
|
("crypto_operation", True, "public"),
|
||||||
("gen_noise32", True, "public"), ("gen_noise64", True, "public"),
|
("gen_noise32", True, "public"), ("gen_noise64", True, "public"),
|
||||||
("gen_noise01", True, "public"), ("gen_int", True, "public"), ("gen_float", True, "public"),
|
("gen_noise01", True, "public"), ("gen_int", True, "public"), ("gen_float", True, "public"),
|
||||||
("draw_int", True, "public"), ("draw_float", True, "public"),
|
("draw_int", True, "public"), ("draw_float", True, "public"),
|
||||||
|
|||||||
@@ -67,6 +67,7 @@ if [[ "$action" == "run" ]]; then
|
|||||||
curl -sS --max-time "$curl_timeout" --fail-with-body --unix-socket "$socket_path" "${base_url}&group=${group}"
|
curl -sS --max-time "$curl_timeout" --fail-with-body --unix-socket "$socket_path" "${base_url}&group=${group}"
|
||||||
done
|
done
|
||||||
scripts/test_dependency_invalidation.sh
|
scripts/test_dependency_invalidation.sh
|
||||||
|
scripts/test_compiler_lock_directory.sh
|
||||||
scripts/test_abi_generation_rollout.sh
|
scripts/test_abi_generation_rollout.sh
|
||||||
scripts/test_parallel_precompile.sh
|
scripts/test_parallel_precompile.sh
|
||||||
timeout --signal=TERM --kill-after=5s 175s scripts/test_parallel_proactive_compile.sh
|
timeout --signal=TERM --kill-after=5s 175s scripts/test_parallel_proactive_compile.sh
|
||||||
@@ -77,8 +78,11 @@ if [[ "$action" == "run" ]]; then
|
|||||||
scripts/test_component_once_prefetch.sh
|
scripts/test_component_once_prefetch.sh
|
||||||
scripts/test_relative_component_cache.sh
|
scripts/test_relative_component_cache.sh
|
||||||
scripts/test_password_hashing.sh
|
scripts/test_password_hashing.sh
|
||||||
|
scripts/test_crypto_operation_native.sh
|
||||||
|
scripts/test_hardened_http_native.sh
|
||||||
scripts/test_mysql_epoch_refresh.sh
|
scripts/test_mysql_epoch_refresh.sh
|
||||||
scripts/test_mysql_persistent_pool.sh
|
scripts/test_mysql_persistent_pool.sh
|
||||||
|
scripts/test_mysql_persistent_pool_idle.sh
|
||||||
scripts/test_log_timeliness.sh
|
scripts/test_log_timeliness.sh
|
||||||
scripts/test_raw_http_request_log.sh
|
scripts/test_raw_http_request_log.sh
|
||||||
scripts/test_component_resolution_ttl.sh
|
scripts/test_component_resolution_ttl.sh
|
||||||
@@ -89,5 +93,6 @@ if [[ "$action" == "run" ]]; then
|
|||||||
timeout --signal=TERM --kill-after=5s 240s scripts/test_dynamic_compile_failures.sh
|
timeout --signal=TERM --kill-after=5s 240s scripts/test_dynamic_compile_failures.sh
|
||||||
scripts/test_wasm_source_locations.sh
|
scripts/test_wasm_source_locations.sh
|
||||||
scripts/test_server_arguments.sh
|
scripts/test_server_arguments.sh
|
||||||
|
scripts/test_cli_worker_isolation.sh
|
||||||
scripts/test_socket_activation.sh
|
scripts/test_socket_activation.sh
|
||||||
fi
|
fi
|
||||||
|
|||||||
Executable
+153
@@ -0,0 +1,153 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
cd "$(dirname "$0")/.."
|
||||||
|
|
||||||
|
if [[ "${1:-}" != "--inside" ]]; then
|
||||||
|
exec timeout --signal=TERM --kill-after=5s 120s unshare --mount --fork --kill-child=TERM "$0" --inside
|
||||||
|
fi
|
||||||
|
|
||||||
|
root="/tmp/uce-cli-worker-isolation-$$"
|
||||||
|
site="$root/site"
|
||||||
|
work="$root/work"
|
||||||
|
settings="$root/settings.cfg"
|
||||||
|
log="$root/service.log"
|
||||||
|
cli_socket="$root/run/cli.sock"
|
||||||
|
fastcgi_socket="$root/run/fastcgi.sock"
|
||||||
|
server_pid=""
|
||||||
|
|
||||||
|
cleanup() {
|
||||||
|
status=$?
|
||||||
|
if [[ -n "$server_pid" ]] && kill -0 "$server_pid" 2>/dev/null; then
|
||||||
|
kill -TERM "$server_pid" 2>/dev/null || true
|
||||||
|
deadline=$((SECONDS + 10))
|
||||||
|
while kill -0 "$server_pid" 2>/dev/null && (( SECONDS < deadline )); do sleep 0.05; done
|
||||||
|
if kill -0 "$server_pid" 2>/dev/null; then kill -KILL "$server_pid" 2>/dev/null || true; fi
|
||||||
|
wait "$server_pid" 2>/dev/null || true
|
||||||
|
fi
|
||||||
|
if (( status != 0 )) && [[ -r "$log" ]]; then cat "$log" >&2; fi
|
||||||
|
rm -rf "$root"
|
||||||
|
return "$status"
|
||||||
|
}
|
||||||
|
trap cleanup EXIT
|
||||||
|
|
||||||
|
mkdir -p "$site" "$work" "$root/run" "$root/session" "$root/upload"
|
||||||
|
cp /etc/uce/settings.cfg "$settings"
|
||||||
|
cat >>"$settings" <<CFG
|
||||||
|
BIN_DIRECTORY=$work
|
||||||
|
PRECOMPILE_FILES_IN=$site
|
||||||
|
SITE_DIRECTORY=$site
|
||||||
|
FCGI_SOCKET_PATH=$fastcgi_socket
|
||||||
|
FCGI_PORT=
|
||||||
|
CLI_SOCKET_PATH=$cli_socket
|
||||||
|
CLI_WORKER_COUNT=2
|
||||||
|
CLI_WORKER_MAX_REQUESTS=3
|
||||||
|
WS_BROKER_SOCKET_PATH=$root/run/ws.sock
|
||||||
|
HTTP_PORT=
|
||||||
|
HTTP_DOCUMENT_ROOT=$site
|
||||||
|
SESSION_PATH=$root/session
|
||||||
|
TMP_UPLOAD_PATH=$root/upload
|
||||||
|
WASM_CORE_PATH=$(pwd)/bin/wasm/core.wasm
|
||||||
|
WORKER_COUNT=2
|
||||||
|
PROACTIVE_COMPILE_ENABLED=0
|
||||||
|
CFG
|
||||||
|
mount --bind "$settings" /etc/uce/settings.cfg
|
||||||
|
|
||||||
|
cat >"$site/isolation.uce" <<'UCE'
|
||||||
|
RENDER(Request& context)
|
||||||
|
{
|
||||||
|
print(request_perf()["worker_pid"].to_string());
|
||||||
|
}
|
||||||
|
|
||||||
|
CLI(Request& context)
|
||||||
|
{
|
||||||
|
print(request_perf()["worker_pid"].to_string());
|
||||||
|
}
|
||||||
|
UCE
|
||||||
|
|
||||||
|
cat >"$site/nested.uce" <<UCE
|
||||||
|
CLI(Request& context)
|
||||||
|
{
|
||||||
|
print(shell_exec("cd $(pwd) && timeout 10s scripts/uce-cli --socket $cli_socket /isolation.uce"));
|
||||||
|
}
|
||||||
|
UCE
|
||||||
|
|
||||||
|
bin/uce_fastcgi.linux.bin >"$log" 2>&1 &
|
||||||
|
server_pid=$!
|
||||||
|
deadline=$((SECONDS + 20))
|
||||||
|
while [[ ! -S "$cli_socket" ]] && (( SECONDS < deadline )); do sleep 0.05; done
|
||||||
|
[[ -S "$cli_socket" ]] || { echo "private UCE CLI socket was not ready" >&2; exit 1; }
|
||||||
|
timeout --signal=TERM --kill-after=1s 30s scripts/uce-cli --socket "$cli_socket" /isolation.uce >/dev/null
|
||||||
|
timeout --signal=TERM --kill-after=1s 30s scripts/uce-cli --socket "$cli_socket" /nested.uce | grep -Eq '^[0-9]+([.]0+)?$'
|
||||||
|
|
||||||
|
fastcgi_request() {
|
||||||
|
SCRIPT_FILENAME="$site/isolation.uce" SCRIPT_NAME=/isolation.uce REQUEST_URI=/isolation.uce REQUEST_METHOD=GET DOCUMENT_ROOT="$site" \
|
||||||
|
timeout --signal=TERM --kill-after=1s 10s cgi-fcgi -bind -connect "$fastcgi_socket" | tr -d '\r' | awk 'NF{last=$0} END{print last}'
|
||||||
|
}
|
||||||
|
fastcgi_request >/dev/null
|
||||||
|
|
||||||
|
cli_request() {
|
||||||
|
local output
|
||||||
|
local deadline=$((SECONDS + 20))
|
||||||
|
while (( SECONDS < deadline )); do
|
||||||
|
if output=$(timeout --signal=TERM --kill-after=1s 10s scripts/uce-cli --socket "$cli_socket" /isolation.uce 2>/dev/null); then
|
||||||
|
printf '%s\n' "$output"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
sleep 0.05
|
||||||
|
done
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
cli_pids=()
|
||||||
|
http_pids=()
|
||||||
|
|
||||||
|
# Concurrent completions must count individually and retire both initial CLI
|
||||||
|
# workers despite neither connection pool necessarily becoming idle first.
|
||||||
|
concurrent_jobs=()
|
||||||
|
for n in $(seq 1 12); do
|
||||||
|
cli_request >"$root/concurrent-$n" & concurrent_jobs+=("$!")
|
||||||
|
done
|
||||||
|
for job in "${concurrent_jobs[@]}"; do wait "$job"; done
|
||||||
|
mapfile -t concurrent_cli < <(cat "$root"/concurrent-* | sort -u)
|
||||||
|
cli_pids+=("${concurrent_cli[@]}")
|
||||||
|
|
||||||
|
for _ in $(seq 1 12); do
|
||||||
|
cli_pids+=("$(cli_request)")
|
||||||
|
http_pids+=("$(fastcgi_request)")
|
||||||
|
done
|
||||||
|
mapfile -t unique_cli < <(printf '%s\n' "${cli_pids[@]}" | sort -u)
|
||||||
|
mapfile -t unique_http < <(printf '%s\n' "${http_pids[@]}" | sort -u)
|
||||||
|
[[ ${#unique_cli[@]} -ge 4 ]] || { echo "CLI workers did not recycle after three requests: ${unique_cli[*]}" >&2; exit 1; }
|
||||||
|
[[ ${#unique_http[@]} -ge 1 ]] || { echo "HTTP requests reached no public worker" >&2; exit 1; }
|
||||||
|
for cli_pid in "${unique_cli[@]}"; do
|
||||||
|
if printf '%s\n' "${unique_http[@]}" | grep -Fxq "$cli_pid"; then
|
||||||
|
echo "CLI worker also served public HTTP: $cli_pid" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
[[ $(grep -c 'wasm FastCGI worker ready' "$log") -eq 2 ]]
|
||||||
|
[[ $(grep -c 'wasm CLI worker ready' "$log") -ge 4 ]]
|
||||||
|
|
||||||
|
echo "CLI worker isolation and recycling passed: CLI ${unique_cli[*]}, public ${unique_http[*]}"
|
||||||
|
|
||||||
|
# A CLI-only runtime must not allocate a listener-less public Wasmtime worker.
|
||||||
|
kill -TERM "$server_pid"
|
||||||
|
wait "$server_pid"
|
||||||
|
server_pid=""
|
||||||
|
rm -f "$cli_socket" "$fastcgi_socket"
|
||||||
|
cat >>"$settings" <<CFG
|
||||||
|
FCGI_SOCKET_PATH=
|
||||||
|
CLI_WORKER_COUNT=1
|
||||||
|
CLI_WORKER_MAX_REQUESTS=0
|
||||||
|
CFG
|
||||||
|
: >"$log"
|
||||||
|
bin/uce_fastcgi.linux.bin >"$log" 2>&1 &
|
||||||
|
server_pid=$!
|
||||||
|
deadline=$((SECONDS + 20))
|
||||||
|
while [[ ! -S "$cli_socket" ]] && (( SECONDS < deadline )); do sleep 0.05; done
|
||||||
|
[[ -S "$cli_socket" ]] || { echo "CLI-only socket was not ready" >&2; exit 1; }
|
||||||
|
cli_request >/dev/null
|
||||||
|
[[ $(grep -c 'wasm CLI worker ready' "$log") -eq 1 ]]
|
||||||
|
[[ $(grep -c 'wasm FastCGI worker ready' "$log") -eq 0 ]]
|
||||||
|
|
||||||
|
echo "CLI-only worker allocation passed"
|
||||||
Executable
+55
@@ -0,0 +1,55 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
cd "$(dirname "$0")/.."
|
||||||
|
|
||||||
|
if [[ "${1:-}" != "--inside" ]]; then
|
||||||
|
exec timeout --signal=TERM --kill-after=5s 90s unshare --mount --fork --kill-child=TERM "$0" --inside
|
||||||
|
fi
|
||||||
|
|
||||||
|
root="/tmp/uce-compiler-lock-directory-$$"
|
||||||
|
site="$root/site"
|
||||||
|
work="$root/missing/parents/work"
|
||||||
|
settings="$root/settings.cfg"
|
||||||
|
log="$root/service.log"
|
||||||
|
server_pid=""
|
||||||
|
|
||||||
|
cleanup() {
|
||||||
|
if [[ -n "$server_pid" ]] && kill -0 "$server_pid" 2>/dev/null; then
|
||||||
|
kill -TERM "$server_pid" 2>/dev/null || true
|
||||||
|
wait "$server_pid" 2>/dev/null || true
|
||||||
|
fi
|
||||||
|
rm -rf "$root"
|
||||||
|
}
|
||||||
|
trap cleanup EXIT
|
||||||
|
mkdir -p "$site" "$root/run" "$root/session" "$root/upload"
|
||||||
|
printf '%s\n' 'CLI(Request& context) { print("recursive-lock-directory-ok"); }' >"$site/test.uce"
|
||||||
|
cp /etc/uce/settings.cfg "$settings"
|
||||||
|
cat >>"$settings" <<CFG
|
||||||
|
BIN_DIRECTORY=$work
|
||||||
|
PRECOMPILE_FILES_IN=$site
|
||||||
|
SITE_DIRECTORY=$site
|
||||||
|
FCGI_SOCKET_PATH=$root/run/fastcgi.sock
|
||||||
|
FCGI_PORT=
|
||||||
|
CLI_SOCKET_PATH=$root/run/cli.sock
|
||||||
|
WS_BROKER_SOCKET_PATH=$root/run/ws.sock
|
||||||
|
HTTP_PORT=
|
||||||
|
HTTP_DOCUMENT_ROOT=$site
|
||||||
|
SESSION_PATH=$root/session
|
||||||
|
TMP_UPLOAD_PATH=$root/upload
|
||||||
|
WASM_CORE_PATH=$(pwd)/bin/wasm/core.wasm
|
||||||
|
WORKER_COUNT=1
|
||||||
|
PROACTIVE_COMPILE_ENABLED=0
|
||||||
|
CFG
|
||||||
|
mount --bind "$settings" /etc/uce/settings.cfg
|
||||||
|
|
||||||
|
bin/uce_fastcgi.linux.bin >"$log" 2>&1 &
|
||||||
|
server_pid=$!
|
||||||
|
deadline=$((SECONDS + 20))
|
||||||
|
while [[ ! -S "$root/run/cli.sock" ]] && (( SECONDS < deadline )); do sleep 0.05; done
|
||||||
|
[[ -S "$root/run/cli.sock" ]] || { cat "$log" >&2; exit 1; }
|
||||||
|
|
||||||
|
rm -rf "$work"
|
||||||
|
response=$(curl -sS --max-time 45 --fail-with-body --unix-socket "$root/run/cli.sock" http://localhost/test.uce) || { cat "$log" >&2; exit 1; }
|
||||||
|
[[ "$response" == *"recursive-lock-directory-ok"* ]] || { printf '%s\n' "$response" >&2; cat "$log" >&2; exit 1; }
|
||||||
|
find "$work" -type f -name '*.lock' -print -quit | grep -q . || { echo "compiler did not recreate a nested lock path" >&2; cat "$log" >&2; exit 1; }
|
||||||
|
echo "compiler recursive lock directory passed"
|
||||||
Executable
+176
@@ -0,0 +1,176 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
cd "$(dirname "$0")/.."
|
||||||
|
|
||||||
|
test_source="/tmp/uce-oauth-es256-native-$$.cpp"
|
||||||
|
test_binary="/tmp/uce-oauth-es256-native-$$"
|
||||||
|
cleanup() { rm -f "$test_source" "$test_binary"; }
|
||||||
|
trap cleanup EXIT
|
||||||
|
|
||||||
|
cat >"$test_source" <<'EOF'
|
||||||
|
#include "src/lib/types.cpp"
|
||||||
|
#include "src/lib/dvalue.cpp"
|
||||||
|
#include "src/lib/functionlib.cpp"
|
||||||
|
#include <openssl/bn.h>
|
||||||
|
#include <openssl/core_names.h>
|
||||||
|
#include <openssl/ecdsa.h>
|
||||||
|
#include <openssl/evp.h>
|
||||||
|
#include <openssl/params.h>
|
||||||
|
#include <memory>
|
||||||
|
|
||||||
|
String base64_encode(String raw)
|
||||||
|
{
|
||||||
|
if(raw.empty()) return("");
|
||||||
|
String out(4 * ((raw.size() + 2) / 3), 0);
|
||||||
|
int size = EVP_EncodeBlock((unsigned char*)out.data(), (const unsigned char*)raw.data(), (int)raw.size());
|
||||||
|
out.resize(size > 0 ? (size_t)size : 0);
|
||||||
|
return(out);
|
||||||
|
}
|
||||||
|
|
||||||
|
String base64_decode(String raw, bool& ok)
|
||||||
|
{
|
||||||
|
ok = false;
|
||||||
|
if(raw.empty() || raw.size() % 4) return("");
|
||||||
|
String out(3 * raw.size() / 4, 0);
|
||||||
|
int size = EVP_DecodeBlock((unsigned char*)out.data(), (const unsigned char*)raw.data(), (int)raw.size());
|
||||||
|
if(size < 0) return("");
|
||||||
|
while(!raw.empty() && raw.back() == '=') { size--; raw.pop_back(); }
|
||||||
|
out.resize((size_t)size); ok = true; return(out);
|
||||||
|
}
|
||||||
|
|
||||||
|
#include "src/lib/hash.cpp"
|
||||||
|
|
||||||
|
static String b64url_encode(String text)
|
||||||
|
{
|
||||||
|
String out=replace(replace(base64_encode(text),"+","-"),"/","_"); while(!out.empty()&&out.back()=='=') out.pop_back(); return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
static String b64url_decode(String text)
|
||||||
|
{
|
||||||
|
text = replace(replace(text, "-", "+"), "_", "/");
|
||||||
|
while(text.size() % 4) text += "=";
|
||||||
|
bool ok = false;
|
||||||
|
String result = base64_decode(text, ok);
|
||||||
|
return(ok ? result : String(""));
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool verify(DValue public_jwk, String jwt)
|
||||||
|
{
|
||||||
|
StringList parts = split(jwt, ".");
|
||||||
|
if(parts.size() != 3) return(false);
|
||||||
|
String x = b64url_decode(public_jwk["x"].to_string());
|
||||||
|
String y = b64url_decode(public_jwk["y"].to_string());
|
||||||
|
String raw = b64url_decode(parts[2]);
|
||||||
|
if(x.size() != 32 || y.size() != 32 || raw.size() != 64) return(false);
|
||||||
|
unsigned char point[65] = {4}; memcpy(point + 1, x.data(), 32); memcpy(point + 33, y.data(), 32);
|
||||||
|
OSSL_PARAM params[] = { OSSL_PARAM_construct_utf8_string(OSSL_PKEY_PARAM_GROUP_NAME, (char*)"prime256v1", 0), OSSL_PARAM_construct_octet_string(OSSL_PKEY_PARAM_PUB_KEY, point, sizeof(point)), OSSL_PARAM_construct_end() };
|
||||||
|
EVP_PKEY_CTX* build = EVP_PKEY_CTX_new_from_name(0, "EC", 0); EVP_PKEY* key = 0;
|
||||||
|
if(!build || EVP_PKEY_fromdata_init(build) <= 0 || EVP_PKEY_fromdata(build, &key, EVP_PKEY_PUBLIC_KEY, params) <= 0) { EVP_PKEY_CTX_free(build); return(false); }
|
||||||
|
EVP_PKEY_CTX_free(build);
|
||||||
|
BIGNUM* r = BN_bin2bn((const unsigned char*)raw.data(), 32, 0); BIGNUM* s = BN_bin2bn((const unsigned char*)raw.data() + 32, 32, 0);
|
||||||
|
ECDSA_SIG* sig = ECDSA_SIG_new(); int der_size = r && s && sig && ECDSA_SIG_set0(sig, r, s) ? i2d_ECDSA_SIG(sig, 0) : 0; r = s = 0;
|
||||||
|
String der(der_size > 0 ? (size_t)der_size : 0, 0); unsigned char* out = (unsigned char*)der.data();
|
||||||
|
bool ok = der_size > 0 && i2d_ECDSA_SIG(sig, &out) == der_size;
|
||||||
|
EVP_MD_CTX* verify_ctx = EVP_MD_CTX_new();
|
||||||
|
String signing_input = parts[0] + "." + parts[1];
|
||||||
|
ok = ok && verify_ctx && EVP_DigestVerifyInit(verify_ctx, 0, EVP_sha256(), 0, key) > 0 && EVP_DigestVerify(verify_ctx, (const unsigned char*)der.data(), der.size(), (const unsigned char*)signing_input.data(), signing_input.size()) == 1;
|
||||||
|
EVP_MD_CTX_free(verify_ctx); ECDSA_SIG_free(sig); EVP_PKEY_free(key); return(ok);
|
||||||
|
}
|
||||||
|
|
||||||
|
int main()
|
||||||
|
{
|
||||||
|
DValue key_request; key_request["operation"] = "key_generate"; key_request["algorithm"] = "ES256";
|
||||||
|
DValue key = crypto_operation_native(key_request);
|
||||||
|
DValue header; header["alg"] = "none"; header["kid"] = key["kid"]; DValue claims; claims["iss"] = "https://client.example";
|
||||||
|
auto sign = [&](DValue private_jwk) { DValue request; request["operation"] = "jwt_sign"; request["algorithm"] = "ES256"; request["private_jwk"] = private_jwk; request["protected_header"] = header; request["claims"] = claims; return(crypto_operation_native(request)); };
|
||||||
|
DValue signed_result = sign(key["private_jwk"]); String jwt = signed_result["jwt"].to_string();
|
||||||
|
DValue wrong_curve = key["private_jwk"]; wrong_curve["crv"] = "P-384";
|
||||||
|
DValue malformed = key["private_jwk"]; malformed["x"] = "bad=";
|
||||||
|
DValue mismatch = key["private_jwk"]; String d = mismatch["d"].to_string(); d[0] = d[0] == 'A' ? 'B' : 'A'; mismatch["d"] = d;
|
||||||
|
const String cose="pQECAyYgASFYIGsX0fLhLEJH-Lzm5WOkQPJ3A32BLeszoPShOUXYmMKWIlggT-NC4v4af5uO5-tKfA-eFivOM1drMV7Oy7ZAaDe_UfU", message="d2ViYXV0aG4gZml4ZWQgbWVzc2FnZQ", signature="MEUCIQCkatZK1VVsjk17uvyzyhjdAkMNWXPjxSOMqWcjmM_8XAIgDaSk3Qufyd0_6r9Dm9A8RQbFco-FdTBulq7bvRGoBC4";
|
||||||
|
auto op = [&](String operation) { DValue r; r["operation"]=operation; r["algorithm"]="ES256"; return r; };
|
||||||
|
auto parse = [&](String key) { DValue r=op("cose_es256_parse"); r["cose_key_base64url"]=key; return crypto_operation_native(r); };
|
||||||
|
auto verify_cose = [&](String key,String msg,String sig) { DValue r=op("es256_verify"); r["cose_key_base64url"]=key; r["message_base64url"]=msg; r["signature_der_base64url"]=sig; return crypto_operation_native(r); };
|
||||||
|
auto decode_cbor = [&](String bytes) { DValue r=op("cbor_decode"); r["cbor_base64url"]=b64url_encode(bytes); return crypto_operation_native(r); };
|
||||||
|
DValue unsupported; unsupported["operation"] = "encrypt"; unsupported["algorithm"] = "ES256";
|
||||||
|
DValue unknown_algorithm; unknown_algorithm["operation"] = "key_generate"; unknown_algorithm["algorithm"] = "none";
|
||||||
|
DValue untyped_algorithm = key_request; untyped_algorithm["algorithm"] = (f64)256;
|
||||||
|
DValue list_header; list_header.set_array(); DValue list_item; list_item = "not-an-object"; list_header.push(list_item); DValue list_request; list_request["operation"] = "jwt_sign"; list_request["algorithm"] = "ES256"; list_request["private_jwk"] = key["private_jwk"]; list_request["protected_header"] = list_header; list_request["claims"] = claims;
|
||||||
|
DValue control_request = list_request; control_request["protected_header"] = header; control_request["claims"] = claims; control_request["claims"]["bad"] = String("control\nbyte");
|
||||||
|
DValue oversized = key_request; oversized["ignored"] = String(33000, 'x');
|
||||||
|
DValue nonfinite = key_request; nonfinite["ignored"] = std::numeric_limits<f64>::quiet_NaN();
|
||||||
|
String tampered = jwt; if(tampered.size() > 2) tampered[tampered.size() - 2] = tampered[tampered.size() - 2] == 'A' ? 'B' : 'A';
|
||||||
|
bool kid_ok = key["ok"].to_bool() && key["kid"].to_string() == key["thumbprint"].to_string();
|
||||||
|
bool signed_ok = signed_result["ok"].to_bool() && jwt != "" && verify(key["public_jwk"], jwt);
|
||||||
|
bool tamper_ok = !verify(key["public_jwk"], tampered);
|
||||||
|
auto cbor_error = [&](String bytes) { return(decode_cbor(bytes)["error"].to_string() == "invalid_cbor"); };
|
||||||
|
auto cose_error = [&](String encoded) { return(parse(encoded)["error"].to_string() == "invalid_cose_key"); };
|
||||||
|
String cose_raw = b64url_decode(cose);
|
||||||
|
String wrong_kty = cose_raw; wrong_kty[2] = 1;
|
||||||
|
String wrong_alg = cose_raw; wrong_alg[4] = 0x27;
|
||||||
|
String wrong_cose_curve = cose_raw; wrong_cose_curve[6] = 2;
|
||||||
|
String missing_label = cose_raw; missing_label[0] = 0xa4; missing_label.resize(missing_label.size() - 35);
|
||||||
|
String short_x = String("\xa5\x01\x02\x03\x26\x20\x01\x21\x58\x1f",10) + String(31, 'x') + String("\x22\x58\x20", 3) + String(32, 'y');
|
||||||
|
String duplicate_label = cose_raw; duplicate_label[0] = 0xa6; duplicate_label += String("\x01\x02", 2);
|
||||||
|
String invalid_point = String("\xa5\x01\x02\x03\x26\x20\x01\x21\x58\x20", 10) + String(32, 'x') + String("\x22\x58\x20", 3) + String(32, 'y');
|
||||||
|
String der = b64url_decode(signature);
|
||||||
|
String tampered_der = der; tampered_der[10] ^= 1;
|
||||||
|
String trailing_der = der + String("\x00", 1);
|
||||||
|
String noncanonical_der = der; noncanonical_der[1]++; noncanonical_der[3]++; noncanonical_der.insert(4, 1, '\0');
|
||||||
|
DValue valid_verify = verify_cose(cose, message, signature);
|
||||||
|
DValue tampered_signature = verify_cose(cose, message, b64url_encode(tampered_der));
|
||||||
|
DValue tampered_message = verify_cose(cose, b64url_encode("tampered"), signature);
|
||||||
|
DValue malformed_der = verify_cose(cose, message, b64url_encode(String("\x30\x00", 2)));
|
||||||
|
DValue noncanonical_der_result = verify_cose(cose, message, b64url_encode(noncanonical_der));
|
||||||
|
DValue trailing_der_result = verify_cose(cose, message, b64url_encode(trailing_der));
|
||||||
|
DValue invalid_point_result = verify_cose(b64url_encode(invalid_point), message, signature);
|
||||||
|
String large_cbor = String("\x58\x81", 2) + String(129, 'x');
|
||||||
|
String oversized_cbor = String("\x5a\x00\x00\x40\x00", 5) + String(16384, 'x');
|
||||||
|
String duplicate_compound = String("\xa2\x82\x41[\x41]\x01\x82\x41[\x41]\x02", 13);
|
||||||
|
String distinct_compound = String("\xa2\x82\x41[\x41]\x01\x82\x41[\x41[\x02", 13);
|
||||||
|
String node_overflow = String("\x99\x01\x00", 3) + String(256, 0);
|
||||||
|
String decoded;
|
||||||
|
bool cbor_valid = decode_cbor(String("\x82\x01\x62ok", 5))["ok"].to_bool();
|
||||||
|
bool cbor_large = decode_cbor(large_cbor)["ok"].to_bool();
|
||||||
|
bool cbor_control_text = decode_cbor(String("\x61\x01", 2))["ok"].to_bool();
|
||||||
|
bool cbor_duplicate = cbor_error(String("\xa2\x01\x02\x01\x03", 5));
|
||||||
|
bool cbor_compound_duplicate = cbor_error(duplicate_compound);
|
||||||
|
bool cbor_compound_distinct = decode_cbor(distinct_compound)["ok"].to_bool();
|
||||||
|
bool cbor_invalid_utf8 = cbor_error(String("\x61\x80", 2));
|
||||||
|
bool cbor_truncated = cbor_error(String("\xa1", 1));
|
||||||
|
bool cbor_trailing = cbor_error(String("\x01\x02", 2));
|
||||||
|
bool cbor_depth = cbor_error(String(17, '\x81') + "\x00");
|
||||||
|
bool cbor_nodes = cbor_error(node_overflow);
|
||||||
|
bool cbor_size = cbor_error(oversized_cbor);
|
||||||
|
bool cbor_indefinite = cbor_error(String("\x9f\x01\xff", 3));
|
||||||
|
bool cbor_integer_overflow = cbor_error(String("\x5b\xff\xff\xff\xff\xff\xff\xff\xff", 9));
|
||||||
|
bool cbor_nonminimal = cbor_error(String("\x18\x17", 2));
|
||||||
|
bool b64_padding = !uce_base64url_decode(cose + "=", decoded, UCE_CBOR_MAX_BASE64URL);
|
||||||
|
bool b64_truncated = !uce_base64url_decode("A", decoded, UCE_CBOR_MAX_BASE64URL);
|
||||||
|
bool b64_trailing_bits = !uce_base64url_decode("AB", decoded, UCE_CBOR_MAX_BASE64URL);
|
||||||
|
bool cose_valid = parse(cose)["ok"].to_bool();
|
||||||
|
bool cose_wrong_kty = cose_error(b64url_encode(wrong_kty));
|
||||||
|
bool cose_wrong_alg = cose_error(b64url_encode(wrong_alg));
|
||||||
|
bool cose_wrong_curve = cose_error(b64url_encode(wrong_cose_curve));
|
||||||
|
bool cose_short_coordinate = cose_error(b64url_encode(short_x));
|
||||||
|
bool cose_missing_label = cose_error(b64url_encode(missing_label));
|
||||||
|
bool cose_duplicate_label = cose_error(b64url_encode(duplicate_label));
|
||||||
|
bool cose_invalid_point = invalid_point_result["error"].to_string() == "invalid_key_or_payload";
|
||||||
|
bool verify_valid = valid_verify["ok"].to_bool() && valid_verify["valid"].to_bool();
|
||||||
|
bool verify_tampered_signature = tampered_signature["ok"].to_bool() && !tampered_signature["valid"].to_bool();
|
||||||
|
bool verify_tampered_message = tampered_message["ok"].to_bool() && !tampered_message["valid"].to_bool();
|
||||||
|
bool verify_malformed_der = malformed_der["error"].to_string() == "invalid_signature";
|
||||||
|
bool verify_noncanonical_der = noncanonical_der_result["error"].to_string() == "invalid_signature";
|
||||||
|
bool verify_trailing_der = trailing_der_result["error"].to_string() == "invalid_signature";
|
||||||
|
bool cbor_ok = cbor_valid && cbor_large && cbor_control_text && cbor_duplicate && cbor_compound_duplicate && cbor_compound_distinct && cbor_invalid_utf8 && cbor_truncated && cbor_trailing && cbor_depth && cbor_nodes && cbor_size && cbor_indefinite && cbor_integer_overflow && cbor_nonminimal;
|
||||||
|
bool cose_ok = cose_valid && cose_wrong_kty && cose_wrong_alg && cose_wrong_curve && cose_short_coordinate && cose_missing_label && cose_duplicate_label && cose_invalid_point;
|
||||||
|
bool cose_negative = verify_valid && verify_tampered_signature && verify_tampered_message && verify_malformed_der && verify_noncanonical_der && verify_trailing_der;
|
||||||
|
bool b64_ok = b64_padding && b64_truncated && b64_trailing_bits;
|
||||||
|
bool negatives_ok = !sign(wrong_curve)["ok"].to_bool() && !sign(malformed)["ok"].to_bool() && !sign(mismatch)["ok"].to_bool() && crypto_operation_native(unsupported)["error"].to_string() == "unsupported_operation" && crypto_operation_native(unknown_algorithm)["error"].to_string() == "unsupported_algorithm" && crypto_operation_native(untyped_algorithm)["error"].to_string() == "invalid_request" && crypto_operation_native(list_request)["error"].to_string() == "invalid_key_or_payload" && crypto_operation_native(control_request)["error"].to_string() == "invalid_request" && crypto_operation_native(oversized)["error"].to_string() == "invalid_request" && crypto_operation_native(nonfinite)["error"].to_string() == "invalid_request";
|
||||||
|
if(!(kid_ok && signed_ok && tamper_ok && negatives_ok && cose_ok && cbor_ok && cose_negative && b64_ok)) std::cerr << "kid=" << kid_ok << " signed=" << signed_ok << " tamper=" << tamper_ok << " negatives=" << negatives_ok << " cbor=" << cbor_ok << " cose=" << cose_ok << " verify=" << cose_negative << " b64=" << b64_ok << " large=" << cbor_large << " compound=" << cbor_compound_duplicate << "/" << cbor_compound_distinct << " der=" << verify_malformed_der << "/" << verify_noncanonical_der << "/" << verify_trailing_der << " cborparts=" << cbor_valid << cbor_control_text << cbor_duplicate << cbor_invalid_utf8 << cbor_truncated << cbor_trailing << cbor_depth << cbor_nodes << cbor_size << cbor_indefinite << cbor_integer_overflow << cbor_nonminimal << "\\n";
|
||||||
|
return(kid_ok && signed_ok && tamper_ok && negatives_ok && cose_ok && cbor_ok && cose_negative && b64_ok ? 0 : 1);
|
||||||
|
}
|
||||||
|
EOF
|
||||||
|
"${CXX:-c++}" -std=c++20 -fpermissive -I. "$test_source" -lpcre2-8 -lcrypto -o "$test_binary"
|
||||||
|
"$test_binary"
|
||||||
|
echo "native structured crypto operation passed"
|
||||||
@@ -450,14 +450,23 @@ fi
|
|||||||
|
|
||||||
worker_count=$(awk -F= '/^[[:space:]]*WORKER_COUNT[[:space:]]*=/ {gsub(/^[[:space:]]+|[[:space:]]+$/, "", $2); print $2; exit}' /etc/uce/settings.cfg 2>/dev/null || true)
|
worker_count=$(awk -F= '/^[[:space:]]*WORKER_COUNT[[:space:]]*=/ {gsub(/^[[:space:]]+|[[:space:]]+$/, "", $2); print $2; exit}' /etc/uce/settings.cfg 2>/dev/null || true)
|
||||||
worker_count="${worker_count:-4}"
|
worker_count="${worker_count:-4}"
|
||||||
|
cli_worker_count=$(awk -F= '/^[[:space:]]*CLI_WORKER_COUNT[[:space:]]*=/ {gsub(/^[[:space:]]+|[[:space:]]+$/, "", $2); value=$2} END{print value}' /etc/uce/settings.cfg 2>/dev/null || true)
|
||||||
|
cli_worker_count="${cli_worker_count:-0}"
|
||||||
|
cli_worker_max_requests=$(awk -F= '/^[[:space:]]*CLI_WORKER_MAX_REQUESTS[[:space:]]*=/ {gsub(/^[[:space:]]+|[[:space:]]+$/, "", $2); value=$2} END{print value}' /etc/uce/settings.cfg 2>/dev/null || true)
|
||||||
|
cli_worker_max_requests="${cli_worker_max_requests:-8}"
|
||||||
worker_pids=""
|
worker_pids=""
|
||||||
for _ in {1..48}; do
|
for _ in {1..48}; do
|
||||||
output=$(scripts/uce-cli "/$test_name/parent.uce")
|
output=$(scripts/uce-cli "/$test_name/parent.uce")
|
||||||
worker_pids+="${output##*:}"$'\n'
|
worker_pids+="${output##*:}"$'\n'
|
||||||
done
|
done
|
||||||
unique_workers=$(printf '%s' "$worker_pids" | sed '/^$/d' | sort -u | wc -l)
|
unique_workers=$(printf '%s' "$worker_pids" | sed '/^$/d' | sort -u | wc -l)
|
||||||
if (( unique_workers > worker_count )); then
|
if (( cli_worker_count > 0 && cli_worker_max_requests > 0 )); then
|
||||||
echo "worker pool recycled during 48 requests: $unique_workers PIDs for $worker_count workers" >&2
|
if (( unique_workers <= cli_worker_count )); then
|
||||||
|
echo "dedicated CLI pool did not recycle during 48 requests: $unique_workers PIDs for $cli_worker_count workers" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
elif (( unique_workers > worker_count )); then
|
||||||
|
echo "shared worker pool recycled during 48 requests: $unique_workers PIDs for $worker_count workers" >&2
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,37 @@
|
|||||||
|
#include "src/lib/types.cpp"
|
||||||
|
#include "src/lib/dvalue.cpp"
|
||||||
|
#include "src/lib/functionlib.cpp"
|
||||||
|
#include "src/wasm/hardened_http_internal.h"
|
||||||
|
#include <iostream>
|
||||||
|
struct Fake { std::vector<String> answers, argv, env; String input; HardenedHttpExecResult result; };
|
||||||
|
static DValue request() { DValue r; r["method"]="POST"; r["url"]="https://api.example.test/token"; r["headers"]["Accept"]="application/json"; r["body"]="client_secret=SECRET"; DValue& s=r["security"]; s["https_only"].set_bool(true); s["public_dns_only"].set_bool(true); s["pin_dns"].set_bool(true); s["isolated_curl"].set_bool(true); s["no_redirects"].set_bool(true); return r; }
|
||||||
|
static DValue run(Fake& f,DValue r) { HardenedHttpHooks h; h.resolve=[&](String){return f.answers;}; h.execute=[&](std::vector<String> a,String i,std::vector<String> e,u64,size_t){f.argv=a;f.input=i;f.env=e;return f.result;}; return hardened_http_request_internal(r,9000,h); }
|
||||||
|
static String headers(int status=200) { return "HTTP/1.1 "+std::to_string(status)+" OK\r\nContent-Type: application/json\r\n\r\n"; }
|
||||||
|
static bool exists(String path) { return access(path.c_str(),F_OK)==0; }
|
||||||
|
static void wait_for(String path) { for(int n=0;n<500&&!exists(path);n++) usleep(1000); }
|
||||||
|
static bool process_live(pid_t pid) { char state=0; String path="/proc/"+std::to_string((long long)pid)+"/stat"; int fd=open(path.c_str(),O_RDONLY); if(fd<0) return false; char text[256]{}; ssize_t n=read(fd,text,sizeof(text)-1); close(fd); if(n<=0) return false; char* close_paren=strrchr(text,')'); return close_paren&&close_paren[2]!='Z'; }
|
||||||
|
int main(int argc,char** argv) {
|
||||||
|
if(argc==3&&String(argv[1])=="--child-timeout") { pid_t child=fork(); if(child==0) { usleep(100000); int fd=open(argv[2],O_WRONLY|O_CREAT|O_TRUNC,0600); if(fd>=0) { write(fd,"leaked",6); close(fd); } _exit(0); } for(;;) pause(); }
|
||||||
|
if(argc==2&&String(argv[1])=="--child-output") { String x(4096,'x'); for(int n=0;n<64;n++) write(1,x.data(),x.size()); return 0; }
|
||||||
|
if(argc==3&&String(argv[1])=="--child-fds") { int inherited=atoi(argv[2]); bool closed=fcntl(inherited,F_GETFD)==-1&&errno==EBADF; bool header=fcntl(3,F_GETFD)!=-1; write(1,closed&&header?"closed":"open",closed&&header?6:4); return closed&&header?0:1; }
|
||||||
|
if(argc==5&&String(argv[1])=="--child-job") { int ready=open(argv[3],O_WRONLY|O_CREAT|O_TRUNC,0600); if(ready>=0) { write(ready,"ready",5); close(ready); } pid_t child=fork(); if(child==0) { int pidfile=open(argv[4],O_WRONLY|O_CREAT|O_TRUNC,0600); if(pidfile>=0) { String pid=std::to_string((long long)getpid()); write(pidfile,pid.data(),pid.size()); close(pidfile); } usleep(150000); int marker=open(argv[2],O_WRONLY|O_CREAT|O_TRUNC,0600); if(marker>=0) { write(marker,"leaked",6); close(marker); } _exit(0); } for(;;) pause(); }
|
||||||
|
if(argc==3&&String(argv[1])=="--child-async-timeout") { setsid(); HardenedHttpExecResult result=hardened_http_exec_argv_capture({"/bin/sleep","1"},"",20,4096,false,false); if(result.timed_out&&result.exit_code==137) { int fd=open(argv[2],O_WRONLY|O_CREAT|O_TRUNC,0600); if(fd>=0) { write(fd,"typed",5); close(fd); } } return 0; }
|
||||||
|
if(argc==3&&String(argv[1])=="--child-async-output") { setsid(); HardenedHttpExecResult result=hardened_http_exec_argv_capture({"/proc/self/exe","--child-output"},"",1000,1024,false,false); if(result.output_limited&&result.exit_code==137) { int fd=open(argv[2],O_WRONLY|O_CREAT|O_TRUNC,0600); if(fd>=0) { write(fd,"typed",5); close(fd); } } return 0; }
|
||||||
|
bool ok=true; auto need=[&](bool x,const char* n){if(!x){std::cerr<<n<<"\n";ok=false;}};
|
||||||
|
Fake f; f.answers={"8.8.8.8"}; f.result.exit_code=0; f.result.stderr_text="SECRET"; f.result.headers_text=headers(); f.result.body_text="one\r\n\r\ntwo"; DValue out=run(f,request());
|
||||||
|
need(out["error"].to_string()==""&&out["error"].to_string().find("SECRET")==String::npos&&out["body"].to_string()=="one\r\n\r\ntwo","body framing / non-secret errors"); need(f.argv.size()>2&&f.argv[0]=="/usr/bin/curl"&&f.argv[1]=="--disable","absolute curl disable"); need(f.env.size()==1&&f.env[0]=="PATH=/usr/bin:/bin","clean env"); need(std::find(f.argv.begin(),f.argv.end(),"SECRET")==f.argv.end()&&f.input.find("SECRET")!=String::npos,"secret stdin only"); need(std::find(f.argv.begin(),f.argv.end(),"--resolve")!=f.argv.end(),"pinned dns");
|
||||||
|
Fake interim; interim.answers={"8.8.8.8"}; interim.result.exit_code=0; interim.result.headers_text="HTTP/1.1 100 Continue\r\n\r\n"+headers(201); interim.result.body_text="{}"; need(run(interim,request())["status"].to_u64()==201,"interim headers");
|
||||||
|
for(String bad:{"127.0.0.1","10.0.0.1","169.254.1.1","192.168.1.1","::1","2001:db8::1"}) { Fake x; x.answers={"8.8.8.8",bad}; need(run(x,request())["error"].to_string()=="unsafe_dns","dns matrix"); }
|
||||||
|
for(auto pair:std::vector<std::pair<int,String>>{{302,headers(302)},{500,headers(500)},{200,"bad\r\n\r\n"}}) { Fake x; x.answers={"8.8.8.8"};x.result.exit_code=0;x.result.headers_text=pair.second;need(run(x,request())["error"].to_string()==(pair.first==302?"redirect_not_allowed":pair.first==500?"http_status":"malformed_output"),"status errors"); }
|
||||||
|
DValue bad=request(); bad["method"]="TRACE"; Fake x; need(run(x,bad)["error"].to_string()=="invalid_request"&&x.argv.empty(),"method validation"); bad=request();bad["headers"]["Host"]="evil";need(run(x,bad)["error"].to_string()=="invalid_request","header validation"); bad=request();bad["headers"]["Accept"]="ok\r\nInjected: x";need(run(x,bad)["error"].to_string()=="invalid_request","header CRLF validation");bad=request();bad["url"]="https://127.0.0.1/";need(run(x,bad)["error"].to_string()=="invalid_request","url validation");bad=request();bad["follow_redirects"].set_bool(true);need(run(x,bad)["error"].to_string()=="invalid_request","redirect composition");
|
||||||
|
for(String key:{"https_only","public_dns_only","pin_dns","isolated_curl","no_redirects"}) { bad=request(); bad["security"].remove(key); need(run(x,bad)["error"].to_string()=="invalid_request","partial hardening fails closed"); bad=request(); bad["security"][key].set_bool(false); need(run(x,bad)["error"].to_string()=="invalid_request","false hardening fails closed"); bad=request(); bad["security"][key]="true"; need(run(x,bad)["error"].to_string()=="invalid_request","non-boolean hardening fails closed"); }
|
||||||
|
DValue legacy=request(); legacy["security"].clear(); legacy["security"]["unrelated"]="value"; need(!hardened_http_security_requested(legacy.key("security")),"unknown security object remains legacy");
|
||||||
|
Fake large;large.answers={"8.8.8.8"};large.result.exit_code=0;large.result.headers_text=headers();large.result.body_text=String(65537,'x');need(run(large,request())["error"].to_string()=="response_too_large","body cap");
|
||||||
|
String base="/tmp/hardened-http-"+std::to_string((long long)getpid()), marker=base+"-marker"; unlink(marker.c_str()); HardenedHttpExecResult timeout=hardened_http_exec_argv_capture({"/proc/self/exe","--child-timeout",marker},"",30,4096,false); usleep(150000); need(timeout.timed_out&&timeout.exit_code==137&&!exists(marker),"timeout kills descendants"); unlink(marker.c_str());
|
||||||
|
HardenedHttpExecResult overflow=hardened_http_exec_argv_capture({"/proc/self/exe","--child-output"},"",1000,1024,false);need(overflow.output_limited&&overflow.body_text.size()<=1024,"output cap/reap");
|
||||||
|
int inherited=open("/dev/null",O_RDONLY), high_inherited=fcntl(inherited,F_DUPFD,10); close(inherited); HardenedHttpExecResult fds=hardened_http_exec_argv_capture({"/proc/self/exe","--child-fds",std::to_string(high_inherited)},"",1000,4096,false); close(high_inherited); need(fds.exit_code==0&&fds.body_text=="closed","only stdio and header fd inherited");
|
||||||
|
String ready=base+"-ready", descendant=base+"-descendant", sentinel=base+"-sentinel"; unlink(marker.c_str()); unlink(ready.c_str()); unlink(descendant.c_str()); unlink(sentinel.c_str()); pid_t worker=fork(); if(worker==0) { setsid(); hardened_http_exec_argv_capture({"/proc/self/exe","--child-job",marker,ready,descendant},"",5000,4096,false,false); _exit(0); } wait_for(ready); wait_for(descendant); pid_t unrelated=fork(); if(unrelated==0) { setsid(); usleep(150000); int fd=open(sentinel.c_str(),O_WRONLY|O_CREAT|O_TRUNC,0600); if(fd>=0) { write(fd,"alive",5); close(fd); } _exit(0); } pid_t descendant_pid=exists(descendant)?(pid_t)strtol([](String path){ int fd=open(path.c_str(),O_RDONLY); char text[32]{}; ssize_t n=fd<0?-1:read(fd,text,sizeof(text)-1); if(fd>=0) close(fd); return String(text,n>0?(size_t)n:0); }(descendant).c_str(),0,10):0; need(exists(ready)&&descendant_pid>0,"async worker and descendant started"); kill(-worker,SIGKILL); waitpid(worker,0,0); usleep(250000); waitpid(unrelated,0,0); need(!process_live(descendant_pid)&&!exists(marker)&&exists(sentinel),"async cancellation leaves no live descendant and kills only its worker group"); unlink(marker.c_str()); unlink(ready.c_str()); unlink(descendant.c_str()); unlink(sentinel.c_str());
|
||||||
|
String async_timeout=base+"-async-timeout"; unlink(async_timeout.c_str()); pid_t timeout_worker=fork(); if(timeout_worker==0) { execl("/proc/self/exe","test_hardened_http_native","--child-async-timeout",async_timeout.c_str(),(char*)0); _exit(127); } int timeout_status=0; waitpid(timeout_worker,&timeout_status,0); need(WIFEXITED(timeout_status)&&WEXITSTATUS(timeout_status)==0&&exists(async_timeout),"async timeout records a typed terminal result before worker exit"); unlink(async_timeout.c_str());
|
||||||
|
String async_output=base+"-async-output"; unlink(async_output.c_str()); pid_t output_worker=fork(); if(output_worker==0) { execl("/proc/self/exe","test_hardened_http_native","--child-async-output",async_output.c_str(),(char*)0); _exit(127); } int output_status=0; waitpid(output_worker,&output_status,0); need(WIFEXITED(output_status)&&WEXITSTATUS(output_status)==0&&exists(async_output),"async output cap records a typed terminal result before worker exit"); unlink(async_output.c_str());
|
||||||
|
return ok?0:1;
|
||||||
|
}
|
||||||
Executable
+8
@@ -0,0 +1,8 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
repo=$(cd "$(dirname "$0")/.." && pwd)
|
||||||
|
bin="${TMPDIR:-/tmp}/test_hardened_http_native.$$"
|
||||||
|
trap 'rm -f "$bin"' EXIT
|
||||||
|
compiler=$(command -v clang++ || command -v g++)
|
||||||
|
"$compiler" -std=c++20 -fpermissive -I"$repo" "$repo/scripts/test_hardened_http_native.cpp" -lpcre2-8 -o "$bin"
|
||||||
|
timeout --signal=TERM --kill-after=2s 15s "$bin"
|
||||||
Executable
+145
@@ -0,0 +1,145 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
cd "$(dirname "$0")/.."
|
||||||
|
|
||||||
|
if [[ "${1:-}" != "--inside" ]]; then
|
||||||
|
exec timeout --signal=TERM --kill-after=5s 150s unshare --mount --fork --kill-child=TERM "$0" --inside
|
||||||
|
fi
|
||||||
|
|
||||||
|
name="mysql-pool-idle-test-$$"
|
||||||
|
root="/tmp/$name"
|
||||||
|
site="$root/site"
|
||||||
|
work="$root/work"
|
||||||
|
settings="$root/settings.cfg"
|
||||||
|
log="$root/service.log"
|
||||||
|
socket="$root/run/cli.sock"
|
||||||
|
test_user="uce_pool_idle_$$"
|
||||||
|
test_database="uce_pool_idle_$$"
|
||||||
|
test_password=$(printf '%s' "$name-$(date +%s%N)" | sha256sum | cut -c1-32)
|
||||||
|
server_pid=""
|
||||||
|
|
||||||
|
cleanup() {
|
||||||
|
status=$?
|
||||||
|
if [[ -n "$server_pid" ]] && kill -0 "$server_pid" 2>/dev/null; then
|
||||||
|
kill -TERM "$server_pid" 2>/dev/null || true
|
||||||
|
deadline=$((SECONDS + 10))
|
||||||
|
while kill -0 "$server_pid" 2>/dev/null && (( SECONDS < deadline )); do sleep 0.05; done
|
||||||
|
if kill -0 "$server_pid" 2>/dev/null; then kill -KILL "$server_pid" 2>/dev/null || true; fi
|
||||||
|
wait "$server_pid" 2>/dev/null || true
|
||||||
|
fi
|
||||||
|
mariadb -e "DROP DATABASE IF EXISTS \`$test_database\`; DROP USER IF EXISTS '$test_user'@'127.0.0.1'" >/dev/null 2>&1 || true
|
||||||
|
if (( status != 0 )) && [[ -r "$log" ]]; then cat "$log" >&2; fi
|
||||||
|
rm -rf "$root"
|
||||||
|
return "$status"
|
||||||
|
}
|
||||||
|
trap cleanup EXIT
|
||||||
|
|
||||||
|
mkdir -p "$site" "$work" "$root/run" "$root/session" "$root/upload"
|
||||||
|
cp /etc/uce/settings.cfg "$settings"
|
||||||
|
cat >>"$settings" <<CFG
|
||||||
|
BIN_DIRECTORY=$work
|
||||||
|
PRECOMPILE_FILES_IN=$site
|
||||||
|
SITE_DIRECTORY=$site
|
||||||
|
FCGI_SOCKET_PATH=$root/run/fastcgi.sock
|
||||||
|
FCGI_PORT=
|
||||||
|
CLI_SOCKET_PATH=$socket
|
||||||
|
CLI_WORKER_COUNT=1
|
||||||
|
CLI_WORKER_MAX_REQUESTS=0
|
||||||
|
WS_BROKER_SOCKET_PATH=$root/run/ws.sock
|
||||||
|
HTTP_PORT=
|
||||||
|
HTTP_DOCUMENT_ROOT=$site
|
||||||
|
SESSION_PATH=$root/session
|
||||||
|
TMP_UPLOAD_PATH=$root/upload
|
||||||
|
WASM_CORE_PATH=$(pwd)/bin/wasm/core.wasm
|
||||||
|
WORKER_COUNT=1
|
||||||
|
PROACTIVE_COMPILE_ENABLED=0
|
||||||
|
MYSQL_PERSISTENT_POOL_SIZE=8
|
||||||
|
MYSQL_PERSISTENT_POOL_IDLE_TIMEOUT_SECONDS=2
|
||||||
|
CFG
|
||||||
|
mount --bind "$settings" /etc/uce/settings.cfg
|
||||||
|
|
||||||
|
mariadb -e "DROP DATABASE IF EXISTS \`$test_database\`; DROP USER IF EXISTS '$test_user'@'127.0.0.1'; CREATE DATABASE \`$test_database\`; CREATE USER '$test_user'@'127.0.0.1' IDENTIFIED BY '$test_password'; GRANT ALL ON \`$test_database\`.* TO '$test_user'@'127.0.0.1'"
|
||||||
|
|
||||||
|
cat >"$site/database.uce" <<UCE
|
||||||
|
CLI(Request& context)
|
||||||
|
{
|
||||||
|
MySQL* db = mysql_connect("127.0.0.1", "$test_user", "$test_password", "$test_database");
|
||||||
|
if(!mysql_connected(db)) { print("connect-failed"); return; }
|
||||||
|
String connection_id;
|
||||||
|
mysql_query(db, "SELECT CONNECTION_ID() AS id").each([&](DValue row, String key) { connection_id = row["id"].to_string(); });
|
||||||
|
String source;
|
||||||
|
DValue perf = request_perf();
|
||||||
|
perf["mysql_operations"].each([&](DValue operation, String key) {
|
||||||
|
if(operation["op"].to_string() == "connect") source = operation["source"].to_string();
|
||||||
|
});
|
||||||
|
print(perf["worker_pid"].to_string(), "|", connection_id, "|", source);
|
||||||
|
mysql_disconnect(db);
|
||||||
|
}
|
||||||
|
UCE
|
||||||
|
printf '%s\n' 'CLI(Request& context) { print(request_perf()["worker_pid"].to_string(), "|plain"); }' >"$site/plain.uce"
|
||||||
|
|
||||||
|
timeout --signal=TERM --kill-after=5s 120s bin/uce_fastcgi.linux.bin >"$log" 2>&1 &
|
||||||
|
server_pid=$!
|
||||||
|
deadline=$((SECONDS + 20))
|
||||||
|
while [[ ! -S "$socket" ]] && (( SECONDS < deadline )); do sleep 0.05; done
|
||||||
|
[[ -S "$socket" ]] || { echo "private UCE CLI socket was not ready" >&2; exit 1; }
|
||||||
|
|
||||||
|
request() {
|
||||||
|
timeout --signal=TERM --kill-after=1s 30s scripts/uce-cli --socket "$socket" "$1"
|
||||||
|
}
|
||||||
|
|
||||||
|
first=$(request /database.uce)
|
||||||
|
second=$(request /database.uce)
|
||||||
|
IFS='|' read -r first_pid first_id first_source <<<"$first"
|
||||||
|
IFS='|' read -r second_pid second_id second_source <<<"$second"
|
||||||
|
[[ "$first_source" == "new" && "$second_source" == "worker" ]]
|
||||||
|
[[ "$first_pid" == "$second_pid" && "$first_id" == "$second_id" ]]
|
||||||
|
|
||||||
|
sleep 3
|
||||||
|
plain=$(request /plain.uce)
|
||||||
|
[[ "$plain" == "$first_pid|plain" ]]
|
||||||
|
deadline=$((SECONDS + 5))
|
||||||
|
while (( SECONDS < deadline )); do
|
||||||
|
connections=$(mariadb --batch --skip-column-names -e "SELECT COUNT(*) FROM information_schema.PROCESSLIST WHERE USER='$test_user'")
|
||||||
|
[[ "$connections" == "0" ]] && break
|
||||||
|
sleep 0.05
|
||||||
|
done
|
||||||
|
[[ "$connections" == "0" ]] || { echo "expired pooled connection remained after a request boundary" >&2; exit 1; }
|
||||||
|
|
||||||
|
third=$(request /database.uce)
|
||||||
|
IFS='|' read -r third_pid third_id third_source <<<"$third"
|
||||||
|
[[ "$third_pid" == "$first_pid" && "$third_source" == "new" && "$third_id" != "$first_id" ]]
|
||||||
|
|
||||||
|
sleep 3
|
||||||
|
plain=$(request /plain.uce)
|
||||||
|
[[ "$plain" == "$first_pid|plain" ]]
|
||||||
|
deadline=$((SECONDS + 5))
|
||||||
|
while (( SECONDS < deadline )); do
|
||||||
|
connections=$(mariadb --batch --skip-column-names -e "SELECT COUNT(*) FROM information_schema.PROCESSLIST WHERE USER='$test_user'")
|
||||||
|
[[ "$connections" == "0" ]] && break
|
||||||
|
sleep 0.05
|
||||||
|
done
|
||||||
|
[[ "$connections" == "0" ]] || { echo "final pooled fixture connection remained after idle eviction" >&2; exit 1; }
|
||||||
|
|
||||||
|
retained=$(request /database.uce)
|
||||||
|
IFS='|' read -r retained_pid retained_id retained_source <<<"$retained"
|
||||||
|
[[ "$retained_pid" == "$first_pid" && "$retained_source" == "new" ]]
|
||||||
|
connections=$(mariadb --batch --skip-column-names -e "SELECT COUNT(*) FROM information_schema.PROCESSLIST WHERE USER='$test_user'")
|
||||||
|
[[ "$connections" == "1" ]] || { echo "shutdown fixture connection was not retained in the worker pool" >&2; exit 1; }
|
||||||
|
|
||||||
|
journal_cursor=$(journalctl -u mariadb.service -n 0 --show-cursor --no-pager | sed -n 's/^-- cursor: //p')
|
||||||
|
[[ -n "$journal_cursor" ]] || { echo "could not capture MariaDB journal cursor" >&2; exit 1; }
|
||||||
|
kill -TERM "$server_pid"
|
||||||
|
deadline=$((SECONDS + 10))
|
||||||
|
while kill -0 "$server_pid" 2>/dev/null && (( SECONDS < deadline )); do sleep 0.05; done
|
||||||
|
[[ ! -e "/proc/$server_pid" ]] || { echo "private UCE server did not stop cleanly" >&2; exit 1; }
|
||||||
|
wait "$server_pid" 2>/dev/null || true
|
||||||
|
server_pid=""
|
||||||
|
sleep 1
|
||||||
|
shutdown_log=$(journalctl -u mariadb.service --after-cursor "$journal_cursor" --no-pager)
|
||||||
|
if grep -F "$test_user" <<<"$shutdown_log" | grep -Fq "Aborted connection"; then
|
||||||
|
echo "clean UCE shutdown aborted a retained MySQL connection" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "Persistent MySQL idle eviction preserved hot reuse, retired expired connections, and closed a retained connection on clean shutdown"
|
||||||
@@ -134,6 +134,15 @@ printf '0\n' >"$root/maximum"
|
|||||||
flock -u 7
|
flock -u 7
|
||||||
mapfile -t scanner_pids < <(awk -F '\t' -v site="$site/" '$3 ~ ("^" site "unit-[0-9]+[.]uce$") { count[$4]++ } END { for(pid in count) if(count[pid] >= 2) print pid }' "$shim_log" | sort -n)
|
mapfile -t scanner_pids < <(awk -F '\t' -v site="$site/" '$3 ~ ("^" site "unit-[0-9]+[.]uce$") { count[$4]++ } END { for(pid in count) if(count[pid] >= 2) print pid }' "$shim_log" | sort -n)
|
||||||
[[ "${#scanner_pids[@]}" -eq 2 ]] || { echo "controlled units were not split between both scanners" >&2; cat "$shim_log" >&2; exit 1; }
|
[[ "${#scanner_pids[@]}" -eq 2 ]] || { echo "controlled units were not split between both scanners" >&2; cat "$shim_log" >&2; exit 1; }
|
||||||
|
assert_scanner_memory_bounded() {
|
||||||
|
for scanner_pid in "${scanner_pids[@]}"; do
|
||||||
|
retained_kb=$(awk '/^(VmRSS|VmSwap):/{total += $2} END{print total + 0}' "/proc/$scanner_pid/status")
|
||||||
|
# Scanner bookkeeping is small; 64 MiB leaves ample headroom while still
|
||||||
|
# catching the measured old 12-serialization retention (~73 MiB).
|
||||||
|
(( retained_kb < 65536 )) || { echo "proactive scanner retained serialized-module arenas: pid=$scanner_pid retained_kb=$retained_kb" >&2; exit 1; }
|
||||||
|
done
|
||||||
|
}
|
||||||
|
assert_scanner_memory_bounded
|
||||||
blockers=()
|
blockers=()
|
||||||
victims=()
|
victims=()
|
||||||
for scanner_pid in "${scanner_pids[@]}"; do
|
for scanner_pid in "${scanner_pids[@]}"; do
|
||||||
@@ -202,4 +211,8 @@ while [[ ! -s "$artifacts/priority.uce.cwasm" ]] && (( SECONDS < deadline )); do
|
|||||||
priority_nice=$(awk -F '\t' -v path="$site/priority.uce" '$3 == path { print $2; exit }' "$shim_log")
|
priority_nice=$(awk -F '\t' -v path="$site/priority.uce" '$3 == path { print $2; exit }' "$shim_log")
|
||||||
[[ "$priority_nice" == "5" ]] || { echo "priority queue was not owned by the nice-5 compiler: $priority_nice" >&2; cat "$shim_log" >&2; exit 1; }
|
[[ "$priority_nice" == "5" ]] || { echo "priority queue was not owned by the nice-5 compiler: $priority_nice" >&2; cat "$shim_log" >&2; exit 1; }
|
||||||
|
|
||||||
|
# Recheck after dependency fanout, failure/retry, recovery, and priority work;
|
||||||
|
# the historical leak accumulated in the long-lived scanners across phases.
|
||||||
|
assert_scanner_memory_bounded
|
||||||
|
|
||||||
printf '%s\n' 'parallel proactive compile passed'
|
printf '%s\n' 'parallel proactive compile passed'
|
||||||
|
|||||||
@@ -38,7 +38,8 @@ socket_inode=$(stat -c %i "$socket_path")
|
|||||||
invoke() {
|
invoke() {
|
||||||
local output="$1"
|
local output="$1"
|
||||||
shift
|
shift
|
||||||
timeout --signal=TERM --kill-after=1s 2s unshare --mount --fork \
|
local timeout_seconds="${INVOKE_TIMEOUT_SECONDS:-2}"
|
||||||
|
timeout --signal=TERM --kill-after=1s "$timeout_seconds" unshare --mount --fork \
|
||||||
bash -c 'mount --bind "$1" /etc/uce/settings.cfg; exec "$2" "${@:3}"' \
|
bash -c 'mount --bind "$1" /etc/uce/settings.cfg; exec "$2" "${@:3}"' \
|
||||||
_ "$cfg" "$binary" "$@" >"$output.stdout" 2>"$output.stderr"
|
_ "$cfg" "$binary" "$@" >"$output.stdout" 2>"$output.stderr"
|
||||||
}
|
}
|
||||||
@@ -47,10 +48,11 @@ for option in --help -h; do
|
|||||||
invoke "$root/help" "$option"
|
invoke "$root/help" "$option"
|
||||||
grep -q '^Usage: uce_fastcgi' "$root/help.stdout"
|
grep -q '^Usage: uce_fastcgi' "$root/help.stdout"
|
||||||
grep -q -- '--precompile' "$root/help.stdout"
|
grep -q -- '--precompile' "$root/help.stdout"
|
||||||
|
grep -q -- '--serialize-module' "$root/help.stdout"
|
||||||
[[ ! -s "$root/help.stderr" ]]
|
[[ ! -s "$root/help.stderr" ]]
|
||||||
done
|
done
|
||||||
|
|
||||||
for arguments in '--unknown' '--precompile extra' '--help extra'; do
|
for arguments in '--unknown' '--precompile extra' '--serialize-module' '--help extra'; do
|
||||||
read -r -a argv <<<"$arguments"
|
read -r -a argv <<<"$arguments"
|
||||||
set +e
|
set +e
|
||||||
invoke "$root/invalid" "${argv[@]}"
|
invoke "$root/invalid" "${argv[@]}"
|
||||||
@@ -62,6 +64,36 @@ for arguments in '--unknown' '--precompile extra' '--help extra'; do
|
|||||||
[[ ! -s "$root/invalid.stdout" ]]
|
[[ ! -s "$root/invalid.stdout" ]]
|
||||||
done
|
done
|
||||||
|
|
||||||
|
cp bin/wasm/core.wasm "$root/serialize.wasm"
|
||||||
|
INVOKE_TIMEOUT_SECONDS=30 invoke "$root/serialize" --serialize-module "$root/serialize.wasm"
|
||||||
|
[[ -s "$root/serialize.cwasm" ]]
|
||||||
|
[[ ! -s "$root/serialize.stderr" ]]
|
||||||
|
|
||||||
|
# Serialization shares the unit compile lock. If the wasm path is replaced
|
||||||
|
# while the serializer is waiting, it must read the replacement and must not
|
||||||
|
# publish a stale native artifact for the old inode.
|
||||||
|
cp bin/wasm/core.wasm "$root/race.wasm"
|
||||||
|
exec 8>"$root/race.wasm.lock"
|
||||||
|
flock 8
|
||||||
|
INVOKE_TIMEOUT_SECONDS=30 invoke "$root/race" --serialize-module "$root/race.wasm" &
|
||||||
|
race_pid=$!
|
||||||
|
sleep 0.1
|
||||||
|
kill -0 "$race_pid"
|
||||||
|
printf 'not wasm\n' >"$root/race.wasm.next"
|
||||||
|
mv "$root/race.wasm.next" "$root/race.wasm"
|
||||||
|
flock -u 8
|
||||||
|
set +e
|
||||||
|
wait "$race_pid"
|
||||||
|
race_rc=$?
|
||||||
|
set -e
|
||||||
|
[[ $race_rc -eq 1 ]]
|
||||||
|
grep -Eqi 'wasm|magic|WebAssembly|compile' "$root/race.stderr"
|
||||||
|
[[ ! -e "$root/race.cwasm" ]]
|
||||||
|
if compgen -G "$root/race.cwasm.*.tmp" >/dev/null; then
|
||||||
|
echo "failed serialization left a temporary artifact" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
[[ -S "$socket_path" ]]
|
[[ -S "$socket_path" ]]
|
||||||
[[ "$(stat -c %i "$socket_path")" == "$socket_inode" ]]
|
[[ "$(stat -c %i "$socket_path")" == "$socket_inode" ]]
|
||||||
kill -0 "$listener_pid"
|
kill -0 "$listener_pid"
|
||||||
|
|||||||
@@ -24,6 +24,9 @@ inode_before=$(stat -c %i "$socket_path")
|
|||||||
load_pid=$!
|
load_pid=$!
|
||||||
sleep 0.05
|
sleep 0.05
|
||||||
systemctl restart "$service_name"
|
systemctl restart "$service_name"
|
||||||
|
systemctl is-active --quiet "$socket_name"
|
||||||
|
inode_during=$(stat -c %i "$socket_path")
|
||||||
|
[[ "$inode_during" = "$inode_before" ]]
|
||||||
wait "$load_pid"
|
wait "$load_pid"
|
||||||
|
|
||||||
inode_after=$(stat -c %i "$socket_path")
|
inode_after=$(stat -c %i "$socket_path")
|
||||||
|
|||||||
@@ -31,7 +31,7 @@ cleanup() {
|
|||||||
}
|
}
|
||||||
trap cleanup EXIT
|
trap cleanup EXIT
|
||||||
mkdir -p "$site/components" "$work" "$root/run" "$root/session" "$root/upload"
|
mkdir -p "$site/components" "$work" "$root/run" "$root/session" "$root/upload"
|
||||||
sed -E '/^[[:space:]]*(BIN_DIRECTORY|PRECOMPILE_FILES_IN|SITE_DIRECTORY|FCGI_SOCKET_PATH|FCGI_PORT|CLI_SOCKET_PATH|WS_BROKER_SOCKET_PATH|HTTP_PORT|HTTP_DOCUMENT_ROOT|SESSION_PATH|TMP_UPLOAD_PATH|WASM_CORE_PATH|WASM_COMPILE_SCRIPT|WASM_INVOCATION_TIMEOUT_MS|WASM_EPOCH_PERIOD_MS|PROACTIVE_COMPILE_ENABLED|WORKER_COUNT|COMPILE_FAILURE_RETRY_SECONDS)[[:space:]]*=/d' \
|
sed -E '/^[[:space:]]*(BIN_DIRECTORY|PRECOMPILE_FILES_IN|SITE_DIRECTORY|FCGI_SOCKET_PATH|FCGI_PORT|CLI_SOCKET_PATH|CLI_WORKER_COUNT|CLI_WORKER_MAX_REQUESTS|WS_BROKER_SOCKET_PATH|HTTP_PORT|HTTP_DOCUMENT_ROOT|SESSION_PATH|TMP_UPLOAD_PATH|WASM_CORE_PATH|WASM_COMPILE_SCRIPT|WASM_INVOCATION_TIMEOUT_MS|WASM_EPOCH_PERIOD_MS|PROACTIVE_COMPILE_ENABLED|WORKER_COUNT|COMPILE_FAILURE_RETRY_SECONDS)[[:space:]]*=/d' \
|
||||||
/etc/uce/settings.cfg >"$settings"
|
/etc/uce/settings.cfg >"$settings"
|
||||||
cat >>"$settings" <<CFG
|
cat >>"$settings" <<CFG
|
||||||
BIN_DIRECTORY=$work
|
BIN_DIRECTORY=$work
|
||||||
@@ -40,12 +40,14 @@ SITE_DIRECTORY=$site
|
|||||||
FCGI_SOCKET_PATH=$root/run/fastcgi.sock
|
FCGI_SOCKET_PATH=$root/run/fastcgi.sock
|
||||||
FCGI_PORT=
|
FCGI_PORT=
|
||||||
CLI_SOCKET_PATH=$root/run/cli.sock
|
CLI_SOCKET_PATH=$root/run/cli.sock
|
||||||
|
CLI_WORKER_COUNT=1
|
||||||
|
CLI_WORKER_MAX_REQUESTS=0
|
||||||
WS_BROKER_SOCKET_PATH=$root/run/ws.sock
|
WS_BROKER_SOCKET_PATH=$root/run/ws.sock
|
||||||
HTTP_PORT=
|
HTTP_PORT=
|
||||||
HTTP_DOCUMENT_ROOT=$site
|
HTTP_DOCUMENT_ROOT=$site
|
||||||
SESSION_PATH=$root/session
|
SESSION_PATH=$root/session
|
||||||
TMP_UPLOAD_PATH=$root/upload
|
TMP_UPLOAD_PATH=$root/upload
|
||||||
WASM_CORE_PATH=$(pwd)/bin/wasm/core.wasm
|
WASM_CORE_PATH=$(pwd -P)/bin/wasm/core.wasm
|
||||||
WASM_COMPILE_SCRIPT=$root/compile
|
WASM_COMPILE_SCRIPT=$root/compile
|
||||||
WASM_INVOCATION_TIMEOUT_MS=2000
|
WASM_INVOCATION_TIMEOUT_MS=2000
|
||||||
WASM_EPOCH_PERIOD_MS=20
|
WASM_EPOCH_PERIOD_MS=20
|
||||||
|
|||||||
@@ -21,11 +21,20 @@ if [[ -r "$settings_file" ]]; then
|
|||||||
[[ -n "${UCE_CLI_SOCKET:-}" ]] || socket_path=$(awk -F= '/^[[:space:]]*CLI_SOCKET_PATH[[:space:]]*=/ {gsub(/^[[:space:]]+|[[:space:]]+$/, "", $2); print $2; exit}' "$settings_file")
|
[[ -n "${UCE_CLI_SOCKET:-}" ]] || socket_path=$(awk -F= '/^[[:space:]]*CLI_SOCKET_PATH[[:space:]]*=/ {gsub(/^[[:space:]]+|[[:space:]]+$/, "", $2); print $2; exit}' "$settings_file")
|
||||||
[[ -n "${BIN_DIRECTORY:-}" ]] || bin_directory=$(awk -F= '/^[[:space:]]*BIN_DIRECTORY[[:space:]]*=/ {gsub(/^[[:space:]]+|[[:space:]]+$/, "", $2); print $2; exit}' "$settings_file")
|
[[ -n "${BIN_DIRECTORY:-}" ]] || bin_directory=$(awk -F= '/^[[:space:]]*BIN_DIRECTORY[[:space:]]*=/ {gsub(/^[[:space:]]+|[[:space:]]+$/, "", $2); print $2; exit}' "$settings_file")
|
||||||
invocation_ms=$(awk -F= '/^[[:space:]]*WASM_INVOCATION_TIMEOUT_MS[[:space:]]*=/ {gsub(/^[[:space:]]+|[[:space:]]+$/, "", $2); print $2; exit}' "$settings_file")
|
invocation_ms=$(awk -F= '/^[[:space:]]*WASM_INVOCATION_TIMEOUT_MS[[:space:]]*=/ {gsub(/^[[:space:]]+|[[:space:]]+$/, "", $2); print $2; exit}' "$settings_file")
|
||||||
|
cli_worker_count=$(awk -F= '/^[[:space:]]*CLI_WORKER_COUNT[[:space:]]*=/ {gsub(/^[[:space:]]+|[[:space:]]+$/, "", $2); value=$2} END{print value}' "$settings_file")
|
||||||
fi
|
fi
|
||||||
site_directory="${site_directory:-site}"
|
site_directory="${site_directory:-site}"
|
||||||
socket_path="${socket_path:-/run/uce/cli.sock}"
|
socket_path="${socket_path:-/run/uce/cli.sock}"
|
||||||
bin_directory="${bin_directory:-/tmp/uce/work}"
|
bin_directory="${bin_directory:-/tmp/uce/work}"
|
||||||
invocation_ms="${invocation_ms:-30000}"
|
invocation_ms="${invocation_ms:-30000}"
|
||||||
|
cli_worker_count="${cli_worker_count:-0}"
|
||||||
|
curl_transport=(--unix-socket "$socket_path")
|
||||||
|
test_base_url="http://localhost"
|
||||||
|
if (( cli_worker_count > 0 )); then
|
||||||
|
test_http_host="${UCE_TEST_HTTP_HOST:-uce.openfu.com}"
|
||||||
|
curl_transport=(--resolve "$test_http_host:80:127.0.0.1")
|
||||||
|
test_base_url="http://$test_http_host"
|
||||||
|
fi
|
||||||
test_name="invocation-timeout-test-$$"
|
test_name="invocation-timeout-test-$$"
|
||||||
source_dir="$site_directory/$test_name"
|
source_dir="$site_directory/$test_name"
|
||||||
pid_file="/tmp/uce-$test_name-worker"
|
pid_file="/tmp/uce-$test_name-worker"
|
||||||
@@ -42,13 +51,15 @@ mkdir -p "$source_dir"
|
|||||||
cache_dir="$(scripts/unit_cache_directory "$bin_directory")$(realpath "$source_dir")"
|
cache_dir="$(scripts/unit_cache_directory "$bin_directory")$(realpath "$source_dir")"
|
||||||
|
|
||||||
printf '%s\n' \
|
printf '%s\n' \
|
||||||
'CLI(Request& context) {' \
|
'void timeout_test_run(Request& context) {' \
|
||||||
' if(context.get["warm"] == "1") { print("warm"); return; }' \
|
' if(context.get["warm"] == "1") { print("warm"); return; }' \
|
||||||
" file_put_contents(\"$pid_file\", std::to_string(request_perf()[\"worker_pid\"].to_u64()));" \
|
" file_put_contents(\"$pid_file\", std::to_string(request_perf()[\"worker_pid\"].to_u64()));" \
|
||||||
' while(true) time_precise();' \
|
' while(true) time_precise();' \
|
||||||
'}' >"$source_dir/hostcall-loop.uce"
|
'}' \
|
||||||
|
'CLI(Request& context) { timeout_test_run(context); }' \
|
||||||
|
'RENDER(Request& context) { timeout_test_run(context); }' >"$source_dir/hostcall-loop.uce"
|
||||||
printf '%s\n' \
|
printf '%s\n' \
|
||||||
'CLI(Request& context) {' \
|
'void timeout_test_run(Request& context) {' \
|
||||||
' if(context.get["warm"] == "1") { print("warm"); return; }' \
|
' if(context.get["warm"] == "1") { print("warm"); return; }' \
|
||||||
' if(context.get["quick"] == "1") { print(shell_exec("printf quick")); return; }' \
|
' if(context.get["quick"] == "1") { print(shell_exec("printf quick")); return; }' \
|
||||||
' if(context.get["status"] == "1") { DValue spec; spec["cmd"] = "exit 7"; spec["timeout_ms"] = (f64)500; print(shell_exec(spec)["exit_code"].to_u64()); return; }' \
|
' if(context.get["status"] == "1") { DValue spec; spec["cmd"] = "exit 7"; spec["timeout_ms"] = (f64)500; print(shell_exec(spec)["exit_code"].to_u64()); return; }' \
|
||||||
@@ -56,46 +67,50 @@ printf '%s\n' \
|
|||||||
' if(context.get["job"] == "1") { DValue spec; spec["cmd"] = "sleep 2"; spec["timeout_ms"] = (f64)5000; u64 job = shell_spawn(spec); f64 started = time_precise(); job_await(job, 300); u64 elapsed = (u64)((time_precise() - started) * 1000); job_cancel(job); print(elapsed); return; }' \
|
' if(context.get["job"] == "1") { DValue spec; spec["cmd"] = "sleep 2"; spec["timeout_ms"] = (f64)5000; u64 job = shell_spawn(spec); f64 started = time_precise(); job_await(job, 300); u64 elapsed = (u64)((time_precise() - started) * 1000); job_cancel(job); print(elapsed); return; }' \
|
||||||
" file_put_contents(\"$pid_file\", std::to_string(request_perf()[\"worker_pid\"].to_u64()));" \
|
" file_put_contents(\"$pid_file\", std::to_string(request_perf()[\"worker_pid\"].to_u64()));" \
|
||||||
' print(shell_exec("printf shell-start; sleep 60 & printf shell-end"));' \
|
' print(shell_exec("printf shell-start; sleep 60 & printf shell-end"));' \
|
||||||
'}' >"$source_dir/legacy-shell.uce"
|
'}' \
|
||||||
|
'CLI(Request& context) { timeout_test_run(context); }' \
|
||||||
|
'RENDER(Request& context) { timeout_test_run(context); }' >"$source_dir/legacy-shell.uce"
|
||||||
printf '%s\n' \
|
printf '%s\n' \
|
||||||
'CLI(Request& context) {' \
|
'void timeout_test_run(Request& context) {' \
|
||||||
' if(context.get["warm"] == "1") { print("warm"); return; }' \
|
' if(context.get["warm"] == "1") { print("warm"); return; }' \
|
||||||
" file_put_contents(\"$pid_file\", std::to_string(request_perf()[\"worker_pid\"].to_u64()));" \
|
" file_put_contents(\"$pid_file\", std::to_string(request_perf()[\"worker_pid\"].to_u64()));" \
|
||||||
' while(true) sleep(60);' \
|
' while(true) sleep(60);' \
|
||||||
' print("sleep-end");' \
|
' print("sleep-end");' \
|
||||||
'}' >"$source_dir/sleep.uce"
|
'}' \
|
||||||
|
'CLI(Request& context) { timeout_test_run(context); }' \
|
||||||
|
'RENDER(Request& context) { timeout_test_run(context); }' >"$source_dir/sleep.uce"
|
||||||
printf '%s\n' \
|
printf '%s\n' \
|
||||||
'CLI(Request& context) { print(request_perf()["worker_pid"].to_u64(), "|health"); }' >"$source_dir/health.uce"
|
'void timeout_test_run(Request& context) { print(request_perf()["worker_pid"].to_u64(), "|health"); }' \
|
||||||
|
'CLI(Request& context) { timeout_test_run(context); }' \
|
||||||
|
'RENDER(Request& context) { timeout_test_run(context); }' >"$source_dir/health.uce"
|
||||||
|
|
||||||
max_seconds=$(( (invocation_ms + 15000) / 1000 ))
|
max_seconds=$(( (invocation_ms + 15000) / 1000 ))
|
||||||
(( max_seconds >= 10 )) || max_seconds=10
|
(( max_seconds >= 10 )) || max_seconds=10
|
||||||
|
|
||||||
request() {
|
request() {
|
||||||
local unit="$1"
|
local unit="$1"
|
||||||
curl -sS --max-time "$max_seconds" -o "$body_file" -w '%{http_code}' --unix-socket "$socket_path" "http://localhost/$test_name/$unit"
|
curl -sS --max-time "$max_seconds" -o "$body_file" -w '%{http_code}' "${curl_transport[@]}" "$test_base_url/$test_name/$unit"
|
||||||
}
|
}
|
||||||
|
|
||||||
same_worker_health() {
|
same_worker_health() {
|
||||||
local expected_pid="$1"
|
local expected_pid="$1"
|
||||||
for _ in $(seq 1 32); do
|
kill -0 "$expected_pid" 2>/dev/null || return 1
|
||||||
local health
|
local health
|
||||||
health=$(curl -sS --max-time 5 --fail-with-body --unix-socket "$socket_path" "http://localhost/$test_name/health.uce")
|
health=$(curl -sS --max-time 5 --fail-with-body "${curl_transport[@]}" "$test_base_url/$test_name/health.uce")
|
||||||
[[ "$health" == "$expected_pid|health" ]] && return 0
|
[[ "$health" =~ ^[0-9]+\|health$ ]]
|
||||||
done
|
|
||||||
return 1
|
|
||||||
}
|
}
|
||||||
|
|
||||||
curl -sS --max-time "$max_seconds" --fail-with-body --unix-socket "$socket_path" "http://localhost/$test_name/hostcall-loop.uce?warm=1" >/dev/null
|
curl -sS --max-time "$max_seconds" --fail-with-body "${curl_transport[@]}" "$test_base_url/$test_name/hostcall-loop.uce?warm=1" >/dev/null
|
||||||
curl -sS --max-time "$max_seconds" --fail-with-body --unix-socket "$socket_path" "http://localhost/$test_name/legacy-shell.uce?warm=1" >/dev/null
|
curl -sS --max-time "$max_seconds" --fail-with-body "${curl_transport[@]}" "$test_base_url/$test_name/legacy-shell.uce?warm=1" >/dev/null
|
||||||
curl -sS --max-time "$max_seconds" --fail-with-body --unix-socket "$socket_path" "http://localhost/$test_name/sleep.uce?warm=1" >/dev/null
|
curl -sS --max-time "$max_seconds" --fail-with-body "${curl_transport[@]}" "$test_base_url/$test_name/sleep.uce?warm=1" >/dev/null
|
||||||
curl -sS --max-time "$max_seconds" --fail-with-body --unix-socket "$socket_path" "http://localhost/$test_name/health.uce" >/dev/null
|
curl -sS --max-time "$max_seconds" --fail-with-body "${curl_transport[@]}" "$test_base_url/$test_name/health.uce" >/dev/null
|
||||||
quick=$(curl -sS --max-time 5 --fail-with-body --unix-socket "$socket_path" "http://localhost/$test_name/legacy-shell.uce?quick=1")
|
quick=$(curl -sS --max-time 5 --fail-with-body "${curl_transport[@]}" "$test_base_url/$test_name/legacy-shell.uce?quick=1")
|
||||||
[[ "$quick" == quick ]] || { echo "quick legacy shell returned: $quick" >&2; exit 1; }
|
[[ "$quick" == quick ]] || { echo "quick legacy shell returned: $quick" >&2; exit 1; }
|
||||||
exit_status=$(curl -sS --max-time 5 --fail-with-body --unix-socket "$socket_path" "http://localhost/$test_name/legacy-shell.uce?status=1")
|
exit_status=$(curl -sS --max-time 5 --fail-with-body "${curl_transport[@]}" "$test_base_url/$test_name/legacy-shell.uce?status=1")
|
||||||
[[ "$exit_status" == 7 ]] || { echo "structured shell lost exit status: $exit_status" >&2; exit 1; }
|
[[ "$exit_status" == 7 ]] || { echo "structured shell lost exit status: $exit_status" >&2; exit 1; }
|
||||||
zero_timeout=$(curl -sS --max-time 5 --fail-with-body --unix-socket "$socket_path" "http://localhost/$test_name/legacy-shell.uce?zero=1")
|
zero_timeout=$(curl -sS --max-time 5 --fail-with-body "${curl_transport[@]}" "$test_base_url/$test_name/legacy-shell.uce?zero=1")
|
||||||
[[ "$zero_timeout" == zero ]] || { echo "structured shell zero-timeout default changed: $zero_timeout" >&2; exit 1; }
|
[[ "$zero_timeout" == zero ]] || { echo "structured shell zero-timeout default changed: $zero_timeout" >&2; exit 1; }
|
||||||
job_elapsed=$(curl -sS --max-time 5 --fail-with-body --unix-socket "$socket_path" "http://localhost/$test_name/legacy-shell.uce?job=1")
|
job_elapsed=$(curl -sS --max-time 5 --fail-with-body "${curl_transport[@]}" "$test_base_url/$test_name/legacy-shell.uce?job=1")
|
||||||
[[ "$job_elapsed" =~ ^[0-9]+$ && "$job_elapsed" -ge 200 && "$job_elapsed" -lt 500 ]] || { echo "job_await two-call duration was ${job_elapsed}ms" >&2; exit 1; }
|
[[ "$job_elapsed" =~ ^[0-9]+$ && "$job_elapsed" -ge 200 && "$job_elapsed" -lt 500 ]] || { echo "job_await two-call duration was ${job_elapsed}ms" >&2; exit 1; }
|
||||||
|
|
||||||
for unit in hostcall-loop.uce legacy-shell.uce sleep.uce; do
|
for unit in hostcall-loop.uce legacy-shell.uce sleep.uce; do
|
||||||
|
|||||||
@@ -29,7 +29,7 @@ cleanup() {
|
|||||||
}
|
}
|
||||||
trap cleanup EXIT
|
trap cleanup EXIT
|
||||||
mkdir -p "$site/components" "$work" "$root/run" "$root/session" "$root/upload"
|
mkdir -p "$site/components" "$work" "$root/run" "$root/session" "$root/upload"
|
||||||
sed -E '/^[[:space:]]*(BIN_DIRECTORY|PRECOMPILE_FILES_IN|SITE_DIRECTORY|FCGI_SOCKET_PATH|FCGI_PORT|CLI_SOCKET_PATH|WS_BROKER_SOCKET_PATH|HTTP_PORT|HTTP_DOCUMENT_ROOT|SESSION_PATH|TMP_UPLOAD_PATH|WASM_CORE_PATH|WASM_INVOCATION_TIMEOUT_MS|WASM_EPOCH_PERIOD_MS|PROACTIVE_COMPILE_ENABLED|WORKER_COUNT)[[:space:]]*=/d' \
|
sed -E '/^[[:space:]]*(BIN_DIRECTORY|PRECOMPILE_FILES_IN|SITE_DIRECTORY|FCGI_SOCKET_PATH|FCGI_PORT|CLI_SOCKET_PATH|CLI_WORKER_COUNT|CLI_WORKER_MAX_REQUESTS|WS_BROKER_SOCKET_PATH|HTTP_PORT|HTTP_DOCUMENT_ROOT|SESSION_PATH|TMP_UPLOAD_PATH|WASM_CORE_PATH|WASM_INVOCATION_TIMEOUT_MS|WASM_EPOCH_PERIOD_MS|PROACTIVE_COMPILE_ENABLED|SERVE_LAST_KNOWN_GOOD|SHOW_DYNAMIC_COMPILE_ERRORS|WORKER_COUNT)[[:space:]]*=/d' \
|
||||||
/etc/uce/settings.cfg >"$settings"
|
/etc/uce/settings.cfg >"$settings"
|
||||||
cat >>"$settings" <<CFG
|
cat >>"$settings" <<CFG
|
||||||
BIN_DIRECTORY=$work
|
BIN_DIRECTORY=$work
|
||||||
@@ -38,15 +38,19 @@ SITE_DIRECTORY=$site
|
|||||||
FCGI_SOCKET_PATH=$root/run/fastcgi.sock
|
FCGI_SOCKET_PATH=$root/run/fastcgi.sock
|
||||||
FCGI_PORT=
|
FCGI_PORT=
|
||||||
CLI_SOCKET_PATH=$socket
|
CLI_SOCKET_PATH=$socket
|
||||||
|
CLI_WORKER_COUNT=1
|
||||||
|
CLI_WORKER_MAX_REQUESTS=0
|
||||||
WS_BROKER_SOCKET_PATH=$root/run/ws.sock
|
WS_BROKER_SOCKET_PATH=$root/run/ws.sock
|
||||||
HTTP_PORT=
|
HTTP_PORT=
|
||||||
HTTP_DOCUMENT_ROOT=$site
|
HTTP_DOCUMENT_ROOT=$site
|
||||||
SESSION_PATH=$root/session
|
SESSION_PATH=$root/session
|
||||||
TMP_UPLOAD_PATH=$root/upload
|
TMP_UPLOAD_PATH=$root/upload
|
||||||
WASM_CORE_PATH=$(pwd)/bin/wasm/core.wasm
|
WASM_CORE_PATH=$(pwd)/bin/wasm/core.wasm
|
||||||
WASM_INVOCATION_TIMEOUT_MS=100
|
WASM_INVOCATION_TIMEOUT_MS=1000
|
||||||
WASM_EPOCH_PERIOD_MS=10
|
WASM_EPOCH_PERIOD_MS=10
|
||||||
PROACTIVE_COMPILE_ENABLED=0
|
PROACTIVE_COMPILE_ENABLED=0
|
||||||
|
SERVE_LAST_KNOWN_GOOD=0
|
||||||
|
SHOW_DYNAMIC_COMPILE_ERRORS=1
|
||||||
WORKER_COUNT=1
|
WORKER_COUNT=1
|
||||||
CFG
|
CFG
|
||||||
mount --bind "$settings" /etc/uce/settings.cfg
|
mount --bind "$settings" /etc/uce/settings.cfg
|
||||||
@@ -259,14 +263,14 @@ wait_ready() {
|
|||||||
prepare_deadline
|
prepare_deadline
|
||||||
deadline_wasm="$cache/components/deadline.uce.wasm"
|
deadline_wasm="$cache/components/deadline.uce.wasm"
|
||||||
deadline_counter="$root/deadline.counter"
|
deadline_counter="$root/deadline.counter"
|
||||||
printf '%s delay 150 - - %s\n' "$deadline_wasm" "$deadline_counter" >"$control"
|
printf '%s delay 1100 - - %s\n' "$deadline_wasm" "$deadline_counter" >"$control"
|
||||||
started=$(date +%s%N)
|
started=$(date +%s%N)
|
||||||
set +e
|
set +e
|
||||||
deadline_output=$(request target=components/deadline.uce 2>&1)
|
deadline_output=$(request target=components/deadline.uce 2>&1)
|
||||||
set -e
|
set -e
|
||||||
elapsed_ms=$(( ($(date +%s%N) - started) / 1000000 ))
|
elapsed_ms=$(( ($(date +%s%N) - started) / 1000000 ))
|
||||||
[[ "$deadline_output" == *UCE_INVOCATION_TIMEOUT:* ]] || { echo "metadata scan lacked canonical timeout: $deadline_output" >&2; exit 1; }
|
[[ "$deadline_output" == *UCE_INVOCATION_TIMEOUT:* ]] || { echo "metadata scan lacked canonical timeout: $deadline_output" >&2; exit 1; }
|
||||||
(( elapsed_ms >= 100 && elapsed_ms < 2000 )) || { echo "metadata timeout took ${elapsed_ms}ms" >&2; exit 1; }
|
(( elapsed_ms >= 1000 && elapsed_ms < 3000 )) || { echo "metadata timeout took ${elapsed_ms}ms" >&2; exit 1; }
|
||||||
[[ $(wc -l <"$deadline_counter") -eq 1 ]] || { echo "metadata scanner read past the first delayed refill" >&2; exit 1; }
|
[[ $(wc -l <"$deadline_counter") -eq 1 ]] || { echo "metadata scanner read past the first delayed refill" >&2; exit 1; }
|
||||||
rm -f "$control"
|
rm -f "$control"
|
||||||
[[ "$(request health=1)" == "$worker_pid|health" ]]
|
[[ "$(request health=1)" == "$worker_pid|health" ]]
|
||||||
@@ -275,14 +279,14 @@ restore deadline
|
|||||||
|
|
||||||
rm -f "$cache/components/deadline.uce.cwasm"
|
rm -f "$cache/components/deadline.uce.cwasm"
|
||||||
touch "$deadline_wasm"
|
touch "$deadline_wasm"
|
||||||
printf '%s delay 150 - - %s\n' "$deadline_wasm" "$root/full-deadline.counter" >"$control"
|
printf '%s delay 1100 - - %s\n' "$deadline_wasm" "$root/full-deadline.counter" >"$control"
|
||||||
started=$(date +%s%N)
|
started=$(date +%s%N)
|
||||||
set +e
|
set +e
|
||||||
full_deadline_output=$(request target=components/deadline.uce 2>&1)
|
full_deadline_output=$(request target=components/deadline.uce 2>&1)
|
||||||
set -e
|
set -e
|
||||||
elapsed_ms=$(( ($(date +%s%N) - started) / 1000000 ))
|
elapsed_ms=$(( ($(date +%s%N) - started) / 1000000 ))
|
||||||
[[ "$full_deadline_output" == *UCE_INVOCATION_TIMEOUT:* ]] || { echo "full artifact read lacked canonical timeout: $full_deadline_output" >&2; exit 1; }
|
[[ "$full_deadline_output" == *UCE_INVOCATION_TIMEOUT:* ]] || { echo "full artifact read lacked canonical timeout: $full_deadline_output" >&2; exit 1; }
|
||||||
(( elapsed_ms >= 100 && elapsed_ms < 2000 )) || { echo "full artifact timeout took ${elapsed_ms}ms" >&2; exit 1; }
|
(( elapsed_ms >= 1000 && elapsed_ms < 3000 )) || { echo "full artifact timeout took ${elapsed_ms}ms" >&2; exit 1; }
|
||||||
[[ $(wc -l <"$root/full-deadline.counter") -eq 1 ]] || { echo "full artifact reader continued after the first delayed chunk" >&2; exit 1; }
|
[[ $(wc -l <"$root/full-deadline.counter") -eq 1 ]] || { echo "full artifact reader continued after the first delayed chunk" >&2; exit 1; }
|
||||||
rm -f "$control"
|
rm -f "$control"
|
||||||
[[ "$(request health=1)" == "$worker_pid|health" ]]
|
[[ "$(request health=1)" == "$worker_pid|health" ]]
|
||||||
|
|||||||
@@ -14,6 +14,7 @@ hmac_sha256
|
|||||||
hmac_sha256_hex
|
hmac_sha256_hex
|
||||||
random_bytes
|
random_bytes
|
||||||
crypto_equal
|
crypto_equal
|
||||||
|
crypto_operation
|
||||||
password_hash
|
password_hash
|
||||||
password_verify
|
password_verify
|
||||||
password_needs_rehash
|
password_needs_rehash
|
||||||
|
|||||||
@@ -0,0 +1,28 @@
|
|||||||
|
:sig
|
||||||
|
DValue crypto_operation(DValue request)
|
||||||
|
|
||||||
|
:params
|
||||||
|
request : structured operation, algorithm, and operation-specific fields
|
||||||
|
return value : map with ok, bounded error code, and operation-specific output
|
||||||
|
|
||||||
|
:content
|
||||||
|
Runs one explicitly supported structured asymmetric cryptographic operation. The initial allowlist is `key_generate` with `ES256` and `jwt_sign` with `ES256`. Unknown operations and algorithms fail closed.
|
||||||
|
|
||||||
|
ES256 signing validates that `x`, `y`, and `d` form one P-256 key, forces the protected `alg` to `ES256`, and emits a compact JWT with a 64-byte JOSE signature. Requests are capped at 32 KiB; `cbor_decode` accepts at most 16 KiB decoded CBOR (21,846 canonical base64url characters), 256 nodes, and depth 16. CBOR is definite-length only, validates UTF-8 text, preserves typed map keys, rejects structurally duplicate keys and trailing bytes, and reports malformed CBOR as `invalid_cbor`. Header and claims roots must be JSON objects containing valid UTF-8 without raw control bytes.
|
||||||
|
|
||||||
|
This function does not replace typed digest, HMAC, password, randomness, or constant-time comparison APIs. It exposes no raw signing, arbitrary curve/digest selection, encryption, or generic OpenSSL access. Keep returned private JWKs secret.
|
||||||
|
|
||||||
|
:example
|
||||||
|
DValue key_request;
|
||||||
|
key_request["operation"] = "key_generate";
|
||||||
|
key_request["algorithm"] = "ES256";
|
||||||
|
DValue key = crypto_operation(key_request);
|
||||||
|
|
||||||
|
DValue sign_request;
|
||||||
|
sign_request["operation"] = "jwt_sign";
|
||||||
|
sign_request["algorithm"] = "ES256";
|
||||||
|
sign_request["private_jwk"] = key["private_jwk"];
|
||||||
|
sign_request["protected_header"]["typ"] = "JWT";
|
||||||
|
sign_request["claims"]["iss"] = "https://client.example";
|
||||||
|
DValue signed_jwt = crypto_operation(sign_request);
|
||||||
|
print(signed_jwt["ok"].to_bool() ? "signed" : "failed", "\n");
|
||||||
@@ -1,8 +1,18 @@
|
|||||||
# http_request
|
# http_request
|
||||||
|
|
||||||
Performs a bounded outbound HTTP(S) request using the runtime `curl` binary. Request fields: `method`, `url`, `headers`, `body`, `timeout_ms`, `follow_redirects`.
|
Performs an outbound HTTP(S) request using the runtime `curl` binary. Existing request fields remain backward-compatible: `method`, `url`, `headers`, `body`, `timeout_ms`, and `follow_redirects`.
|
||||||
|
|
||||||
Returns `{ status, headers, body, error }`. `headers` is a name/value map. A missing `curl` binary returns a clear `error` string.
|
An absent `security` map, or a map containing none of these recognized keys, keeps the legacy request behavior. If a `security` map contains any recognized key, it selects hardening and **every** recognized field below must be explicitly boolean `true`; missing, partial, non-true, or `false` fields fail closed with `invalid_request`.
|
||||||
|
|
||||||
|
- `https_only` rejects non-HTTPS URLs, IP-literal hosts, and URL userinfo.
|
||||||
|
- `public_dns_only` validates **every** DNS answer against the public IPv4 policy. IPv6 answers currently fail closed.
|
||||||
|
- `pin_dns` pins curl to one validated answer with `--resolve`, retaining the URL hostname for TLS/SNI.
|
||||||
|
- `isolated_curl` uses absolute `/usr/bin/curl`, `--disable` as argv[1], cleared environment, no proxy/config/netrc/HSTS/Alt-Svc inheritance, no redirects, a three-second connect bound, and a ten-second total bound.
|
||||||
|
- `no_redirects` makes redirect following and `follow_redirects=true` invalid composition.
|
||||||
|
|
||||||
|
Hardened requests bound body input/output to 64 KiB and response headers to 8 KiB. Async hardened requests keep curl and its descendants in the job worker process group, so cancelling that job kills that group only. Methods are limited to `GET`, `POST`, `PUT`, `PATCH`, `DELETE`, `HEAD`, and `OPTIONS`; header names/values are validated and caller-controlled `Host`, framing, connection, and expectation headers are rejected. Sensitive request bodies go to curl stdin, never argv. Errors are typed non-secret values such as `invalid_request`, `unsafe_dns`, `timeout`, `response_too_large`, `redirect_not_allowed`, `http_status`, and `network_failure`.
|
||||||
|
|
||||||
|
Returns `{ status, headers, body, error }`.
|
||||||
|
|
||||||
:see
|
:see
|
||||||
>socket
|
>socket
|
||||||
@@ -12,3 +22,10 @@ DValue req; req["method"] = "GET"; req["url"] = "http://127.0.0.1/doc/index.uce"
|
|||||||
req["headers"]["Host"] = "uce.openfu.com"; req["timeout_ms"] = (f64)2000;
|
req["headers"]["Host"] = "uce.openfu.com"; req["timeout_ms"] = (f64)2000;
|
||||||
DValue resp = http_request(req);
|
DValue resp = http_request(req);
|
||||||
print("HTTP ", resp["status"].to_u64(), ", ", resp["body"].to_string().length(), " bytes returned\n");
|
print("HTTP ", resp["status"].to_u64(), ", ", resp["body"].to_string().length(), " bytes returned\n");
|
||||||
|
|
||||||
|
:example
|
||||||
|
// Hardened requests reject IP-literal hosts before they start curl.
|
||||||
|
DValue hardened; hardened["method"]="GET"; hardened["url"]="https://127.0.0.1/";
|
||||||
|
for(String key:{"https_only","public_dns_only","pin_dns","isolated_curl","no_redirects"}) hardened["security"][key].set_bool(true);
|
||||||
|
DValue rejected=http_request(hardened);
|
||||||
|
print(rejected["error"].to_string(), "\n");
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# http_request_async
|
# http_request_async
|
||||||
|
|
||||||
Starts the same bounded curl-backed request as `http_request()` in the file-backed async job registry and returns a job id. Use `job_await()` or `job_result()` to retrieve the HTTP result.
|
Starts the same request shape as `http_request()`, including its opt-in `security` hardening fields, in the file-backed async job registry and returns a job id. An unknown `security` object remains legacy; any recognized hardening key requires all five keys to be explicit `true` or the request fails closed. Hardened curl remains in this job's worker process group, so `job_cancel()` kills curl and its descendants without signalling other jobs. Use `job_await()` or `job_result()` to retrieve the HTTP result.
|
||||||
|
|
||||||
:see
|
:see
|
||||||
>socket
|
>socket
|
||||||
|
|||||||
@@ -13,6 +13,7 @@ return value : string containing the Memcache server's response
|
|||||||
Executes a raw command on an open Memcache connection and returns the server response as a string.
|
Executes a raw command on an open Memcache connection and returns the server response as a string.
|
||||||
|
|
||||||
This is the low-level escape hatch for Memcache operations that are not covered by the dedicated helpers.
|
This is the low-level escape hatch for Memcache operations that are not covered by the dedicated helpers.
|
||||||
|
UCE reads the complete framed ASCII response, including multi-value retrievals and `stats`, rather than returning a socket chunk. The complete write and read share a one-second deadline and responses are limited to 8 MiB. A timeout, malformed frame, or oversized response returns an empty string and makes that connection unusable so unread bytes cannot affect a later command. Commands ending in `noreply` and `quit` return immediately with an empty string.
|
||||||
|
|
||||||
:example
|
:example
|
||||||
u64 conn = memcache_connect();
|
u64 conn = memcache_connect();
|
||||||
|
|||||||
@@ -16,7 +16,7 @@ Establishes a connection to a MySQL server and returns a request-owned pointer t
|
|||||||
|
|
||||||
This connection handle is then used with helpers such as `mysql_query()`, `mysql_error()`, and `mysql_disconnect()`.
|
This connection handle is then used with helpers such as `mysql_query()`, `mysql_error()`, and `mysql_disconnect()`.
|
||||||
|
|
||||||
MySQL handles are request-scoped framework resources. Repeated `mysql_connect()` calls with the same host, credentials, and database reuse one server connection within the current request. Database identity is part of both request-local and worker-persistent pool keys. Each call creates a lease and `mysql_disconnect()` releases that lease. At request cleanup UCE returns the server connection to the current worker's persistent pool, then resets it before cross-request reuse. The pool holds up to `MYSQL_PERSISTENT_POOL_SIZE` connections per worker (default 8); set the size to 0 to disable cross-request reuse. Never store a `MySQL*` in globals, sessions, or other state that can outlive the current request.
|
MySQL handles are request-scoped framework resources. Repeated `mysql_connect()` calls with the same host, credentials, and database reuse one server connection within the current request. Database identity is part of both request-local and worker-persistent pool keys. Each call creates a lease and `mysql_disconnect()` releases that lease. At request cleanup UCE returns the server connection to the current worker's persistent pool, then resets it before cross-request reuse. The pool holds up to `MYSQL_PERSISTENT_POOL_SIZE` connections per worker (default 8); set the size to 0 to disable cross-request reuse. `MYSQL_PERSISTENT_POOL_IDLE_TIMEOUT_SECONDS` retires idle entries at the first later request boundary (default 300, 0 keeps unlimited idle retention). A completely idle worker has no boundary and does not proactively close sockets. Never store a `MySQL*` in globals, sessions, or other state that can outlive the current request.
|
||||||
|
|
||||||
:example
|
:example
|
||||||
MySQL* db = mysql_connect();
|
MySQL* db = mysql_connect();
|
||||||
|
|||||||
@@ -16,6 +16,8 @@ Executes a MySQL query and returns the resulting data, if any.
|
|||||||
|
|
||||||
`params` provides the query parameter values used by the statement. Use named `:name` placeholders only; positional `?` placeholders are rejected.
|
`params` provides the query parameter values used by the statement. Use named `:name` placeholders only; positional `?` placeholders are rejected.
|
||||||
|
|
||||||
|
Ordinary placeholders are escaped and quoted as SQL string values. For grammar positions that require an unquoted non-negative integer, such as `LIMIT` and `OFFSET`, append `!` to the placeholder (`:limit!`). Unsigned placeholders fail before query execution unless their value is a non-empty sequence of decimal digits. They never accept signs, whitespace, expressions, identifiers, or other SQL fragments.
|
||||||
|
|
||||||
The result is returned as a `DValue`, which makes it easy to iterate through rows and read fields with the usual `DValue` accessors.
|
The result is returned as a `DValue`, which makes it easy to iterate through rows and read fields with the usual `DValue` accessors.
|
||||||
|
|
||||||
After an insert, update, or delete, use `mysql_affected_rows()` to inspect how many rows changed.
|
After an insert, update, or delete, use `mysql_affected_rows()` to inspect how many rows changed.
|
||||||
@@ -24,7 +26,8 @@ After an insert, update, or delete, use `mysql_affected_rows()` to inspect how m
|
|||||||
MySQL* db = mysql_connect();
|
MySQL* db = mysql_connect();
|
||||||
if(db != 0)
|
if(db != 0)
|
||||||
{
|
{
|
||||||
DValue rows = mysql_query(db, "select 'ada@example.test' as email, 1 + 1 as total");
|
StringMap params; params["limit"] = "1";
|
||||||
|
DValue rows = mysql_query(db, "select 'ada@example.test' as email, 1 + 1 as total limit :limit!", params);
|
||||||
String email = "none"; String total = "?";
|
String email = "none"; String total = "?";
|
||||||
rows.each([&](DValue r, String key) { email = r["email"].to_string(); total = r["total"].to_string(); });
|
rows.each([&](DValue r, String key) { email = r["email"].to_string(); total = r["total"].to_string(); });
|
||||||
print(email, " / total=", total, "\n");
|
print(email, " / total=", total, "\n");
|
||||||
|
|||||||
@@ -27,7 +27,7 @@ The request log's `wasm-ready`, `wasm`, `workspace`, `invoke`, `collect`, and `p
|
|||||||
|
|
||||||
Workspace birth is subdivided into `birth_policy_us`, `birth_import_us`, `birth_instantiate_us`, `birth_exports_us`, and `birth_initialize_us`. Request-context transfer reports `context_bytes`, `context_encode_us`, `context_allocate_us`, `context_write_us`, `context_guest_apply_us`, and `context_free_us`. These bounded aggregate fields expose sizes and timing only, never request values.
|
Workspace birth is subdivided into `birth_policy_us`, `birth_import_us`, `birth_instantiate_us`, `birth_exports_us`, and `birth_initialize_us`. Request-context transfer reports `context_bytes`, `context_encode_us`, `context_allocate_us`, `context_write_us`, `context_guest_apply_us`, and `context_free_us`. These bounded aggregate fields expose sizes and timing only, never request values.
|
||||||
|
|
||||||
Wasm FastCGI workers retain up to `MYSQL_PERSISTENT_POOL_SIZE` credential-keyed MySQL connections (default `8`; set `0` to disable). UCE calls the client library's connection-reset operation before another request receives a cached connection, clearing transactions, temporary tables, session variables, and selected databases while avoiding a new authentication handshake. Same-request leases continue to share state until request cleanup.
|
Wasm FastCGI workers retain up to `MYSQL_PERSISTENT_POOL_SIZE` credential-keyed MySQL connections (default `8`; set `0` to disable). `MYSQL_PERSISTENT_POOL_IDLE_TIMEOUT_SECONDS` retires an idle entry at the first later request boundary (default `300`; `0` disables age eviction). UCE calls the client library's connection-reset operation before another request receives a cached connection, clearing transactions, temporary tables, session variables, and selected databases while avoiding a new authentication handshake. Same-request leases continue to share state until request cleanup.
|
||||||
|
|
||||||
:example
|
:example
|
||||||
DValue perf = request_perf();
|
DValue perf = request_perf();
|
||||||
|
|||||||
@@ -21,7 +21,7 @@ Starts a repeating background worker process.
|
|||||||
|
|
||||||
If a process with the same `key` is already running anywhere in the runtime instance, `task_repeat()` does not start a second worker and instead returns the PID of the existing one. Coordination is through the same shared task state used by `task()`.
|
If a process with the same `key` is already running anywhere in the runtime instance, `task_repeat()` does not start a second worker and instead returns the PID of the existing one. Coordination is through the same shared task state used by `task()`.
|
||||||
|
|
||||||
`timeout` bounds the lifetime of the repeating worker. The default is ten minutes. Pass `0` only for workers that have another shutdown path.
|
`timeout` bounds the lifetime of the repeating worker. The default is ten minutes. Pass `0` only for workers that have another shutdown path. In Wasm, each individual callback invocation remains bounded by the runtime invocation timeout even when the repeating process lifetime is unbounded.
|
||||||
|
|
||||||
:example
|
:example
|
||||||
task_repeat("doc-demo-repeat", 60.0, []() { usleep(10000); });
|
task_repeat("doc-demo-repeat", 60.0, []() { usleep(10000); });
|
||||||
|
|||||||
@@ -33,3 +33,9 @@ The example uses query-string routing in the same style as the PHP starter, but
|
|||||||
Direct requests to `/examples/uce-starter/index.uce` still work, but self-links are canonicalized back to `/examples/uce-starter/`.
|
Direct requests to `/examples/uce-starter/index.uce` still work, but self-links are canonicalized back to `/examples/uce-starter/`.
|
||||||
|
|
||||||
The demo account pages use a small file-backed user store under `/tmp/uce-starter-data/` with session-based login state.
|
The demo account pages use a small file-backed user store under `/tmp/uce-starter-data/` with session-based login state.
|
||||||
|
|
||||||
|
|
||||||
|
## Optional Datastar Assets
|
||||||
|
|
||||||
|
This starter vendors Datastar v1.0.2 as `js/datastar.js` for future server-rendered interactive islands. It is not loaded by the default page shell. `lib/datastar.uce` contains small Datastar SSE formatting helpers, and `views/datastar-example.uce` is an unlinked opt-in example route.
|
||||||
|
|
||||||
|
|||||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,38 @@
|
|||||||
|
String datastar_data_lines(String field, String value)
|
||||||
|
{
|
||||||
|
String out = "";
|
||||||
|
for(String line : split(value, "\n"))
|
||||||
|
out += "data: " + field + " " + line + "\n";
|
||||||
|
return(out);
|
||||||
|
}
|
||||||
|
|
||||||
|
void datastar_sse_headers(Request& context)
|
||||||
|
{
|
||||||
|
context.call["app"]["page_type"] = "blank";
|
||||||
|
context.header["Content-Type"] = "text/event-stream; charset=utf-8";
|
||||||
|
context.header["Cache-Control"] = "no-cache, no-store, must-revalidate";
|
||||||
|
context.header["X-Accel-Buffering"] = "no";
|
||||||
|
}
|
||||||
|
|
||||||
|
String datastar_event(String event_name, StringMap fields)
|
||||||
|
{
|
||||||
|
String out = "event: " + event_name + "\n";
|
||||||
|
for(const auto& field : fields)
|
||||||
|
out += datastar_data_lines(field.first, field.second);
|
||||||
|
out += "\n";
|
||||||
|
return(out);
|
||||||
|
}
|
||||||
|
|
||||||
|
String datastar_patch_elements(String html)
|
||||||
|
{
|
||||||
|
StringMap fields;
|
||||||
|
fields["elements"] = html;
|
||||||
|
return(datastar_event("datastar-patch-elements", fields));
|
||||||
|
}
|
||||||
|
|
||||||
|
String datastar_patch_signals(String signals_json)
|
||||||
|
{
|
||||||
|
StringMap fields;
|
||||||
|
fields["signals"] = signals_json;
|
||||||
|
return(datastar_event("datastar-patch-signals", fields));
|
||||||
|
}
|
||||||
@@ -0,0 +1,20 @@
|
|||||||
|
#load "../lib/app.uce"
|
||||||
|
#load "../lib/datastar.uce"
|
||||||
|
|
||||||
|
COMPONENT(Request& context)
|
||||||
|
{
|
||||||
|
if(context.get["stream"] == "1")
|
||||||
|
{
|
||||||
|
datastar_sse_headers(context);
|
||||||
|
print(datastar_patch_elements("<div id=\"datastar-example-result\">Server-rendered UCE Datastar fragment</div>"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
<>
|
||||||
|
<h1>Datastar optional example</h1>
|
||||||
|
<p>Datastar is vendored beside the starter client libraries for future server-rendered islands. It is not loaded globally.</p>
|
||||||
|
<script type="module" src="<?= app_asset_url("js/datastar.js", context) ?>"></script>
|
||||||
|
<button class="btn" data-on:click="@get('<?= app_link("datastar-example", context) ?>&stream=1')">Patch a fragment</button>
|
||||||
|
<div id="datastar-example-result">Waiting for a server patch.</div>
|
||||||
|
</>;
|
||||||
|
}
|
||||||
File diff suppressed because one or more lines are too long
@@ -20,6 +20,37 @@ CLI(Request& context)
|
|||||||
print(cli_arg(context, "missing", "fallback"), "\n");
|
print(cli_arg(context, "missing", "fallback"), "\n");
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
if(action == "task_repeat_unbounded")
|
||||||
|
{
|
||||||
|
String marker = "/tmp/uce-task-repeat-unbounded.txt";
|
||||||
|
file_unlink(marker);
|
||||||
|
pid_t pid = task_repeat("uce-task-repeat-unbounded", 0.05, [marker]() { file_put_contents(marker, "ran"); }, 0);
|
||||||
|
usleep(200000);
|
||||||
|
bool ran = file_get_contents(marker) == "ran";
|
||||||
|
if(pid > 0) task_kill(pid, 15);
|
||||||
|
file_unlink(marker);
|
||||||
|
print(pid > 0 && ran ? "task repeat unbounded ok\n" : "task repeat unbounded failed\n");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if(action == "mysql_unsigned_params")
|
||||||
|
{
|
||||||
|
MySQL db;
|
||||||
|
StringMap params; params["limit"] = "1";
|
||||||
|
String parsed = db.parse_query_parameters("SELECT 7 LIMIT :limit!", params);
|
||||||
|
StringMap comparison; comparison["left"] = "7"; comparison["right"] = "8";
|
||||||
|
String inequality = db.parse_query_parameters("SELECT :left!=:right", comparison);
|
||||||
|
StringMap empty_name; empty_name[""] = "1";
|
||||||
|
db.parameter_error = false; db.statement_info = "";
|
||||||
|
db.parse_query_parameters("SELECT 1 LIMIT :!", empty_name);
|
||||||
|
bool empty_rejected = db.parameter_error;
|
||||||
|
StringMap invalid; invalid["limit"] = "1;SELECT 9";
|
||||||
|
db.parameter_error = false;
|
||||||
|
db.statement_info = "";
|
||||||
|
db.parse_query_parameters("SELECT 1 LIMIT :limit!", invalid);
|
||||||
|
bool ok = parsed.find("LIMIT 1") != String::npos && inequality.find("'7'!='8'") != String::npos && empty_rejected && db.parameter_error && db.statement_info.find("must contain only decimal digits") != String::npos;
|
||||||
|
print(ok ? "mysql unsigned params ok\n" : "mysql unsigned params failed: parsed=" + parsed + ", parameter_error=" + (db.parameter_error ? "1" : "0") + ", statement=" + db.statement_info + "\n");
|
||||||
|
return;
|
||||||
|
}
|
||||||
context.set_status(400, "CLI Error");
|
context.set_status(400, "CLI Error");
|
||||||
print("unknown cli action: ", action, "\n");
|
print("unknown cli action: ", action, "\n");
|
||||||
}
|
}
|
||||||
|
|||||||
+11
-3
@@ -31,7 +31,7 @@ RENDER(Request& context)
|
|||||||
StringMap colon_uri_headers = split_http_headers("GET /clock.uce?t=12:30 HTTP/1.1\r\nHost: colon.example\r\n");
|
StringMap colon_uri_headers = split_http_headers("GET /clock.uce?t=12:30 HTTP/1.1\r\nHost: colon.example\r\n");
|
||||||
check("split_http_headers() request line with colon in URI", colon_uri_headers["REQUEST_METHOD"] == "GET" && colon_uri_headers["DOCUMENT_URI"] == "/clock.uce" && colon_uri_headers["QUERY_STRING"] == "t=12:30" && colon_uri_headers["HTTP_HOST"] == "colon.example", var_dump(colon_uri_headers));
|
check("split_http_headers() request line with colon in URI", colon_uri_headers["REQUEST_METHOD"] == "GET" && colon_uri_headers["DOCUMENT_URI"] == "/clock.uce" && colon_uri_headers["QUERY_STRING"] == "t=12:30" && colon_uri_headers["HTTP_HOST"] == "colon.example", var_dump(colon_uri_headers));
|
||||||
check("trim() / split_kv() / split_http_headers()", trim(" padded value ") == "padded value" && kv["alpha"] == "one" && kv["empty"] == "" && http_headers["REQUEST_METHOD"] == "GET" && http_headers["DOCUMENT_URI"] == "/demo.uce" && http_headers["QUERY_STRING"] == "x=1" && http_headers["HTTP_X_EMPTY"] == "" && leading_crlf_headers["REQUEST_METHOD"] == "GET" && leading_crlf_headers["DOCUMENT_URI"] == "/lead.uce" && leading_crlf_headers["HTTP_HOST"] == "lead.example" && header_only["REQUEST_METHOD"] == "" && header_only["HTTP_HOST"] == "example.test" && header_only["HTTP_X_TOKEN"] == "abc", trim(" padded value ") + " / " + var_dump(kv) + " / " + var_dump(http_headers) + " / " + var_dump(leading_crlf_headers) + " / " + var_dump(header_only));
|
check("trim() / split_kv() / split_http_headers()", trim(" padded value ") == "padded value" && kv["alpha"] == "one" && kv["empty"] == "" && http_headers["REQUEST_METHOD"] == "GET" && http_headers["DOCUMENT_URI"] == "/demo.uce" && http_headers["QUERY_STRING"] == "x=1" && http_headers["HTTP_X_EMPTY"] == "" && leading_crlf_headers["REQUEST_METHOD"] == "GET" && leading_crlf_headers["DOCUMENT_URI"] == "/lead.uce" && leading_crlf_headers["HTTP_HOST"] == "lead.example" && header_only["REQUEST_METHOD"] == "" && header_only["HTTP_HOST"] == "example.test" && header_only["HTTP_X_TOKEN"] == "abc", trim(" padded value ") + " / " + var_dump(kv) + " / " + var_dump(http_headers) + " / " + var_dump(leading_crlf_headers) + " / " + var_dump(header_only));
|
||||||
check("replace()", replace("alpha-beta-beta", "beta", "done") == "alpha-done-done", replace("alpha-beta-beta", "beta", "done"));
|
check("replace()", replace("alpha-beta-beta", "beta", "done") == "alpha-done-done" && replace("hello", "", "X") == "hello", replace("alpha-beta-beta", "beta", "done"));
|
||||||
check("html_escape() attribute-safe quotes", html_escape("<&>\"Don't") == "<&>"Don't", html_escape("<&>\"Don't"));
|
check("html_escape() attribute-safe quotes", html_escape("<&>\"Don't") == "<&>"Don't", html_escape("<&>\"Don't"));
|
||||||
check("regex_match()", regex_match("[A-Z][a-z]+", "Alice") && !regex_match("[A-Z][a-z]+", "Alice!"), "full-string validation");
|
check("regex_match()", regex_match("[A-Z][a-z]+", "Alice") && !regex_match("[A-Z][a-z]+", "Alice!"), "full-string validation");
|
||||||
|
|
||||||
@@ -212,11 +212,19 @@ RENDER(Request& context)
|
|||||||
String decoded_base64 = base64_decode(encoded_base64, base64_ok);
|
String decoded_base64 = base64_decode(encoded_base64, base64_ok);
|
||||||
bool invalid_base64_ok = true;
|
bool invalid_base64_ok = true;
|
||||||
base64_decode("AA=A", invalid_base64_ok);
|
base64_decode("AA=A", invalid_base64_ok);
|
||||||
check("base64_encode() / base64_decode() binary-safe", base64_ok && decoded_base64 == binary_payload && decoded_base64.size() == binary_payload.size() && !invalid_base64_ok, encoded_base64 + " bytes=" + std::to_string((u64)decoded_base64.size()));
|
bool early_padding_base64_ok = true;
|
||||||
|
base64_decode("A=AA", early_padding_base64_ok);
|
||||||
|
check("base64_encode() / base64_decode() binary-safe", base64_ok && decoded_base64 == binary_payload && decoded_base64.size() == binary_payload.size() && !invalid_base64_ok && !early_padding_base64_ok, encoded_base64 + " bytes=" + std::to_string((u64)decoded_base64.size()));
|
||||||
|
|
||||||
String utf8_sample = "A\xC3\xA9";
|
String utf8_sample = "A\xC3\xA9";
|
||||||
auto utf8_parts = split_utf8(utf8_sample);
|
auto utf8_parts = split_utf8(utf8_sample);
|
||||||
check("split_utf8()", utf8_parts.size() == 2, "count=" + std::to_string(utf8_parts.size()));
|
String truncated_utf8 = "abc";
|
||||||
|
truncated_utf8.push_back((char)0xE2);
|
||||||
|
truncated_utf8.push_back((char)0x82);
|
||||||
|
auto truncated_utf8_parts = split_utf8(truncated_utf8);
|
||||||
|
String lone_utf8_lead(1, (char)0xF0);
|
||||||
|
auto compound_lone_utf8_parts = split_utf8(lone_utf8_lead, true);
|
||||||
|
check("split_utf8()", utf8_parts.size() == 2 && truncated_utf8_parts.size() == 4 && truncated_utf8_parts[3].size() == 2 && compound_lone_utf8_parts.size() == 1 && compound_lone_utf8_parts[0] == lone_utf8_lead, "count=" + std::to_string(utf8_parts.size()) + " truncated=" + std::to_string(truncated_utf8_parts.size()));
|
||||||
|
|
||||||
DValue payload;
|
DValue payload;
|
||||||
payload["name"] = "uce";
|
payload["name"] = "uce";
|
||||||
|
|||||||
@@ -0,0 +1,75 @@
|
|||||||
|
#include "testlib.h"
|
||||||
|
|
||||||
|
RENDER(Request& context)
|
||||||
|
{
|
||||||
|
u64 passed = 0;
|
||||||
|
u64 failed = 0;
|
||||||
|
u64 skipped = 0;
|
||||||
|
auto check = [&](String name, bool ok, String detail)
|
||||||
|
{
|
||||||
|
site_tests_case(name, ok ? "pass" : "fail", detail);
|
||||||
|
if(ok) passed++; else failed++;
|
||||||
|
};
|
||||||
|
site_tests_page_start("Structured crypto operations", "Algorithm-selected P-256 JWK creation and ES256 JWT signing.");
|
||||||
|
DValue key_request;
|
||||||
|
key_request["operation"] = "key_generate";
|
||||||
|
key_request["algorithm"] = "ES256";
|
||||||
|
DValue key = crypto_operation(key_request);
|
||||||
|
DValue header;
|
||||||
|
header["alg"] = "none";
|
||||||
|
header["kid"] = key["kid"];
|
||||||
|
DValue claims;
|
||||||
|
claims["iss"] = "https://client.example";
|
||||||
|
claims["sub"] = "client.example";
|
||||||
|
DValue sign_request;
|
||||||
|
sign_request["operation"] = "jwt_sign";
|
||||||
|
sign_request["algorithm"] = "ES256";
|
||||||
|
sign_request["private_jwk"] = key["private_jwk"];
|
||||||
|
sign_request["protected_header"] = header;
|
||||||
|
sign_request["claims"] = claims;
|
||||||
|
DValue signed_result = crypto_operation(sign_request);
|
||||||
|
String jwt = signed_result["jwt"].to_string();
|
||||||
|
StringList parts = split(jwt, ".");
|
||||||
|
String header_b64 = parts.size() == 3 ? parts[0] : "";
|
||||||
|
while(header_b64.size() % 4) header_b64 += "=";
|
||||||
|
String decoded_header = base64_decode(header_b64);
|
||||||
|
DValue wrong_curve = key["private_jwk"];
|
||||||
|
wrong_curve["crv"] = "P-384";
|
||||||
|
DValue malformed = key["private_jwk"];
|
||||||
|
malformed["x"] = "not_base64url=";
|
||||||
|
DValue wrong_request = sign_request;
|
||||||
|
wrong_request["private_jwk"] = wrong_curve;
|
||||||
|
DValue malformed_request = sign_request;
|
||||||
|
malformed_request["private_jwk"] = malformed;
|
||||||
|
DValue cose_request;
|
||||||
|
cose_request["operation"] = "cose_es256_parse";
|
||||||
|
cose_request["algorithm"] = "ES256";
|
||||||
|
cose_request["cose_key_base64url"] = "pQECAyYgASFYIGsX0fLhLEJH-Lzm5WOkQPJ3A32BLeszoPShOUXYmMKWIlggT-NC4v4af5uO5-tKfA-eFivOM1drMV7Oy7ZAaDe_UfU";
|
||||||
|
DValue cose = crypto_operation(cose_request);
|
||||||
|
DValue verify_request;
|
||||||
|
verify_request["operation"] = "es256_verify";
|
||||||
|
verify_request["algorithm"] = "ES256";
|
||||||
|
verify_request["cose_key_base64url"] = cose_request["cose_key_base64url"];
|
||||||
|
verify_request["message_base64url"] = "d2ViYXV0aG4gZml4ZWQgbWVzc2FnZQ";
|
||||||
|
verify_request["signature_der_base64url"] = "MEUCIQCkatZK1VVsjk17uvyzyhjdAkMNWXPjxSOMqWcjmM_8XAIgDaSk3Qufyd0_6r9Dm9A8RQbFco-FdTBulq7bvRGoBC4";
|
||||||
|
DValue unsupported;
|
||||||
|
unsupported["operation"] = "encrypt";
|
||||||
|
unsupported["algorithm"] = "ES256";
|
||||||
|
DValue list_header;
|
||||||
|
list_header.set_array();
|
||||||
|
DValue list_item;
|
||||||
|
list_item = "not-an-object";
|
||||||
|
list_header.push(list_item);
|
||||||
|
DValue list_request = sign_request;
|
||||||
|
list_request["protected_header"] = list_header;
|
||||||
|
DValue control_request = sign_request;
|
||||||
|
control_request["claims"]["bad"] = String("control\nbyte");
|
||||||
|
check("crypto_operation() generates an ES256 key", key["ok"].to_bool() && key["private_jwk"]["kty"].to_string() == "EC" && key["private_jwk"]["crv"].to_string() == "P-256" && key["public_jwk"]["d"].to_string() == "" && key["kid"].to_string() == key["thumbprint"].to_string(), key["kid"].to_string());
|
||||||
|
check("crypto_operation() signs ES256 JWTs", signed_result["ok"].to_bool() && parts.size() == 3 && parts[2].size() == 86 && decoded_header.find("ES256") != String::npos, signed_result["error"].to_string());
|
||||||
|
check("crypto_operation() rejects invalid signing requests", !crypto_operation(wrong_request)["ok"].to_bool() && !crypto_operation(malformed_request)["ok"].to_bool() && !crypto_operation(list_request)["ok"].to_bool() && crypto_operation(control_request)["error"].to_string() == "invalid_request", "signing validation");
|
||||||
|
check("crypto_operation() parses an ES256 COSE key", cose["ok"].to_bool(), cose["error"].to_string());
|
||||||
|
check("crypto_operation() verifies ES256 DER signatures", crypto_operation(verify_request)["ok"].to_bool() && crypto_operation(verify_request)["valid"].to_bool(), crypto_operation(verify_request)["error"].to_string());
|
||||||
|
check("crypto_operation() rejects unsupported operations", crypto_operation(unsupported)["error"].to_string() == "unsupported_operation", crypto_operation(unsupported)["error"].to_string());
|
||||||
|
site_tests_summary(passed, failed, skipped, "Structured crypto tests generate ephemeral P-256 keys and retain no key material.");
|
||||||
|
site_tests_page_end();
|
||||||
|
}
|
||||||
@@ -8,6 +8,7 @@ markdown.uce|Markdown|Markdown parsing, rendering, and component hooks.|http sui
|
|||||||
units.uce|Units|unit_call(), lifecycle hooks, and unit metadata.|http suite uce public|Units|1|1
|
units.uce|Units|unit_call(), lifecycle hooks, and unit metadata.|http suite uce public|Units|1|1
|
||||||
websockets.ws.uce|WebSockets|Browser-driven WebSocket helper checks.|http suite uce public websocket|WebSockets|1|1
|
websockets.ws.uce|WebSockets|Browser-driven WebSocket helper checks.|http suite uce public websocket|WebSockets|1|1
|
||||||
io.uce|Filesystem|Filesystem helpers that are restricted outside trusted networks.|http suite uce internal|Filesystem|1|1
|
io.uce|Filesystem|Filesystem helpers that are restricted outside trusted networks.|http suite uce internal|Filesystem|1|1
|
||||||
|
crypto_operation.uce|Structured Crypto|Algorithm-selected key generation and JWT signing coverage.|http suite uce public crypto|Structured crypto operations|1|1
|
||||||
sqlite.uce|SQLite|SQLite connector with prepared named parameters and DValue rows.|http suite uce internal sqlite|SQLite|1|1
|
sqlite.uce|SQLite|SQLite connector with prepared named parameters and DValue rows.|http suite uce internal sqlite|SQLite|1|1
|
||||||
zip.uce|ZIP|Archive helpers that create and extract temporary server-side files.|http suite uce internal|ZIP|1|1
|
zip.uce|ZIP|Archive helpers that create and extract temporary server-side files.|http suite uce internal|ZIP|1|1
|
||||||
services.uce|Sockets And Services|Network/service helpers that are restricted outside trusted networks.|http suite uce internal|Sockets And Services|1|1
|
services.uce|Sockets And Services|Network/service helpers that are restricted outside trusted networks.|http suite uce internal|Sockets And Services|1|1
|
||||||
|
|||||||
@@ -31,6 +31,8 @@ RENDER(Request& context)
|
|||||||
};
|
};
|
||||||
|
|
||||||
String nested = preprocessor_nested_literal();
|
String nested = preprocessor_nested_literal();
|
||||||
|
String trailing_backslash = "C:\\Users\\";
|
||||||
|
String escaped_quote = "before\"after";
|
||||||
|
|
||||||
site_tests_page_start("Preprocessor", "Regression coverage for literal output rewriting and parser edge cases.");
|
site_tests_page_start("Preprocessor", "Regression coverage for literal output rewriting and parser edge cases.");
|
||||||
?>
|
?>
|
||||||
@@ -43,6 +45,8 @@ RENDER(Request& context)
|
|||||||
check("raw string terminator in nested literal", contains(nested, "nested )\" marker"), nested);
|
check("raw string terminator in nested literal", contains(nested, "nested )\" marker"), nested);
|
||||||
check("entrypoint @fragment attribute captures output", context.call["fragments"]["preprocessor-test"].to_string() == "fragment attr once", context.call["fragments"]["preprocessor-test"].to_string());
|
check("entrypoint @fragment attribute captures output", context.call["fragments"]["preprocessor-test"].to_string() == "fragment attr once", context.call["fragments"]["preprocessor-test"].to_string());
|
||||||
check("inline code island after dangerous literal", true, "parser returned to C++ after rendering literal content containing )\"");
|
check("inline code island after dangerous literal", true, "parser returned to C++ after rendering literal content containing )\"");
|
||||||
|
check("quote scanner handles trailing escaped backslash", trailing_backslash == "C:\\Users\\", trailing_backslash);
|
||||||
|
check("quote scanner retains escaped quote content", escaped_quote == "before\"after", escaped_quote);
|
||||||
|
|
||||||
site_tests_summary(passed, failed, skipped, "Literal content containing the C++ raw-string terminator sequence must compile and render unchanged.");
|
site_tests_summary(passed, failed, skipped, "Literal content containing the C++ raw-string terminator sequence must compile and render unchanged.");
|
||||||
site_tests_page_end();
|
site_tests_page_end();
|
||||||
|
|||||||
+43
-5
@@ -28,14 +28,20 @@ RENDER(Request& context)
|
|||||||
u64 sockfd = socket_connect("127.0.0.1", 80);
|
u64 sockfd = socket_connect("127.0.0.1", 80);
|
||||||
if(sockfd != 0)
|
if(sockfd != 0)
|
||||||
{
|
{
|
||||||
|
u64 closed_handle = socket_connect("127.0.0.1", 80);
|
||||||
|
bool opaque_handles = sockfd == 1 && closed_handle == 2;
|
||||||
|
socket_close(closed_handle);
|
||||||
|
socket_close(closed_handle);
|
||||||
|
socket_close(999999);
|
||||||
|
bool closed_write_rejected = !socket_write(closed_handle, "invalid");
|
||||||
bool write_ok = socket_write(sockfd, "GET /tests/index.uce HTTP/1.0\r\nHost: uce.openfu.com\r\n\r\n");
|
bool write_ok = socket_write(sockfd, "GET /tests/index.uce HTTP/1.0\r\nHost: uce.openfu.com\r\n\r\n");
|
||||||
String response = socket_read(sockfd, 4096, 2);
|
String response = socket_read(sockfd, 4096, 2);
|
||||||
socket_close(sockfd);
|
socket_close(sockfd);
|
||||||
bool has_nul = response.find(String("\0", 1)) != String::npos;
|
bool has_nul = response.find(String("\0", 1)) != String::npos;
|
||||||
mark(
|
mark(
|
||||||
"socket_connect() / socket_write() / socket_read()",
|
"socket handles reject invalid and closed indices",
|
||||||
(write_ok && response.find("200 OK") != String::npos && !has_nul) ? "pass" : "fail",
|
(opaque_handles && closed_write_rejected && write_ok && response.find("200 OK") != String::npos && !has_nul) ? "pass" : "fail",
|
||||||
response.substr(0, response.length() > 220 ? 220 : response.length()) + (has_nul ? " [unexpected NUL]" : "")
|
"handles=" + std::to_string(sockfd) + "/" + std::to_string(closed_handle) + " " + response.substr(0, response.length() > 180 ? 180 : response.length()) + (has_nul ? " [unexpected NUL]" : "")
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
else
|
else
|
||||||
@@ -90,10 +96,25 @@ RENDER(Request& context)
|
|||||||
StringMap mem_multi = memcache_get_multiple(memfd, {"site-tests-key", "site-tests-key2"});
|
StringMap mem_multi = memcache_get_multiple(memfd, {"site-tests-key", "site-tests-key2"});
|
||||||
memcache_delete(memfd, "site-tests-key");
|
memcache_delete(memfd, "site-tests-key");
|
||||||
memcache_delete(memfd, "site-tests-key2");
|
memcache_delete(memfd, "site-tests-key2");
|
||||||
|
String large_mem_value(200 * 1024, 'x');
|
||||||
|
large_mem_value.replace(1024, 7, "\r\nEND\r\n");
|
||||||
|
bool large_mem_stored = memcache_set(memfd, "site-tests-large", large_mem_value);
|
||||||
|
String large_mem_loaded = memcache_get(memfd, "site-tests-large");
|
||||||
|
String second_large_mem_value(160 * 1024, 'y');
|
||||||
|
bool second_large_mem_stored = memcache_set(memfd, "site-tests-large-2", second_large_mem_value);
|
||||||
|
StringMap large_mem_multi = memcache_get_multiple(memfd, {"site-tests-large", "site-tests-large-2"});
|
||||||
|
String no_reply = memcache_command(memfd, "set site-tests-noreply 0 60 5 noreply\r\nvalue");
|
||||||
|
String no_reply_value = memcache_get(memfd, "site-tests-noreply");
|
||||||
|
String stats_mem = memcache_command(memfd, "stats");
|
||||||
|
String stats_reset_mem = memcache_command(memfd, "stats reset");
|
||||||
|
String after_stats_reset = memcache_get(memfd, "site-tests-noreply");
|
||||||
|
bool large_mem_deleted = memcache_delete(memfd, "site-tests-large");
|
||||||
|
bool second_large_mem_deleted = memcache_delete(memfd, "site-tests-large-2");
|
||||||
|
bool no_reply_deleted = memcache_delete(memfd, "site-tests-noreply");
|
||||||
mark(
|
mark(
|
||||||
"memcache_connect() / memcache_command() / memcache_get_multiple()",
|
"memcache_connect() / memcache_command() / memcache_get_multiple()",
|
||||||
mem_value == "value-1" && contains(raw_mem, "VALUE site-tests-key") && mem_multi["site-tests-key"] == "value-1" && mem_multi["site-tests-key2"] == "value-2" ? "pass" : "fail",
|
mem_value == "value-1" && contains(raw_mem, "VALUE site-tests-key") && mem_multi["site-tests-key"] == "value-1" && mem_multi["site-tests-key2"] == "value-2" && large_mem_stored && large_mem_loaded == large_mem_value && second_large_mem_stored && large_mem_multi["site-tests-large"] == large_mem_value && large_mem_multi["site-tests-large-2"] == second_large_mem_value && no_reply == "" && no_reply_value == "value" && contains(stats_mem, "STAT ") && trim(stats_reset_mem) == "RESET" && after_stats_reset == "value" && large_mem_deleted && second_large_mem_deleted && no_reply_deleted ? "pass" : "fail",
|
||||||
"memcache value=" + mem_value + " raw=" + raw_mem + " multi=" + var_dump(mem_multi)
|
"memcache value=" + mem_value + " raw=" + raw_mem + " multi=" + var_dump(mem_multi) + " large=" + std::to_string(large_mem_loaded.size()) + " multi-large=" + std::to_string(large_mem_multi["site-tests-large"].size() + large_mem_multi["site-tests-large-2"].size()) + " noreply=" + no_reply_value + " stats=" + std::to_string(stats_mem.size()) + "/" + trim(stats_reset_mem) + " after-reset=" + after_stats_reset + " deleted=" + (large_mem_deleted && second_large_mem_deleted && no_reply_deleted ? "true" : "false")
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -115,6 +136,15 @@ RENDER(Request& context)
|
|||||||
parsed_underscore.find("'Ada'") != String::npos && parsed_underscore.find("_name") == String::npos ? "pass" : "fail",
|
parsed_underscore.find("'Ada'") != String::npos && parsed_underscore.find("_name") == String::npos ? "pass" : "fail",
|
||||||
parsed_underscore
|
parsed_underscore
|
||||||
);
|
);
|
||||||
|
StringMap quoted_params;
|
||||||
|
quoted_params["name_looking_text"] = "SUBSTITUTED";
|
||||||
|
String escaped_quote_query = "SELECT 'a\\':name_looking_text'";
|
||||||
|
String parsed_escaped_quote = placeholder_guard.parse_query_parameters(escaped_quote_query, quoted_params);
|
||||||
|
mark(
|
||||||
|
"mysql named placeholders stay opaque after escaped quotes",
|
||||||
|
parsed_escaped_quote.find(":name_looking_text") != String::npos && parsed_escaped_quote.find("SUBSTITUTED") == String::npos ? "pass" : "fail",
|
||||||
|
parsed_escaped_quote
|
||||||
|
);
|
||||||
MySQL unavailable_mysql;
|
MySQL unavailable_mysql;
|
||||||
unavailable_mysql.connect("127.0.0.1", "__uce_intentionally_missing__", "not-a-real-password");
|
unavailable_mysql.connect("127.0.0.1", "__uce_intentionally_missing__", "not-a-real-password");
|
||||||
mark(
|
mark(
|
||||||
@@ -158,6 +188,14 @@ RENDER(Request& context)
|
|||||||
inserted == 2 && updated == 1 && selected == 0 ? "pass" : "fail",
|
inserted == 2 && updated == 1 && selected == 0 ? "pass" : "fail",
|
||||||
"inserted=" + std::to_string((u64)inserted) + " updated=" + std::to_string((u64)updated) + " selected=" + std::to_string((u64)selected)
|
"inserted=" + std::to_string((u64)inserted) + " updated=" + std::to_string((u64)updated) + " selected=" + std::to_string((u64)selected)
|
||||||
);
|
);
|
||||||
|
mysql.query("INSERT INTO uce_affected_rows_test VALUES (1,30)");
|
||||||
|
u32 duplicate_error_code = mysql._preload_next_error_code;
|
||||||
|
String duplicate_error = mysql.error();
|
||||||
|
mark(
|
||||||
|
"mysql query failures retain server diagnostics",
|
||||||
|
duplicate_error_code > 1 && duplicate_error != "" && duplicate_error != "Unknown server error" ? "pass" : "fail",
|
||||||
|
"code=" + std::to_string((u64)duplicate_error_code) + " error=" + duplicate_error
|
||||||
|
);
|
||||||
}
|
}
|
||||||
DValue mysql_perf = request_perf();
|
DValue mysql_perf = request_perf();
|
||||||
String mysql_operations = json_encode(mysql_perf["mysql_operations"]);
|
String mysql_operations = json_encode(mysql_perf["mysql_operations"]);
|
||||||
|
|||||||
@@ -18,6 +18,16 @@ RENDER(Request& context)
|
|||||||
auto unit_paths = units_list();
|
auto unit_paths = units_list();
|
||||||
DValue info = unit_info("call_helpers.uce");
|
DValue info = unit_info("call_helpers.uce");
|
||||||
DValue relative_info = unit_info("components/../relative-child.uce");
|
DValue relative_info = unit_info("components/../relative-child.uce");
|
||||||
|
String outside_unit = "/tmp/uce-site-tests-outside-unit.uce";
|
||||||
|
file_put_contents(outside_unit, "RENDER(Request& context) {}");
|
||||||
|
DValue outside_info = unit_info(outside_unit);
|
||||||
|
file_unlink(outside_unit);
|
||||||
|
String generated_dir = path_join(context.params["UCE_BIN_DIRECTORY"], "site-tests-generated");
|
||||||
|
mkdir(generated_dir);
|
||||||
|
String generated_unit = path_join(generated_dir, "generated.uce");
|
||||||
|
file_put_contents(generated_unit, "RENDER(Request& context) {}");
|
||||||
|
DValue generated_info = unit_info(generated_unit);
|
||||||
|
file_unlink(generated_unit);
|
||||||
|
|
||||||
ob_start();
|
ob_start();
|
||||||
unit_call("call_helpers.uce", "emit_marker");
|
unit_call("call_helpers.uce", "emit_marker");
|
||||||
@@ -32,6 +42,8 @@ RENDER(Request& context)
|
|||||||
check("units_list()", unit_paths.size() > 0, "count=" + std::to_string(unit_paths.size()));
|
check("units_list()", unit_paths.size() > 0, "count=" + std::to_string(unit_paths.size()));
|
||||||
check("unit_info()", info["path"].to_string() != "", json_encode(info));
|
check("unit_info()", info["path"].to_string() != "", json_encode(info));
|
||||||
check("compiler canonicalizes relative unit paths", relative_info["path"].to_string().find("/../") == String::npos && str_ends_with(relative_info["path"].to_string(), "/site/tests/relative-child.uce"), json_encode(relative_info));
|
check("compiler canonicalizes relative unit paths", relative_info["path"].to_string().find("/../") == String::npos && str_ends_with(relative_info["path"].to_string(), "/site/tests/relative-child.uce"), json_encode(relative_info));
|
||||||
|
check("compiler rejects units outside document root", outside_info["path"].to_string() == "", json_encode(outside_info));
|
||||||
|
check("compiler accepts generated units in BIN_DIRECTORY", generated_info["path"].to_string() == generated_unit, json_encode(generated_info));
|
||||||
check("unit_compile()", unit_compile("call_helpers.uce"), "call_helpers.uce");
|
check("unit_compile()", unit_compile("call_helpers.uce"), "call_helpers.uce");
|
||||||
check("unit_call()", call_output.find("UNIT_CALL_EXPORT_OK") != String::npos, call_output);
|
check("unit_call()", call_output.find("UNIT_CALL_EXPORT_OK") != String::npos, call_output);
|
||||||
check("unit_render()", render_output.find("data-unit-render=\"ok\"") != String::npos, render_output);
|
check("unit_render()", render_output.find("data-unit-render=\"ok\"") != String::npos, render_output);
|
||||||
|
|||||||
@@ -362,6 +362,7 @@ FastCGIServer::listen(const std::string& local_path)
|
|||||||
close(server_socket);
|
close(server_socket);
|
||||||
throw;
|
throw;
|
||||||
}
|
}
|
||||||
|
server_socket_types[server_socket] = 'F';
|
||||||
return server_socket;
|
return server_socket;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -400,6 +401,23 @@ FastCGIServer::close_http_listeners()
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
void
|
||||||
|
FastCGIServer::close_listeners_except(char type)
|
||||||
|
{
|
||||||
|
for(std::vector<int>::iterator it = server_sockets.begin(); it != server_sockets.end();)
|
||||||
|
{
|
||||||
|
int socket_handle = *it;
|
||||||
|
if(server_socket_types[socket_handle] != type)
|
||||||
|
{
|
||||||
|
close(socket_handle);
|
||||||
|
server_socket_types.erase(socket_handle);
|
||||||
|
it = server_sockets.erase(it);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
++it;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
bool
|
bool
|
||||||
FastCGIServer::is_http_like_type(char type)
|
FastCGIServer::is_http_like_type(char type)
|
||||||
{
|
{
|
||||||
@@ -672,10 +690,12 @@ FastCGIServer::process(int timeout_ms)
|
|||||||
Connection* doomed_connection = it->second;
|
Connection* doomed_connection = it->second;
|
||||||
client_sockets.erase(it++);
|
client_sockets.erase(it++);
|
||||||
delete doomed_connection;
|
delete doomed_connection;
|
||||||
if(calls_until_termination != -1 && client_sockets.size() == 0)
|
if(calls_until_termination > 0)
|
||||||
{
|
|
||||||
calls_until_termination -= 1;
|
calls_until_termination -= 1;
|
||||||
if(calls_until_termination <= 0)
|
if(calls_until_termination == 0)
|
||||||
|
{
|
||||||
|
close_listeners_except(0);
|
||||||
|
if(client_sockets.empty())
|
||||||
exit(0);
|
exit(0);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -96,6 +96,7 @@ public:
|
|||||||
std::map<int, Connection*> client_sockets;
|
std::map<int, Connection*> client_sockets;
|
||||||
|
|
||||||
void close_http_listeners();
|
void close_http_listeners();
|
||||||
|
void close_listeners_except(char type);
|
||||||
void read_fgci(Connection&);
|
void read_fgci(Connection&);
|
||||||
static bool is_http_like_type(char type);
|
static bool is_http_like_type(char type);
|
||||||
Connection* open_client_connection(int server_socket, int client_socket);
|
Connection* open_client_connection(int server_socket, int client_socket);
|
||||||
|
|||||||
@@ -74,8 +74,14 @@ void compiler_code_state_consume(CompilerCodeState& state, String& buffer, const
|
|||||||
|
|
||||||
if(state.inside_quote)
|
if(state.inside_quote)
|
||||||
{
|
{
|
||||||
if(state.quote_char == c && (i == 0 || content[i-1] != '\\'))
|
if(state.quote_char == c)
|
||||||
state.inside_quote = false;
|
{
|
||||||
|
u32 backslashes = 0;
|
||||||
|
for(u32 j = i; j > 0 && content[j - 1] == '\\'; j -= 1)
|
||||||
|
backslashes += 1;
|
||||||
|
if(backslashes % 2 == 0)
|
||||||
|
state.inside_quote = false;
|
||||||
|
}
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
+39
-6
@@ -646,7 +646,10 @@ int compiler_open_lock_file(String file_name, String purpose, bool nonblocking =
|
|||||||
(void)purpose;
|
(void)purpose;
|
||||||
auto lock_dir = dirname(file_name);
|
auto lock_dir = dirname(file_name);
|
||||||
if(lock_dir != "")
|
if(lock_dir != "")
|
||||||
mkdir(lock_dir);
|
{
|
||||||
|
std::error_code error;
|
||||||
|
std::filesystem::create_directories(lock_dir, error);
|
||||||
|
}
|
||||||
int fdlock = open(file_name.c_str(), O_RDWR | O_CREAT, 0666);
|
int fdlock = open(file_name.c_str(), O_RDWR | O_CREAT, 0666);
|
||||||
if(fdlock == -1 && (errno == EACCES || errno == EPERM))
|
if(fdlock == -1 && (errno == EACCES || errno == EPERM))
|
||||||
fdlock = open(file_name.c_str(), O_RDONLY | O_CLOEXEC);
|
fdlock = open(file_name.c_str(), O_RDONLY | O_CLOEXEC);
|
||||||
@@ -676,7 +679,10 @@ int compiler_open_lock_file_bounded(String file_name, String purpose, CompilerDe
|
|||||||
return(compiler_open_lock_file(file_name, purpose));
|
return(compiler_open_lock_file(file_name, purpose));
|
||||||
auto lock_dir = dirname(file_name);
|
auto lock_dir = dirname(file_name);
|
||||||
if(lock_dir != "")
|
if(lock_dir != "")
|
||||||
mkdir(lock_dir);
|
{
|
||||||
|
std::error_code error;
|
||||||
|
std::filesystem::create_directories(lock_dir, error);
|
||||||
|
}
|
||||||
int fdlock = open(file_name.c_str(), O_RDWR | O_CREAT, 0666);
|
int fdlock = open(file_name.c_str(), O_RDWR | O_CREAT, 0666);
|
||||||
if(fdlock == -1 && (errno == EACCES || errno == EPERM))
|
if(fdlock == -1 && (errno == EACCES || errno == EPERM))
|
||||||
fdlock = open(file_name.c_str(), O_RDONLY | O_CLOEXEC);
|
fdlock = open(file_name.c_str(), O_RDONLY | O_CLOEXEC);
|
||||||
@@ -734,14 +740,41 @@ static void compiler_mark_source_generation_nonblocking(Request* context)
|
|||||||
String compiler_normalize_unit_path(Request* context, String file_name)
|
String compiler_normalize_unit_path(Request* context, String file_name)
|
||||||
{
|
{
|
||||||
file_name = trim(file_name);
|
file_name = trim(file_name);
|
||||||
if(file_name == "")
|
if(file_name == "" || !context || !context->server)
|
||||||
return("");
|
return("");
|
||||||
if(file_name[0] != '/')
|
if(file_name[0] != '/')
|
||||||
file_name = expand_path(file_name, context->server->config["COMPILER_SYS_PATH"]);
|
file_name = expand_path(file_name, context->server->config["COMPILER_SYS_PATH"]);
|
||||||
String canonical = path_real(file_name);
|
String canonical = path_real(file_name);
|
||||||
if(canonical != "")
|
if(canonical == "")
|
||||||
return(canonical);
|
{
|
||||||
return(file_name);
|
std::error_code error;
|
||||||
|
canonical = std::filesystem::weakly_canonical(file_name, error).string();
|
||||||
|
if(error || canonical == "")
|
||||||
|
return("");
|
||||||
|
}
|
||||||
|
StringList allowed_roots = {
|
||||||
|
context->params["DOCUMENT_ROOT"],
|
||||||
|
context->server->config["PRECOMPILE_FILES_IN"],
|
||||||
|
context->server->config["HTTP_DOCUMENT_ROOT"],
|
||||||
|
path_join(context->server->config["COMPILER_SYS_PATH"], context->server->config["SITE_DIRECTORY"]),
|
||||||
|
context->server->config["BIN_DIRECTORY"]
|
||||||
|
};
|
||||||
|
for(String allowed_root : allowed_roots)
|
||||||
|
{
|
||||||
|
allowed_root = trim(allowed_root);
|
||||||
|
if(allowed_root == "")
|
||||||
|
continue;
|
||||||
|
if(allowed_root[0] != '/')
|
||||||
|
allowed_root = expand_path(allowed_root, context->server->config["COMPILER_SYS_PATH"]);
|
||||||
|
allowed_root = path_real(allowed_root);
|
||||||
|
if(allowed_root == "")
|
||||||
|
continue;
|
||||||
|
if(allowed_root[allowed_root.length() - 1] != '/')
|
||||||
|
allowed_root += "/";
|
||||||
|
if(canonical + "/" == allowed_root || str_starts_with(canonical, allowed_root))
|
||||||
|
return(canonical);
|
||||||
|
}
|
||||||
|
return("");
|
||||||
}
|
}
|
||||||
|
|
||||||
bool compiler_is_known_unit_file(String file_name)
|
bool compiler_is_known_unit_file(String file_name)
|
||||||
|
|||||||
@@ -140,6 +140,8 @@ bool contains(String haystack, String needle)
|
|||||||
|
|
||||||
String replace(String s, String search, String replace_with)
|
String replace(String s, String search, String replace_with)
|
||||||
{
|
{
|
||||||
|
if(search == "")
|
||||||
|
return(s);
|
||||||
s64 last_spos = 0;
|
s64 last_spos = 0;
|
||||||
auto spos = s.find(search);
|
auto spos = s.find(search);
|
||||||
if(spos == std::string::npos)
|
if(spos == std::string::npos)
|
||||||
@@ -791,7 +793,7 @@ DValue array_merge(DValue a, DValue b)
|
|||||||
StringList split_utf8(String s, bool compound_characters)
|
StringList split_utf8(String s, bool compound_characters)
|
||||||
{
|
{
|
||||||
StringList result;
|
StringList result;
|
||||||
auto len = s.size();
|
s64 len = (s64)s.size();
|
||||||
String codepoint = "";
|
String codepoint = "";
|
||||||
for(s64 i = 0; i < len; i++)
|
for(s64 i = 0; i < len; i++)
|
||||||
{
|
{
|
||||||
@@ -800,13 +802,13 @@ StringList split_utf8(String s, bool compound_characters)
|
|||||||
{
|
{
|
||||||
codepoint = "";
|
codepoint = "";
|
||||||
codepoint.append(1, c);
|
codepoint.append(1, c);
|
||||||
if(is_bit_set(c, 6))
|
if(is_bit_set(c, 6) && i + 1 < len)
|
||||||
{
|
{
|
||||||
codepoint.append(1, s[++i]);
|
codepoint.append(1, s[++i]);
|
||||||
if(is_bit_set(c, 5))
|
if(is_bit_set(c, 5) && i + 1 < len)
|
||||||
{
|
{
|
||||||
codepoint.append(1, s[++i]);
|
codepoint.append(1, s[++i]);
|
||||||
if(is_bit_set(c, 4))
|
if(is_bit_set(c, 4) && i + 1 < len)
|
||||||
{
|
{
|
||||||
codepoint.append(1, s[++i]);
|
codepoint.append(1, s[++i]);
|
||||||
}
|
}
|
||||||
@@ -837,7 +839,7 @@ StringList split_utf8(String s, bool compound_characters)
|
|||||||
join_next = true;
|
join_next = true;
|
||||||
last_was_regional = false;
|
last_was_regional = false;
|
||||||
}
|
}
|
||||||
else if(s[0] == '\xF0' && s[1] == '\x9F' && s[2] == '\x87' && s[3] >= '\xA6' && s[3] <= '\xBF') // Regional indicator letters
|
else if(s.size() == 4 && s[0] == '\xF0' && s[1] == '\x9F' && s[2] == '\x87' && s[3] >= '\xA6' && s[3] <= '\xBF') // Regional indicator letters
|
||||||
{
|
{
|
||||||
if(last_was_regional)
|
if(last_was_regional)
|
||||||
{
|
{
|
||||||
@@ -850,7 +852,7 @@ StringList split_utf8(String s, bool compound_characters)
|
|||||||
last_was_regional = true;
|
last_was_regional = true;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
else if(s[0] == '\xEF' && s[1] == '\xB8' && s[2] >= '\x80' && s[2] <= '\x8F') // Variation selector
|
else if(s.size() == 3 && s[0] == '\xEF' && s[1] == '\xB8' && s[2] >= '\x80' && s[2] <= '\x8F') // Variation selector
|
||||||
{
|
{
|
||||||
compound_result[compound_result.size()-1] += s;
|
compound_result[compound_result.size()-1] += s;
|
||||||
last_was_regional = false;
|
last_was_regional = false;
|
||||||
|
|||||||
@@ -41,6 +41,7 @@ void SHA1Final(unsigned char digest[20], SHA1_CTX* context);
|
|||||||
#include <string.h>
|
#include <string.h>
|
||||||
#include <sys/types.h> /* for u_int*_t */
|
#include <sys/types.h> /* for u_int*_t */
|
||||||
#include "hash.h"
|
#include "hash.h"
|
||||||
|
#include "uri.h"
|
||||||
|
|
||||||
#ifndef BYTE_ORDER
|
#ifndef BYTE_ORDER
|
||||||
#if (BSD >= 199103)
|
#if (BSD >= 199103)
|
||||||
@@ -422,8 +423,69 @@ bool crypto_equal_native(String a, String b)
|
|||||||
return(diff == 0);
|
return(diff == 0);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
static bool uce_crypto_utf8_string(String value)
|
||||||
|
{
|
||||||
|
for(size_t i = 0; i < value.size();)
|
||||||
|
{
|
||||||
|
u8 c = (u8)value[i];
|
||||||
|
if(c < 0x80) { i++; continue; }
|
||||||
|
size_t need = c >= 0xC2 && c <= 0xDF ? 1 : (c >= 0xE0 && c <= 0xEF ? 2 : (c >= 0xF0 && c <= 0xF4 ? 3 : 0));
|
||||||
|
if(need == 0 || i + need >= value.size()) return(false);
|
||||||
|
for(size_t j = 1; j <= need; j++) if(((u8)value[i + j] & 0xC0) != 0x80) return(false);
|
||||||
|
u8 second = (u8)value[i + 1];
|
||||||
|
if((c == 0xE0 && second < 0xA0) || (c == 0xED && second >= 0xA0) || (c == 0xF0 && second < 0x90) || (c == 0xF4 && second >= 0x90)) return(false);
|
||||||
|
i += need + 1;
|
||||||
|
}
|
||||||
|
return(true);
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool uce_crypto_utf8_json_string(String value)
|
||||||
|
{
|
||||||
|
if(!uce_crypto_utf8_string(value)) return(false);
|
||||||
|
for(unsigned char c : value) if(c < 0x20) return(false);
|
||||||
|
return(true);
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool uce_crypto_value_valid(const DValue& value, size_t depth, size_t& nodes, size_t& bytes)
|
||||||
|
{
|
||||||
|
if(depth > 16 || ++nodes > 256) return(false);
|
||||||
|
const DValue& item = value.deref();
|
||||||
|
if(item.type == 'S')
|
||||||
|
{
|
||||||
|
bytes += item._String.size();
|
||||||
|
return(bytes <= 32768 && uce_crypto_utf8_json_string(item._String));
|
||||||
|
}
|
||||||
|
if(item.type == 'F') return(std::isfinite(item._float));
|
||||||
|
if(item.type == 'B') return(true);
|
||||||
|
if(item.type != 'M') return(false);
|
||||||
|
bool list = item.is_list();
|
||||||
|
for(const auto& child : item._map)
|
||||||
|
{
|
||||||
|
if(!list)
|
||||||
|
{
|
||||||
|
bytes += child.first.size();
|
||||||
|
if(bytes > 32768 || !uce_crypto_utf8_json_string(child.first)) return(false);
|
||||||
|
}
|
||||||
|
if(!uce_crypto_value_valid(child.second, depth + 1, nodes, bytes)) return(false);
|
||||||
|
}
|
||||||
|
return(true);
|
||||||
|
}
|
||||||
|
|
||||||
|
bool crypto_operation_request_valid(DValue request)
|
||||||
|
{
|
||||||
|
const DValue& root = request.deref();
|
||||||
|
if(root.type != 'M' || root.is_list()) return(false);
|
||||||
|
size_t nodes = 0, bytes = 0;
|
||||||
|
return(uce_crypto_value_valid(root, 0, nodes, bytes));
|
||||||
|
}
|
||||||
|
|
||||||
#ifndef __UCE_WASM_CORE__
|
#ifndef __UCE_WASM_CORE__
|
||||||
|
#include <openssl/bn.h>
|
||||||
|
#include <openssl/core_names.h>
|
||||||
|
#include <openssl/ecdsa.h>
|
||||||
#include <openssl/evp.h>
|
#include <openssl/evp.h>
|
||||||
|
#include <openssl/params.h>
|
||||||
|
#include <openssl/param_build.h>
|
||||||
#include <openssl/rand.h>
|
#include <openssl/rand.h>
|
||||||
|
|
||||||
namespace {
|
namespace {
|
||||||
@@ -525,4 +587,385 @@ bool password_needs_rehash_native(String encoded)
|
|||||||
String salt, digest;
|
String salt, digest;
|
||||||
return(!uce_password_parts(encoded, n, r, p, salt, digest) || n != UCE_PASSWORD_SCRYPT_N || r != UCE_PASSWORD_SCRYPT_R || p != UCE_PASSWORD_SCRYPT_P);
|
return(!uce_password_parts(encoded, n, r, p, salt, digest) || n != UCE_PASSWORD_SCRYPT_N || r != UCE_PASSWORD_SCRYPT_R || p != UCE_PASSWORD_SCRYPT_P);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
namespace {
|
||||||
|
static constexpr size_t UCE_ES256_COORDINATE_BYTES = 32;
|
||||||
|
static constexpr size_t UCE_ES256_SIGNATURE_BYTES = 64;
|
||||||
|
static constexpr size_t UCE_ES256_JSON_MAX = 16 * 1024;
|
||||||
|
static constexpr size_t UCE_ES256_VALUE_MAX = 256;
|
||||||
|
static constexpr size_t UCE_ES256_DEPTH_MAX = 16;
|
||||||
|
static constexpr size_t UCE_ES256_COORDINATE_BASE64URL_MAX = 43;
|
||||||
|
static constexpr size_t UCE_CBOR_MAX_BYTES = 16 * 1024;
|
||||||
|
static constexpr size_t UCE_CBOR_MAX_BASE64URL = 21846;
|
||||||
|
static constexpr size_t UCE_ES256_DER_MAX_BYTES = 144;
|
||||||
|
static constexpr size_t UCE_ES256_DER_BASE64URL_MAX = 192;
|
||||||
|
|
||||||
|
struct UcePkeyDeleter { void operator()(EVP_PKEY* value) const { EVP_PKEY_free(value); } };
|
||||||
|
struct UcePkeyCtxDeleter { void operator()(EVP_PKEY_CTX* value) const { EVP_PKEY_CTX_free(value); } };
|
||||||
|
struct UceEcdsaSigDeleter { void operator()(ECDSA_SIG* value) const { ECDSA_SIG_free(value); } };
|
||||||
|
struct UceBnDeleter { void operator()(BIGNUM* value) const { BN_clear_free(value); } };
|
||||||
|
struct UceParamBldDeleter { void operator()(OSSL_PARAM_BLD* value) const { OSSL_PARAM_BLD_free(value); } };
|
||||||
|
struct UceParamsDeleter { void operator()(OSSL_PARAM* value) const { OSSL_PARAM_free(value); } };
|
||||||
|
|
||||||
|
static String uce_base64url_encode(const unsigned char* bytes, size_t size)
|
||||||
|
{
|
||||||
|
String encoded = base64_encode(String((const char*)bytes, size));
|
||||||
|
encoded = replace(replace(encoded, "+", "-"), "/", "_");
|
||||||
|
while(!encoded.empty() && encoded.back() == '=') encoded.pop_back();
|
||||||
|
return(encoded);
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool uce_base64url_decode(String encoded, String& decoded, size_t max_encoded)
|
||||||
|
{
|
||||||
|
if(encoded.empty() || encoded.size() > max_encoded || encoded.find('=') != String::npos || encoded.size() % 4 == 1)
|
||||||
|
return(false);
|
||||||
|
for(char c : encoded)
|
||||||
|
if(!(c >= 'A' && c <= 'Z') && !(c >= 'a' && c <= 'z') && !(c >= '0' && c <= '9') && c != '-' && c != '_')
|
||||||
|
return(false);
|
||||||
|
String padded = replace(replace(encoded, "-", "+"), "_", "/");
|
||||||
|
while(padded.size() % 4) padded += "=";
|
||||||
|
bool ok = false;
|
||||||
|
decoded = base64_decode(padded, ok);
|
||||||
|
return(ok && uce_base64url_encode((const unsigned char*)decoded.data(), decoded.size()) == encoded);
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool uce_es256_json_value(const DValue& value, size_t depth, size_t& values, size_t& bytes)
|
||||||
|
{
|
||||||
|
if(depth > UCE_ES256_DEPTH_MAX || ++values > UCE_ES256_VALUE_MAX)
|
||||||
|
return(false);
|
||||||
|
const DValue& item = value.deref();
|
||||||
|
if(item.type == 'S')
|
||||||
|
return((bytes += item._String.size()) <= UCE_ES256_JSON_MAX);
|
||||||
|
if(item.type == 'F' || item.type == 'B')
|
||||||
|
return(true);
|
||||||
|
if(item.type != 'M')
|
||||||
|
return(false);
|
||||||
|
for(const auto& child : item._map)
|
||||||
|
{
|
||||||
|
if((bytes += child.first.size()) > UCE_ES256_JSON_MAX || !uce_es256_json_value(child.second, depth + 1, values, bytes))
|
||||||
|
return(false);
|
||||||
|
}
|
||||||
|
return(true);
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool uce_es256_json_map(const DValue& value)
|
||||||
|
{
|
||||||
|
if(value.deref().type != 'M' || value.deref().is_list())
|
||||||
|
return(false);
|
||||||
|
size_t values = 0, bytes = 0;
|
||||||
|
return(uce_es256_json_value(value, 0, values, bytes));
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool uce_es256_jwk_string(const DValue& jwk, const String& field, String& value)
|
||||||
|
{
|
||||||
|
const DValue* found = jwk.key(field);
|
||||||
|
if(!found)
|
||||||
|
return(false);
|
||||||
|
const DValue& item = found->deref();
|
||||||
|
if(item.type != 'S' || item._String.empty() || item._String.size() > 32768)
|
||||||
|
return(false);
|
||||||
|
value = item._String;
|
||||||
|
return(true);
|
||||||
|
}
|
||||||
|
|
||||||
|
static std::unique_ptr<EVP_PKEY, UcePkeyDeleter> uce_es256_key_from_jwk(const DValue& jwk)
|
||||||
|
{
|
||||||
|
if(jwk.deref().type != 'M')
|
||||||
|
return(nullptr);
|
||||||
|
String kty, crv, x64, y64, d64, x, y, d;
|
||||||
|
if(!uce_es256_jwk_string(jwk, "kty", kty) || !uce_es256_jwk_string(jwk, "crv", crv) || !uce_es256_jwk_string(jwk, "x", x64) || !uce_es256_jwk_string(jwk, "y", y64) || !uce_es256_jwk_string(jwk, "d", d64) ||
|
||||||
|
kty != "EC" || crv != "P-256" || !uce_base64url_decode(x64, x, UCE_ES256_COORDINATE_BASE64URL_MAX) || !uce_base64url_decode(y64, y, UCE_ES256_COORDINATE_BASE64URL_MAX) || !uce_base64url_decode(d64, d, UCE_ES256_COORDINATE_BASE64URL_MAX) ||
|
||||||
|
x.size() != UCE_ES256_COORDINATE_BYTES || y.size() != UCE_ES256_COORDINATE_BYTES || d.size() != UCE_ES256_COORDINATE_BYTES)
|
||||||
|
return(nullptr);
|
||||||
|
unsigned char public_key[65];
|
||||||
|
public_key[0] = 4;
|
||||||
|
memcpy(public_key + 1, x.data(), x.size());
|
||||||
|
memcpy(public_key + 33, y.data(), y.size());
|
||||||
|
std::unique_ptr<BIGNUM, UceBnDeleter> private_bn(BN_bin2bn((const unsigned char*)d.data(), d.size(), 0));
|
||||||
|
std::unique_ptr<OSSL_PARAM_BLD, UceParamBldDeleter> builder(OSSL_PARAM_BLD_new());
|
||||||
|
if(!private_bn || !builder || OSSL_PARAM_BLD_push_utf8_string(builder.get(), OSSL_PKEY_PARAM_GROUP_NAME, "prime256v1", 0) <= 0 ||
|
||||||
|
OSSL_PARAM_BLD_push_octet_string(builder.get(), OSSL_PKEY_PARAM_PUB_KEY, public_key, sizeof(public_key)) <= 0 ||
|
||||||
|
OSSL_PARAM_BLD_push_BN(builder.get(), OSSL_PKEY_PARAM_PRIV_KEY, private_bn.get()) <= 0)
|
||||||
|
return(nullptr);
|
||||||
|
std::unique_ptr<OSSL_PARAM, UceParamsDeleter> params(OSSL_PARAM_BLD_to_param(builder.get()));
|
||||||
|
std::unique_ptr<EVP_PKEY_CTX, UcePkeyCtxDeleter> ctx(EVP_PKEY_CTX_new_from_name(0, "EC", 0));
|
||||||
|
EVP_PKEY* raw = 0;
|
||||||
|
if(!params || !ctx || EVP_PKEY_fromdata_init(ctx.get()) <= 0 || EVP_PKEY_fromdata(ctx.get(), &raw, EVP_PKEY_KEYPAIR, params.get()) <= 0)
|
||||||
|
return(nullptr);
|
||||||
|
std::unique_ptr<EVP_PKEY, UcePkeyDeleter> key(raw);
|
||||||
|
std::unique_ptr<EVP_PKEY_CTX, UcePkeyCtxDeleter> check(EVP_PKEY_CTX_new(key.get(), 0));
|
||||||
|
return(check && EVP_PKEY_public_check(check.get()) > 0 && EVP_PKEY_private_check(check.get()) > 0 && EVP_PKEY_pairwise_check(check.get()) > 0 ? std::move(key) : nullptr);
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool uce_es256_key_coordinates(EVP_PKEY* key, String& x, String& y, String& d)
|
||||||
|
{
|
||||||
|
unsigned char public_key[65]; size_t public_key_size = sizeof(public_key);
|
||||||
|
BIGNUM* private_bn = 0;
|
||||||
|
if(EVP_PKEY_get_octet_string_param(key, OSSL_PKEY_PARAM_PUB_KEY, public_key, sizeof(public_key), &public_key_size) <= 0 || public_key_size != sizeof(public_key) || public_key[0] != 4 ||
|
||||||
|
EVP_PKEY_get_bn_param(key, OSSL_PKEY_PARAM_PRIV_KEY, &private_bn) <= 0)
|
||||||
|
return(false);
|
||||||
|
std::unique_ptr<BIGNUM, UceBnDeleter> private_key(private_bn);
|
||||||
|
if(BN_bn2binpad(private_key.get(), (unsigned char*)d.data(), UCE_ES256_COORDINATE_BYTES) != UCE_ES256_COORDINATE_BYTES)
|
||||||
|
return(false);
|
||||||
|
x.assign((const char*)public_key + 1, UCE_ES256_COORDINATE_BYTES);
|
||||||
|
y.assign((const char*)public_key + 33, UCE_ES256_COORDINATE_BYTES);
|
||||||
|
return(true);
|
||||||
|
}
|
||||||
|
|
||||||
|
enum class UceCborKind { Unsigned, Negative, Bytes, Text, Array, Map };
|
||||||
|
struct UceCbor { UceCborKind kind; u64 number = 0; String bytes; std::vector<UceCbor> items; };
|
||||||
|
static constexpr size_t UCE_CBOR_MAX_NODES = 256, UCE_CBOR_MAX_DEPTH = 16;
|
||||||
|
|
||||||
|
static bool uce_cbor_uint(const String& in, size_t& p, u8 ai, u64& n)
|
||||||
|
{
|
||||||
|
if(ai < 24) { n = ai; return(true); }
|
||||||
|
size_t count = ai == 24 ? 1 : ai == 25 ? 2 : ai == 26 ? 4 : ai == 27 ? 8 : 0;
|
||||||
|
if(!count || p > in.size() || count > in.size() - p) return(false);
|
||||||
|
n = 0;
|
||||||
|
for(size_t i = 0; i < count; i++)
|
||||||
|
{
|
||||||
|
if(n > (UINT64_MAX - (u8)in[p + i]) / 256) return(false);
|
||||||
|
n = n * 256 + (u8)in[p + i];
|
||||||
|
}
|
||||||
|
p += count;
|
||||||
|
return((count == 1 && n >= 24) || (count == 2 && n > 0xff) || (count == 4 && n > 0xffff) || (count == 8 && n > 0xffffffff));
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool uce_cbor_equal(const UceCbor& left, const UceCbor& right)
|
||||||
|
{
|
||||||
|
if(left.kind != right.kind || left.number != right.number || left.bytes != right.bytes || left.items.size() != right.items.size()) return(false);
|
||||||
|
for(size_t i = 0; i < left.items.size(); i++) if(!uce_cbor_equal(left.items[i], right.items[i])) return(false);
|
||||||
|
return(true);
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool uce_cbor_read(const String& in, size_t& p, UceCbor& out, size_t depth, size_t& nodes)
|
||||||
|
{
|
||||||
|
if(p >= in.size() || depth > UCE_CBOR_MAX_DEPTH || ++nodes > UCE_CBOR_MAX_NODES) return(false);
|
||||||
|
u8 header = (u8)in[p++], major = header >> 5, ai = header & 31;
|
||||||
|
u64 n = 0;
|
||||||
|
if(ai == 31 || !uce_cbor_uint(in, p, ai, n)) return(false);
|
||||||
|
if(major == 0 || major == 1)
|
||||||
|
{
|
||||||
|
out.kind = major ? UceCborKind::Negative : UceCborKind::Unsigned;
|
||||||
|
out.number = n;
|
||||||
|
return(true);
|
||||||
|
}
|
||||||
|
if(major == 2 || major == 3)
|
||||||
|
{
|
||||||
|
if(n > UCE_CBOR_MAX_BYTES || n > in.size() - p || (major == 3 && !uce_crypto_utf8_string(String(in.data() + p, (size_t)n)))) return(false);
|
||||||
|
out.kind = major == 2 ? UceCborKind::Bytes : UceCborKind::Text;
|
||||||
|
out.bytes.assign(in.data() + p, (size_t)n);
|
||||||
|
p += (size_t)n;
|
||||||
|
return(true);
|
||||||
|
}
|
||||||
|
if((major != 4 && major != 5) || n > UCE_CBOR_MAX_NODES) return(false);
|
||||||
|
size_t item_count = (size_t)n * (major == 5 ? 2 : 1);
|
||||||
|
if(item_count > UCE_CBOR_MAX_NODES - nodes) return(false);
|
||||||
|
out.kind = major == 4 ? UceCborKind::Array : UceCborKind::Map;
|
||||||
|
out.items.reserve(item_count);
|
||||||
|
for(size_t i = 0; i < item_count; i++)
|
||||||
|
{
|
||||||
|
UceCbor child;
|
||||||
|
if(!uce_cbor_read(in, p, child, depth + 1, nodes)) return(false);
|
||||||
|
if(major == 5 && !(i & 1))
|
||||||
|
for(size_t previous = 0; previous < out.items.size(); previous += 2)
|
||||||
|
if(uce_cbor_equal(out.items[previous], child)) return(false);
|
||||||
|
out.items.push_back(std::move(child));
|
||||||
|
}
|
||||||
|
return(true);
|
||||||
|
}
|
||||||
|
static DValue uce_cbor_value(const UceCbor& value)
|
||||||
|
{
|
||||||
|
DValue out; switch(value.kind) { case UceCborKind::Unsigned: out["type"]="unsigned"; out["value"]=std::to_string(value.number); break; case UceCborKind::Negative: out["type"]="negative"; out["value"]=value.number==UINT64_MAX?"-18446744073709551616":"-"+std::to_string(value.number+1); break; case UceCborKind::Bytes: out["type"]="bytes"; out["base64url"]=uce_base64url_encode((const unsigned char*)value.bytes.data(),value.bytes.size()); break; case UceCborKind::Text: out["type"]="text"; out["value"]=value.bytes; break; case UceCborKind::Array: out["type"]="array"; out["items"].set_array(); for(const auto& x:value.items) out["items"].push(uce_cbor_value(x)); break; case UceCborKind::Map: out["type"]="map"; out["entries"].set_array(); for(size_t i=0;i<value.items.size();i+=2) { DValue entry; entry["key"]=uce_cbor_value(value.items[i]); entry["value"]=uce_cbor_value(value.items[i+1]); out["entries"].push(entry); } } return out;
|
||||||
|
}
|
||||||
|
static bool uce_cbor_es256(const UceCbor& cose,String& x,String& y)
|
||||||
|
{
|
||||||
|
if(cose.kind!=UceCborKind::Map) return false; const UceCbor *kty=0,*alg=0,*crv=0,*xx=0,*yy=0;
|
||||||
|
for(size_t i=0;i<cose.items.size();i+=2) { const UceCbor& k=cose.items[i]; const UceCbor& v=cose.items[i+1]; if(k.kind!=UceCborKind::Unsigned&&k.kind!=UceCborKind::Negative) continue; bool neg=k.kind==UceCborKind::Negative; if(!neg&&k.number==1) kty=&v; else if(!neg&&k.number==3) alg=&v; else if(neg&&k.number==0) crv=&v; else if(neg&&k.number==1) xx=&v; else if(neg&&k.number==2) yy=&v; }
|
||||||
|
return(kty&&alg&&crv&&xx&&yy&&kty->kind==UceCborKind::Unsigned&&kty->number==2&&alg->kind==UceCborKind::Negative&&alg->number==6&&crv->kind==UceCborKind::Unsigned&&crv->number==1&&xx->kind==UceCborKind::Bytes&&yy->kind==UceCborKind::Bytes&&xx->bytes.size()==32&&yy->bytes.size()==32&&(x=xx->bytes,true)&&(y=yy->bytes,true));
|
||||||
|
}
|
||||||
|
static std::unique_ptr<EVP_PKEY,UcePkeyDeleter> uce_es256_public_key(String x,String y)
|
||||||
|
{
|
||||||
|
if(x.size()!=32||y.size()!=32) return nullptr; unsigned char point[65]={4}; memcpy(point+1,x.data(),32); memcpy(point+33,y.data(),32); OSSL_PARAM p[]={OSSL_PARAM_construct_utf8_string(OSSL_PKEY_PARAM_GROUP_NAME,(char*)"prime256v1",0),OSSL_PARAM_construct_octet_string(OSSL_PKEY_PARAM_PUB_KEY,point,sizeof(point)),OSSL_PARAM_construct_end()}; std::unique_ptr<EVP_PKEY_CTX,UcePkeyCtxDeleter> ctx(EVP_PKEY_CTX_new_from_name(0,"EC",0)); EVP_PKEY* raw=0; if(!ctx||EVP_PKEY_fromdata_init(ctx.get())<=0||EVP_PKEY_fromdata(ctx.get(),&raw,EVP_PKEY_PUBLIC_KEY,p)<=0) return nullptr; std::unique_ptr<EVP_PKEY,UcePkeyDeleter> key(raw); std::unique_ptr<EVP_PKEY_CTX,UcePkeyCtxDeleter> check(EVP_PKEY_CTX_new(key.get(),0)); return check&&EVP_PKEY_public_check(check.get())>0?std::move(key):nullptr;
|
||||||
|
}
|
||||||
|
static bool uce_es256_cose(String encoded,String& x,String& y)
|
||||||
|
{
|
||||||
|
String raw; UceCbor cose; size_t p=0,nodes=0; return uce_base64url_decode(encoded, raw, UCE_CBOR_MAX_BASE64URL) && raw.size() <= UCE_CBOR_MAX_BYTES && uce_cbor_read(raw, p, cose, 0, nodes) && p == raw.size() && uce_cbor_es256(cose, x, y) && uce_es256_public_key(x, y);
|
||||||
|
}
|
||||||
|
|
||||||
|
static DValue uce_es256_jwk(String x, String y, String d = "")
|
||||||
|
{
|
||||||
|
DValue jwk;
|
||||||
|
jwk["kty"] = "EC";
|
||||||
|
jwk["crv"] = "P-256";
|
||||||
|
jwk["x"] = uce_base64url_encode((const unsigned char*)x.data(), x.size());
|
||||||
|
jwk["y"] = uce_base64url_encode((const unsigned char*)y.data(), y.size());
|
||||||
|
if(d != "") jwk["d"] = uce_base64url_encode((const unsigned char*)d.data(), d.size());
|
||||||
|
return(jwk);
|
||||||
|
}
|
||||||
|
|
||||||
|
static String uce_es256_thumbprint(const DValue& public_jwk)
|
||||||
|
{
|
||||||
|
const DValue* x = public_jwk.key("x");
|
||||||
|
const DValue* y = public_jwk.key("y");
|
||||||
|
if(!x || !y)
|
||||||
|
return("");
|
||||||
|
String canonical = "{\"crv\":\"P-256\",\"kty\":\"EC\",\"x\":\"" + x->to_string() + "\",\"y\":\"" + y->to_string() + "\"}";
|
||||||
|
String digest = sha256_native(canonical);
|
||||||
|
return(uce_base64url_encode((const unsigned char*)digest.data(), digest.size()));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
static DValue uce_es256_key_create()
|
||||||
|
{
|
||||||
|
std::unique_ptr<EVP_PKEY_CTX, UcePkeyCtxDeleter> ctx(EVP_PKEY_CTX_new_from_name(0, "EC", 0));
|
||||||
|
EVP_PKEY* raw = 0;
|
||||||
|
OSSL_PARAM params[] = { OSSL_PARAM_construct_utf8_string(OSSL_PKEY_PARAM_GROUP_NAME, (char*)"prime256v1", 0), OSSL_PARAM_construct_end() };
|
||||||
|
if(!ctx || EVP_PKEY_keygen_init(ctx.get()) <= 0 || EVP_PKEY_CTX_set_params(ctx.get(), params) <= 0 || EVP_PKEY_generate(ctx.get(), &raw) <= 0)
|
||||||
|
return(DValue());
|
||||||
|
std::unique_ptr<EVP_PKEY, UcePkeyDeleter> key(raw);
|
||||||
|
String x(UCE_ES256_COORDINATE_BYTES, 0), y(UCE_ES256_COORDINATE_BYTES, 0), d(UCE_ES256_COORDINATE_BYTES, 0);
|
||||||
|
if(!uce_es256_key_coordinates(key.get(), x, y, d))
|
||||||
|
return(DValue());
|
||||||
|
DValue result;
|
||||||
|
result["public_jwk"] = uce_es256_jwk(x, y);
|
||||||
|
String kid = uce_es256_thumbprint(result["public_jwk"]);
|
||||||
|
result["public_jwk"]["kid"] = kid;
|
||||||
|
result["private_jwk"] = uce_es256_jwk(x, y, d);
|
||||||
|
result["private_jwk"]["kid"] = kid;
|
||||||
|
result["kid"] = kid;
|
||||||
|
result["thumbprint"] = kid;
|
||||||
|
return(result);
|
||||||
|
}
|
||||||
|
|
||||||
|
static String uce_es256_jwt(DValue private_jwk, DValue protected_header, DValue claims)
|
||||||
|
{
|
||||||
|
if(!uce_es256_json_map(protected_header) || !uce_es256_json_map(claims))
|
||||||
|
return("");
|
||||||
|
std::unique_ptr<EVP_PKEY, UcePkeyDeleter> key = uce_es256_key_from_jwk(private_jwk);
|
||||||
|
if(!key)
|
||||||
|
return("");
|
||||||
|
protected_header["alg"] = "ES256";
|
||||||
|
String header_json = json_encode(protected_header);
|
||||||
|
String claims_json = json_encode(claims);
|
||||||
|
if(header_json.size() > UCE_ES256_JSON_MAX || claims_json.size() > UCE_ES256_JSON_MAX)
|
||||||
|
return("");
|
||||||
|
String signing_input = uce_base64url_encode((const unsigned char*)header_json.data(), header_json.size()) + "." + uce_base64url_encode((const unsigned char*)claims_json.data(), claims_json.size());
|
||||||
|
std::unique_ptr<EVP_MD_CTX, decltype(&EVP_MD_CTX_free)> ctx(EVP_MD_CTX_new(), EVP_MD_CTX_free);
|
||||||
|
size_t der_size = 0;
|
||||||
|
if(!ctx || EVP_DigestSignInit(ctx.get(), 0, EVP_sha256(), 0, key.get()) <= 0 || EVP_DigestSign(ctx.get(), 0, &der_size, (const unsigned char*)signing_input.data(), signing_input.size()) <= 0 || der_size == 0 || der_size > 256)
|
||||||
|
return("");
|
||||||
|
String der(der_size, 0);
|
||||||
|
if(EVP_DigestSign(ctx.get(), (unsigned char*)der.data(), &der_size, (const unsigned char*)signing_input.data(), signing_input.size()) <= 0)
|
||||||
|
return("");
|
||||||
|
const unsigned char* cursor = (const unsigned char*)der.data();
|
||||||
|
std::unique_ptr<ECDSA_SIG, UceEcdsaSigDeleter> signature(d2i_ECDSA_SIG(0, &cursor, der_size));
|
||||||
|
const BIGNUM *r = 0, *s = 0;
|
||||||
|
if(!signature || cursor != (const unsigned char*)der.data() + der_size)
|
||||||
|
return("");
|
||||||
|
ECDSA_SIG_get0(signature.get(), &r, &s);
|
||||||
|
String jose(UCE_ES256_SIGNATURE_BYTES, 0);
|
||||||
|
if(!r || !s || BN_bn2binpad(r, (unsigned char*)jose.data(), UCE_ES256_COORDINATE_BYTES) != UCE_ES256_COORDINATE_BYTES || BN_bn2binpad(s, (unsigned char*)jose.data() + UCE_ES256_COORDINATE_BYTES, UCE_ES256_COORDINATE_BYTES) != UCE_ES256_COORDINATE_BYTES)
|
||||||
|
return("");
|
||||||
|
return(signing_input + "." + uce_base64url_encode((const unsigned char*)jose.data(), jose.size()));
|
||||||
|
}
|
||||||
|
|
||||||
|
DValue crypto_operation_native(DValue request)
|
||||||
|
{
|
||||||
|
DValue result;
|
||||||
|
result["ok"].set_bool(false);
|
||||||
|
if(!crypto_operation_request_valid(request))
|
||||||
|
{
|
||||||
|
result["error"] = "invalid_request";
|
||||||
|
return(result);
|
||||||
|
}
|
||||||
|
String operation, algorithm;
|
||||||
|
if(!uce_es256_jwk_string(request, "operation", operation) || !uce_es256_jwk_string(request, "algorithm", algorithm))
|
||||||
|
{
|
||||||
|
result["error"] = "invalid_request";
|
||||||
|
return(result);
|
||||||
|
}
|
||||||
|
if(algorithm != "ES256")
|
||||||
|
{
|
||||||
|
result["error"] = "unsupported_algorithm";
|
||||||
|
return(result);
|
||||||
|
}
|
||||||
|
if(operation == "key_generate")
|
||||||
|
{
|
||||||
|
DValue key = uce_es256_key_create();
|
||||||
|
if(key["private_jwk"]["d"].to_string() == "")
|
||||||
|
{
|
||||||
|
result["error"] = "operation_failed";
|
||||||
|
return(result);
|
||||||
|
}
|
||||||
|
key["ok"].set_bool(true);
|
||||||
|
key["operation"] = operation;
|
||||||
|
key["algorithm"] = algorithm;
|
||||||
|
return(key);
|
||||||
|
}
|
||||||
|
if(operation == "cbor_decode")
|
||||||
|
{
|
||||||
|
String encoded, raw; UceCbor value; size_t p=0,nodes=0;
|
||||||
|
if(!uce_es256_jwk_string(request, "cbor_base64url", encoded) || !uce_base64url_decode(encoded, raw, UCE_CBOR_MAX_BASE64URL) || raw.size() > UCE_CBOR_MAX_BYTES || !uce_cbor_read(raw, p, value, 0, nodes) || p != raw.size()) { result["error"] = "invalid_cbor"; return(result); }
|
||||||
|
result["ok"].set_bool(true); result["operation"]=operation; result["value"]=uce_cbor_value(value); return result;
|
||||||
|
}
|
||||||
|
if(operation == "cose_es256_parse")
|
||||||
|
{
|
||||||
|
String encoded,x,y; if(!uce_es256_jwk_string(request,"cose_key_base64url",encoded)||!uce_es256_cose(encoded,x,y)) { result["error"]="invalid_cose_key"; return result; }
|
||||||
|
result["ok"].set_bool(true); result["operation"]=operation; result["algorithm"]=algorithm; result["x_base64url"]=uce_base64url_encode((const unsigned char*)x.data(),x.size()); result["y_base64url"]=uce_base64url_encode((const unsigned char*)y.data(),y.size()); return result;
|
||||||
|
}
|
||||||
|
if(operation == "es256_verify")
|
||||||
|
{
|
||||||
|
String encoded, message64, signature64, x, y, message, der;
|
||||||
|
if(!uce_es256_jwk_string(request, "cose_key_base64url", encoded) || !uce_es256_jwk_string(request, "message_base64url", message64) || !uce_es256_jwk_string(request, "signature_der_base64url", signature64) || !uce_es256_cose(encoded, x, y) || !uce_base64url_decode(message64, message, UCE_CBOR_MAX_BASE64URL) || !uce_base64url_decode(signature64, der, UCE_ES256_DER_BASE64URL_MAX) || message.size() > UCE_CBOR_MAX_BYTES || der.empty() || der.size() > UCE_ES256_DER_MAX_BYTES)
|
||||||
|
{
|
||||||
|
result["error"] = "invalid_key_or_payload";
|
||||||
|
return(result);
|
||||||
|
}
|
||||||
|
const unsigned char* cursor = (const unsigned char*)der.data();
|
||||||
|
std::unique_ptr<ECDSA_SIG, UceEcdsaSigDeleter> signature(d2i_ECDSA_SIG(0, &cursor, der.size()));
|
||||||
|
int canonical = signature ? i2d_ECDSA_SIG(signature.get(), 0) : 0;
|
||||||
|
String reencoded(canonical > 0 ? (size_t)canonical : 0, 0);
|
||||||
|
unsigned char* dest = (unsigned char*)reencoded.data();
|
||||||
|
if(!signature || cursor != (const unsigned char*)der.data() + der.size() || canonical <= 0 || i2d_ECDSA_SIG(signature.get(), &dest) != canonical || reencoded != der)
|
||||||
|
{
|
||||||
|
result["error"] = "invalid_signature";
|
||||||
|
return(result);
|
||||||
|
}
|
||||||
|
std::unique_ptr<EVP_PKEY, UcePkeyDeleter> key = uce_es256_public_key(x, y);
|
||||||
|
std::unique_ptr<EVP_MD_CTX, decltype(&EVP_MD_CTX_free)> ctx(EVP_MD_CTX_new(), EVP_MD_CTX_free);
|
||||||
|
if(!key || !ctx || EVP_DigestVerifyInit(ctx.get(), 0, EVP_sha256(), 0, key.get()) <= 0)
|
||||||
|
{
|
||||||
|
result["error"] = "operation_failed";
|
||||||
|
return(result);
|
||||||
|
}
|
||||||
|
int verified = EVP_DigestVerify(ctx.get(), (const unsigned char*)der.data(), der.size(), (const unsigned char*)message.data(), message.size());
|
||||||
|
if(verified < 0)
|
||||||
|
{
|
||||||
|
result["error"] = "operation_failed";
|
||||||
|
return(result);
|
||||||
|
}
|
||||||
|
result["ok"].set_bool(true);
|
||||||
|
result["operation"] = operation;
|
||||||
|
result["algorithm"] = algorithm;
|
||||||
|
result["valid"].set_bool(verified == 1);
|
||||||
|
return(result);
|
||||||
|
}
|
||||||
|
if(operation == "jwt_sign")
|
||||||
|
{
|
||||||
|
String jwt = uce_es256_jwt(request["private_jwk"], request["protected_header"], request["claims"]);
|
||||||
|
if(jwt == "")
|
||||||
|
{
|
||||||
|
result["error"] = "invalid_key_or_payload";
|
||||||
|
return(result);
|
||||||
|
}
|
||||||
|
result["ok"].set_bool(true);
|
||||||
|
result["operation"] = operation;
|
||||||
|
result["algorithm"] = algorithm;
|
||||||
|
result["jwt"] = jwt;
|
||||||
|
return(result);
|
||||||
|
}
|
||||||
|
result["error"] = "unsupported_operation";
|
||||||
|
return(result);
|
||||||
|
}
|
||||||
#endif
|
#endif
|
||||||
|
|||||||
@@ -1,5 +1,7 @@
|
|||||||
#pragma once
|
#pragma once
|
||||||
|
|
||||||
|
struct DValue;
|
||||||
|
|
||||||
/* ================ sha1.h ================ */
|
/* ================ sha1.h ================ */
|
||||||
/*
|
/*
|
||||||
SHA-1 in C
|
SHA-1 in C
|
||||||
@@ -27,6 +29,8 @@ bool crypto_equal_native(String a, String b);
|
|||||||
String password_hash_native(String password);
|
String password_hash_native(String password);
|
||||||
bool password_verify_native(String password, String encoded);
|
bool password_verify_native(String password, String encoded);
|
||||||
bool password_needs_rehash_native(String encoded);
|
bool password_needs_rehash_native(String encoded);
|
||||||
|
bool crypto_operation_request_valid(DValue request);
|
||||||
|
DValue crypto_operation_native(DValue request);
|
||||||
String sha256(String data);
|
String sha256(String data);
|
||||||
String sha256_hex(String data);
|
String sha256_hex(String data);
|
||||||
String hmac_sha256(String key, String data);
|
String hmac_sha256(String key, String data);
|
||||||
|
|||||||
+48
-10
@@ -246,9 +246,10 @@ DValue MySQL::query(String q)
|
|||||||
statement_info = "mysql connection is not open";
|
statement_info = "mysql connection is not open";
|
||||||
return(DValue());
|
return(DValue());
|
||||||
}
|
}
|
||||||
_preload_next_error_code = mysql_query((MYSQL*)connection, q.c_str());
|
int query_status = mysql_query((MYSQL*)connection, q.c_str());
|
||||||
|
_preload_next_error_code = query_status == 0 ? 0 : mysql_errno((MYSQL*)connection);
|
||||||
DValue result;
|
DValue result;
|
||||||
if(_preload_next_error_code == 0)
|
if(query_status == 0)
|
||||||
result = get_pending_result();
|
result = get_pending_result();
|
||||||
return(result);
|
return(result);
|
||||||
}
|
}
|
||||||
@@ -291,11 +292,16 @@ static bool mysql_has_unquoted_positional_placeholder(String query)
|
|||||||
|
|
||||||
DValue MySQL::query(String q, StringMap params)
|
DValue MySQL::query(String q, StringMap params)
|
||||||
{
|
{
|
||||||
// Positional ? placeholders survive named substitution (values are always
|
// Positional ? placeholders survive named substitution (ordinary values
|
||||||
// quoted by escape()), so the check in query(String) covers this path too.
|
// are quoted by escape()), so the check in query(String) covers this path.
|
||||||
return(query(
|
parameter_error = false;
|
||||||
parse_query_parameters(q, params).c_str()
|
String parsed = parse_query_parameters(q, params);
|
||||||
));
|
if(parameter_error)
|
||||||
|
{
|
||||||
|
_preload_next_error_code = CR_UNKNOWN_ERROR;
|
||||||
|
return(DValue());
|
||||||
|
}
|
||||||
|
return(query(parsed));
|
||||||
}
|
}
|
||||||
|
|
||||||
String MySQL::parse_query_parameters(String query, StringMap map)
|
String MySQL::parse_query_parameters(String query, StringMap map)
|
||||||
@@ -305,6 +311,7 @@ String MySQL::parse_query_parameters(String query, StringMap map)
|
|||||||
|
|
||||||
u8 mode = 0;
|
u8 mode = 0;
|
||||||
char quote;
|
char quote;
|
||||||
|
bool escaped = false;
|
||||||
String identifier;
|
String identifier;
|
||||||
for(u32 i = 0; i < query.length(); i++)
|
for(u32 i = 0; i < query.length(); i++)
|
||||||
{
|
{
|
||||||
@@ -321,6 +328,7 @@ String MySQL::parse_query_parameters(String query, StringMap map)
|
|||||||
result.append(1, c);
|
result.append(1, c);
|
||||||
mode = 2;
|
mode = 2;
|
||||||
quote = c;
|
quote = c;
|
||||||
|
escaped = false;
|
||||||
}
|
}
|
||||||
else
|
else
|
||||||
{
|
{
|
||||||
@@ -329,10 +337,25 @@ String MySQL::parse_query_parameters(String query, StringMap map)
|
|||||||
}
|
}
|
||||||
else if(mode == 1) // identifier mode
|
else if(mode == 1) // identifier mode
|
||||||
{
|
{
|
||||||
if(isalnum(c) || c == '_')
|
if(isalnum((unsigned char)c) || c == '_')
|
||||||
{
|
{
|
||||||
identifier.append(1, c);
|
identifier.append(1, c);
|
||||||
}
|
}
|
||||||
|
else if(c == '!' && query[i + 1] != '=')
|
||||||
|
{
|
||||||
|
String value = map[identifier];
|
||||||
|
bool valid = identifier != "" && value != "";
|
||||||
|
for(char digit : value)
|
||||||
|
if(!isdigit((unsigned char)digit)) valid = false;
|
||||||
|
if(!valid)
|
||||||
|
{
|
||||||
|
parameter_error = true;
|
||||||
|
statement_info = "mysql unsigned parameter :" + identifier + "! must contain only decimal digits";
|
||||||
|
return("");
|
||||||
|
}
|
||||||
|
result.append(value);
|
||||||
|
mode = 0;
|
||||||
|
}
|
||||||
else
|
else
|
||||||
{
|
{
|
||||||
result.append(escape(map[identifier]));
|
result.append(escape(map[identifier]));
|
||||||
@@ -342,9 +365,19 @@ String MySQL::parse_query_parameters(String query, StringMap map)
|
|||||||
}
|
}
|
||||||
else if(mode == 2) // quoted mode
|
else if(mode == 2) // quoted mode
|
||||||
{
|
{
|
||||||
|
result.append(1, c);
|
||||||
|
if(escaped)
|
||||||
|
{
|
||||||
|
escaped = false;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if(c == '\\')
|
||||||
|
{
|
||||||
|
escaped = true;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
if(c == quote)
|
if(c == quote)
|
||||||
mode = 0;
|
mode = 0;
|
||||||
result.append(1, c);
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -390,10 +423,15 @@ String MySQL::error()
|
|||||||
case(CR_OUT_OF_MEMORY):
|
case(CR_OUT_OF_MEMORY):
|
||||||
p = "Out of memory";
|
p = "Out of memory";
|
||||||
break;
|
break;
|
||||||
default:
|
|
||||||
case(CR_UNKNOWN_ERROR):
|
case(CR_UNKNOWN_ERROR):
|
||||||
p = "Unknown server error";
|
p = "Unknown server error";
|
||||||
break;
|
break;
|
||||||
|
default:
|
||||||
|
if(connection && mysql_error((MYSQL*)connection)[0] != '\0')
|
||||||
|
p = mysql_error((MYSQL*)connection);
|
||||||
|
else
|
||||||
|
p = "Unknown server error";
|
||||||
|
break;
|
||||||
}
|
}
|
||||||
_preload_next_error_code = 0;
|
_preload_next_error_code = 0;
|
||||||
return(p);
|
return(p);
|
||||||
|
|||||||
@@ -20,6 +20,7 @@ struct MySQL {
|
|||||||
u32 row_count = 0;
|
u32 row_count = 0;
|
||||||
u64 insert_id = 0;
|
u64 insert_id = 0;
|
||||||
String statement_info = ""; //
|
String statement_info = ""; //
|
||||||
|
bool parameter_error = false;
|
||||||
bool request_cleanup_delete = false;
|
bool request_cleanup_delete = false;
|
||||||
bool request_pooled = false;
|
bool request_pooled = false;
|
||||||
bool worker_persistent = false;
|
bool worker_persistent = false;
|
||||||
|
|||||||
+240
-21
@@ -64,6 +64,7 @@ int uce_host_crypto_equal(const char* a, size_t a_len, const char* b, size_t b_l
|
|||||||
size_t uce_host_password_hash(const char* password, size_t password_len, char* out, size_t cap);
|
size_t uce_host_password_hash(const char* password, size_t password_len, char* out, size_t cap);
|
||||||
int uce_host_password_verify(const char* password, size_t password_len, const char* encoded, size_t encoded_len);
|
int uce_host_password_verify(const char* password, size_t password_len, const char* encoded, size_t encoded_len);
|
||||||
int uce_host_password_needs_rehash(const char* encoded, size_t encoded_len);
|
int uce_host_password_needs_rehash(const char* encoded, size_t encoded_len);
|
||||||
|
size_t uce_host_crypto_operation(const char* in, size_t in_len, char* out, size_t cap);
|
||||||
size_t uce_host_http_request(const char* in, size_t in_len, char* out, size_t cap);
|
size_t uce_host_http_request(const char* in, size_t in_len, char* out, size_t cap);
|
||||||
uint64_t uce_host_http_request_async(const char* in, size_t in_len);
|
uint64_t uce_host_http_request_async(const char* in, size_t in_len);
|
||||||
size_t uce_host_shell_exec_dv(const char* in, size_t in_len, char* out, size_t cap);
|
size_t uce_host_shell_exec_dv(const char* in, size_t in_len, char* out, size_t cap);
|
||||||
@@ -284,6 +285,25 @@ String password_hash(String password)
|
|||||||
}
|
}
|
||||||
bool password_verify(String password, String encoded) { return(uce_host_password_verify(password.data(), password.size(), encoded.data(), encoded.size()) != 0); }
|
bool password_verify(String password, String encoded) { return(uce_host_password_verify(password.data(), password.size(), encoded.data(), encoded.size()) != 0); }
|
||||||
bool password_needs_rehash(String encoded) { return(uce_host_password_needs_rehash(encoded.data(), encoded.size()) != 0); }
|
bool password_needs_rehash(String encoded) { return(uce_host_password_needs_rehash(encoded.data(), encoded.size()) != 0); }
|
||||||
|
DValue crypto_operation(DValue request)
|
||||||
|
{
|
||||||
|
if(!crypto_operation_request_valid(request))
|
||||||
|
{
|
||||||
|
DValue result;
|
||||||
|
result["ok"].set_bool(false);
|
||||||
|
result["error"] = "invalid_request";
|
||||||
|
return(result);
|
||||||
|
}
|
||||||
|
String encoded = ucb_encode(request);
|
||||||
|
size_t required = uce_host_crypto_operation(encoded.data(), encoded.size(), 0, 0);
|
||||||
|
if(required == 0 || required > 64 * 1024)
|
||||||
|
return(DValue());
|
||||||
|
String out(required, 0);
|
||||||
|
size_t got = uce_host_crypto_operation(encoded.data(), encoded.size(), &out[0], required);
|
||||||
|
DValue result;
|
||||||
|
String error;
|
||||||
|
return(got <= required && ucb_decode(String(out.data(), got), result, &error) ? result : DValue());
|
||||||
|
}
|
||||||
String base64_decode(String raw) { return(wasm_string_hostcall_1(uce_host_base64_decode, raw)); }
|
String base64_decode(String raw) { return(wasm_string_hostcall_1(uce_host_base64_decode, raw)); }
|
||||||
String random_bytes(u64 n)
|
String random_bytes(u64 n)
|
||||||
{
|
{
|
||||||
@@ -305,6 +325,7 @@ u64 http_request_async(DValue req)
|
|||||||
return((u64)uce_host_http_request_async(encoded.data(), encoded.size()));
|
return((u64)uce_host_http_request_async(encoded.data(), encoded.size()));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
DValue shell_exec(DValue spec)
|
DValue shell_exec(DValue spec)
|
||||||
{
|
{
|
||||||
String encoded = ucb_encode(spec);
|
String encoded = ucb_encode(spec);
|
||||||
@@ -737,6 +758,7 @@ StringMap default_config()
|
|||||||
#include <fcntl.h>
|
#include <fcntl.h>
|
||||||
#include <limits.h>
|
#include <limits.h>
|
||||||
#include <sys/file.h>
|
#include <sys/file.h>
|
||||||
|
#include <poll.h>
|
||||||
#include <errno.h>
|
#include <errno.h>
|
||||||
#include "sys.h"
|
#include "sys.h"
|
||||||
#include "hash.h"
|
#include "hash.h"
|
||||||
@@ -752,10 +774,12 @@ bool crypto_equal(String a, String b) { return(crypto_equal_native(a, b)); }
|
|||||||
String password_hash(String password) { return(password_hash_native(password)); }
|
String password_hash(String password) { return(password_hash_native(password)); }
|
||||||
bool password_verify(String password, String encoded) { return(password_verify_native(password, encoded)); }
|
bool password_verify(String password, String encoded) { return(password_verify_native(password, encoded)); }
|
||||||
bool password_needs_rehash(String encoded) { return(password_needs_rehash_native(encoded)); }
|
bool password_needs_rehash(String encoded) { return(password_needs_rehash_native(encoded)); }
|
||||||
|
DValue crypto_operation(DValue request) { return(crypto_operation_native(request)); }
|
||||||
|
|
||||||
// Single definitions for the native split build (declared extern in sys.h).
|
// Single definitions for the native split build (declared extern in sys.h).
|
||||||
pid_t parent_pid = 0;
|
pid_t parent_pid = 0;
|
||||||
pid_t my_pid = 0;
|
pid_t my_pid = 0;
|
||||||
|
bool task_child_process = false;
|
||||||
|
|
||||||
namespace {
|
namespace {
|
||||||
|
|
||||||
@@ -1291,6 +1315,185 @@ String socket_read(u64 sockfd, u32 max_length, u32 timeout)
|
|||||||
return("");
|
return("");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
static bool memcache_decimal(String value, u64& result)
|
||||||
|
{
|
||||||
|
if(value == "")
|
||||||
|
return(false);
|
||||||
|
result = 0;
|
||||||
|
for(char c : value)
|
||||||
|
{
|
||||||
|
if(c < '0' || c > '9' || result > (UINT64_MAX - (u64)(c - '0')) / 10)
|
||||||
|
return(false);
|
||||||
|
result = result * 10 + (u64)(c - '0');
|
||||||
|
}
|
||||||
|
return(true);
|
||||||
|
}
|
||||||
|
|
||||||
|
static String memcache_command_line(const String& command)
|
||||||
|
{
|
||||||
|
size_t end = command.find("\r\n");
|
||||||
|
return(trim(command.substr(0, end)));
|
||||||
|
}
|
||||||
|
|
||||||
|
bool uce_memcache_command_has_no_reply(const String& command)
|
||||||
|
{
|
||||||
|
StringList words = split(memcache_command_line(command), " ");
|
||||||
|
if(words.size() == 0)
|
||||||
|
return(false);
|
||||||
|
String verb = to_lower(words[0]);
|
||||||
|
return(verb == "quit" || to_lower(words.back()) == "noreply");
|
||||||
|
}
|
||||||
|
|
||||||
|
MemcacheResponseState uce_memcache_response_advance(const String& command, const String& response, MemcacheResponseParser& parser)
|
||||||
|
{
|
||||||
|
while(true)
|
||||||
|
{
|
||||||
|
if(parser.mode == MemcacheResponseParser::Mode::FirstLine)
|
||||||
|
{
|
||||||
|
size_t line_end = response.find("\r\n");
|
||||||
|
if(line_end == String::npos)
|
||||||
|
return(MemcacheResponseState::Incomplete);
|
||||||
|
String line = response.substr(0, line_end);
|
||||||
|
if(line == "ERROR" || line.rfind("CLIENT_ERROR", 0) == 0 || line.rfind("SERVER_ERROR", 0) == 0 || line == "RESET" || line == "OK")
|
||||||
|
return(MemcacheResponseState::Complete);
|
||||||
|
if(line.rfind("VALUE ", 0) == 0)
|
||||||
|
parser.mode = MemcacheResponseParser::Mode::RetrievalHeader;
|
||||||
|
else if(line.rfind("VA ", 0) == 0)
|
||||||
|
{
|
||||||
|
StringList fields = split(line, " ");
|
||||||
|
u64 length = 0;
|
||||||
|
if(fields.size() < 2 || !memcache_decimal(fields[1], length) || length > SIZE_MAX - line_end - 4)
|
||||||
|
return(MemcacheResponseState::Malformed);
|
||||||
|
parser.data_end = line_end + 2 + (size_t)length;
|
||||||
|
parser.mode = MemcacheResponseParser::Mode::MetaData;
|
||||||
|
}
|
||||||
|
else
|
||||||
|
{
|
||||||
|
StringList request_words = split(memcache_command_line(command), " ");
|
||||||
|
String request_verb = request_words.size() ? to_lower(request_words[0]) : String("");
|
||||||
|
bool end_terminated = line.rfind("STAT ", 0) == 0 || line.rfind("ITEM ", 0) == 0 ||
|
||||||
|
request_verb == "stats" || (request_verb == "lru_crawler" && request_words.size() > 1 && to_lower(request_words[1]) == "metadump");
|
||||||
|
if(!end_terminated)
|
||||||
|
return(MemcacheResponseState::Complete);
|
||||||
|
parser.mode = MemcacheResponseParser::Mode::Lines;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if(parser.mode == MemcacheResponseParser::Mode::RetrievalHeader)
|
||||||
|
{
|
||||||
|
size_t line_end = response.find("\r\n", parser.offset);
|
||||||
|
if(line_end == String::npos)
|
||||||
|
return(MemcacheResponseState::Incomplete);
|
||||||
|
String line = response.substr(parser.offset, line_end - parser.offset);
|
||||||
|
if(line == "END")
|
||||||
|
return(MemcacheResponseState::Complete);
|
||||||
|
StringList fields = split(line, " ");
|
||||||
|
u64 length = 0;
|
||||||
|
if(fields.size() < 4 || fields[0] != "VALUE" || !memcache_decimal(fields[3], length) || length > SIZE_MAX - line_end - 4)
|
||||||
|
return(MemcacheResponseState::Malformed);
|
||||||
|
parser.data_end = line_end + 2 + (size_t)length;
|
||||||
|
parser.mode = MemcacheResponseParser::Mode::RetrievalData;
|
||||||
|
}
|
||||||
|
|
||||||
|
if(parser.mode == MemcacheResponseParser::Mode::RetrievalData)
|
||||||
|
{
|
||||||
|
if(response.size() < parser.data_end + 2)
|
||||||
|
return(MemcacheResponseState::Incomplete);
|
||||||
|
if(response.compare(parser.data_end, 2, "\r\n") != 0)
|
||||||
|
return(MemcacheResponseState::Malformed);
|
||||||
|
parser.offset = parser.data_end + 2;
|
||||||
|
parser.mode = MemcacheResponseParser::Mode::RetrievalHeader;
|
||||||
|
}
|
||||||
|
|
||||||
|
if(parser.mode == MemcacheResponseParser::Mode::MetaData)
|
||||||
|
{
|
||||||
|
if(response.size() < parser.data_end + 2)
|
||||||
|
return(MemcacheResponseState::Incomplete);
|
||||||
|
return(response.compare(parser.data_end, 2, "\r\n") == 0 ? MemcacheResponseState::Complete : MemcacheResponseState::Malformed);
|
||||||
|
}
|
||||||
|
|
||||||
|
if(parser.mode == MemcacheResponseParser::Mode::Lines)
|
||||||
|
{
|
||||||
|
size_t line_end = response.find("\r\n", parser.offset);
|
||||||
|
if(line_end == String::npos)
|
||||||
|
return(MemcacheResponseState::Incomplete);
|
||||||
|
if(response.compare(parser.offset, line_end - parser.offset, "END") == 0)
|
||||||
|
return(MemcacheResponseState::Complete);
|
||||||
|
parser.offset = line_end + 2;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool native_socket_wait(int fd, short events, u64 deadline)
|
||||||
|
{
|
||||||
|
while(true)
|
||||||
|
{
|
||||||
|
u64 now = monotonic_ms();
|
||||||
|
if(now >= deadline)
|
||||||
|
return(false);
|
||||||
|
pollfd item = {fd, events, 0};
|
||||||
|
int timeout = (int)std::min<u64>(deadline - now, INT_MAX);
|
||||||
|
int ready = poll(&item, 1, timeout);
|
||||||
|
if(ready > 0)
|
||||||
|
return((item.revents & events) != 0);
|
||||||
|
if(ready == 0 || errno != EINTR)
|
||||||
|
return(false);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
static String native_memcache_exchange(u64 connection, String command)
|
||||||
|
{
|
||||||
|
static constexpr size_t max_response = 8 * 1024 * 1024;
|
||||||
|
int fd = (int)connection;
|
||||||
|
u64 deadline = monotonic_ms() + 1000;
|
||||||
|
String request = command + "\r\n";
|
||||||
|
size_t sent = 0;
|
||||||
|
while(sent < request.size())
|
||||||
|
{
|
||||||
|
if(!native_socket_wait(fd, POLLOUT, deadline))
|
||||||
|
break;
|
||||||
|
ssize_t count = send(fd, request.data() + sent, request.size() - sent, MSG_DONTWAIT | MSG_NOSIGNAL);
|
||||||
|
if(count > 0)
|
||||||
|
sent += (size_t)count;
|
||||||
|
else if(count < 0 && (errno == EINTR || errno == EAGAIN || errno == EWOULDBLOCK))
|
||||||
|
continue;
|
||||||
|
else
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
if(sent != request.size())
|
||||||
|
{
|
||||||
|
shutdown(fd, SHUT_RDWR);
|
||||||
|
return("");
|
||||||
|
}
|
||||||
|
if(uce_memcache_command_has_no_reply(command))
|
||||||
|
return("");
|
||||||
|
|
||||||
|
String response;
|
||||||
|
response.reserve(16 * 1024);
|
||||||
|
MemcacheResponseParser parser;
|
||||||
|
while(response.size() < max_response)
|
||||||
|
{
|
||||||
|
MemcacheResponseState state = uce_memcache_response_advance(command, response, parser);
|
||||||
|
if(state == MemcacheResponseState::Complete)
|
||||||
|
return(response);
|
||||||
|
if(state == MemcacheResponseState::Malformed || !native_socket_wait(fd, POLLIN, deadline))
|
||||||
|
break;
|
||||||
|
char buffer[64 * 1024];
|
||||||
|
size_t remaining = max_response - response.size();
|
||||||
|
ssize_t count = recv(fd, buffer, std::min(remaining, sizeof(buffer)), MSG_DONTWAIT);
|
||||||
|
if(count > 0)
|
||||||
|
response.append(buffer, (size_t)count);
|
||||||
|
else if(count < 0 && (errno == EINTR || errno == EAGAIN || errno == EWOULDBLOCK))
|
||||||
|
continue;
|
||||||
|
else
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
if(uce_memcache_response_advance(command, response, parser) == MemcacheResponseState::Complete)
|
||||||
|
return(response);
|
||||||
|
shutdown(fd, SHUT_RDWR);
|
||||||
|
return("");
|
||||||
|
}
|
||||||
|
|
||||||
String memcache_escape_key(String key)
|
String memcache_escape_key(String key)
|
||||||
{
|
{
|
||||||
String result;
|
String result;
|
||||||
@@ -1320,26 +1523,18 @@ u64 memcache_connect(String host, u16 port)
|
|||||||
|
|
||||||
String memcache_command(u64 connection, String command)
|
String memcache_command(u64 connection, String command)
|
||||||
{
|
{
|
||||||
socket_write(connection, command+"\r\n");
|
return(native_memcache_exchange(connection, command));
|
||||||
return(socket_read(connection)); // FIXME: do multi-chunk until END line is received!
|
|
||||||
}
|
}
|
||||||
|
|
||||||
bool memcache_set(u64 connection, String key, String value, u64 expires_in)
|
bool memcache_set(u64 connection, String key, String value, u64 expires_in)
|
||||||
{
|
{
|
||||||
socket_write(connection,
|
return("STORED" == trim(memcache_command(connection,
|
||||||
// set KEY META_DATA EXPIRY_TIME LENGTH_IN_BYTES
|
String("set ") + memcache_escape_key(key) + " 0 " + std::to_string(expires_in) + " " + std::to_string(value.length()) + "\r\n" + value)));
|
||||||
String("set ") + memcache_escape_key(key) + " 0 " + std::to_string(expires_in) + " " + std::to_string(value.length()) + "\r\n" +
|
|
||||||
value + "\r\n");
|
|
||||||
return("STORED" == trim(socket_read(connection)));
|
|
||||||
}
|
}
|
||||||
|
|
||||||
bool memcache_delete(u64 connection, String key)
|
bool memcache_delete(u64 connection, String key)
|
||||||
{
|
{
|
||||||
socket_write(connection,
|
return("DELETED" == trim(memcache_command(connection, String("delete ") + memcache_escape_key(key))));
|
||||||
// set KEY META_DATA EXPIRY_TIME LENGTH_IN_BYTES
|
|
||||||
String("delete ") + memcache_escape_key(key) + "\r\n"
|
|
||||||
);
|
|
||||||
return("DELETED" == trim(socket_read(connection)));
|
|
||||||
}
|
}
|
||||||
|
|
||||||
String memcache_get(u64 connection, String key, String default_value)
|
String memcache_get(u64 connection, String key, String default_value)
|
||||||
@@ -1389,6 +1584,7 @@ void on_segfault(int sig)
|
|||||||
|
|
||||||
struct Worker {
|
struct Worker {
|
||||||
pid_t pid;
|
pid_t pid;
|
||||||
|
char listener_type = 0;
|
||||||
};
|
};
|
||||||
|
|
||||||
std::map<pid_t, Worker> workers;
|
std::map<pid_t, Worker> workers;
|
||||||
@@ -1617,27 +1813,41 @@ DValue process_exec(String cmd, String input, StringMap env, u64 timeout_ms, u64
|
|||||||
return(result);
|
return(result);
|
||||||
}
|
}
|
||||||
|
|
||||||
pid_t spawn_subprocess(std::function<void()> exec_after_spawn)
|
pid_t spawn_subprocess(std::function<void()> exec_after_spawn, char listener_type = 0)
|
||||||
{
|
{
|
||||||
parent_pid = getpid();
|
parent_pid = getpid();
|
||||||
pid_t p;
|
sigset_t blocked;
|
||||||
p = fork();
|
sigset_t previous;
|
||||||
|
sigemptyset(&blocked);
|
||||||
|
sigaddset(&blocked, SIGCHLD);
|
||||||
|
if(sigprocmask(SIG_BLOCK, &blocked, &previous) != 0)
|
||||||
|
{
|
||||||
|
perror("sigprocmask");
|
||||||
|
return(0);
|
||||||
|
}
|
||||||
|
pid_t p = fork();
|
||||||
if(p == 0)
|
if(p == 0)
|
||||||
{
|
{
|
||||||
|
sigprocmask(SIG_SETMASK, &previous, 0);
|
||||||
my_pid = getpid();
|
my_pid = getpid();
|
||||||
//printf("(C) child procress started, PID:%i\n", my_pid);
|
//printf("(C) child procress started, PID:%i\n", my_pid);
|
||||||
prctl(PR_SET_PDEATHSIG, SIGHUP);
|
prctl(PR_SET_PDEATHSIG, SIGHUP);
|
||||||
exec_after_spawn();
|
exec_after_spawn();
|
||||||
return(0);
|
return(0);
|
||||||
}
|
}
|
||||||
else
|
if(p < 0)
|
||||||
{
|
{
|
||||||
Worker w;
|
perror("fork worker");
|
||||||
w.pid = p;
|
sigprocmask(SIG_SETMASK, &previous, 0);
|
||||||
workers[w.pid] = w;
|
return(0);
|
||||||
printf("(P) child procress spawned: PID %i\n", p);
|
|
||||||
return(p);
|
|
||||||
}
|
}
|
||||||
|
Worker w;
|
||||||
|
w.pid = p;
|
||||||
|
w.listener_type = listener_type;
|
||||||
|
workers[w.pid] = w;
|
||||||
|
sigprocmask(SIG_SETMASK, &previous, 0);
|
||||||
|
printf("(P) child procress spawned: PID %i\n", p);
|
||||||
|
return(p);
|
||||||
}
|
}
|
||||||
|
|
||||||
String runtime_safe_key(String key, String label)
|
String runtime_safe_key(String key, String label)
|
||||||
@@ -1788,6 +1998,7 @@ pid_t task(String key, std::function<void()> exec_after_spawn, u64 timeout)
|
|||||||
{
|
{
|
||||||
close_locked_file(lock_fd);
|
close_locked_file(lock_fd);
|
||||||
my_pid = getpid();
|
my_pid = getpid();
|
||||||
|
task_child_process = true;
|
||||||
// The FastCGI worker handles termination to drain accepted requests.
|
// The FastCGI worker handles termination to drain accepted requests.
|
||||||
// Generic task children do not run that drain loop, so inheriting those
|
// Generic task children do not run that drain loop, so inheriting those
|
||||||
// handlers would turn task_kill(SIGTERM) into a no-op.
|
// handlers would turn task_kill(SIGTERM) into a no-op.
|
||||||
@@ -1904,6 +2115,9 @@ StringMap make_server_settings()
|
|||||||
cfg["WASM_MEMORY_LIMIT_BYTES"] = std::to_string(512ull * 1024 * 1024);
|
cfg["WASM_MEMORY_LIMIT_BYTES"] = std::to_string(512ull * 1024 * 1024);
|
||||||
cfg["WASM_EPOCH_DEADLINE_TICKS"] = "200";
|
cfg["WASM_EPOCH_DEADLINE_TICKS"] = "200";
|
||||||
cfg["WASM_EPOCH_PERIOD_MS"] = "50";
|
cfg["WASM_EPOCH_PERIOD_MS"] = "50";
|
||||||
|
cfg["WASM_SERIALIZE_TIMEOUT_SECONDS"] = "120";
|
||||||
|
cfg["MYSQL_PERSISTENT_POOL_SIZE"] = "8";
|
||||||
|
cfg["MYSQL_PERSISTENT_POOL_IDLE_TIMEOUT_SECONDS"] = "300";
|
||||||
cfg["SETUP_TEMPLATE"] = "scripts/setup.h.template";
|
cfg["SETUP_TEMPLATE"] = "scripts/setup.h.template";
|
||||||
cfg["LIT_ESC"] = "3d5b5_1";
|
cfg["LIT_ESC"] = "3d5b5_1";
|
||||||
cfg["CONTENT_TYPE"] = "text/html; charset=utf-8";
|
cfg["CONTENT_TYPE"] = "text/html; charset=utf-8";
|
||||||
@@ -1911,6 +2125,11 @@ StringMap make_server_settings()
|
|||||||
cfg["FCGI_SOCKET_MODE"] = "0666";
|
cfg["FCGI_SOCKET_MODE"] = "0666";
|
||||||
cfg["CLI_SOCKET_PATH"] = "/run/uce/cli.sock";
|
cfg["CLI_SOCKET_PATH"] = "/run/uce/cli.sock";
|
||||||
cfg["CLI_SOCKET_MODE"] = "0600";
|
cfg["CLI_SOCKET_MODE"] = "0600";
|
||||||
|
// Zero preserves the legacy shared renderer pool. A positive count keeps
|
||||||
|
// trusted CLI/test module caches out of public FastCGI workers.
|
||||||
|
cfg["CLI_WORKER_COUNT"] = "0";
|
||||||
|
// Bound retained CLI/test module state. Zero disables recycling.
|
||||||
|
cfg["CLI_WORKER_MAX_REQUESTS"] = "8";
|
||||||
// Command socket the WS broker listens on; workers flush ws_* dispatch
|
// Command socket the WS broker listens on; workers flush ws_* dispatch
|
||||||
// command batches here at workspace teardown.
|
// command batches here at workspace teardown.
|
||||||
cfg["WS_BROKER_SOCKET_PATH"] = "/run/uce/ws-broker.sock";
|
cfg["WS_BROKER_SOCKET_PATH"] = "/run/uce/ws-broker.sock";
|
||||||
|
|||||||
@@ -51,6 +51,7 @@ bool crypto_equal(String a, String b);
|
|||||||
String password_hash(String password);
|
String password_hash(String password);
|
||||||
bool password_verify(String password, String encoded);
|
bool password_verify(String password, String encoded);
|
||||||
bool password_needs_rehash(String encoded);
|
bool password_needs_rehash(String encoded);
|
||||||
|
DValue crypto_operation(DValue request);
|
||||||
String shell_escape(String raw);
|
String shell_escape(String raw);
|
||||||
String basename(String fn);
|
String basename(String fn);
|
||||||
String dirname(String fn);
|
String dirname(String fn);
|
||||||
@@ -128,6 +129,15 @@ String signal_name(s32 sig);
|
|||||||
String memcache_escape_key(String key);
|
String memcache_escape_key(String key);
|
||||||
StringList memcache_escape_keys(StringList keys);
|
StringList memcache_escape_keys(StringList keys);
|
||||||
u64 memcache_connect(String host = "127.0.0.1", u16 port = 11211);
|
u64 memcache_connect(String host = "127.0.0.1", u16 port = 11211);
|
||||||
|
enum class MemcacheResponseState { Incomplete, Complete, Malformed };
|
||||||
|
struct MemcacheResponseParser
|
||||||
|
{
|
||||||
|
enum class Mode { FirstLine, RetrievalHeader, RetrievalData, Lines, MetaData } mode = Mode::FirstLine;
|
||||||
|
size_t offset = 0;
|
||||||
|
size_t data_end = 0;
|
||||||
|
};
|
||||||
|
__attribute__((visibility("hidden"))) bool uce_memcache_command_has_no_reply(const String& command);
|
||||||
|
__attribute__((visibility("hidden"))) MemcacheResponseState uce_memcache_response_advance(const String& command, const String& response, MemcacheResponseParser& parser);
|
||||||
String memcache_command(u64 connection, String command);
|
String memcache_command(u64 connection, String command);
|
||||||
bool memcache_set(u64 connection, String key, String value, u64 expires_in = 60*60);
|
bool memcache_set(u64 connection, String key, String value, u64 expires_in = 60*60);
|
||||||
bool memcache_delete(u64 connection, String key);
|
bool memcache_delete(u64 connection, String key);
|
||||||
@@ -139,9 +149,11 @@ StringMap memcache_get_multiple(u64 connection, StringList keys);
|
|||||||
#if defined(__UCE_WASM_CORE__) || defined(__UCE_WASM_UNIT__)
|
#if defined(__UCE_WASM_CORE__) || defined(__UCE_WASM_UNIT__)
|
||||||
pid_t parent_pid = 0;
|
pid_t parent_pid = 0;
|
||||||
pid_t my_pid = 0;
|
pid_t my_pid = 0;
|
||||||
|
bool task_child_process = false;
|
||||||
#else
|
#else
|
||||||
extern pid_t parent_pid;
|
extern pid_t parent_pid;
|
||||||
extern pid_t my_pid;
|
extern pid_t my_pid;
|
||||||
|
extern bool task_child_process;
|
||||||
#endif
|
#endif
|
||||||
|
|
||||||
void on_segfault(int sig);
|
void on_segfault(int sig);
|
||||||
|
|||||||
+5
-1
@@ -97,7 +97,11 @@ String base64_decode(String raw, bool& ok)
|
|||||||
return("");
|
return("");
|
||||||
if(padding > 0 && i + 4 != cleaned.length())
|
if(padding > 0 && i + 4 != cleaned.length())
|
||||||
return("");
|
return("");
|
||||||
if(cleaned[i + 2] == '=' && cleaned[i + 3] != '=')
|
if(cleaned[i] == '=' || cleaned[i + 1] == '=')
|
||||||
|
return("");
|
||||||
|
if(padding == 1 && (cleaned[i + 2] == '=' || cleaned[i + 3] != '='))
|
||||||
|
return("");
|
||||||
|
if(padding == 2 && (cleaned[i + 2] != '=' || cleaned[i + 3] != '='))
|
||||||
return("");
|
return("");
|
||||||
|
|
||||||
result.append(1, (char)((values[0] << 2) | (values[1] >> 4)));
|
result.append(1, (char)((values[0] << 2) | (values[1] >> 4)));
|
||||||
|
|||||||
+112
-16
@@ -1132,7 +1132,8 @@ void run_ws_broker()
|
|||||||
if(server_state.config["WS_BROKER_SOCKET_PATH"] != "")
|
if(server_state.config["WS_BROKER_SOCKET_PATH"] != "")
|
||||||
{
|
{
|
||||||
ws_broker.listen(server_state.config["WS_BROKER_SOCKET_PATH"]);
|
ws_broker.listen(server_state.config["WS_BROKER_SOCKET_PATH"]);
|
||||||
chmod(server_state.config["WS_BROKER_SOCKET_PATH"].c_str(), S_IRWXU | S_IRGRP | S_IWGRP);
|
if(chmod(server_state.config["WS_BROKER_SOCKET_PATH"].c_str(), S_IRWXU | S_IRGRP | S_IWGRP) != 0)
|
||||||
|
fprintf(stderr, "(!) Could not chmod socket %s to %04o: %s\n", server_state.config["WS_BROKER_SOCKET_PATH"].c_str(), 0760, strerror(errno));
|
||||||
}
|
}
|
||||||
while(!termination_signal_received)
|
while(!termination_signal_received)
|
||||||
{
|
{
|
||||||
@@ -1265,6 +1266,58 @@ bool proactive_compile_queue_has(StringList& queue, String file_name)
|
|||||||
return(std::find(queue.begin(), queue.end(), file_name) != queue.end());
|
return(std::find(queue.begin(), queue.end(), file_name) != queue.end());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
String serialized_module_path(String wasm_path)
|
||||||
|
{
|
||||||
|
if(wasm_path.size() >= 5 && wasm_path.rfind(".wasm") == wasm_path.size() - 5)
|
||||||
|
return(wasm_path.substr(0, wasm_path.size() - 5) + ".cwasm");
|
||||||
|
return(wasm_path + ".cwasm");
|
||||||
|
}
|
||||||
|
|
||||||
|
void cleanup_dead_serialization_temps(String wasm_path)
|
||||||
|
{
|
||||||
|
String cached_path = serialized_module_path(wasm_path);
|
||||||
|
String directory = dirname(cached_path);
|
||||||
|
String prefix = basename(cached_path) + ".";
|
||||||
|
std::error_code ec;
|
||||||
|
for(auto const& entry : std::filesystem::directory_iterator(directory, ec))
|
||||||
|
{
|
||||||
|
if(ec)
|
||||||
|
break;
|
||||||
|
String name = entry.path().filename().string();
|
||||||
|
if(!str_starts_with(name, prefix) || name.size() <= prefix.size() + 4 || name.rfind(".tmp") != name.size() - 4)
|
||||||
|
continue;
|
||||||
|
String pid_text = name.substr(prefix.size(), name.size() - prefix.size() - 4);
|
||||||
|
char* end = 0;
|
||||||
|
errno = 0;
|
||||||
|
long pid = strtol(pid_text.c_str(), &end, 10);
|
||||||
|
if(pid <= 0 || errno == ERANGE || !end || *end != '\0')
|
||||||
|
continue;
|
||||||
|
if(kill((pid_t)pid, 0) != 0 && errno == ESRCH)
|
||||||
|
file_unlink(entry.path().string());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
String proactive_serialize_module(String wasm_path)
|
||||||
|
{
|
||||||
|
String executable = compiler_source_path_real("/proc/self/exe");
|
||||||
|
if(executable == "")
|
||||||
|
return("cannot resolve UCE executable for isolated serialization");
|
||||||
|
u64 timeout_seconds = std::max<u64>(1, std::min<u64>(3600,
|
||||||
|
to_u64(server_state.config["WASM_SERIALIZE_TIMEOUT_SECONDS"], 120)));
|
||||||
|
DValue execution = process_exec(
|
||||||
|
"exec " + shell_escape(executable) + " --serialize-module " + shell_escape(wasm_path),
|
||||||
|
"", StringMap(), timeout_seconds * 1000, 64 * 1024);
|
||||||
|
cleanup_dead_serialization_temps(wasm_path);
|
||||||
|
if(execution["timed_out"].to_bool())
|
||||||
|
return("serialized-module child timed out after " + std::to_string(timeout_seconds) + " seconds");
|
||||||
|
if(execution["exit_code"].to_s64() != 0)
|
||||||
|
{
|
||||||
|
String diagnostic = trim(first(execution["stderr"].to_string(), execution["stdout"].to_string()));
|
||||||
|
return(diagnostic == "" ? "serialized-module child failed" : diagnostic);
|
||||||
|
}
|
||||||
|
return("");
|
||||||
|
}
|
||||||
|
|
||||||
u64 bounded_compile_jobs(String value, u64 fallback = 2)
|
u64 bounded_compile_jobs(String value, u64 fallback = 2)
|
||||||
{
|
{
|
||||||
value = trim(value);
|
value = trim(value);
|
||||||
@@ -1318,7 +1371,7 @@ bool proactive_compile_unit(Request& context, String file_name, bool& source_mis
|
|||||||
if(!source_missing && !failed && wasm_serialized_module_needs_refresh(wasm_path))
|
if(!source_missing && !failed && wasm_serialized_module_needs_refresh(wasm_path))
|
||||||
{
|
{
|
||||||
printf("(i) proactive serialize %s\n", file_name.c_str());
|
printf("(i) proactive serialize %s\n", file_name.c_str());
|
||||||
String serialize_error = wasm_serialize_module_artifact(wasm_path);
|
String serialize_error = proactive_serialize_module(wasm_path);
|
||||||
if(serialize_error != "")
|
if(serialize_error != "")
|
||||||
{
|
{
|
||||||
printf("(!) proactive serialize failed for %s: %s\n", file_name.c_str(), serialize_error.c_str());
|
printf("(!) proactive serialize failed for %s: %s\n", file_name.c_str(), serialize_error.c_str());
|
||||||
@@ -1558,16 +1611,26 @@ void ensure_proactive_compiler()
|
|||||||
priority_compiler_pid = spawn_compiler("priority compiler", run_priority_compiler);
|
priority_compiler_pid = spawn_compiler("priority compiler", run_priority_compiler);
|
||||||
}
|
}
|
||||||
|
|
||||||
void listen_for_connections()
|
void listen_for_connections(char listener_type = 0)
|
||||||
{
|
{
|
||||||
install_process_fault_handlers();
|
install_process_fault_handlers();
|
||||||
// Workers are uniform FastCGI/CLI renderers; the WS broker owns the HTTP/WS
|
// The WS broker owns the HTTP/WS port. A configured dedicated CLI pool keeps
|
||||||
// port and every connection, so workers never accept raw HTTP themselves.
|
// test/admin module-cache churn out of public FastCGI renderers; zero keeps
|
||||||
server.close_http_listeners();
|
// the legacy shared listener set.
|
||||||
// The transport's legacy eight-connection recycle predates persistent
|
if(listener_type == 'F' || listener_type == 'C')
|
||||||
// Wasmtime engines. Recycling makes every ninth request pay engine/module
|
server.close_listeners_except(listener_type);
|
||||||
// startup; request-scoped workspaces already isolate and release user state.
|
else
|
||||||
|
server.close_http_listeners();
|
||||||
|
// Public workers keep their hot engines and modules. Dedicated CLI workers
|
||||||
|
// recycle so broad test/admin runs cannot retain an unbounded module set.
|
||||||
server.calls_until_termination = -1;
|
server.calls_until_termination = -1;
|
||||||
|
if(listener_type == 'C')
|
||||||
|
{
|
||||||
|
u64 max_requests = std::min<u64>(1024,
|
||||||
|
to_u64(server_state.config["CLI_WORKER_MAX_REQUESTS"], 8));
|
||||||
|
if(max_requests > 0)
|
||||||
|
server.calls_until_termination = (int)max_requests;
|
||||||
|
}
|
||||||
server.on_request = &handle_request;
|
server.on_request = &handle_request;
|
||||||
server.on_data = &handle_data;
|
server.on_data = &handle_data;
|
||||||
server.on_complete = &handle_complete;
|
server.on_complete = &handle_complete;
|
||||||
@@ -1578,7 +1641,8 @@ void listen_for_connections()
|
|||||||
String wasm_error = wasm_backend_start(startup_context);
|
String wasm_error = wasm_backend_start(startup_context);
|
||||||
f64 wasm_ms = (time_precise() - wasm_start) * 1000.0;
|
f64 wasm_ms = (time_precise() - wasm_start) * 1000.0;
|
||||||
if(wasm_error == "")
|
if(wasm_error == "")
|
||||||
printf("(P) wasm worker ready: PID %i in %.3f ms\n", getpid(), wasm_ms);
|
printf("(P) wasm %s worker ready: PID %i in %.3f ms\n",
|
||||||
|
listener_type == 'F' ? "FastCGI" : listener_type == 'C' ? "CLI" : "shared", getpid(), wasm_ms);
|
||||||
else
|
else
|
||||||
fprintf(stderr, "(!) wasm worker initialization failed: PID %i in %.3f ms: %s\n", getpid(), wasm_ms, wasm_error.c_str());
|
fprintf(stderr, "(!) wasm worker initialization failed: PID %i in %.3f ms: %s\n", getpid(), wasm_ms, wasm_error.c_str());
|
||||||
while(!termination_signal_received)
|
while(!termination_signal_received)
|
||||||
@@ -1870,9 +1934,11 @@ void print_fastcgi_usage(FILE* stream)
|
|||||||
{
|
{
|
||||||
fprintf(stream,
|
fprintf(stream,
|
||||||
"Usage: uce_fastcgi [--precompile]\n"
|
"Usage: uce_fastcgi [--precompile]\n"
|
||||||
|
" uce_fastcgi --serialize-module PATH\n"
|
||||||
" uce_fastcgi --help\n\n"
|
" uce_fastcgi --help\n\n"
|
||||||
"Without options, start the FastCGI server.\n"
|
"Without options, start the FastCGI server.\n"
|
||||||
" --precompile Compile the current source generation without starting listeners.\n"
|
" --precompile Compile the current source generation without starting listeners.\n"
|
||||||
|
" --serialize-module PATH Serialize one Wasm artifact in an isolated process.\n"
|
||||||
" -h, --help Show this help and exit.\n");
|
" -h, --help Show this help and exit.\n");
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1888,7 +1954,8 @@ int main(int argc, char** argv)
|
|||||||
return(0);
|
return(0);
|
||||||
}
|
}
|
||||||
bool precompile = argc == 2 && String(argv[1]) == "--precompile";
|
bool precompile = argc == 2 && String(argv[1]) == "--precompile";
|
||||||
if(argc != 1 && !precompile)
|
bool serialize_module = argc == 3 && String(argv[1]) == "--serialize-module";
|
||||||
|
if(argc != 1 && !precompile && !serialize_module)
|
||||||
{
|
{
|
||||||
fprintf(stderr, "invalid arguments\n");
|
fprintf(stderr, "invalid arguments\n");
|
||||||
print_fastcgi_usage(stderr);
|
print_fastcgi_usage(stderr);
|
||||||
@@ -1898,6 +1965,16 @@ int main(int argc, char** argv)
|
|||||||
// after a fault does not allocate.
|
// after a fault does not allocate.
|
||||||
backtrace(request_fault_frames, 4);
|
backtrace(request_fault_frames, 4);
|
||||||
process_start_directory();
|
process_start_directory();
|
||||||
|
if(serialize_module)
|
||||||
|
{
|
||||||
|
String error = wasm_serialize_module_artifact(argv[2]);
|
||||||
|
if(error != "")
|
||||||
|
{
|
||||||
|
fprintf(stderr, "%s\n", error.c_str());
|
||||||
|
return(1);
|
||||||
|
}
|
||||||
|
return(0);
|
||||||
|
}
|
||||||
if(precompile)
|
if(precompile)
|
||||||
return(precompile_unit_generation());
|
return(precompile_unit_generation());
|
||||||
|
|
||||||
@@ -1923,11 +2000,30 @@ int main(int argc, char** argv)
|
|||||||
if(!termination_signal_received)
|
if(!termination_signal_received)
|
||||||
ensure_ws_broker();
|
ensure_ws_broker();
|
||||||
|
|
||||||
while(workers.size() < int_val(server_state.config["WORKER_COUNT"]))
|
u64 cli_worker_count = server_state.config["CLI_SOCKET_PATH"] == "" ? 0 :
|
||||||
{
|
std::min<u64>(16, to_u64(server_state.config["CLI_WORKER_COUNT"], 0));
|
||||||
if(!termination_signal_received)
|
bool has_public_listener = false;
|
||||||
spawn_subprocess(listen_for_connections);
|
for(auto& listener : server.server_socket_types)
|
||||||
}
|
if(listener.second == 'F')
|
||||||
|
has_public_listener = true;
|
||||||
|
u64 configured_worker_count = std::max<s64>(1, int_val(server_state.config["WORKER_COUNT"]));
|
||||||
|
u64 public_worker_count = cli_worker_count > 0 && !has_public_listener ? 0 :
|
||||||
|
(server.server_sockets.empty() ? 0 : configured_worker_count);
|
||||||
|
auto role_count = [&](char listener_type) {
|
||||||
|
u64 count = 0;
|
||||||
|
for(auto& worker : workers)
|
||||||
|
if(worker.second.listener_type == listener_type)
|
||||||
|
count++;
|
||||||
|
return(count);
|
||||||
|
};
|
||||||
|
auto spawn_listener_worker = [&](char listener_type) {
|
||||||
|
return(spawn_subprocess([listener_type]() { listen_for_connections(listener_type); }, listener_type));
|
||||||
|
};
|
||||||
|
char public_listener_type = cli_worker_count > 0 ? 'F' : 0;
|
||||||
|
while(!termination_signal_received && role_count(public_listener_type) < public_worker_count)
|
||||||
|
if(spawn_listener_worker(public_listener_type) <= 0) { sleep(1); break; }
|
||||||
|
while(!termination_signal_received && role_count('C') < cli_worker_count)
|
||||||
|
if(spawn_listener_worker('C') <= 0) { sleep(1); break; }
|
||||||
sleep(1);
|
sleep(1);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
+11
-2
@@ -121,6 +121,13 @@ static String wasm_backend_ensure_started(Request* context)
|
|||||||
return(g_wasm_init_error);
|
return(g_wasm_init_error);
|
||||||
}
|
}
|
||||||
wc.mysql_persistent_pool_size = std::min<u64>(to_u64(cfg["MYSQL_PERSISTENT_POOL_SIZE"], 8), 64);
|
wc.mysql_persistent_pool_size = std::min<u64>(to_u64(cfg["MYSQL_PERSISTENT_POOL_SIZE"], 8), 64);
|
||||||
|
u64 mysql_idle_timeout = to_u64(first(cfg["MYSQL_PERSISTENT_POOL_IDLE_TIMEOUT_SECONDS"], "300"), UINT64_MAX);
|
||||||
|
if(mysql_idle_timeout == UINT64_MAX || mysql_idle_timeout > 86400)
|
||||||
|
{
|
||||||
|
g_wasm_init_error = "MYSQL_PERSISTENT_POOL_IDLE_TIMEOUT_SECONDS must be an integer no greater than 86400";
|
||||||
|
return(g_wasm_init_error);
|
||||||
|
}
|
||||||
|
wc.mysql_persistent_pool_idle_timeout_seconds = mysql_idle_timeout;
|
||||||
wc.profile_hostcall_cpu = to_bool(cfg["WASM_PROFILE_HOSTCALL_CPU"], false);
|
wc.profile_hostcall_cpu = to_bool(cfg["WASM_PROFILE_HOSTCALL_CPU"], false);
|
||||||
wc.profile_thread_runtime = to_bool(cfg["WASM_PROFILE_THREAD_RUNTIME"], false);
|
wc.profile_thread_runtime = to_bool(cfg["WASM_PROFILE_THREAD_RUNTIME"], false);
|
||||||
wc.verbose = to_bool(cfg["WASM_BACKEND_VERBOSE"], false);
|
wc.verbose = to_bool(cfg["WASM_BACKEND_VERBOSE"], false);
|
||||||
@@ -422,8 +429,8 @@ String wasm_backend_serve(Request& request, const String& entry_unit, const Stri
|
|||||||
return("");
|
return("");
|
||||||
}
|
}
|
||||||
|
|
||||||
// Stop the ticker before the worker process exits (best-effort; forked workers
|
// Stop the ticker before destroying the worker state it references. Deleting
|
||||||
// are usually killed, but a clean ager-out path should join the thread).
|
// the worker also closes retained connector resources on a clean process exit.
|
||||||
void wasm_backend_shutdown()
|
void wasm_backend_shutdown()
|
||||||
{
|
{
|
||||||
if(g_wasm_epoch_ticker)
|
if(g_wasm_epoch_ticker)
|
||||||
@@ -434,4 +441,6 @@ void wasm_backend_shutdown()
|
|||||||
delete g_wasm_epoch_ticker;
|
delete g_wasm_epoch_ticker;
|
||||||
g_wasm_epoch_ticker = 0;
|
g_wasm_epoch_ticker = 0;
|
||||||
}
|
}
|
||||||
|
delete g_wasm_worker;
|
||||||
|
g_wasm_worker = 0;
|
||||||
}
|
}
|
||||||
|
|||||||
+42
-3
@@ -207,6 +207,7 @@ String MySQL::parse_query_parameters(String query, StringMap map)
|
|||||||
|
|
||||||
u8 mode = 0;
|
u8 mode = 0;
|
||||||
char quote = 0;
|
char quote = 0;
|
||||||
|
bool escaped = false;
|
||||||
String identifier;
|
String identifier;
|
||||||
for(u32 i = 0; i < query.length(); i++)
|
for(u32 i = 0; i < query.length(); i++)
|
||||||
{
|
{
|
||||||
@@ -223,14 +224,30 @@ String MySQL::parse_query_parameters(String query, StringMap map)
|
|||||||
result.append(1, c);
|
result.append(1, c);
|
||||||
mode = 2;
|
mode = 2;
|
||||||
quote = c;
|
quote = c;
|
||||||
|
escaped = false;
|
||||||
}
|
}
|
||||||
else
|
else
|
||||||
result.append(1, c);
|
result.append(1, c);
|
||||||
}
|
}
|
||||||
else if(mode == 1)
|
else if(mode == 1)
|
||||||
{
|
{
|
||||||
if(isalnum(c) || c == '_')
|
if(isalnum((unsigned char)c) || c == '_')
|
||||||
identifier.append(1, c);
|
identifier.append(1, c);
|
||||||
|
else if(c == '!' && query[i + 1] != '=')
|
||||||
|
{
|
||||||
|
String value = map[identifier];
|
||||||
|
bool valid = identifier != "" && value != "";
|
||||||
|
for(char digit : value)
|
||||||
|
if(!isdigit((unsigned char)digit)) valid = false;
|
||||||
|
if(!valid)
|
||||||
|
{
|
||||||
|
parameter_error = true;
|
||||||
|
statement_info = "mysql unsigned parameter :" + identifier + "! must contain only decimal digits";
|
||||||
|
return("");
|
||||||
|
}
|
||||||
|
result.append(value);
|
||||||
|
mode = 0;
|
||||||
|
}
|
||||||
else
|
else
|
||||||
{
|
{
|
||||||
result.append(escape(map[identifier]));
|
result.append(escape(map[identifier]));
|
||||||
@@ -240,9 +257,19 @@ String MySQL::parse_query_parameters(String query, StringMap map)
|
|||||||
}
|
}
|
||||||
else if(mode == 2)
|
else if(mode == 2)
|
||||||
{
|
{
|
||||||
|
result.append(1, c);
|
||||||
|
if(escaped)
|
||||||
|
{
|
||||||
|
escaped = false;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if(c == '\\')
|
||||||
|
{
|
||||||
|
escaped = true;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
if(c == quote)
|
if(c == quote)
|
||||||
mode = 0;
|
mode = 0;
|
||||||
result.append(1, c);
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -315,7 +342,17 @@ DValue MySQL::query(String q)
|
|||||||
return(result ? *result : DValue());
|
return(result ? *result : DValue());
|
||||||
}
|
}
|
||||||
|
|
||||||
DValue MySQL::query(String q, StringMap params) { return(query(parse_query_parameters(q, params))); }
|
DValue MySQL::query(String q, StringMap params)
|
||||||
|
{
|
||||||
|
parameter_error = false;
|
||||||
|
String parsed = parse_query_parameters(q, params);
|
||||||
|
if(parameter_error)
|
||||||
|
{
|
||||||
|
_preload_next_error_code = 2000;
|
||||||
|
return(DValue());
|
||||||
|
}
|
||||||
|
return(query(parsed));
|
||||||
|
}
|
||||||
DValue MySQL::get_pending_result() { return(DValue()); }
|
DValue MySQL::get_pending_result() { return(DValue()); }
|
||||||
|
|
||||||
// sqlite runs host-side (the host links libsqlite and owns the connections in
|
// sqlite runs host-side (the host links libsqlite and owns the connections in
|
||||||
@@ -461,6 +498,8 @@ extern "C" void uce_wasm_link_anchors()
|
|||||||
(void*)(double (*)(double))&cos,
|
(void*)(double (*)(double))&cos,
|
||||||
(void*)(double (*)(double))&sin,
|
(void*)(double (*)(double))&sin,
|
||||||
(void*)(double (*)(double))&round,
|
(void*)(double (*)(double))&round,
|
||||||
|
// structured crypto dispatcher is hostcall-backed and otherwise unreferenced by core
|
||||||
|
(void*)&crypto_operation,
|
||||||
};
|
};
|
||||||
(void)libc_anchors;
|
(void)libc_anchors;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -13,6 +13,7 @@ uce_host_crypto_equal
|
|||||||
uce_host_password_hash
|
uce_host_password_hash
|
||||||
uce_host_password_verify
|
uce_host_password_verify
|
||||||
uce_host_password_needs_rehash
|
uce_host_password_needs_rehash
|
||||||
|
uce_host_crypto_operation
|
||||||
uce_host_task_spawn
|
uce_host_task_spawn
|
||||||
uce_host_task_pid
|
uce_host_task_pid
|
||||||
uce_host_task_kill
|
uce_host_task_kill
|
||||||
|
|||||||
@@ -0,0 +1,146 @@
|
|||||||
|
// Internal worker-local implementation for opt-in generic hardened HTTP.
|
||||||
|
#pragma once
|
||||||
|
#include <arpa/inet.h>
|
||||||
|
#include <cctype>
|
||||||
|
#include <cerrno>
|
||||||
|
#include <fcntl.h>
|
||||||
|
#include <functional>
|
||||||
|
#include <signal.h>
|
||||||
|
#include <sys/wait.h>
|
||||||
|
#include <sys/syscall.h>
|
||||||
|
#include <unistd.h>
|
||||||
|
|
||||||
|
static u64 hardened_http_monotonic_ms() { timespec ts{}; clock_gettime(CLOCK_MONOTONIC,&ts); return (u64)ts.tv_sec*1000ull+(u64)ts.tv_nsec/1000000ull; }
|
||||||
|
static void hardened_http_close_inherited_fds()
|
||||||
|
{
|
||||||
|
#ifdef SYS_close_range
|
||||||
|
if(syscall(SYS_close_range,4u,~0u,0u)==0) return;
|
||||||
|
#endif
|
||||||
|
long max_fd=sysconf(_SC_OPEN_MAX); if(max_fd<4) max_fd=4;
|
||||||
|
for(int fd=4;fd<max_fd;fd++) close(fd);
|
||||||
|
}
|
||||||
|
|
||||||
|
struct HardenedHttpExecResult { int exit_code=-1; bool timed_out=false, output_limited=false; String headers_text, body_text, stderr_text; };
|
||||||
|
struct HardenedHttpHooks {
|
||||||
|
std::function<std::vector<String>(String)> resolve;
|
||||||
|
std::function<HardenedHttpExecResult(std::vector<String>, String, std::vector<String>, u64, size_t)> execute;
|
||||||
|
};
|
||||||
|
|
||||||
|
// Hardened requests deliberately support public IPv4 answers only. Every IPv6
|
||||||
|
// answer fails closed until this generic transport has a reviewed IPv6 policy.
|
||||||
|
// IPv4 must be globally unicast: reject IANA special-purpose 0/8, 10/8,
|
||||||
|
// 100.64/10, 127/8, 169.254/16, 172.16/12, 192.0.0/24, 192.0.2/24,
|
||||||
|
// 192.31.196/24, 192.52.193/24, 192.88.99/24, 192.175.48/24, 192.168/16,
|
||||||
|
// 198.18/15, 198.51.100/24, 203.0.113/24, and all 224/4 multicast/reserved.
|
||||||
|
static bool hardened_http_public_address(String text)
|
||||||
|
{
|
||||||
|
in_addr v4{};
|
||||||
|
if(inet_pton(AF_INET,text.c_str(),&v4)!=1) return false; // Includes every AAAA candidate.
|
||||||
|
u32 x=ntohl(v4.s_addr); u8 a=x>>24,b=x>>16,c=x>>8;
|
||||||
|
if(a==0||a==10||a==127||a>=224||(a==100&&b>=64&&b<=127)||(a==169&&b==254)||(a==172&&b>=16&&b<=31)||(a==192&&(b==0||b==2||b==168))||(a==192&&b==31&&c==196)||(a==192&&b==52&&c==193)||(a==192&&b==88&&c==99)||(a==192&&b==175&&c==48)||(a==198&&(b==18||b==19||b==51))||(a==203&&b==0&&c==113)) return false;
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
// Three private pipes keep curl's -D headers and -o body distinct. The child
|
||||||
|
// inherits only stdin/stdout/stderr and the header pipe on fd 3. Async workers
|
||||||
|
// retain their own process group so job_cancel can kill curl and its descendants.
|
||||||
|
static HardenedHttpExecResult hardened_http_exec_argv_capture(std::vector<String> argv, String input, u64 timeout_ms, size_t output_limit, bool clean_env=true, bool own_process_group=true)
|
||||||
|
{
|
||||||
|
HardenedHttpExecResult r; if(argv.empty()) return r;
|
||||||
|
int in[2], body[2], headers[2], err[2];
|
||||||
|
if(pipe(in)||pipe(body)||pipe(headers)||pipe(err)) return r;
|
||||||
|
pid_t pid=fork();
|
||||||
|
if(pid==0) {
|
||||||
|
if(own_process_group) setpgid(0,0);
|
||||||
|
dup2(in[0],0); dup2(body[1],1); dup2(err[1],2); dup2(headers[1],3);
|
||||||
|
hardened_http_close_inherited_fds();
|
||||||
|
std::vector<char*> args; for(String& a:argv) args.push_back((char*)a.c_str()); args.push_back(0);
|
||||||
|
if(clean_env) { clearenv(); setenv("PATH","/usr/bin:/bin",1); execv(args[0],args.data()); } else execvp(args[0],args.data());
|
||||||
|
_exit(127);
|
||||||
|
}
|
||||||
|
auto close_all=[&](){ for(int fd: {in[0],in[1],body[0],body[1],headers[0],headers[1],err[0],err[1]}) close(fd); };
|
||||||
|
if(pid<0) { close_all(); return r; }
|
||||||
|
if(own_process_group) setpgid(pid,pid);
|
||||||
|
close(in[0]); close(body[1]); close(headers[1]); close(err[1]);
|
||||||
|
for(int fd: {in[1],body[0],headers[0],err[0]}) fcntl(fd,F_SETFL,fcntl(fd,F_GETFL,0)|O_NONBLOCK);
|
||||||
|
size_t input_off=0, total=0; bool in_open=true,body_open=true,headers_open=true,err_open=true,exited=false,killed=false; int status=0; u64 deadline=hardened_http_monotonic_ms()+timeout_ms;
|
||||||
|
auto terminate=[&](bool timeout){ if(timeout) r.timed_out=true; if(!killed) { if(own_process_group) kill(-pid,SIGKILL); kill(pid,SIGKILL); killed=true; } };
|
||||||
|
auto drain=[&](int fd,bool& open,String& dst) { char buf[4096]; ssize_t n; while((n=read(fd,buf,sizeof(buf)))>0) { if(total+(size_t)n>output_limit) { r.output_limited=true; terminate(false); } else { dst+=String(buf,n); total+=(size_t)n; } } if(n==0) { close(fd); open=false; } };
|
||||||
|
while(in_open||body_open||headers_open||err_open||!exited) {
|
||||||
|
if(!exited) { pid_t w=waitpid(pid,&status,WNOHANG); if(w==pid) exited=true; }
|
||||||
|
if(in_open) { if(input_off<input.size()) { ssize_t n=write(in[1],input.data()+input_off,input.size()-input_off); if(n>0) input_off+=(size_t)n; else if(n<0&&errno!=EINTR&&errno!=EAGAIN&&errno!=EWOULDBLOCK) { close(in[1]); in_open=false; } } else { close(in[1]); in_open=false; } }
|
||||||
|
drain(body[0],body_open,r.body_text); drain(headers[0],headers_open,r.headers_text); drain(err[0],err_open,r.stderr_text);
|
||||||
|
if(!killed&&hardened_http_monotonic_ms()>=deadline) terminate(true);
|
||||||
|
if(killed&&!exited) { while(waitpid(pid,&status,0)<0&&errno==EINTR) {} exited=true; }
|
||||||
|
if(in_open||body_open||headers_open||err_open||!exited) usleep(1000);
|
||||||
|
}
|
||||||
|
if(WIFEXITED(status)) r.exit_code=WEXITSTATUS(status); else if(WIFSIGNALED(status)) r.exit_code=128+WTERMSIG(status);
|
||||||
|
return r;
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool hardened_http_parse_headers(String text, u64& status, DValue& filtered)
|
||||||
|
{
|
||||||
|
status=0; filtered.set_array(); size_t pos=0; bool found=false;
|
||||||
|
while(pos<text.size()) {
|
||||||
|
size_t end=text.find("\r\n\r\n",pos); size_t sep=4; if(end==String::npos) { end=text.find("\n\n",pos); sep=2; } if(end==String::npos) return false;
|
||||||
|
String block=replace(text.substr(pos,end-pos),"\r",""); size_t nl=block.find('\n'); String first=nl==String::npos?block:block.substr(0,nl);
|
||||||
|
if(first.size()<12||first.rfind("HTTP/",0)!=0||first[8]!=' '||!isdigit((unsigned char)first[9])||!isdigit((unsigned char)first[10])||!isdigit((unsigned char)first[11])) return false;
|
||||||
|
status=strtoull(first.substr(9,3).c_str(),0,10); DValue current; current.set_array();
|
||||||
|
if(nl!=String::npos) for(String line:split(block.substr(nl+1),"\n")) { size_t c=line.find(':'); if(c==String::npos) return false; String k=to_lower(trim(line.substr(0,c))); if(k=="content-type"||k=="cache-control") current[k]=trim(line.substr(c+1)); }
|
||||||
|
filtered=current; found=true; pos=end+sep;
|
||||||
|
}
|
||||||
|
return found;
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool hardened_http_token(String text)
|
||||||
|
{
|
||||||
|
if(text=="") return false;
|
||||||
|
for(unsigned char c:text) if(!(isalnum(c)||c=='-'||c=='_')) return false;
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
static bool hardened_http_header_value(String text)
|
||||||
|
{
|
||||||
|
if(text.size()>8192||text.find('\0')!=String::npos||text.find('\r')!=String::npos||text.find('\n')!=String::npos) return false;
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
static bool hardened_http_ip_literal(String text)
|
||||||
|
{
|
||||||
|
in_addr v4{}; in6_addr v6{};
|
||||||
|
return inet_pton(AF_INET,text.c_str(),&v4)==1 || inet_pton(AF_INET6,text.c_str(),&v6)==1;
|
||||||
|
}
|
||||||
|
static bool hardened_http_url(String url, String& host, String& port)
|
||||||
|
{
|
||||||
|
if(url.find('\0')!=String::npos||url.find_first_of("\\ \r\n\t")!=String::npos||url.rfind("https://",0)!=0) return false;
|
||||||
|
String rest=url.substr(8); size_t slash=rest.find('/'); String authority=slash==String::npos?rest:rest.substr(0,slash);
|
||||||
|
if(authority==""||authority.find('@')!=String::npos||authority.find('[')!=String::npos||authority.find(']')!=String::npos) return false;
|
||||||
|
size_t colon=authority.rfind(':'); host=colon==String::npos?authority:authority.substr(0,colon); port=colon==String::npos?"443":authority.substr(colon+1);
|
||||||
|
if(host==""||hardened_http_ip_literal(host)||!hardened_http_token(port)) return false;
|
||||||
|
for(unsigned char c:host) if(!(isalnum(c)||c=='.'||c=='-')) return false;
|
||||||
|
char* end=0; unsigned long n=strtoul(port.c_str(),&end,10); return end!=port.c_str()&&*end==0&&n>0&&n<=65535;
|
||||||
|
}
|
||||||
|
static bool hardened_http_security_requested(const DValue* security)
|
||||||
|
{
|
||||||
|
return security && (security->key("https_only") || security->key("public_dns_only") || security->key("pin_dns") || security->key("isolated_curl") || security->key("no_redirects"));
|
||||||
|
}
|
||||||
|
static bool hardened_http_true(const DValue* value)
|
||||||
|
{
|
||||||
|
return value && value->get_type_name()=="bool" && value->to_bool();
|
||||||
|
}
|
||||||
|
static DValue hardened_http_request_internal(const DValue& req, u64 timeout_ms, const HardenedHttpHooks& hooks)
|
||||||
|
{
|
||||||
|
DValue r; r["status"]=(f64)0; r["headers"].set_array(); r["body"]=""; r["error"]="";
|
||||||
|
const DValue* sec=req.key("security");
|
||||||
|
if(req.key("follow_redirects") && req.key("follow_redirects")->to_bool()) { r["error"]="invalid_request"; return r; }
|
||||||
|
timeout_ms=std::min<u64>(std::max<u64>(1,timeout_ms),10000);
|
||||||
|
if(!sec || !hardened_http_true(sec->key("https_only")) || !hardened_http_true(sec->key("public_dns_only")) || !hardened_http_true(sec->key("pin_dns")) || !hardened_http_true(sec->key("isolated_curl")) || !hardened_http_true(sec->key("no_redirects"))) { r["error"]="invalid_request"; return r; }
|
||||||
|
String method=req.key("method")?req.key("method")->to_string():"GET"; String url=req.key("url")?req.key("url")->to_string():""; String host,port;
|
||||||
|
if((method!="GET"&&method!="POST"&&method!="PUT"&&method!="PATCH"&&method!="DELETE"&&method!="HEAD"&&method!="OPTIONS")||!hardened_http_url(url,host,port)) { r["error"]="invalid_request"; return r; }
|
||||||
|
String body=req.key("body")?req.key("body")->to_string():""; if(body.size()>65536||body.find('\0')!=String::npos) { r["error"]="invalid_request"; return r; }
|
||||||
|
std::vector<String> argv={"/usr/bin/curl","--disable","-sS","--http1.1","--proto","=https","--proto-redir","=https","--noproxy","*","--proxy","","--alt-svc","","--hsts","","--cacert","/etc/ssl/certs/ca-certificates.crt","--connect-timeout","3","--max-time",std::to_string(std::max<u64>(1,timeout_ms/1000)),"--max-filesize","65536","-X",method,"-D","/proc/self/fd/3","-o","/proc/self/fd/1"};
|
||||||
|
const DValue* hs=req.key("headers"); if(hs) { bool valid=true; hs->each([&](const DValue& v,String k) { String value=v.to_string(), lower=to_lower(k); if(!hardened_http_token(k)||!hardened_http_header_value(value)||lower=="host"||lower=="content-length"||lower=="transfer-encoding"||lower=="connection"||lower=="proxy-connection"||lower=="expect") valid=false; else { argv.push_back("-H"); argv.push_back(k+": "+value); } }); if(!valid) { r["error"]="invalid_request"; return r; } }
|
||||||
|
std::vector<String> answers=hooks.resolve(host); String address; for(String a:answers) { if(!hardened_http_public_address(a)) { r["error"]="unsafe_dns"; return r; } if(address=="") address=a; } if(address=="") { r["error"]="unsafe_dns"; return r; }
|
||||||
|
argv.push_back("--resolve"); argv.push_back(host+":"+port+":"+address); if(req.key("body")) { argv.push_back("--data-binary"); argv.push_back("@-"); } argv.push_back(url);
|
||||||
|
HardenedHttpExecResult pr=hooks.execute(argv,body,{"PATH=/usr/bin:/bin"},timeout_ms,72*1024); if(pr.output_limited||pr.body_text.size()>65536||pr.headers_text.size()>8192) { r["error"]="response_too_large"; return r; }
|
||||||
|
u64 status; DValue headers; if(!hardened_http_parse_headers(pr.headers_text,status,headers)) { r["error"]=pr.timed_out?"timeout":"malformed_output"; return r; }
|
||||||
|
r["status"]=(f64)status; r["headers"]=headers; r["body"]=pr.body_text;
|
||||||
|
if(status/100==3) r["error"]="redirect_not_allowed"; else if(pr.exit_code!=0) r["error"]=pr.timed_out?"timeout":"network_failure"; else if(status/100!=2) r["error"]="http_status"; return r;
|
||||||
|
}
|
||||||
+302
-60
@@ -39,6 +39,7 @@
|
|||||||
#include <string>
|
#include <string>
|
||||||
#include <arpa/inet.h>
|
#include <arpa/inet.h>
|
||||||
#include <netinet/in.h>
|
#include <netinet/in.h>
|
||||||
|
#include <netdb.h>
|
||||||
#include <sys/socket.h>
|
#include <sys/socket.h>
|
||||||
#include <sys/stat.h>
|
#include <sys/stat.h>
|
||||||
#include <sys/file.h>
|
#include <sys/file.h>
|
||||||
@@ -55,6 +56,7 @@
|
|||||||
#include <sys/wait.h>
|
#include <sys/wait.h>
|
||||||
#include <signal.h>
|
#include <signal.h>
|
||||||
#include <poll.h>
|
#include <poll.h>
|
||||||
|
#include "hardened_http_internal.h"
|
||||||
|
|
||||||
struct WasmDylinkInfo
|
struct WasmDylinkInfo
|
||||||
{
|
{
|
||||||
@@ -153,6 +155,7 @@ struct WasmWorkerConfig
|
|||||||
u64 epoch_period_ms = 50;
|
u64 epoch_period_ms = 50;
|
||||||
u64 invocation_timeout_ms = 30000;
|
u64 invocation_timeout_ms = 30000;
|
||||||
u64 mysql_persistent_pool_size = 8;
|
u64 mysql_persistent_pool_size = 8;
|
||||||
|
u64 mysql_persistent_pool_idle_timeout_seconds = 300;
|
||||||
bool profile_hostcall_cpu = false;
|
bool profile_hostcall_cpu = false;
|
||||||
bool profile_thread_runtime = false;
|
bool profile_thread_runtime = false;
|
||||||
bool verbose = false;
|
bool verbose = false;
|
||||||
@@ -389,15 +392,12 @@ static u64 wasm_socket_connect_bounded(const String& host, u16 port, u64 timeout
|
|||||||
}
|
}
|
||||||
if(fd <= 0)
|
if(fd <= 0)
|
||||||
return(0);
|
return(0);
|
||||||
if(context)
|
|
||||||
context->resources.sockets.push_back(fd);
|
|
||||||
return((u64)fd);
|
return((u64)fd);
|
||||||
}
|
}
|
||||||
|
|
||||||
static bool wasm_socket_write_bounded(u64 socket_fd, const String& data, u64 timeout_ms)
|
static bool wasm_socket_write_until(u64 socket_fd, const String& data, u64 deadline)
|
||||||
{
|
{
|
||||||
int fd = (int)socket_fd;
|
int fd = (int)socket_fd;
|
||||||
u64 deadline = wasm_deadline_after_ms(timeout_ms);
|
|
||||||
size_t offset = 0;
|
size_t offset = 0;
|
||||||
while(offset < data.size())
|
while(offset < data.size())
|
||||||
{
|
{
|
||||||
@@ -412,6 +412,11 @@ static bool wasm_socket_write_bounded(u64 socket_fd, const String& data, u64 tim
|
|||||||
return(true);
|
return(true);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
static bool wasm_socket_write_bounded(u64 socket_fd, const String& data, u64 timeout_ms)
|
||||||
|
{
|
||||||
|
return(wasm_socket_write_until(socket_fd, data, wasm_deadline_after_ms(timeout_ms)));
|
||||||
|
}
|
||||||
|
|
||||||
static String wasm_socket_read_bounded(u64 socket_fd, u32 max_length, u64 timeout_ms)
|
static String wasm_socket_read_bounded(u64 socket_fd, u32 max_length, u64 timeout_ms)
|
||||||
{
|
{
|
||||||
if(max_length == 0 || !wasm_socket_wait((int)socket_fd, POLLIN, wasm_deadline_after_ms(timeout_ms)))
|
if(max_length == 0 || !wasm_socket_wait((int)socket_fd, POLLIN, wasm_deadline_after_ms(timeout_ms)))
|
||||||
@@ -421,6 +426,42 @@ static String wasm_socket_read_bounded(u64 socket_fd, u32 max_length, u64 timeou
|
|||||||
return(count > 0 ? String(buffer.data(), (size_t)count) : String(""));
|
return(count > 0 ? String(buffer.data(), (size_t)count) : String(""));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
static String wasm_memcache_exchange(u64 socket_fd, const String& command, u64 timeout_ms)
|
||||||
|
{
|
||||||
|
static constexpr size_t max_response = 8 * 1024 * 1024;
|
||||||
|
int fd = (int)socket_fd;
|
||||||
|
u64 deadline = wasm_deadline_after_ms(timeout_ms);
|
||||||
|
if(!wasm_socket_write_until(socket_fd, command + "\r\n", deadline))
|
||||||
|
{
|
||||||
|
shutdown(fd, SHUT_RDWR);
|
||||||
|
return("");
|
||||||
|
}
|
||||||
|
if(uce_memcache_command_has_no_reply(command))
|
||||||
|
return("");
|
||||||
|
String response;
|
||||||
|
response.reserve(16 * 1024);
|
||||||
|
MemcacheResponseParser parser;
|
||||||
|
while(response.size() < max_response)
|
||||||
|
{
|
||||||
|
MemcacheResponseState state = uce_memcache_response_advance(command, response, parser);
|
||||||
|
if(state == MemcacheResponseState::Complete)
|
||||||
|
return(response);
|
||||||
|
if(state == MemcacheResponseState::Malformed || !wasm_socket_wait(fd, POLLIN, deadline))
|
||||||
|
break;
|
||||||
|
char buffer[64 * 1024];
|
||||||
|
size_t remaining = max_response - response.size();
|
||||||
|
ssize_t count = recv(fd, buffer, std::min(remaining, sizeof(buffer)), MSG_DONTWAIT);
|
||||||
|
if(count > 0)
|
||||||
|
response.append(buffer, (size_t)count);
|
||||||
|
else if(count == 0 || (errno != EINTR && errno != EAGAIN && errno != EWOULDBLOCK))
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
if(uce_memcache_response_advance(command, response, parser) == MemcacheResponseState::Complete)
|
||||||
|
return(response);
|
||||||
|
shutdown(fd, SHUT_RDWR);
|
||||||
|
return("");
|
||||||
|
}
|
||||||
|
|
||||||
static f64 wasm_thread_cpu_time()
|
static f64 wasm_thread_cpu_time()
|
||||||
{
|
{
|
||||||
struct timespec ts;
|
struct timespec ts;
|
||||||
@@ -581,7 +622,7 @@ static u64 uce_shell_spawn_spec(const DValue& spec)
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
static DValue uce_exec_argv_capture(std::vector<String> argv, String input, u64 timeout_ms)
|
static DValue uce_exec_argv_capture(std::vector<String> argv, String input, u64 timeout_ms, size_t output_limit=0, bool clean_env=false)
|
||||||
{
|
{
|
||||||
DValue r; r["exit_code"]=(f64)-1; r["stdout"]=""; r["stderr"]=""; r["timed_out"].set_bool(false);
|
DValue r; r["exit_code"]=(f64)-1; r["stdout"]=""; r["stderr"]=""; r["timed_out"].set_bool(false);
|
||||||
if(argv.empty()) { r["stderr"]="empty argv"; return(r); }
|
if(argv.empty()) { r["stderr"]="empty argv"; return(r); }
|
||||||
@@ -598,7 +639,9 @@ static DValue uce_exec_argv_capture(std::vector<String> argv, String input, u64
|
|||||||
dup2(inpipe[0],0); dup2(outpipe[1],1); dup2(errpipe[1],2);
|
dup2(inpipe[0],0); dup2(outpipe[1],1); dup2(errpipe[1],2);
|
||||||
close(inpipe[0]); close(inpipe[1]); close(outpipe[0]); close(outpipe[1]); close(errpipe[0]); close(errpipe[1]);
|
close(inpipe[0]); close(inpipe[1]); close(outpipe[0]); close(outpipe[1]); close(errpipe[0]); close(errpipe[1]);
|
||||||
std::vector<char*> args; for(auto& a: argv) args.push_back((char*)a.c_str()); args.push_back(0);
|
std::vector<char*> args; for(auto& a: argv) args.push_back((char*)a.c_str()); args.push_back(0);
|
||||||
execvp(args[0], args.data()); _exit(127);
|
if(clean_env) { clearenv(); setenv("PATH", "/usr/bin:/bin", 1); execv(args[0], args.data()); }
|
||||||
|
else execvp(args[0], args.data());
|
||||||
|
_exit(127);
|
||||||
}
|
}
|
||||||
if(pid < 0)
|
if(pid < 0)
|
||||||
{
|
{
|
||||||
@@ -614,8 +657,8 @@ static DValue uce_exec_argv_capture(std::vector<String> argv, String input, u64
|
|||||||
{
|
{
|
||||||
if(!exited) { pid_t w=waitpid(pid,&status,WNOHANG); if(w==pid) { exited=true; status_valid=true; } else if(w<0&&errno==ECHILD) { u64 transfer_deadline=wasm_deadline_after_ms(50); do { status_valid=child_exit_status_take(pid,status,child_status_snapshot); if(!status_valid)sched_yield(); } while(!status_valid&&wasm_monotonic_ms()<transfer_deadline); exited=true; if(!status_valid)r["stderr"]=r["stderr"].to_string()+"lost child exit status"; } }
|
if(!exited) { pid_t w=waitpid(pid,&status,WNOHANG); if(w==pid) { exited=true; status_valid=true; } else if(w<0&&errno==ECHILD) { u64 transfer_deadline=wasm_deadline_after_ms(50); do { status_valid=child_exit_status_take(pid,status,child_status_snapshot); if(!status_valid)sched_yield(); } while(!status_valid&&wasm_monotonic_ms()<transfer_deadline); exited=true; if(!status_valid)r["stderr"]=r["stderr"].to_string()+"lost child exit status"; } }
|
||||||
if(in_open) { if(input_off<input.size()) { ssize_t n=write(inpipe[1], input.data()+input_off, input.size()-input_off); if(n>0) input_off+=(size_t)n; else if(n<0 && errno!=EINTR && errno!=EAGAIN && errno!=EWOULDBLOCK) { close(inpipe[1]); in_open=false; } } else { close(inpipe[1]); in_open=false; } }
|
if(in_open) { if(input_off<input.size()) { ssize_t n=write(inpipe[1], input.data()+input_off, input.size()-input_off); if(n>0) input_off+=(size_t)n; else if(n<0 && errno!=EINTR && errno!=EAGAIN && errno!=EWOULDBLOCK) { close(inpipe[1]); in_open=false; } } else { close(inpipe[1]); in_open=false; } }
|
||||||
char buf[4096]; ssize_t n; while((n=read(outpipe[0],buf,sizeof(buf)))>0) r["stdout"] = r["stdout"].to_string()+String(buf,n); if(n==0&&out_open){close(outpipe[0]);out_open=false;}
|
char buf[4096]; ssize_t n; while((n=read(outpipe[0],buf,sizeof(buf)))>0) { if(output_limit && r["stdout"].to_string().size()+(size_t)n>output_limit) { r["output_limited"].set_bool(true); kill(-pid,SIGKILL); kill(pid,SIGKILL); } else r["stdout"] = r["stdout"].to_string()+String(buf,n); } if(n==0&&out_open){close(outpipe[0]);out_open=false;}
|
||||||
while((n=read(errpipe[0],buf,sizeof(buf)))>0) r["stderr"] = r["stderr"].to_string()+String(buf,n); if(n==0&&err_open){close(errpipe[0]);err_open=false;}
|
while((n=read(errpipe[0],buf,sizeof(buf)))>0) { if(output_limit && r["stderr"].to_string().size()+(size_t)n>output_limit) { r["output_limited"].set_bool(true); kill(-pid,SIGKILL); kill(pid,SIGKILL); } else r["stderr"] = r["stderr"].to_string()+String(buf,n); } if(n==0&&err_open){close(errpipe[0]);err_open=false;}
|
||||||
if((out_open || err_open || !exited) && wasm_monotonic_ms() >= deadline) { r["timed_out"].set_bool(true); kill(-pid,SIGKILL); kill(pid,SIGKILL); if(!exited) status_valid=waitpid(pid,&status,0)==pid; exited=true; if(in_open){close(inpipe[1]);in_open=false;} if(out_open){close(outpipe[0]);out_open=false;} if(err_open){close(errpipe[0]);err_open=false;} }
|
if((out_open || err_open || !exited) && wasm_monotonic_ms() >= deadline) { r["timed_out"].set_bool(true); kill(-pid,SIGKILL); kill(pid,SIGKILL); if(!exited) status_valid=waitpid(pid,&status,0)==pid; exited=true; if(in_open){close(inpipe[1]);in_open=false;} if(out_open){close(outpipe[0]);out_open=false;} if(err_open){close(errpipe[0]);err_open=false;} }
|
||||||
if(out_open || err_open || !exited) usleep(10000);
|
if(out_open || err_open || !exited) usleep(10000);
|
||||||
}
|
}
|
||||||
@@ -630,8 +673,11 @@ static bool uce_header_name_safe(String name)
|
|||||||
return(true);
|
return(true);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
static DValue uce_hardened_http_request_value(const DValue& req, u64 timeout_ms, bool keep_worker_process_group=false);
|
||||||
|
|
||||||
static DValue uce_http_request_value(const DValue& req)
|
static DValue uce_http_request_value(const DValue& req)
|
||||||
{
|
{
|
||||||
|
const DValue* security=req.key("security"); if(hardened_http_security_requested(security)) { u64 requested=req.key("timeout_ms")?req.key("timeout_ms")->to_u64(5000):5000; return(uce_hardened_http_request_value(req,std::max<u64>(1,requested))); }
|
||||||
DValue r; r["status"]=(f64)0; r["headers"].set_array(); r["body"]=""; r["error"]="";
|
DValue r; r["status"]=(f64)0; r["headers"].set_array(); r["body"]=""; r["error"]="";
|
||||||
const DValue* method_value = req.key("method");
|
const DValue* method_value = req.key("method");
|
||||||
const DValue* url_value = req.key("url");
|
const DValue* url_value = req.key("url");
|
||||||
@@ -665,19 +711,35 @@ static DValue uce_http_request_value(const DValue& req)
|
|||||||
return(r);
|
return(r);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
static DValue uce_hardened_http_request_value(const DValue& req, u64 timeout_ms, bool keep_worker_process_group)
|
||||||
|
{
|
||||||
|
HardenedHttpHooks hooks;
|
||||||
|
hooks.resolve=[](String host) { std::vector<String> answers; addrinfo hints{}; hints.ai_socktype=SOCK_STREAM; hints.ai_family=AF_UNSPEC; addrinfo* result=0; if(getaddrinfo(host.c_str(), "443", &hints, &result)!=0) return answers; for(addrinfo* p=result;p;p=p->ai_next) { char text[INET6_ADDRSTRLEN]; if(p->ai_family==AF_INET && inet_ntop(AF_INET,&((sockaddr_in*)p->ai_addr)->sin_addr,text,sizeof(text))) answers.push_back(text); else if(p->ai_family==AF_INET6 && inet_ntop(AF_INET6,&((sockaddr_in6*)p->ai_addr)->sin6_addr,text,sizeof(text))) answers.push_back(text); else answers.push_back(""); } freeaddrinfo(result); return answers; };
|
||||||
|
hooks.execute=[keep_worker_process_group](std::vector<String> argv, String input, std::vector<String>, u64 deadline, size_t limit) { return hardened_http_exec_argv_capture(argv,input,deadline,limit,true,!keep_worker_process_group); };
|
||||||
|
return hardened_http_request_internal(req,timeout_ms,hooks);
|
||||||
|
}
|
||||||
|
|
||||||
static u64 uce_http_spawn_spec(const DValue& req)
|
static u64 uce_http_spawn_spec(const DValue& req)
|
||||||
{
|
{
|
||||||
uce_job_reap(); u64 id=uce_job_new("http"); if(!id) return(0);
|
uce_job_reap(); u64 id=uce_job_new("http"); if(!id) return(0);
|
||||||
|
int ready[2]; if(pipe(ready)) { DValue r; r["error"]="pipe failed"; uce_job_finish(id,r,"failed"); return(id); }
|
||||||
pid_t pid=fork();
|
pid_t pid=fork();
|
||||||
if(pid==0) { setsid(); uce_write_text(uce_job_path(id)+"/worker_pid", std::to_string((long long)getpid())); uce_write_text(uce_job_path(id)+"/state", "running"); DValue result=uce_http_request_value(req); uce_job_finish(id,result,result["error"].to_string()==""?"done":"failed"); _exit(0); }
|
if(pid==0) { close(ready[0]); if(setsid()<0) _exit(127); char ok='1'; if(write(ready[1],&ok,1)!=1) _exit(127); close(ready[1]); uce_write_text(uce_job_path(id)+"/worker_pid", std::to_string((long long)getpid())); uce_write_text(uce_job_path(id)+"/state", "running"); const DValue* security=req.key("security"); bool hardened=hardened_http_security_requested(security); DValue result=hardened ? uce_hardened_http_request_value(req,req.key("timeout_ms")?std::max<u64>(1,req.key("timeout_ms")->to_u64(5000)):5000,true) : uce_http_request_value(req); uce_job_finish(id,result,result["error"].to_string()==""?"done":"failed"); _exit(0); }
|
||||||
if(pid<0) { DValue r; r["error"]="fork failed"; uce_job_finish(id,r,"failed"); return(id); }
|
close(ready[1]); char ok=0; ssize_t started; do { started=read(ready[0],&ok,1); } while(started<0&&errno==EINTR); close(ready[0]);
|
||||||
|
if(pid<0 || started!=1 || ok!='1') { DValue r; r["error"]="async worker start failed"; uce_job_finish(id,r,"failed"); return(id); }
|
||||||
uce_write_text(uce_job_path(id)+"/worker_pid", std::to_string((long long)pid)); uce_write_text(uce_job_path(id)+"/state", "running"); return(id);
|
uce_write_text(uce_job_path(id)+"/worker_pid", std::to_string((long long)pid)); uce_write_text(uce_job_path(id)+"/state", "running"); return(id);
|
||||||
}
|
}
|
||||||
|
|
||||||
static DValue uce_job_status_value(u64 id)
|
static DValue uce_job_status_value(u64 id)
|
||||||
{
|
{
|
||||||
DValue r; String dir=uce_job_path(id); r["job_id"]=(f64)id;
|
DValue r; String dir=uce_job_path(id); r["job_id"]=(f64)id;
|
||||||
if(id==0 || !std::filesystem::is_directory(dir)) { r["state"]="missing"; return(r); }
|
bool is_directory = false;
|
||||||
|
if(id != 0)
|
||||||
|
{
|
||||||
|
try { is_directory = std::filesystem::is_directory(dir); }
|
||||||
|
catch(...) { is_directory = false; }
|
||||||
|
}
|
||||||
|
if(!is_directory) { r["state"]="missing"; return(r); }
|
||||||
String state=trim(uce_read_text(dir+"/state")); if(state=="") state="pending"; r["state"]=state;
|
String state=trim(uce_read_text(dir+"/state")); if(state=="") state="pending"; r["state"]=state;
|
||||||
r["kind"]=trim(uce_read_text(dir+"/kind")); r["pid"]=(f64)strtoull(uce_read_text(dir+"/worker_pid").c_str(),0,10);
|
r["kind"]=trim(uce_read_text(dir+"/kind")); r["pid"]=(f64)strtoull(uce_read_text(dir+"/worker_pid").c_str(),0,10);
|
||||||
r["done"].set_bool(state=="done"||state=="failed"||state=="cancelled");
|
r["done"].set_bool(state=="done"||state=="failed"||state=="cancelled");
|
||||||
@@ -1271,6 +1333,14 @@ static bool wasm_read_metadata_file(const String& path, std::vector<u8>& metadat
|
|||||||
|
|
||||||
class WasmWorkspace;
|
class WasmWorkspace;
|
||||||
|
|
||||||
|
#ifdef UCE_WASM_HOST_CONNECTORS
|
||||||
|
struct WasmMySQLPersistentConnection
|
||||||
|
{
|
||||||
|
MySQL* db = 0;
|
||||||
|
std::chrono::steady_clock::time_point idle_since;
|
||||||
|
};
|
||||||
|
#endif
|
||||||
|
|
||||||
class WasmWorker
|
class WasmWorker
|
||||||
{
|
{
|
||||||
public:
|
public:
|
||||||
@@ -1283,25 +1353,59 @@ public:
|
|||||||
#ifdef UCE_WASM_HOST_CONNECTORS
|
#ifdef UCE_WASM_HOST_CONNECTORS
|
||||||
~WasmWorker()
|
~WasmWorker()
|
||||||
{
|
{
|
||||||
for(auto* db : mysql_persistent_pool)
|
for(auto& entry : mysql_persistent_pool)
|
||||||
delete db;
|
delete entry.db;
|
||||||
|
}
|
||||||
|
|
||||||
|
void mysql_evict_idle(std::chrono::steady_clock::time_point now = std::chrono::steady_clock::now())
|
||||||
|
{
|
||||||
|
if(cfg.mysql_persistent_pool_idle_timeout_seconds == 0)
|
||||||
|
return;
|
||||||
|
auto timeout = std::chrono::seconds(cfg.mysql_persistent_pool_idle_timeout_seconds);
|
||||||
|
for(size_t i = 0; i < mysql_persistent_pool.size();)
|
||||||
|
{
|
||||||
|
auto& entry = mysql_persistent_pool[i];
|
||||||
|
if(entry.idle_since != std::chrono::steady_clock::time_point() && now - entry.idle_since >= timeout)
|
||||||
|
{
|
||||||
|
delete entry.db;
|
||||||
|
mysql_persistent_pool.erase(mysql_persistent_pool.begin() + i);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
i++;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
void mysql_release(MySQL* db)
|
||||||
|
{
|
||||||
|
if(!db)
|
||||||
|
return;
|
||||||
|
db->request_leases = 0;
|
||||||
|
for(auto& entry : mysql_persistent_pool)
|
||||||
|
if(entry.db == db)
|
||||||
|
{
|
||||||
|
entry.idle_since = std::chrono::steady_clock::now();
|
||||||
|
return;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
MySQL* mysql_checkout(const String& host, const String& username, const String& password, const String& database, bool& reused, bool& persistent)
|
MySQL* mysql_checkout(const String& host, const String& username, const String& password, const String& database, bool& reused, bool& persistent)
|
||||||
{
|
{
|
||||||
|
mysql_evict_idle();
|
||||||
reused = false;
|
reused = false;
|
||||||
persistent = false;
|
persistent = false;
|
||||||
for(size_t i = 0; i < mysql_persistent_pool.size(); i++)
|
for(size_t i = 0; i < mysql_persistent_pool.size(); i++)
|
||||||
{
|
{
|
||||||
MySQL* db = mysql_persistent_pool[i];
|
MySQL* db = mysql_persistent_pool[i].db;
|
||||||
if(!db || !db->connection || db->request_host != host || db->request_username != username || db->request_password != password || db->request_database != database)
|
if(!db || !db->connection || db->request_host != host || db->request_username != username || db->request_password != password || db->request_database != database)
|
||||||
continue;
|
continue;
|
||||||
if(db->reset_connection())
|
if(db->reset_connection())
|
||||||
{
|
{
|
||||||
|
mysql_persistent_pool[i].idle_since = std::chrono::steady_clock::time_point();
|
||||||
if(i + 1 < mysql_persistent_pool.size())
|
if(i + 1 < mysql_persistent_pool.size())
|
||||||
{
|
{
|
||||||
|
auto entry = mysql_persistent_pool[i];
|
||||||
mysql_persistent_pool.erase(mysql_persistent_pool.begin() + i);
|
mysql_persistent_pool.erase(mysql_persistent_pool.begin() + i);
|
||||||
mysql_persistent_pool.push_back(db);
|
mysql_persistent_pool.push_back(entry);
|
||||||
}
|
}
|
||||||
reused = true;
|
reused = true;
|
||||||
persistent = true;
|
persistent = true;
|
||||||
@@ -1326,15 +1430,17 @@ public:
|
|||||||
{
|
{
|
||||||
while(mysql_persistent_pool.size() >= cfg.mysql_persistent_pool_size)
|
while(mysql_persistent_pool.size() >= cfg.mysql_persistent_pool_size)
|
||||||
{
|
{
|
||||||
delete mysql_persistent_pool.front();
|
delete mysql_persistent_pool.front().db;
|
||||||
mysql_persistent_pool.erase(mysql_persistent_pool.begin());
|
mysql_persistent_pool.erase(mysql_persistent_pool.begin());
|
||||||
}
|
}
|
||||||
mysql_persistent_pool.push_back(db);
|
WasmMySQLPersistentConnection entry;
|
||||||
|
entry.db = db;
|
||||||
|
mysql_persistent_pool.push_back(entry);
|
||||||
}
|
}
|
||||||
return(db);
|
return(db);
|
||||||
}
|
}
|
||||||
|
|
||||||
std::vector<MySQL*> mysql_persistent_pool;
|
std::vector<WasmMySQLPersistentConnection> mysql_persistent_pool;
|
||||||
#endif
|
#endif
|
||||||
|
|
||||||
String init()
|
String init()
|
||||||
@@ -1557,14 +1663,30 @@ public:
|
|||||||
|
|
||||||
static String serialize_module_artifact(const String& wasm_path)
|
static String serialize_module_artifact(const String& wasm_path)
|
||||||
{
|
{
|
||||||
|
int lock_fd = open((wasm_path + ".lock").c_str(), O_RDWR | O_CREAT | O_CLOEXEC, 0666);
|
||||||
|
if(lock_fd < 0 || flock(lock_fd, LOCK_EX) != 0)
|
||||||
|
{
|
||||||
|
if(lock_fd >= 0)
|
||||||
|
close(lock_fd);
|
||||||
|
return("cannot lock " + wasm_path);
|
||||||
|
}
|
||||||
|
struct SerializationLock
|
||||||
|
{
|
||||||
|
int fd;
|
||||||
|
~SerializationLock() { flock(fd, LOCK_UN); close(fd); }
|
||||||
|
} lock{lock_fd};
|
||||||
if(!serialized_module_needs_refresh(wasm_path))
|
if(!serialized_module_needs_refresh(wasm_path))
|
||||||
return("");
|
return("");
|
||||||
|
struct stat initial_stat;
|
||||||
|
if(stat(wasm_path.c_str(), &initial_stat) != 0 || !S_ISREG(initial_stat.st_mode))
|
||||||
|
return("cannot stat " + wasm_path);
|
||||||
std::vector<u8> bytes;
|
std::vector<u8> bytes;
|
||||||
if(!wasm_read_file(wasm_path, bytes))
|
if(!wasm_read_file(wasm_path, bytes))
|
||||||
return("cannot read " + wasm_path);
|
return("cannot read " + wasm_path);
|
||||||
wasmtime::Engine engine = make_engine();
|
wasmtime::Engine engine = make_engine();
|
||||||
String error;
|
String error;
|
||||||
auto module = compile_and_cache_module(engine, cached_wasm_path(wasm_path), bytes, error);
|
auto module = compile_and_cache_module(engine, cached_wasm_path(wasm_path), bytes, error,
|
||||||
|
wasm_path, &initial_stat);
|
||||||
if(!module)
|
if(!module)
|
||||||
return(error);
|
return(error);
|
||||||
if(serialized_module_needs_refresh(wasm_path))
|
if(serialized_module_needs_refresh(wasm_path))
|
||||||
@@ -1612,8 +1734,15 @@ private:
|
|||||||
return(std::nullopt);
|
return(std::nullopt);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
static bool same_artifact(const struct stat& left, const struct stat& right)
|
||||||
|
{
|
||||||
|
return(left.st_dev == right.st_dev && left.st_ino == right.st_ino && left.st_size == right.st_size &&
|
||||||
|
left.st_mtim.tv_sec == right.st_mtim.tv_sec && left.st_mtim.tv_nsec == right.st_mtim.tv_nsec &&
|
||||||
|
left.st_ctim.tv_sec == right.st_ctim.tv_sec && left.st_ctim.tv_nsec == right.st_ctim.tv_nsec);
|
||||||
|
}
|
||||||
|
|
||||||
static std::optional<wasmtime::Module> compile_and_cache_module(wasmtime::Engine& engine, const String& cached_path,
|
static std::optional<wasmtime::Module> compile_and_cache_module(wasmtime::Engine& engine, const String& cached_path,
|
||||||
std::vector<u8>& bytes, String& compile_error)
|
std::vector<u8>& bytes, String& compile_error, const String& source_path = "", const struct stat* expected_source = 0)
|
||||||
{
|
{
|
||||||
auto compiled = wasmtime::Module::compile(engine, bytes);
|
auto compiled = wasmtime::Module::compile(engine, bytes);
|
||||||
if(!compiled)
|
if(!compiled)
|
||||||
@@ -1636,6 +1765,14 @@ private:
|
|||||||
{
|
{
|
||||||
out.flush();
|
out.flush();
|
||||||
out.close();
|
out.close();
|
||||||
|
struct stat current_source;
|
||||||
|
if(expected_source && (stat(source_path.c_str(), ¤t_source) != 0 ||
|
||||||
|
!same_artifact(*expected_source, current_source)))
|
||||||
|
{
|
||||||
|
(void)std::remove(tmp.c_str());
|
||||||
|
compile_error = "wasm artifact changed during serialization: " + source_path;
|
||||||
|
return(std::nullopt);
|
||||||
|
}
|
||||||
if(std::rename(tmp.c_str(), cached_path.c_str()) != 0)
|
if(std::rename(tmp.c_str(), cached_path.c_str()) != 0)
|
||||||
(void)std::remove(tmp.c_str());
|
(void)std::remove(tmp.c_str());
|
||||||
}
|
}
|
||||||
@@ -1752,6 +1889,7 @@ public:
|
|||||||
bool writable = false;
|
bool writable = false;
|
||||||
};
|
};
|
||||||
std::vector<FileHandle> file_handles;
|
std::vector<FileHandle> file_handles;
|
||||||
|
std::vector<int> socket_fds;
|
||||||
|
|
||||||
struct RequestPerfSnapshot
|
struct RequestPerfSnapshot
|
||||||
{
|
{
|
||||||
@@ -1909,7 +2047,9 @@ public:
|
|||||||
// the wasm-side enforcement of request-scoped DB lifecycle; app code should
|
// the wasm-side enforcement of request-scoped DB lifecycle; app code should
|
||||||
// never cache these opaque handles across requests.
|
// never cache these opaque handles across requests.
|
||||||
std::vector<SQLite*> sqlite_handles;
|
std::vector<SQLite*> sqlite_handles;
|
||||||
std::vector<MySQL*> mysql_handles;
|
std::map<u64, MySQL*> mysql_handles;
|
||||||
|
std::set<u64> mysql_task_handles;
|
||||||
|
u64 mysql_next_handle = 1;
|
||||||
std::vector<MySQL*> mysql_request_pool;
|
std::vector<MySQL*> mysql_request_pool;
|
||||||
std::vector<MySQL*> mysql_request_owned;
|
std::vector<MySQL*> mysql_request_owned;
|
||||||
#endif
|
#endif
|
||||||
@@ -1926,13 +2066,20 @@ public:
|
|||||||
h.fd = -1;
|
h.fd = -1;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
for(auto& fd : socket_fds)
|
||||||
|
{
|
||||||
|
if(fd >= 0)
|
||||||
|
{
|
||||||
|
::socket_close((u64)fd);
|
||||||
|
fd = -1;
|
||||||
|
}
|
||||||
|
}
|
||||||
#ifdef UCE_WASM_HOST_CONNECTORS
|
#ifdef UCE_WASM_HOST_CONNECTORS
|
||||||
for(auto* db : sqlite_handles)
|
for(auto* db : sqlite_handles)
|
||||||
if(db)
|
if(db)
|
||||||
delete db; // ~SQLite disconnects
|
delete db; // ~SQLite disconnects
|
||||||
for(auto* db : mysql_request_pool)
|
for(auto* db : mysql_request_pool)
|
||||||
if(db)
|
worker.mysql_release(db);
|
||||||
db->request_leases = 0;
|
|
||||||
for(auto* db : mysql_request_owned)
|
for(auto* db : mysql_request_owned)
|
||||||
if(db)
|
if(db)
|
||||||
delete db; // ~MySQL disconnects
|
delete db; // ~MySQL disconnects
|
||||||
@@ -3559,6 +3706,33 @@ private:
|
|||||||
return(add([self](Caller, Span<const Val> args, Span<Val> results) -> Result<std::monostate, Trap> { String password,encoded; self->hostcall_read(args[0].i32(), args[1].i32(), password); self->hostcall_read(args[2].i32(), args[3].i32(), encoded); bool valid=password_verify_native(password,encoded); results[0]=Val((int32_t)(valid?1:0)); return(std::monostate()); }));
|
return(add([self](Caller, Span<const Val> args, Span<Val> results) -> Result<std::monostate, Trap> { String password,encoded; self->hostcall_read(args[0].i32(), args[1].i32(), password); self->hostcall_read(args[2].i32(), args[3].i32(), encoded); bool valid=password_verify_native(password,encoded); results[0]=Val((int32_t)(valid?1:0)); return(std::monostate()); }));
|
||||||
if(mod == "env" && name == "uce_host_password_needs_rehash")
|
if(mod == "env" && name == "uce_host_password_needs_rehash")
|
||||||
return(add([self](Caller, Span<const Val> args, Span<Val> results) -> Result<std::monostate, Trap> { String encoded; self->hostcall_read(args[0].i32(), args[1].i32(), encoded); results[0]=Val((int32_t)(password_needs_rehash_native(encoded)?1:0)); return(std::monostate()); }));
|
return(add([self](Caller, Span<const Val> args, Span<Val> results) -> Result<std::monostate, Trap> { String encoded; self->hostcall_read(args[0].i32(), args[1].i32(), encoded); results[0]=Val((int32_t)(password_needs_rehash_native(encoded)?1:0)); return(std::monostate()); }));
|
||||||
|
if(mod == "env" && name == "uce_host_crypto_operation")
|
||||||
|
return(add([self](Caller, Span<const Val> args, Span<Val> results) -> Result<std::monostate, Trap> {
|
||||||
|
String encoded, out;
|
||||||
|
DValue request, response;
|
||||||
|
String error;
|
||||||
|
int32_t input_size = args[1].i32();
|
||||||
|
u32 cap = (u32)args[3].i32();
|
||||||
|
int32_t buf = args[2].i32();
|
||||||
|
if(input_size > 0 && input_size <= 64 * 1024 && self->hostcall_read(args[0].i32(), input_size, encoded) == "" && ucb_decode(encoded, request, &error))
|
||||||
|
{
|
||||||
|
String stage_key = "crypto_operation:" + encoded;
|
||||||
|
if(!self->hostcall_staged(stage_key, out))
|
||||||
|
{
|
||||||
|
response = crypto_operation_native(request);
|
||||||
|
out = ucb_encode(response);
|
||||||
|
if(buf == 0) self->hostcall_stage(stage_key, out);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
else
|
||||||
|
{
|
||||||
|
response["error"] = "invalid_request";
|
||||||
|
out = ucb_encode(response);
|
||||||
|
}
|
||||||
|
if(buf && cap >= out.size()) self->hostcall_write(buf, out);
|
||||||
|
results[0] = Val((int32_t)out.size());
|
||||||
|
return(std::monostate());
|
||||||
|
}));
|
||||||
if(mod == "env" && name == "uce_host_log")
|
if(mod == "env" && name == "uce_host_log")
|
||||||
return(add([self](Caller, Span<const Val> args, Span<Val>) -> Result<std::monostate, Trap> {
|
return(add([self](Caller, Span<const Val> args, Span<Val>) -> Result<std::monostate, Trap> {
|
||||||
String text;
|
String text;
|
||||||
@@ -3985,7 +4159,7 @@ private:
|
|||||||
if(mod == "env" && name == "uce_host_file_truncate")
|
if(mod == "env" && name == "uce_host_file_truncate")
|
||||||
return(add([self](Caller, Span<const Val> args, Span<Val> results) -> Result<std::monostate, Trap> { String path,current; self->hostcall_read(args[0].i32(),args[1].i32(),path); self->hostcall_read(args[2].i32(),args[3].i32(),current); String r=self->resolve_guest_write(path,current); results[0]=Val((int32_t)(r!=""&&truncate(r.c_str(),(off_t)args[4].i64())==0)); return(std::monostate()); }));
|
return(add([self](Caller, Span<const Val> args, Span<Val> results) -> Result<std::monostate, Trap> { String path,current; self->hostcall_read(args[0].i32(),args[1].i32(),path); self->hostcall_read(args[2].i32(),args[3].i32(),current); String r=self->resolve_guest_write(path,current); results[0]=Val((int32_t)(r!=""&&truncate(r.c_str(),(off_t)args[4].i64())==0)); return(std::monostate()); }));
|
||||||
if(mod == "env" && name == "uce_host_dir_remove")
|
if(mod == "env" && name == "uce_host_dir_remove")
|
||||||
return(add([self](Caller, Span<const Val> args, Span<Val> results) -> Result<std::monostate, Trap> { String path,current; self->hostcall_read(args[0].i32(),args[1].i32(),path); self->hostcall_read(args[2].i32(),args[3].i32(),current); String r=self->resolve_guest_write(path,current); bool rec=args[4].i32()!=0; bool ok=false; if(r!="") { if(rec) ok=std::filesystem::remove_all(r)>0; else ok=::rmdir(r.c_str())==0; } results[0]=Val((int32_t)ok); return(std::monostate()); }));
|
return(add([self](Caller, Span<const Val> args, Span<Val> results) -> Result<std::monostate, Trap> { String path,current; self->hostcall_read(args[0].i32(),args[1].i32(),path); self->hostcall_read(args[2].i32(),args[3].i32(),current); String r=self->resolve_guest_write(path,current); bool rec=args[4].i32()!=0; bool ok=false; if(r!="") { if(rec) { try { ok=std::filesystem::remove_all(r)>0; } catch(...) { ok=false; } } else ok=::rmdir(r.c_str())==0; } results[0]=Val((int32_t)ok); return(std::monostate()); }));
|
||||||
if(mod == "env" && name == "uce_host_file_temp")
|
if(mod == "env" && name == "uce_host_file_temp")
|
||||||
return(add([self](Caller, Span<const Val> args, Span<Val> results) -> Result<std::monostate, Trap> { String prefix,current; self->hostcall_read(args[0].i32(),args[1].i32(),prefix); self->hostcall_read(args[2].i32(),args[3].i32(),current); u32 cap=(u32)args[5].i32(); int32_t buf=args[4].i32(); String out; String stage_key="file_temp:"+prefix+"\0"+current; if(!self->hostcall_staged(stage_key,out)) { if(prefix=="") prefix="/tmp/uce-temp"; String templ=self->resolve_guest_write(prefix+"XXXXXX",current); if(templ!="") { std::vector<char> t(templ.begin(), templ.end()); t.push_back(0); int fd=mkstemp(t.data()); if(fd>=0) { close(fd); out=t.data(); } } if(buf==0) self->hostcall_stage(stage_key,out); } if(buf&&cap>=out.size()) self->hostcall_write(buf,out); results[0]=Val((int32_t)out.size()); return(std::monostate()); }));
|
return(add([self](Caller, Span<const Val> args, Span<Val> results) -> Result<std::monostate, Trap> { String prefix,current; self->hostcall_read(args[0].i32(),args[1].i32(),prefix); self->hostcall_read(args[2].i32(),args[3].i32(),current); u32 cap=(u32)args[5].i32(); int32_t buf=args[4].i32(); String out; String stage_key="file_temp:"+prefix+"\0"+current; if(!self->hostcall_staged(stage_key,out)) { if(prefix=="") prefix="/tmp/uce-temp"; String templ=self->resolve_guest_write(prefix+"XXXXXX",current); if(templ!="") { std::vector<char> t(templ.begin(), templ.end()); t.push_back(0); int fd=mkstemp(t.data()); if(fd>=0) { close(fd); out=t.data(); } } if(buf==0) self->hostcall_stage(stage_key,out); } if(buf&&cap>=out.size()) self->hostcall_write(buf,out); results[0]=Val((int32_t)out.size()); return(std::monostate()); }));
|
||||||
if(mod == "env" && name == "uce_host_file_chmod")
|
if(mod == "env" && name == "uce_host_file_chmod")
|
||||||
@@ -4195,9 +4369,15 @@ private:
|
|||||||
}));
|
}));
|
||||||
if(mod == "env" && name == "uce_host_memcache_command")
|
if(mod == "env" && name == "uce_host_memcache_command")
|
||||||
return(add([self](Caller caller, Span<const Val> args, Span<Val> results) -> Result<std::monostate, Trap> {
|
return(add([self](Caller caller, Span<const Val> args, Span<Val> results) -> Result<std::monostate, Trap> {
|
||||||
|
u64 handle = (u64)args[0].i64();
|
||||||
|
if(handle < 1 || handle > self->socket_fds.size() || self->socket_fds[(size_t)handle - 1] < 0)
|
||||||
|
{
|
||||||
|
results[0] = Val((int32_t)0);
|
||||||
|
return(std::monostate());
|
||||||
|
}
|
||||||
String command;
|
String command;
|
||||||
self->hostcall_read(args[1].i32(), args[2].i32(), command);
|
self->hostcall_read(args[1].i32(), args[2].i32(), command);
|
||||||
String key = "memcache:" + std::to_string((u64)args[0].i64()) + ":" + command;
|
String key = "memcache:" + std::to_string(handle) + ":" + command;
|
||||||
u32 cap = (u32)args[4].i32();
|
u32 cap = (u32)args[4].i32();
|
||||||
int32_t buf = args[3].i32();
|
int32_t buf = args[3].i32();
|
||||||
String out;
|
String out;
|
||||||
@@ -4209,9 +4389,8 @@ private:
|
|||||||
}
|
}
|
||||||
else
|
else
|
||||||
{
|
{
|
||||||
u64 socket_fd = (u64)args[0].i64();
|
u64 socket_fd = (u64)self->socket_fds[(size_t)handle - 1];
|
||||||
wasm_socket_write_bounded(socket_fd, command + "\r\n", self->bounded_hostcall_timeout_ms(1000));
|
out = wasm_memcache_exchange(socket_fd, command, self->bounded_hostcall_timeout_ms(1000));
|
||||||
out = wasm_socket_read_bounded(socket_fd, 1024 * 128, self->bounded_hostcall_timeout_ms(1000));
|
|
||||||
if(buf == 0)
|
if(buf == 0)
|
||||||
{
|
{
|
||||||
self->staged_memcache_key = key;
|
self->staged_memcache_key = key;
|
||||||
@@ -4248,41 +4427,60 @@ private:
|
|||||||
String password = request["password"].to_string();
|
String password = request["password"].to_string();
|
||||||
String database = request["database"].to_string();
|
String database = request["database"].to_string();
|
||||||
MySQL* db = 0;
|
MySQL* db = 0;
|
||||||
for(auto* pooled : self->mysql_request_pool)
|
bool ok = false;
|
||||||
if(pooled && pooled->connection && pooled->request_host == host && pooled->request_username == username && pooled->request_password == password && pooled->request_database == database)
|
if(task_child_process)
|
||||||
{
|
|
||||||
db = pooled;
|
|
||||||
connection_source = "request";
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
bool ok = db != 0;
|
|
||||||
if(!db)
|
|
||||||
{
|
{
|
||||||
bool reused = false;
|
// task() closes inherited descriptors after fork. Never reuse
|
||||||
bool persistent = false;
|
// request/worker MySQL objects whose sockets were just closed.
|
||||||
db = self->worker.mysql_checkout(host, username, password, database, reused, persistent);
|
db = new MySQL();
|
||||||
connection_source = reused ? "worker" : "new";
|
ok = db->connect(host, username, password, database);
|
||||||
ok = db && db->connection;
|
connection_source = "task";
|
||||||
if(ok && db->connection)
|
if(ok)
|
||||||
{
|
{
|
||||||
self->mysql_request_pool.push_back(db);
|
self->mysql_request_pool.push_back(db);
|
||||||
if(!persistent)
|
self->mysql_request_owned.push_back(db);
|
||||||
self->mysql_request_owned.push_back(db);
|
}
|
||||||
|
}
|
||||||
|
else
|
||||||
|
{
|
||||||
|
for(auto* pooled : self->mysql_request_pool)
|
||||||
|
if(pooled && pooled->connection && pooled->request_host == host && pooled->request_username == username && pooled->request_password == password && pooled->request_database == database)
|
||||||
|
{
|
||||||
|
db = pooled;
|
||||||
|
connection_source = "request";
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
ok = db != 0;
|
||||||
|
if(!db)
|
||||||
|
{
|
||||||
|
bool reused = false;
|
||||||
|
bool persistent = false;
|
||||||
|
db = self->worker.mysql_checkout(host, username, password, database, reused, persistent);
|
||||||
|
connection_source = reused ? "worker" : "new";
|
||||||
|
ok = db && db->connection;
|
||||||
|
if(ok)
|
||||||
|
{
|
||||||
|
self->mysql_request_pool.push_back(db);
|
||||||
|
if(!persistent)
|
||||||
|
self->mysql_request_owned.push_back(db);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
u64 handle = 0;
|
u64 handle = 0;
|
||||||
if(ok && db->connection)
|
if(ok && db->connection && self->mysql_next_handle != 0)
|
||||||
{
|
{
|
||||||
db->request_leases++;
|
db->request_leases++;
|
||||||
self->mysql_handles.push_back(db);
|
handle = self->mysql_next_handle;
|
||||||
handle = self->mysql_handles.size();
|
self->mysql_next_handle = handle == UINT64_MAX ? 0 : handle + 1;
|
||||||
|
self->mysql_handles[handle] = db;
|
||||||
|
if(task_child_process) self->mysql_task_handles.insert(handle);
|
||||||
if(connection_source == "new") self->mysql_connection_open_count++;
|
if(connection_source == "new") self->mysql_connection_open_count++;
|
||||||
else if(connection_source == "worker") self->mysql_connection_reuse_count++;
|
else if(connection_source == "worker") self->mysql_connection_reuse_count++;
|
||||||
else if(connection_source == "request") self->mysql_request_pool_hit_count++;
|
else if(connection_source == "request") self->mysql_request_pool_hit_count++;
|
||||||
}
|
}
|
||||||
response["handle"] = (f64)handle;
|
response["handle"] = (f64)handle;
|
||||||
response["error_code"] = (f64)db->_preload_next_error_code;
|
response["error_code"] = (f64)(handle == 0 ? 2000 : db->_preload_next_error_code);
|
||||||
response["statement_info"] = db->error();
|
response["statement_info"] = handle == 0 ? String("mysql handle space exhausted") : db->error();
|
||||||
if(handle == 0 && db)
|
if(handle == 0 && db)
|
||||||
delete db;
|
delete db;
|
||||||
}
|
}
|
||||||
@@ -4294,8 +4492,8 @@ private:
|
|||||||
else
|
else
|
||||||
{
|
{
|
||||||
u64 handle = request["handle"].to_u64();
|
u64 handle = request["handle"].to_u64();
|
||||||
MySQL* db = (handle >= 1 && handle <= self->mysql_handles.size())
|
auto handle_it = self->mysql_handles.find(handle);
|
||||||
? self->mysql_handles[(size_t)handle - 1] : 0;
|
MySQL* db = handle_it == self->mysql_handles.end() || (task_child_process && self->mysql_task_handles.find(handle) == self->mysql_task_handles.end()) ? 0 : handle_it->second;
|
||||||
if(op == "query" && db)
|
if(op == "query" && db)
|
||||||
{
|
{
|
||||||
response["result"] = db->query(request["query"].to_string());
|
response["result"] = db->query(request["query"].to_string());
|
||||||
@@ -4308,7 +4506,20 @@ private:
|
|||||||
{
|
{
|
||||||
if(db->request_leases > 0)
|
if(db->request_leases > 0)
|
||||||
db->request_leases--;
|
db->request_leases--;
|
||||||
self->mysql_handles[(size_t)handle - 1] = 0;
|
self->mysql_handles.erase(handle);
|
||||||
|
self->mysql_task_handles.erase(handle);
|
||||||
|
if(task_child_process && db->request_leases == 0)
|
||||||
|
{
|
||||||
|
auto erase_db = [db](auto& pool) { pool.erase(std::remove(pool.begin(), pool.end(), db), pool.end()); };
|
||||||
|
erase_db(self->mysql_request_pool);
|
||||||
|
erase_db(self->mysql_request_owned);
|
||||||
|
delete db;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
else if(op == "query" || op == "disconnect")
|
||||||
|
{
|
||||||
|
response["error_code"] = (f64)2000;
|
||||||
|
response["statement_info"] = "mysql handle is invalid or unavailable in this task";
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -4350,24 +4561,52 @@ private:
|
|||||||
String host;
|
String host;
|
||||||
self->hostcall_read(args[0].i32(), args[1].i32(), host);
|
self->hostcall_read(args[0].i32(), args[1].i32(), host);
|
||||||
u64 fd = wasm_socket_connect_bounded(host, (u16)args[2].i32(), self->bounded_hostcall_timeout_ms(self->worker.cfg.invocation_timeout_ms));
|
u64 fd = wasm_socket_connect_bounded(host, (u16)args[2].i32(), self->bounded_hostcall_timeout_ms(self->worker.cfg.invocation_timeout_ms));
|
||||||
results[0] = Val((int64_t)fd);
|
u64 handle = 0;
|
||||||
|
if(fd > 0)
|
||||||
|
{
|
||||||
|
self->socket_fds.push_back((int)fd);
|
||||||
|
handle = self->socket_fds.size();
|
||||||
|
}
|
||||||
|
results[0] = Val((int64_t)handle);
|
||||||
return(std::monostate());
|
return(std::monostate());
|
||||||
}));
|
}));
|
||||||
if(mod == "env" && name == "uce_host_socket_close")
|
if(mod == "env" && name == "uce_host_socket_close")
|
||||||
return(add([](Caller, Span<const Val> args, Span<Val>) -> Result<std::monostate, Trap> {
|
return(add([self](Caller, Span<const Val> args, Span<Val>) -> Result<std::monostate, Trap> {
|
||||||
::socket_close((u64)args[0].i64());
|
u64 handle = (u64)args[0].i64();
|
||||||
|
if(handle >= 1 && handle <= self->socket_fds.size())
|
||||||
|
{
|
||||||
|
int& fd = self->socket_fds[(size_t)handle - 1];
|
||||||
|
if(fd >= 0)
|
||||||
|
{
|
||||||
|
::socket_close((u64)fd);
|
||||||
|
fd = -1;
|
||||||
|
}
|
||||||
|
}
|
||||||
return(std::monostate());
|
return(std::monostate());
|
||||||
}));
|
}));
|
||||||
if(mod == "env" && name == "uce_host_socket_write")
|
if(mod == "env" && name == "uce_host_socket_write")
|
||||||
return(add([self](Caller, Span<const Val> args, Span<Val> results) -> Result<std::monostate, Trap> {
|
return(add([self](Caller, Span<const Val> args, Span<Val> results) -> Result<std::monostate, Trap> {
|
||||||
|
u64 handle = (u64)args[0].i64();
|
||||||
String data;
|
String data;
|
||||||
self->hostcall_read(args[1].i32(), args[2].i32(), data);
|
self->hostcall_read(args[1].i32(), args[2].i32(), data);
|
||||||
results[0] = Val(wasm_socket_write_bounded((u64)args[0].i64(), data, self->bounded_hostcall_timeout_ms(self->worker.cfg.invocation_timeout_ms)) ? (int32_t)1 : (int32_t)0);
|
bool ok = false;
|
||||||
|
if(handle >= 1 && handle <= self->socket_fds.size())
|
||||||
|
{
|
||||||
|
int fd = self->socket_fds[(size_t)handle - 1];
|
||||||
|
ok = fd >= 0 && wasm_socket_write_bounded((u64)fd, data, self->bounded_hostcall_timeout_ms(self->worker.cfg.invocation_timeout_ms));
|
||||||
|
}
|
||||||
|
results[0] = Val(ok ? (int32_t)1 : (int32_t)0);
|
||||||
return(std::monostate());
|
return(std::monostate());
|
||||||
}));
|
}));
|
||||||
if(mod == "env" && name == "uce_host_socket_read")
|
if(mod == "env" && name == "uce_host_socket_read")
|
||||||
return(add([self](Caller caller, Span<const Val> args, Span<Val> results) -> Result<std::monostate, Trap> {
|
return(add([self](Caller caller, Span<const Val> args, Span<Val> results) -> Result<std::monostate, Trap> {
|
||||||
u64 sockfd = (u64)args[0].i64();
|
u64 handle = (u64)args[0].i64();
|
||||||
|
if(handle < 1 || handle > self->socket_fds.size() || self->socket_fds[(size_t)handle - 1] < 0)
|
||||||
|
{
|
||||||
|
results[0] = Val((int32_t)0);
|
||||||
|
return(std::monostate());
|
||||||
|
}
|
||||||
|
u64 sockfd = (u64)self->socket_fds[(size_t)handle - 1];
|
||||||
u32 max_length = (u32)args[1].i32();
|
u32 max_length = (u32)args[1].i32();
|
||||||
u32 requested_timeout = (u32)args[2].i32();
|
u32 requested_timeout = (u32)args[2].i32();
|
||||||
u64 requested_ms = requested_timeout == 0 ? self->invocation_remaining_ms() : (u64)requested_timeout * 1000;
|
u64 requested_ms = requested_timeout == 0 ? self->invocation_remaining_ms() : (u64)requested_timeout * 1000;
|
||||||
@@ -4375,7 +4614,7 @@ private:
|
|||||||
int32_t buf = args[3].i32();
|
int32_t buf = args[3].i32();
|
||||||
u32 cap = (u32)args[4].i32();
|
u32 cap = (u32)args[4].i32();
|
||||||
// The size and fetch calls share this key, while the remaining budget may change between them.
|
// The size and fetch calls share this key, while the remaining budget may change between them.
|
||||||
String key = std::to_string(sockfd) + ":" + std::to_string(max_length) + ":" + std::to_string(requested_timeout);
|
String key = std::to_string(handle) + ":" + std::to_string(max_length) + ":" + std::to_string(requested_timeout);
|
||||||
String out;
|
String out;
|
||||||
if(buf != 0 && self->staged_socket_read_key == key)
|
if(buf != 0 && self->staged_socket_read_key == key)
|
||||||
{
|
{
|
||||||
@@ -4442,7 +4681,7 @@ private:
|
|||||||
// before the hostcall stack unwinds, so `self` points to the child's
|
// before the hostcall stack unwinds, so `self` points to the child's
|
||||||
// copy of this per-request workspace. The parent request can return and
|
// copy of this per-request workspace. The parent request can return and
|
||||||
// destroy its workspace without invalidating the child copy.
|
// destroy its workspace without invalidating the child copy.
|
||||||
u64 task_timeout_ms = timeout > UINT64_MAX / 1000 ? UINT64_MAX : timeout * 1000;
|
u64 task_timeout_ms = timeout == 0 ? UINT64_MAX : (timeout > UINT64_MAX / 1000 ? UINT64_MAX : timeout * 1000);
|
||||||
auto run_callback = [self, callback_id, task_timeout_ms]() {
|
auto run_callback = [self, callback_id, task_timeout_ms]() {
|
||||||
String error = self->run_task_callback(callback_id, task_timeout_ms);
|
String error = self->run_task_callback(callback_id, task_timeout_ms);
|
||||||
if(error != "")
|
if(error != "")
|
||||||
@@ -4597,6 +4836,9 @@ inline WasmResponse wasm_worker_serve(WasmWorker& worker, const Request& request
|
|||||||
WasmResponse response;
|
WasmResponse response;
|
||||||
f64 serve_started = time_precise();
|
f64 serve_started = time_precise();
|
||||||
auto workspace_start = std::chrono::steady_clock::now();
|
auto workspace_start = std::chrono::steady_clock::now();
|
||||||
|
#ifdef UCE_WASM_HOST_CONNECTORS
|
||||||
|
worker.mysql_evict_idle(workspace_start);
|
||||||
|
#endif
|
||||||
f64 cpu_started = wasm_thread_cpu_time();
|
f64 cpu_started = wasm_thread_cpu_time();
|
||||||
struct rusage thread_runtime_start = {};
|
struct rusage thread_runtime_start = {};
|
||||||
bool thread_runtime_profiled = worker.cfg.profile_thread_runtime && getrusage(RUSAGE_THREAD, &thread_runtime_start) == 0;
|
bool thread_runtime_profiled = worker.cfg.profile_thread_runtime && getrusage(RUSAGE_THREAD, &thread_runtime_start) == 0;
|
||||||
|
|||||||
Reference in New Issue
Block a user