Hosted realm on Fastly Compute: implementation plan

Plan, 2026-09-26. C++ on Fastly Compute (fastlyc), state in Fastly KV. Builds on hosted-realms.md (the why) and the protocol spec (the what).

1. Push or pull? Both, and the store is the truth

Fastly Compute runs a request handler per HTTP request. It cannot hold a listening socket, keep a TCP connection between requests, or run a background loop. Three consequences:

1. Every message rests in KV first. Whoever delivers it later reads it from there. This is the email property and it is what makes offline laptops work: the store is the queue.
2. Push is possible, through Fanout. Fastly Fanout is the edge holding long-lived WebSocket, SSE or long-poll connections on Compute's behalf. A handler hands a connection off to Fanout with a channel name; later, any handler publishes into that channel through Fastly's publish API and Fanout writes to the open socket. So a connected relay is pushed to within milliseconds of a message landing in KV, and Compute never holds the socket itself. This is the pattern Slack, Firebase and push notification services use: durable queue, notify if connected, sync on connect.
3. Pull remains for everyone. A relay that just connected (or a client without Fanout) fetches its backlog by prefix from KV. Push is an optimisation on top of a correct pull.

There is a third delivery mode that costs nothing extra on Compute: webhook. A workspace that is reachable (a home server with a port) registers an HTTPS endpoint, and Compute POSTs each message to it as it arrives, which is exactly the HTTPS peer binding below. So per workspace, delivery is one of: fanout (connected), webhook (reachable), pull (neither), all backed by the same KV queue, all acked the same way.

2. A second binding of the protocol: HTTPS

The spec's only binding today is a JSONL stream. Compute cannot originate or accept one, so the hosted realm needs an HTTPS binding, and because a federated peer must be able to reach airc.dev and vice versa, the binding belongs in the spec, not in the app. Proposed for draft 0.3:

Authentication maps one-to-one onto what the stream binding already has:

• Peer requests carry the signed msg; no hello is needed because each request is authenticated by the message signature (§9.3). The welcome signature's job (proving the destination holds the realm key) is done by a signed capabilities document: GET /.well-known/airc returns the realm's kids and a signature over the document and a request-supplied nonce.
• Client requests carry a signed header: Authorization: AIRC-Key <kid>:<ts>:<nonce>:<sig> with sig over "airc-client\n<realm>\n<method>\n<path>\n<ts>\n<nonce>\n<sha256(body)>", verified against the workspace's registered key, 300 s skew, nonce kept in KV (ADD mode, short TTL) against replay. This is the "key-signed client hello" of hosted-realms.md §3.1 in per-request form.

Discovery: _airc-https._tcp.<realm> SRV → host:443, or a t=https field in the _airc.<realm> TXT record; a server that publishes both bindings lets the peer choose. Decision for Rafael: whether spec 0.3 makes the HTTPS binding mandatory for federated servers (recommended: yes, since a Compute realm cannot reach a stream-only realm, and HTTPS is what every hosting platform can do), with the stream binding staying for the client face and for servers that prefer it.

The Python aircd implements both sides of the HTTPS binding first. That gives the C++ service an interoperability target and lets oroboro.com talk to airc.dev on day one.

3. Architecture on Compute

 relay/laptop ──HTTPS/WSS──▶ Fastly edge ── Compute handler (C++) ──┬── KV: msgs, queues, seen, links, passports, workspaces, nonces
 other realm  ──HTTPS──────▶      │                                 ├── Secret Store: realm key, Fastly API token
                                  │   ◀── Fanout publish API ◀──────┤── Config Store: policy rules, limits
                                  ▼                                 └── Log endpoint: events.jsonl → our ingest
                            Fanout (WS/SSE held at the edge)

One Compute service, one realm (airc.dev), workspaces as namespaces (//airc.dev/joe/…). The handler is a router over the paths in §2, in the style of the existing C++ Compute apps (single main(), route by path, fsut* helpers).

3.1 KV data model

KV is eventually consistent and has no transactions, but fastlyc exposes what a queue needs: ADD (insert only if absent), APPEND, TTL, IF_GENERATION_MATCH (compare-and-swap), prefix LIST, DELETE.

Guarantees: at-least-once delivery with dedupe at the receiver (seen/<id> at the destination server, plus the relay's own dedupe on the id header) and exactly-once acknowledgement (deleting q/… is idempotent). Ordering: per namespace, by key, which is by origin timestamp then id; good enough for messages, and the relay's one-in-flight rule keeps delivery ordered.

3.2 Request flows

Inbound from a peer (POST /airc/v0/msg): parse; canonicalize; check to.realm == airc.dev; fetch peer's keys (_airc.<realm> TXT through a DNS backend, or the peer's /.well-known/airc, cached in KV with TTL); verify signature; ADD seen/<id> (exists → 200 delivered duplicate); policy (config rules → links → default); write m/<id>; write q/<ns>/…; then by workspace mode: fanout → publish the frame to channel ns:<ns> and answer queued with code pushed (the relay's ack will arrive on /ack); webhook → POST to the workspace URL with a short timeout, answer its ack (delete q/… on delivered); pull → answer queued endpoint_offline. Log an event.

Note the change from the stream binding: a Compute handler cannot wait 15 s for a relay's ack over a socket it does not hold, so push mode acks queued, and delivered is recorded when the relay acks. Senders already treat queued as success; the audit stream shows the delivery. (A future option: hold the request up to N seconds and poll for the ack; costs compute time, buys a delivered in the synchronous reply.)

Relay connects (GET /airc/v0/ws): authenticate; hand off to Fanout on channel ns:<ns>; the relay then calls GET /inbox to drain the backlog, acks each, and thereafter receives pushes.

Relay sends (POST /airc/v0/send): authenticate; from must be inside the workspace; local destination → same as inbound; remote → sign with the realm key, resolve the peer, POST to its /airc/v0/msg (dynamic backend), return its ack; on failure write q/<realm>/… and answer queued.

Retries and expiry. With no background loop, retries happen when something pokes the realm: a relay connecting drains its queue; a peer becoming reachable is retried by a tick request (POST /airc/v0/tick, authenticated, called every minute by an external scheduler such as our co cron --exec) that lists q/<realm>/ prefixes and re-sends. Expiry is free: TTLs on m/ and q/.

3.3 Push with Fanout

• Fanout must be enabled on the service (Fastly add-on). fastlyc needs one wrapper for the hand-off hostcall (fastly_http_req_redirect_to_grip_proxy / handoff_fanout); everything else is HTTP (GRIP headers on the hand-off response, POST https://api.fastly.com/service/<id>/publish/ with the API token from the Secret Store).
• WebSocket-over-HTTP: each WS event from the relay (open, text, close) arrives as an HTTP request to the handler, so client frames (ack, link, passport, ping) work over the socket unchanged.
• If Fanout is unavailable or refused, the relay falls back to pull with long-poll (GET /inbox?wait=20) which the handler serves by re-listing the queue a few times before answering empty; acceptable for a trial, wasteful at scale.

3.4 Crypto and libraries (as built in H1)

• ed25519: added to libs/crypto (ed25519.cpp, TweetNaCl-derived, constant-time field arithmetic, SHA-512 from kx). Verified against RFC 8032 and the spec §18.2 vectors, all regenerated with python-cryptography. Release speed is about 1.6 ms per sign and 3.2 ms per verify, fine for a message service; a ref10 port behind the same three functions is the upgrade if volume demands it. Two kx bugs surfaced and were fixed on the way: SHA-512 encoded the padded rather than the message length (every digest wrong; HMAC-SHA512 and any HS512 JWT were affected), and KxCbufBase::operator== was inverted.
• Frames: dl atoms (Dl, DlAtom), parsed with DlParser in JSON mode. Known limit: dl holds numbers as float32 and s32, so a message's fractional ts cannot be read exactly from the tree; the core scans the raw JSON literal for ts (aircFrameTs), creates its own frames with whole-second timestamps, and forwards received frames verbatim (never re-serialised), so signatures always survive. Link records' created ordering has the same precision limit. The proper fix is a 64-bit number type in dl (an out-of-line double, as objects and arrays are stored), which also removes the 2038 limit for integer timestamps; proposed to Rafael.
• Strings and containers: interned KxSymbol for every recurring name (realms, segments, keys, kids, signatures), KxCbufBase for built text, KxVector / KxHashDict for records; no STL.
• The protocol logic has no Fastly dependency and builds for wasi (wkjam -s PLATFORM=wasi produces airc_test.wasm); H2 adds the Compute app on top of it.

3.5 Limits to design against

Compute request time (tens of seconds), no threads, 128 MB; KV values up to 25 MB (frames are ≤ 1 MiB), KV rate limits per key; eventual consistency (seconds at worst) so a pull right after a write may miss it once, which the next push or poll covers; publish API latency (~100 ms). Costs scale with requests, not idle workspaces: an idle Fanout connection is cheap, a message is roughly five KV operations and one publish.

4. Registration and identity

• A workspace is POST /airc/v0/workspaces with a public key and a requested name; the realm answers with the namespace. Signup policy is a config choice: open with rate limits and a proof-of-work or e-mail step, or invitation by passport-like token. Keys rotate through POST /workspaces/<ns>/keys.
• Registrar mode (hosted-realms.md §4) is a separate small API that writes DNS records for <ns>.airc.dev through the DNS provider's API; not a Compute concern beyond hosting the endpoint.

5. What changes elsewhere

• Spec 0.3: the HTTPS binding (§2 here), per-request client authentication, the pushed ack code, capabilities document, discovery field. Everything else is unchanged.
• Python aircd: server and client sides of the HTTPS binding (the server side as a small HTTP/1.1 handler on the existing asyncio loop or http.server in a thread; stdlib either way), a --server https://… mode for airc relay/airc send with a workspace key, and the interop tests against a local mock of the Compute app's routes.
• Site: the spec update, and a "run your agents from a laptop" page once airc.dev exists.

6. Phases

H0 and H1 can run in parallel; H2 is the bulk. A first useful milestone is H2 without Fanout: a laptop relay that polls every few seconds already gives a person offline-tolerant agent messaging under //airc.dev/<ns>/….

6a. As built (H2)

src/aircd_edge/ is the realm: airc.cpp (routes), edge_store.cpp (KV model, policy, delivery), edge_net.cpp (DNS over HTTPS discovery, peer verification, peer and webhook posting). Findings that changed the plan:

• Queues are one KV key per message (q/<ns>/<arrival ms>-<id>, value = the raw frame as received, qi/<id> pointing back) rather than an appended log: viceroy 0.16 rejects the KV append mode, per-key TTLs expire messages individually, and settlement is a delete. Events use the same shape (ev/<ns>/...). Listing is done in strong mode; an eventual listing can lag a write by seconds, which is wrong for a queue.
• Received frames are stored and forwarded byte for byte. Signatures therefore survive dl's float32/s32 numbers; only frames this realm originates are built here (whole-second ts).
• Discovery runs over DNS-over-HTTPS (a doh backend, cloudflare-dns.com) because Compute has no resolver; results are cached in KV for an hour. A peers entry in the config store overrides discovery for static or plaintext peers (tests, private federations).
• Outbound connections are dynamic backends, one per host:port, registered at request time (fsutRegisterDynamicBackend, new in fastlyc).
• KV keys may not contain |; key parts are separated with /.
• Two fastlyc additions: FsutKvStore::list (prefix listing) and fsutRegisterDynamicBackend.

7. Decisions needed

1. HTTPS binding mandatory for federated servers in spec 0.3? (recommended)
2. Fanout: enable the add-on for the airc.dev service, or start with pull and webhook only?
3. Domain: airc.dev (to acquire) or a subdomain we already own for the first version.
4. Signup policy for workspaces: open with limits, or invitation only.
5. Push-mode acks: queued pushed immediately (recommended), or hold the request for a few seconds to return delivered.