Development UI and execution telemetry

A history entry keeps sequence, kind, and recorded_at_millis, and gains four optional parts, written in the same transaction as the transition from state that transaction already loaded:

An explicit retry’s successor is not recorded on the resolution: the successor generation’s attempts carry their own generation, which is how the trace links them to the attempt the operator retried.

A lease acquisition already rewrites its work record, so the record also keeps the time the current attempt was first dispatched. This applies to discovery work and step work alike. Renewing the lease keeps it. The settling transition copies it into history with no additional read or write: attribution grows the history entry the transition was already writing and adds nothing else to the dispatch path.

Entries committed before attribution decode with none of these parts. The trace places them on the run root as unattributed points rather than assigning them to a step.

The trace already draws every attempt that committed a fact, with its ready and dispatch times. It cannot say where the attempt ran. An operator looking at an ambiguous or slow attempt needs the transport and the worker that held it, and needs that answer to survive the worker disconnecting and the bounded worker history overwriting the session.

Where work goes is a property of a connected session or an HTTP endpoint, not of one attempt, so the server records it once as a dispatch:

The worker reports every identity field, so each is bounded when recorded: a string longer than 256 bytes is cut at a character boundary and the dispatch is marked truncated. A dispatch is named by the first 8 bytes of SHA-256 over a fixed domain tag and its encoding, as 16 hexadecimal characters, and stored once under the application. Inserting that immutable record succeeds when the key is absent or already contains the same dispatch. If the key contains different content, the insertion reports a collision and never replaces either value.

A connected session makes one recording attempt after it becomes ready; failure does not end the session and every lease that session takes stays unattributed. The HTTP dispatcher attempts to record its endpoint before each lease until one attempt succeeds, then caches the identifier for the process. Storage transaction retries inside either attempt use the ordinary bounded retry policy. Exhausting it, an encoding failure, or an identifier collision degrades only telemetry: dispatch continues without a dispatch_id. Recording is outside the lease transaction, and no lease names an identifier unless its immutable record was confirmed first.

A lease acquisition already rewrites the work record, and that write is where the server decides who receives the work. The same write therefore records the dispatch_id beside dispatched_at_millis, adding no read, no write, and only the identifier’s bytes to the dispatch path. Work the server runs itself, such as a timer or a fan-out page, names none. Renewing a lease keeps the identifier, and so do Unknown and completion. Work that returns to the queue clears it with the dispatch time, so it always names the current attempt’s lease.

The settling transition copies the identifier into the history entry’s attempt or pass from the work record it already loaded, and a resolution entry copies it from the unknown work it resolves. The identifier appears wherever an attempt is read: the run detail’s current work, each listed Unknown attempt, and the trace’s attempt and pass spans. The run detail and the unknown-attempt list also carry each dispatch their attempts name, read once per distinct identifier in the same read transaction, so the cost grows with the sessions that served a run rather than with its attempts. Records and entries written before dispatches were recorded name none, and the page says the dispatch was not recorded rather than guessing.

Recorded dispatches grow with connected sessions and HTTP endpoints, not with attempts. This slice defines no dispatch-record collector; the development server keeps them indefinitely. A future hosted retention policy may remove one only after no retained history, current or Unknown work, live connected session, or running HTTP dispatcher can name its identifier.

There is no separate attempt endpoint. A step’s attempt history is the attempt spans under its step span, each with its generation, attempt number, status, dispatch and settlement times, and dispatch. The page reads readiness from the matching history attempt or current work’s vested time, never from the trace span’s start: the projector may use dispatch or settlement only as a drawing anchor when recorded readiness is absent. That history is bounded by the run detail’s history page and flagged as incomplete when the page is truncated.

A dispatch says where the server sent the work, not that callback code ran there or how long it ran. Ready-to-dispatch is queue wait. Dispatch-to-settlement includes delivery, execution, and the response, so it is not labelled as execution time. A release explains which part of the fleet received the work. It does not pin later attempts of the run to that release, and the page never implies affinity.

Some lease deliveries commit no history entry. When the run still permits the work, an effect delivery released for redelivery before callback start returns the same physical application attempt to the queue: its next delivery has a new lease generation but the same attempt number, so it leaves no numbered attempt gap. Other pre-start outcomes retain the retry, obsolescence, skip, and stop-intent semantics defined by RFD 4 and RFD 6. A discovery delivery whose response is lost is recovered after lease expiry as the next discovery attempt; only the later accepted pass is recorded, so its larger attempt number exposes the gap. Recording every such delivery would add a write on release and recovery paths, so it is not proposed unless those invisible deliveries prove confusing in practice. An effect attempt whose lease expires after start authorization becomes Unknown and keeps its dispatch until it is resolved.

The run page shows a step’s attempts in its Steps view, oldest first. For each it shows the generation and attempt, the status, when the attempt became ready, the queue wait, and the dispatch-to-settlement interval, and says where the attempt was dispatched in words. A connected worker is named by its instance and release, and the session links to the worker history. An HTTP endpoint is shown as recorded. An unresolved attempt also says where it was dispatched, beside the resolution controls. A selected attempt span in the trace shows the same facts. Activating an attempt with a pointer or keyboard opens Trace and moves focus to that visible detail heading.

The page does not turn missing evidence into activity. Current work supplies an exact queued, leased, authorized, unknown, fenced, or completed state; an open span without matching current work is only open, not in flight. If its ending may fall beyond a truncated history page, it is open in partial history. A server that supplies no trace makes attempt history unavailable, which is different from a complete trace with no recorded attempts. An incomplete trace qualifies its counts, empty state, and open endings as partial.

Copying the whole dispatch onto every lease and history entry made each lease, renewal, completion, and attributed entry grow by about 300 bytes, and an exploratory local comparison showed a material throughput regression. That prototype result selected the content-identifier design; it is not a portable performance claim. The implementation evaluates the final design instead: the lease and settlement paths are measured with the common seeded benchmark workloads against the revision before dispatch facts, with every lease naming a representative connected dispatch and the reproducible profile and raw results reported with the implementation.

A dispatch says where an unresolved attempt went. It does not say why its outcome is unknown, and the operator choosing between a supplied success, a declared failure, and an explicit retry needs that: an endpoint that answered 500 after the callback was authorized, a connection that dropped while a worker held the attempt, and a worker that never answered are different facts about the same unknown outcome. Today the dispatcher observes each of them and discards it, and the attempt becomes Unknown only when its lease later expires, so the only durable trace of the cause is the expiry.

The dispatcher therefore records what it observed on the work record, as the attempt’s ambiguity: a cause and the server time it was observed. RFD 6 classifies HTTP outcomes by what they prove about callback execution; an ambiguity is recorded only for the class that proves nothing, after start authorization of effectful work. Repeatable discovery is re-vested after the same observations and records none, and the other three HTTP classes either return the attempt to the queue or settle it.

Recording the ambiguity is one write on the failure path only, outside the lease transaction, and it degrades only telemetry: a recording failure leaves the attempt as it was, and expiry recovery still makes it Unknown with lease_expired. The write applies only while the record still names the dispatching lease with start authorized, or is Unknown under that same lease with lease_expired, because a slow HTTP response can return after expiry recovery. A recorded cause other than lease_expired is never replaced. The write publishes a run change, so an open run page shows the cause on the current work before the lease expires. Work that returns to the queue clears the ambiguity with its dispatch, and Unknown and completion keep it, so a late result that settles an unknown attempt still says why it had been unknown.

The ambiguity travels with the dispatch identifier: on the run detail’s current work, each listed Unknown attempt, the history attempt of every settling or resolution entry that copies it from the work record, and the trace’s attempt spans. Records written before ambiguity was recorded decode with none, and the page says the cause was not recorded rather than guessing. An attempt that settles without any recorded ambiguity omits the field, so the ordinary success path writes no additional bytes. A late or operator-supplied success retains any ambiguity already recorded.

The page explains an unresolved attempt with its cause and dispatch in one sentence beside the resolution controls, such as "The HTTP endpoint answered 500 after the callback was authorized to start", followed by what that cannot prove. A resolution entry in the durable history, and the selected span of an unknown or resolved attempt, carry the same explanation, so the decision stays legible after the attempt leaves the projection. Where the attempt went over HTTP, the page lists the four RFD 6 outcome classes with what each proves and marks the ambiguous class as applied only when the attempt’s current state or retained history records ambiguity. An ordinary canonical result or a pre-start outcome carries no such marker. A pre-start delivery returned to the queue has no separate historical delivery entry, while a pre-start outcome that settles or skips a target keeps its normal attributed history; the page distinguishes those cases rather than implying either was observed for this attempt.

The run detail projection gains a trace, computed in the same read transaction as the rest of that projection, so it shares its cursor and its live updates. A span has a stable ID, a parent, a kind, a name, the step, target, attempt, and generation it describes, a status, and a start and an end. A history-backed boundary carries the entry’s sequence and recorded time. A work-backed ready or dispatch time has no history sequence; neither does a scheduled deadline. An open span has no end. Spans can carry links to other spans.

Missing provenance stays missing. An attributed settlement or current work may name a step whose declaration predates attribution. Its container is synthetic and marked incomplete, anchored to the earliest available attributed fact, not to an inferred declaration time. Legacy entries remain unattributed points on the root; they do not prove a start or a duration for that container.

The hierarchy is:

Stop intent (cancelling, timing-out) is an event on the run span, not a span: the run keeps settling work after it. Steps that the stop fences close as obsolete. The existing stop entry names the actual fenced target from its loaded work, without another history write; a retained settlement token is not fenced and therefore is not attributed as obsolete. That same entry closes a pending retry delay at the stop, never at its planned ready time.

A discovery attempt whose response is lost, or whose worker dies, commits nothing. Recovery requeues the work with a new ready time and a higher attempt number, and only the pass that commits is drawn. The time lost to the earlier attempt is a gap before that pass, and its attempt number shows that it was not the first, rather than the trace claiming an attempt it cannot prove.

Two other kinds of pass commit no entry of their own and are not drawn. A pass that re-declares a step or an emission the run already accepted is idempotent and writes nothing new. A pass that finds the run’s deadline elapsed commits the run’s timeout instead. Recording either would add a write, which is not proposed unless it proves common.

The trace ID is the first 16 bytes of SHA-256 over a fixed domain tag and the run’s organization, workspace, application, and ID. A span ID is the first 8 bytes of SHA-256 over the trace ID and the span’s path, such as step/charge/0/forward/g2/a1, where the number after the step ID counts the times the step was declared. An all-zero result has its low bit set. Both IDs are OpenTelemetry-compatible and stay the same across reads, restarts, and storage backends, so a later OTLP export can map the same spans without becoming the source of truth. A discovery pass’s path names its discovery work.

Spans are ordered by the sequence of the durable fact that opened or scheduled them, independently of their boundary timestamps. Work-only or synthetic containers use the latest available relevant history fact as an ordering anchor, not as proof of their start time. Recorded times give widths only. Every recorded time comes from the server that committed the transition, never from a worker. A span whose end time precedes its start is drawn at zero width, and its order still comes from sequence. When history_truncated is set, a span whose end lies beyond the page stays open and is marked truncated, and the trace says that it is incomplete.

The run detail draws the trace as a waterfall by default. Selecting a span shows that step’s input, output, or structured error from the step records and the attempt’s identity, so the trace and the existing step panels never disagree.

Attribution makes every history entry larger and the trace makes the run detail read do more work. Both are measured with the common seeded benchmark workloads, with revision-relative regression bounds on the settlement path and on the run detail read.

The server does not decide on its own who may call it. An Authenticator is injected into the server state, and middleware asks it about every API and gateway request before a handler can look a resource up; an admitted request carries the resulting Principal. A refusal is 401 for a caller with no acceptable identity and 403 for one that may not use the server. A hosted deployment injects its own implementation — API keys, an identity provider — without changing the handlers, and the "authorize before lookup" guarantee above is a property of that seam rather than of one credential format.

ryusen dev injects LocalAuthenticator, which admits every caller: the local development server has no identities, so the UI, the API, and the worker gateway need no credential, and a header from an earlier credential is ignored rather than refused. It is protected by its network boundary instead. It binds loopback by default, and a guard in front of every route, including the UI, refuses the two browser requests that can reach a loopback server from elsewhere: an Origin that is not a loopback host, which is another site’s page calling the server from the developer’s browser, and a Host that is not a loopback host, which is a DNS-rebinding attack. A request with neither header is not a browser request and is admitted, and any loopback port is admitted so the Vite dev server’s proxy works. Binding a non-loopback address requires --allow-remote and prints a warning; the guard then admits any Host, and an Origin that is a loopback host or the same origin as the request’s Host. It still refuses another site’s page by its origin, but no longer one whose hostname was rebound to this server’s address.

The tests that prove authorization before lookup inject a bearer Authenticator, so they exercise the seam a hosted deployment uses rather than a code path the development server no longer has.

Every live page starts from one page-level bootstrap response containing all initial projections needed by that view and a snapshot_cursor. The server reads the projections and cursor in one database transaction. A page never combines the maximum cursor from unrelated requests; if a view needs several resource types, its composite bootstrap endpoint returns them together. This avoids persistent snapshot sessions and cross-request database transactions.

The UI then subscribes to server-sent events strictly after snapshot_cursor. Any commit racing between the snapshot transaction and subscription is replayed from durable history before live delivery, and duplicate projection updates are idempotent. SSE is sufficient because browser updates are one-way; UI commands remain normal HTTP requests. On reconnect, the server replays durable history after the client’s last applied cursor before sending live changes. Polling is the fallback and missed SSE notifications never hide committed transitions.

Ephemeral connection observations carry a separate session sequence and may be refreshed from a snapshot when continuity is lost.

The bootstrap endpoints are GET /v1/bootstrap/workspace, GET /v1/bootstrap/events/{event_id}, and GET /v1/bootstrap/runs/{run_id}. Each reads its projections and one snapshot_cursor in a single storage transaction. The cursor is the inclusive upper bound of the commit positions that transaction observed, so it bounds exactly the projections returned with it: a commit that races the page is always strictly after the cursor.

Catch-up is available as a bounded poll (GET /v1/changes/workspace, /v1/changes/events/{event_id}, /v1/changes/runs/{run_id}) and as a server-sent event stream (…​/stream). Both take the same cursor and apply the same view filter and authorization, so a client that cannot hold a stream reads the same changes from the poll endpoint. Polling and streaming therefore never disagree about what is missing.

A change names the projection that changed rather than shipping it: a kind (event or run), the resource identifier, and, for a run, the event it was created from. The client re-reads the authoritative projection, so applying a change twice is safe and a change that no longer resolves is simply stale. The next_cursor of a page is always present, so a client that receives nothing still knows where it resumed from, and a bounded page advances it even when one page cannot deliver everything.

A cursor names one journal entry, not one commit: the commit position plus the entry’s append order. A transaction routinely appends several relevant changes — a run materialization appends a dispatch change and a run change for each matched run — so a cursor that named only the commit would skip the rest of that transaction on reconnect. A bootstrap cursor is the position after every entry of the commit its snapshot observed, so it still bounds exactly the projections returned with it.

Cursors are authenticated with a per-authority key that is created once and read back on restart, and bound to one view, organization, workspace, application, and optional resource. The signature is verified before the payload is parsed, so a client cannot edit the position a cursor names, rewrite its scope, or replay it against another view, and the accepted/rejected difference is not an oracle for the authority’s committed bound. A malformed, oversized, unknown-version, cross-view, cross-scope, or ahead-of-the-journal cursor is rejected before any journal or resource lookup, so a cursor can never disclose whether another workspace or application holds state. Two things are necessarily read first: the signing key, without which no cursor can be authenticated, and the current snapshot bound, which the ahead-of-the-journal check compares against. The server resolves the key once before it starts serving, so a rejected cursor never creates it; a library or router instance that is built without the server resolves it lazily on first use instead. A stream validates its cursor before it upgrades, so a rejected cursor is an ordinary HTTP error instead of a stream that fails on its first frame.

Authentication is not confidentiality. A cursor is signed, not encrypted, so a client that decodes its own token can read the commit bound it carries. That bound is authority-wide and attributable to no workspace or application, so it reveals only that the authority’s position moved; two cursors compared over time reveal aggregate movement, and a bound change does not prove that a journal entry was appended, because creating the signing key also commits a position. Hiding the bound as well would need authenticated encryption with the same durable key, which this slice does not implement.

The stream is ordinary text/event-stream, which is what the authenticated streaming fetch client consumes. The first frame is event: ready and echoes the cursor the stream started from. Each delivered change is an event: change frame whose id field and data payload both carry the cursor to resume from. Idle periods emit a comment heartbeat. No bidirectional carrier is required.

Durable scans are authoritative and the in-process notification is only a wake-up hint: every step re-scans strictly after the cursor the scan reached, so a missed notification delays a change until the next scan instead of losing it. Two cursors move at different times. The scan cursor advances when a page is read and its frames are queued, which is what lets a view whose page was filled by entries it filters out make progress instead of re-reading the same window. The client-applied cursor advances only as frames are read, so a client that stops reading stalls its own stream and applies backpressure instead of dropping changes.

Three implementation discoveries are worth recording against the proposed design:

A review of the implemented slice found four defects that the initial fixtures did not reach, all now fixed with regression tests:

A cursor is a position in the authority’s commit order, so it survives a server restart: after restart the same cursor resumes the same feed with nothing repeated and nothing skipped. The signing key is durable for the same reason, so a cursor issued before a restart is still accepted afterwards.

POST /v1/events accepts an optional caller event identifier, an optional workspace-scoped idempotency key, the event name, the payload as bytes, an optional caller timestamp, and optional routing metadata. The acknowledgement is versioned, so a client can rely on fields rather than on a message: submission_version is 1, and disposition is Created for the submission that committed the event or Replayed for an identical retry under the same key. The remaining fields are the durable event identity, the workspace routing generation the event was routed against, and the expansion state the acknowledging transaction observed.

A refused submission is a versioned JSON body rather than plain text, so a client can act on a code:

idempotency_key_reused is the only code that forbids a retry: the key already names a different canonical request, and the body carries neither the key nor the stored fingerprint. conflict covers the environment refusals a retry cannot fix either — no active routing generation, or a duplicate event identifier — while invalid, limit_exceeded, and server keep their own statuses. The codes are distinct because presenting an environment conflict as a reused key would tell an operator to abandon a submission that was never claimed.

Both envelopes are read only at the version this client implements: an acknowledgement or refusal whose submission_version is missing or unknown, or whose code is not one of the documented ones, is not described as if it were the contract. The HTTP status still classifies the outcome, and the client uses its own wording for the rest rather than quoting a body it cannot attribute.

The embedded UI models one deliberate submission as one intent. The key is generated when the form is submitted and reused by every retry of the same canonical request — the event name and the payload bytes the server fingerprints — so reformatting a byte array or moving the same bytes between the text and byte-array modes does not rotate it. It is rotated when that request changes, when the operator starts a new submission, or once the intent has completed. A double activation is one intent even when the first acknowledgement returns between the two clicks of the gesture, because the form guards on the activation’s own click count as well as on an in-flight flag, not only on the disabled button.

A transport failure — the request left the browser and no response came back — is reported as ambiguous, and so is a successful status whose acknowledgement cannot be read, because the event may be durable while its identity is unknown. The browser’s own transport message is shown only as a labelled local diagnostic stating that it does not determine the server outcome, so it cannot be read as a second, contradictory verdict on the submission. A non-success status is classified from the status itself even when its body cannot be read, and the server-error wording claims only that the submission was not acknowledged, because a proxy failure does not prove the event was not committed. The ambiguous and server states keep the key so that a retry replays an event that may have committed. The form inserts no row of its own, so a created event reaches the overview through the same durable live updates as any other change and the page cannot show a best-effort history that disagrees with the server.

Payloads are collected in the wire format the endpoint accepts. The text mode sends the typed characters as their UTF-8 bytes and the byte-array mode reads the literal JSON array of integers from 0 to 255, so an opaque payload can be sent exactly; the encoded byte count is shown next to the field. Local validation refuses only a request that cannot be canonical at all — an empty event name, or a byte array that is not one — and never replaces the server’s own refusal.

The idempotency key lives only as long as the intent it was generated for. It is never rendered, logged, placed in a URL, or written to browser storage, and a page reload discards the form rather than resubmitting it, which the form states. Server-provided diagnostic text is treated as untrusted for the same reason. The response document is classified exactly as it arrived — redacting it before it is parsed would corrupt a protocol field that happens to contain the key — and redaction is applied only to the decoded values the form renders: the message of a refusal envelope, and the browser’s own transport message. Matching is case-insensitive, because an echo is free to change the case of what it repeats, and an identity field that carries a request secret is refused rather than rewritten, because a rewritten identifier would link to a different event. No other foreign text is rendered: an unstructured response body is reported through its status and the client’s own wording, never quoted, and the credential is captured when the request is sent, so a session that rotates or clears it while a response is outstanding cannot leave what was sent unredacted. What remains outside this guarantee is a responder that fabricates the versioned envelope itself and embeds a partial secret in its message: a partial echo cannot be matched exactly, which is why the client renders no text it cannot attribute to a documented field. Two limits are deliberate here: the form does not persist unsent edits, because persisting them would either keep a request-scoped key beyond its intent or introduce a second storage contract for a development form; and it does not expose the caller event identifier, caller timestamp, or routing metadata, which remain documented API fields exercised by the server and benchmark fixtures rather than by this form.