Durable run state machine and work queue

Define the transactional state machine that turns protocol declarations into durable forward and compensation work. Internal state and resulting work must commit together so retries repair ambiguous acknowledgements without cross-store cleanup.

The initial engine persists versioned records for application-manifest generations, workspace routing generations, connected release registrations, events, fan-out progress and matches, runs, steps, attempts, immutable history entries, work items, leases, and idempotency claims. Secondary indexes are derived durable data maintained in the same transaction as their source records.

A run moves through Running, DefinitionBlocked, Cancelling, Compensating, and Terminal phases. Fan-out creates it as Running together with its first discovery work. Running includes work queued for dispatch or waiting for a compatible worker; those are operational statuses rather than lifecycle phases. The initial model has no generic Pending phase. Future scheduling, admission, or approval features should introduce states with their precise semantics. DefinitionBlocked is a recoverable pause caused by a workflow-definition conflict, not an execution failure. Cancelling also settles work after run timeout; the durable stop reason distinguishes cancellation from timeout. Terminal records preserve two orthogonal outcomes: execution (Completed, Failed, Cancelled, or TimedOut) and recovery (NotRequired, Compensated, PartiallyCompensated, or CompensationFailed). Steps and attempts use a more detailed state machine defined by RFD 1, including Obsolete and Unknown. The recovery value is assigned only when a run terminalizes; a Compensating run blocked by Unknown has no terminal recovery outcome yet.

Every accepted transition appends a history entry with the same commit. History is sufficient to explain why work was created, retried, skipped, made obsolete, or compensated; it is not a second asynchronously updated observability store.

At minimum, the following are single transactions:

An HTTP response is sent only after its authoritative transaction commits. Wake-up publication occurs after commit and is best effort.

An optional caller idempotency key is scoped by workspace and the event.submit operation. It cannot collide with another workspace’s key or an idempotency key for a different API operation. The submission transaction stores a versioned cryptographic fingerprint of the canonical caller-controlled fields that affect the event record or routing, including event name, payload, caller-supplied timestamp, and routing metadata. It excludes credentials, transport headers, generated event identity, and the server-selected workspace routing generation.

The claim, event, selected workspace routing generation, and first fan-out obligation commit together. An identical retry returns the original event identity and its current durable expansion state without creating another event or fan-out item. Reuse of the scoped key with a different fingerprint returns an idempotency conflict and mutates nothing. Concurrent identical submissions serialize on the claim and converge on the same event. Without a caller key, each accepted submission is intentionally a new event.

Event acknowledgement does not depend on a live worker or an external event stream. Its transaction reads the workspace’s active routing-generation head and commits the event plus one durable obligation to evaluate that immutable generation. The response returns the event ID, selected routing generation, and expansion state. A later application activation never changes an acknowledged event’s routing set, and worker disconnect never changes trigger registration.

Fan-out work traverses the selected generation’s application-manifest references and evaluates a bounded range of trigger bindings. Its transaction records each binding’s match result, creates a run with a deterministic identity derived from workspace, event, application, and binding IDs as Running with its first discovery item when matched, advances the durable cursor, and queues the next chunk. Retrying any chunk is idempotent. Runs may become visible incrementally, but expansion reaches Complete only after every candidate binding for the event name across every referenced application manifest has a durable evaluated record. Zero matches is a successful complete outcome rather than a lost event.

The initial exact-name matcher can usually finish in one small chunk. The same state machine permits later CEL predicates and hosted fan-out without requiring one unbounded transaction. Every created run records its source event, trigger binding, workspace and application IDs, application-manifest generation, and workspace routing generation so the association remains explainable after new manifests are published.

A work item contains workspace, optional application, kind, logical target, priority, vesting time, an immutable application-attempt ordinal, a mutable delivery count, generation, and stable operation ID. Application SDK work always carries exact workspace and application scope. Workspace-wide server work such as event fan-out carries its workspace and corresponding durable owner without pretending to target one application. Lease recovery and redelivery change only delivery identity and fencing; they do not consume or renumber application attempts. A request ID may therefore change across deliveries of one physical item, while protocol attempt and operation_id preserve its physical and logical identities respectively. Ready indexes are ordered by vesting time and priority. Dispatchers scan vested work, acquire a random-token lease with an expiry, and renew long attempts.

The durable queue is shared infrastructure for user execution and internal control-plane operations. Its work kind selects one of three dispatch domains:

Server-owned handlers must be idempotent transactional state transitions. Their lease loss is safely requeued and never becomes Unknown; no connected or HTTP worker advertises capacity for them. Work kinds share persistence, vesting, leasing, wake-up, and retry machinery while retaining independent dispatch and later capacity or fairness policies. This avoids a second scheduler without pretending internal and user work have the same failure boundary.

Lease expiry before start authorization makes an item eligible again. The same is true after authorization for server-owned idempotent work and protocol work classified as repeatable, including workflow discovery. An authorized effectful forward or compensation callback instead becomes Unknown; it is not blindly dispatched again. Result acceptance requires the current lease or an unresolved settlement identity, plus the run and step generation. A stale worker cannot commit state after its work has been retried, cancelled, resolved by an operator, or made obsolete.

The initial SQLite dispatcher may use one global ready index and one process. The record model must permit later tenant partitions and bounded fair dequeue, but hosted fairness is not implemented in this RFD.

Connected dispatch evaluates release eligibility when assigning each work item; the run does not retain a release requirement. It hard-filters the target workspace and application authorization scope, function, current execution contract, protocol and value-codec support, and features required by durable history. Among eligible live groups it prefers the greatest server-assigned release position, uses instance capacity for balancing, and excludes draining connections. The accepted start records scoped release and instance metadata for explanation, not future affinity. HTTP dispatch bypasses release selection but requires the configured endpoint’s credential-derived scope to match before invocation.

A run has no deadline unless its function declaration or a visible deployment policy supplies one. There is no universal engine ceiling, and local development does not impose one. A hosted policy may define a maximum accepted duration; an exceeding manifest or invocation is rejected instead of clamped. Attempt timeouts and leases remain independent from the whole-run deadline.

When a deadline exists, the fan-out transaction stores its absolute value and source on the run and creates one generation-fenced, server-owned RunTimeout work item vested at that instant. The deadline is authoritative; the item is the durable wake-up mechanism. Every run in Running or DefinitionBlocked retains that item until terminalization or timeout intent consumes it. Queue notification remains best effort because delayed-index scanning rediscovers the item after restart.

The timeout handler and every authoritative forward result or discovery response compare the engine’s current time with the stored deadline in their transaction. A transition evaluated before the deadline may commit normally and atomically remove stale timeout work when it terminalizes the run. At or after the deadline, it first materializes TimedOut stop intent; delayed timer dispatch cannot let completion bypass the deadline. A worker-supplied clock never controls this decision, and conformance tests use an injected engine clock.

Materializing stop intent and recording the reported outcome are one transaction, so a result that arrives at or after the deadline settles the attempt under stopping semantics whether the timer or the result commits first. The run’s execution outcome stays TimedOut in either order, and a retryable reported failure settles without scheduling the retry its declaration would otherwise allow.

An operator resolution of an unresolved forward attempt follows the same rule. A supplied success or a declared failure on a still-running attempt at or after the deadline commits TimedOut stop intent and settles the attempt under stopping semantics in the same transaction: the supplied success completes the step and joins the unwind, the declared failure settles without any forward retry, and the run compensates or terminalizes an empty unwind only once every authorized attempt has settled. Explicit retry is refused as deadline_elapsed with nothing mutated, whether the deadline has elapsed on a still-running attempt or the run has already committed timeout intent. The refusal is decided from durable state, not from the sampled clock alone, so it never depends on whether the timer has been dispatched: waiting cannot make the retry available, and a supplied success or a declared failure can still settle the attempt. The operator’s recorded resolution owns replay of that combined transaction, so a repeated request returns the response the first one committed rather than re-deriving it from the run’s later state.

The first committed stop reason decides. Cancellation and timeout still race normally, and a later clock sample never replaces the outcome that committed: a run stopped as Cancelled keeps refusing explicit retry as retry_forbidden after the deadline elapses, and a run stopped as TimedOut keeps answering deadline_elapsed even if the deadline has since moved below a sampled clock.

If discovery has not been authorized, timeout fences it and terminalizes an empty unwind. Authorized repeatable discovery is discarded. An authorized effectful callback enters normal Cancelling settlement: a late success is accepted, included in the unwind, and can produce TimedOut + Compensated; an ambiguous outcome becomes Unknown. Once timeout intent exists, the original run timer is consumed and compensation uses its own attempt timeout policies.

A declaration may be a durable timer rather than a callback. The accepting transaction records the instant the wait may settle at — the acceptance instant plus a relative delay, or the absolute deadline the workflow named, never earlier than acceptance — with the step, and creates one workspace-scoped Timer item vested at that instant on the shared delayed queue. The anchor is durable, so a rediscovery, a replay, or a restart never measures the delay again, and a repeated declaration of the same step keeps the recorded anchor and its wake work.

Firing the item records the accepted wake as an ordinary successful step result and schedules the continuation discovery in the same transaction that completes the work, so a redelivered timer can only find work the first transition consumed. A timer that is not due is deferred rather than consumed. Stop intent fences a pending timer in the transaction that commits the stop, so a cancelled or timed-out run never wakes it, and a settled timer is never part of an unwind: the engine refuses a compensable timer declaration, so a timer can never have a compensator.

The run’s own deadline is authoritative over dispatch order. A due timer that is settled only after that deadline already elapsed does not record a wake: the transition commits the deadline’s stop intent instead, in the transaction that consumes the timer, so no wake and no continuation exist for it whatever order the dispatcher happened to settle its work in.

A declaration may instead be a durable external event wait: the same stable step identity with the exact event name, the correlation the event must carry, and a required relative timeout. The accepting transaction records the registration — the event name, the correlation, the timeout, and the instant the wait may time out at, the acceptance instant plus the timeout — with the step, writes the registration into a wait index keyed by event name and registration ordinal, and creates one workspace-scoped Timer item vested at that instant, exactly as a durable timer does. The anchor is durable, so a rediscovery, a replay, or a restart never measures the timeout again, and a repeated declaration of the same step keeps the recorded registration and anchor.

The registration commit is the registration boundary. An event submission records the current per-(workspace, event name) registration ordinal as its exclusive eligibility bound in the same transaction that accepts the event, so a wait registered later holds an ordinal at or beyond that bound and can never be settled by an earlier event. Matching runs in bounded pages of that bound inside the event’s own fan-out obligation, so an event acknowledgement still creates one fan-out obligation and one bounded expansion; an accepted event settles every eligible registration it satisfies, and an event is never consumed by a wait, so ordinary trigger fan-out is unaffected. Both pages of the expansion transaction are bounded by their count and by the bytes their records will add, because a match copies the matched payload into the settling step record and a materialization copies the scope and trigger identifiers into every run record it writes: a count-bounded page could exceed the storage transaction budget and never commit. Each settlement and each materialization is charged its record, the fixed fields of the mutations that surround it, and the identifiers its keys and values repeat, with every occurrence charged for the escaping a tuple key, a nested journal key, or a JSON value applies to it, with the bytes a backend that namespaces its physical keys prepends to each of them, and with the conflict range every candidate inspection reads — reserved before the page reads the candidate, so a page of registrations the event does not match is bounded too. The binding page spends at most half the transaction budget, the wait page spends what the binding page and the expansion’s own finalization leave, and the finalization reserve covers the obligation or the completed event record with its payload, so the three fit one transaction by construction on a backend whose accounting is its logical mutations and the reads the pages perform. The wait page’s first eligible settlement may borrow the binding page’s allowance when its own share cannot carry it, and the binding page then stops before the borrowed bytes — deferring a binding that no longer fits to the next dispatch with the cursor it had — so a wait whose scope fills the transaction still settles in time instead of starving until its timeout, and the deferral never rolls the settlement back. Either page therefore resumes at the next dispatch instead of failing to commit, and at least one registration is always examined so its cursor keeps advancing.

A registration may be settled by a match only when it is eligible by durable facts: the run must still be running and before its own deadline, the event’s durable acceptance instant must be strictly before the wait’s anchor, and the matching transition’s own sampled instant must also be strictly before it. The timeout is therefore authoritative over dispatch order, exactly as the run deadline is for a timer: a delayed match does not extend the wait past its own deadline, and an event accepted at or after the anchor never matches. A zero timeout is due at acceptance itself, so no event can settle it and only the timeout can. Every correlation pair the registration declared must appear in the matched event’s correlation metadata with the same value; the event may carry more pairs than the registration declared.

A match records the matched event’s payload as an ordinary successful step result and schedules the continuation discovery in the same transaction that removes the registration and fences the pending timeout, so a redelivered match or timeout can only find a settled step. The payload is chosen by the event’s sender, not by the waiting run, so a match whose payload the durable record cannot carry — the step record’s value bound, or the run’s replay budget — settles the wait as an accepted sdk.invalid_result step failure naming the limit instead: the workflow learns immediately that the matched payload could not be handed to it, and one unrecordable match neither strands the run until its timeout nor aborts the event’s fan-out page. A settlement no record can carry at all leaves the wait outstanding for its own timeout, because the event must still complete. Registration itself is refused when the run’s replay budget cannot carry the wait’s mandatory timeout outcome, so an exhausted run never commits an unfinishable registration and its anchor. A wait that reaches its timeout records the accepted sdk.timeout step failure and its continuation the same way. Stop intent fences a pending wait and removes its registration in the transaction that commits the stop, so a cancelled or timed-out run never settles it, and a settled wait is never part of an unwind: the engine refuses a compensable wait declaration.

A declaration may be a durable child invocation: the same stable step identity with the invoked function of the same application, the exact input the child receives, and an optional relative invoke timeout. The accepting transaction records the invocation with the step and materializes the child through the same materialization an event trigger uses — one run record, list entry, location, creation history, initial discovery item, and deadline timer — so the parent step, the child’s existence, and the child’s initial work are one durable decision and the two creation paths cannot drift apart.

The child’s identity is derived from durable facts, never minted per declaration: it is a digest over the child’s full storage scope, the parent run, and the step. The materialization refuses an existing record whose recorded invocation provenance or scope disagrees, so a lost acknowledgement, a rediscovery, a replay, or a restart can only ever name the same child; a repeated declaration of the same step keeps the recorded invocation and its child, and a declaration that disagrees with the recorded one is a conflict.

The child is an ordinary run in every respect: its own discovery, attempts, replay, compensators, and terminal outcome. It is bounded by the tightest of the invoked function’s declared run timeout, the invoke timeout anchored at acceptance, and the parent’s remaining deadline, and the durable record names the bound that applied. A zero invoke timeout is due at acceptance itself: the child exists durably, but its first authorization refuses instead of authorizing, so no effect of the child can run before its timeout settles it, and it settles its parent’s step with the sdk.timeout category. The engine refuses an invocation that names no function, one the parent’s pinned manifest generation does not declare, a compensable declaration, a declaration whose retry policy asks for more than one attempt (the child run is the retry unit, and the parent’s recorded policy must not disagree with what the engine does), one whose step record the parent’s remaining replay budget cannot carry together with the small failure that would settle it, and a chain deeper than the documented depth limit.

A child’s terminal transition schedules the settlement of its parent’s step as one durable work item of its own, so a chain of stopping ancestors is settled one level per dispatch instead of recursively inside a single transaction and every transaction stays bounded by its own run’s records. The settlement records the child’s terminal outcome on the parent’s step — its output, its accepted failure, or the server-authored category of a cancelled or timed-out child — and the parent then continues exactly as it does after any other server-owned step. What the parent observes is the outcome the child terminalized with, so a child’s failure reaches its parent only after the child’s own compensation finishes and a parent never continues while its child is still recovering. A settled invocation is never part of an unwind: the engine refuses a compensable invocation declaration, so an invoke step can never have a compensator.

Stop intent propagates one level per transaction and it propagates intent, not termination. A parent’s stop enqueues one durable intent item for its child; the child’s own decision wins when it is already stopping or terminal; the intent fences every new forward authorization of the child — and its pending timers and waits, exactly as a directly stopped run’s are fenced — without cutting off an authorized effect or the child’s own child; and the parent cannot terminalize before its child does. The reason is read from the parent’s committed stop intent rather than carried in the item, so a parent’s cancellation and a parent’s deadline reach the child as the same reason and a redelivered intent propagates the same one. The child’s stop intent is not prompt termination: a child may still hold an authorized effect, a durable wait, or its own child, and it stays Cancelling until every one of those settles.

A declaration may be a durable event emission: the same stable step identity with the exact event name, the exact payload bytes, and the routing metadata the accepted event carries. The accepting transaction accepts the event through the same transactional event acceptance every submitted event uses — the ordinary event record, the workspace-scoped idempotency claim, the fingerprint of the caller-controlled fields, the workspace list entry, and the first fan-out obligation — and settles the declaring step with the acceptance receipt in the same transaction, so no committed state holds an accepted event without the step that emitted it, or a settled emission step without its event.

The emitted event’s identity is derived from durable facts, never minted per declaration: it is a domain-separated digest over the full storage scope, the run, and the step, and the idempotency claim the acceptance uses is derived from the same facts with its own domain separator. The claim is written in the emission’s own internal namespace, distinct from the public submission’s, so a caller who recomputes the derived key cannot pre-claim it and make the emission adopt the caller’s event, and the acceptance refuses a returned identity that disagrees with the derived one before anything is recorded. A lost acknowledgement, a rediscovery, a replay, or a restart therefore re-derives the same logical event: a repeated declaration of the same step keeps the recorded emission and answers with its receipt, and it still leaves the run a continuation discovery, so a declaration delivered on fresh discovery work can never strand the run on a completed item. A declaration that disagrees with the recorded one — a different event name, payload, routing metadata, or step declaration — is a conflict rather than a second event. A step record that is absent is a fresh declaration, and the durable idempotency claim is what keeps the identity when the step record is not there: an identical repeat replays the original acceptance, and a divergent repeat under the same derived claim is refused with the ordinary idempotency conflict.

The payload is the exact event payload bytes, carried as the portable bytes value and recorded without interpretation; the server refuses a payload that is not bytes rather than reinterpreting it. The routing metadata is the caller-supplied string-keyed, string-valued correlation metadata the ordinary event submission carries: it is empty when the workflow declares none, it is never inherited from the run’s own triggering event, and it is part of both the accepted event record and the declaration equality, so an emission cannot silently satisfy a durable wait the workflow did not declare.

The acceptance is what settles the step, and the accepted result is the receipt — the event’s identity, and never the submission disposition, so a deduplicated acceptance cannot change the workflow’s result. The engine then continues the run exactly as it does after any other server-owned step, and the workflow never waits for the event’s fan-out, trigger runs, or matching waits: acceptance is not delivery. The engine refuses a compensable emission declaration, an unnamed event, an over-limit event name, a payload that is not bytes, and a declaration whose step record the run’s replay budget cannot carry together with its receipt. A retry policy that asks for more than one attempt is refused rather than ignored, because the acceptance is not retried by its parent: there is no attempt for a second one to describe. A settled emission is never part of an unwind, and a stop that fences a run’s unresolved work never rolls back an accepted event or its recorded receipt.

An operator or a tool may start one function directly with an input, without the input being broadcast to every binding that listens for an event name. A direct invocation is an addressed event: it is accepted through the same transactional event acceptance as any submitted event, so it has the same acknowledgement, idempotency, routing-generation pinning, durable fan-out obligation, and workspace event list entry, and the run it creates receives its input exactly as a triggered run does.

The accepted event carries a target, the application and function it is addressed to, and the reserved name ryusen/function.invoked. The ryusen/ prefix is reserved for events the server authors: a public submission naming it is refused, so a caller cannot submit an addressed event through the broadcast path or make a broadcast event look addressed. The payload is the exact input bytes, recorded without interpretation, and the event carries no routing metadata.

The idempotency key of an invocation is scoped by workspace and its own function.invoke operation, so it cannot collide with an event.submit key. Its fingerprint covers the target, the payload, and the caller-supplied timestamp. An identical retry returns the original event and its expansion state; reuse of the key with a different target or payload is the ordinary idempotency conflict and mutates nothing.

Fan-out of an addressed event evaluates only its target application in the pinned routing generation, and only the targeted function. When that application’s manifest in the generation declares the function, the expansion records one match under the reserved binding identifier invoke and creates the run with the deterministic identity derived from the workspace, the event, the application, and that binding, exactly as a trigger match would. The function does not need a trigger binding of its own. When the generation does not reference the application, or its manifest does not declare the function, the expansion records a durable no-match that names which, and completes: the invocation is acknowledged and explained rather than lost or silently retargeted at a newer manifest. An addressed event never evaluates trigger bindings and never matches a durable wait, because it is addressed to one function rather than broadcast.

The created run is an ordinary run. Its source event is the invocation event and its trigger binding is invoke; its deadline comes from the function declaration in the pinned manifest; and its input is the payload bytes, so a workflow decodes a directly invoked input and a triggered one the same way.

Discovery and callback execution are separate work kinds. Discovery against the selected eligible function either completes the function or declares its next unresolved step. The declaration transaction records whether compensation currently exists before targeted execution begins. That fact is diagnostic and does not gate future unwind eligibility.

A successful result is immutable for the logical step ID. Duplicate submission of the same accepted result is idempotent. A conflicting result, stale generation, or stale lease is rejected and retained as diagnostic attempt data without replacing the accepted result.

Retryable failures atomically complete the current physical work item and create a distinct delayed successor with the next application-attempt ordinal and deterministic backoff. The logical operation ID remains stable across those physical items. Exhaustion durably records a terminal failure for the logical step and creates discovery work. Replay reproduces that failure at the matching step.run; it does not dispatch the callback again. If current code handles the failure, discovery may declare fallback work or complete the run. If the failure reaches the workflow entry adapter, the resulting uncaught-failure transaction marks the run failed and starts compensation. If current code no longer emits the failed step, normal discovery makes it Obsolete.

An authorized effectful attempt that loses contact without proving its external outcome becomes Unknown; operator resolution may accept a supplied result, mark it failed, or explicitly retry it. At or after the run deadline the first two still settle the attempt under TimedOut stop intent, while an explicit retry is refused as deadline_elapsed. Discovery loss is retried because its protocol contract forbids external effects outside step.run callbacks.

The SDK returns definition_blocked when one discovery pass encounters the same step ID twice. The transaction records one active blocker keyed by run and reason, including the duplicate ID and optional source metadata, and indexes the blocked run by logical function. It invalidates unresolved discovery work and does not create or authorize the second callback. Repeated observations of the same conflict update diagnostics and probe history without appending repeated failure events or changing accepted step results.

The first registration of a newly preferred compatible connected release wakes blocked runs for each function it advertises. HTTP endpoints and unversioned workers do not provide trustworthy code-change identity, so their blockers use a capped low-frequency delayed probe. An operator may request immediate rediscovery in either mode. Wake and manual requests are deduplicated against existing discovery work.

While blocked, the engine authorizes no new forward callback work. A discovery pass that still reports a conflict remains blocked. Any non-blocked response atomically clears the blocker and follows its ordinary transition using normal memoization: accepted step results are reused, and only a newly unique unresolved ID creates callback work. Cancellation or timeout may move the run directly from DefinitionBlocked to normal Cancelling settlement and compensation; the definition conflict itself never triggers compensation.

The run’s history says what happened to the conflict without repeating it as a failure. The pass that first reports it appends definition-blocked; an operator retry appends definition-retry-requested; a pass that still reports a conflict appends definition-still-blocked and updates the blocker’s last observation and count, or replaces the blocker when it names a different step; and the pass that clears it appends definition-unblocked before the entries of its ordinary transition, which carry the pass. Replaying an already accepted emission appends emission-replayed rather than pretending that it accepted the event again. A stop clears the blocker and its index entry in the stop’s own transaction.

Every resolution request names the exact attempt ID and fencing generation, carries an idempotency key, and contains exactly one command: supplied success, declared failure, or explicit retry. One transaction compare-and-sets the attempt from unresolved Unknown. The transaction stores the idempotency key, command fingerprint, resolution, history entry, and successor state or work together. A supplied settlement of an expired forward attempt also materializes the TimedOut stop intent the deadline implies, so the timeout and the settlement are one durable decision with one recorded response.

An identical repeat returns the recorded response. Reusing an idempotency key with a different command conflicts. Any different resolution after the attempt settles also conflicts. A late worker result competes on the same unresolved record; exactly one result or operator command commits, and losing late results remain diagnostic stale submissions.

Resolution successors are normative:

An unresolved compensation attempt ignores the forward deadline: the deadline stops forward progress, and recovery has its own attempt policies. Refusals are decided in one order, so a client can act on the distinction: the request itself, then a recorded idempotency key, then the exact unresolved attempt and fencing generation, and only then the deadline policy. A settled or stale attempt is therefore stale at any clock, and a recorded command replays even when the state that would now refuse it has since committed.

Explicit retry is not a claim that the previous effect failed. It is durable operator acceptance of possible duplication. Incrementing the fencing generation prevents generation G from committing workflow state after G+1 is created, while the stable operation ID lets an idempotent external system recognize both physical attempts as one logical operation.

A generation has one immutable settlement: worker result, operator-supplied result, declared failure, or RetryAuthorized. Resolving G as RetryAuthorized does not consume the logical target’s eventual result; it authorizes G+1 to settle that target. If G+1 also becomes Unknown, another explicit command may repeat the same transition to G+2. After any restart, durable settlement and generation records reconstruct the same fence: an older generation can append diagnostic stale-result data but cannot mutate authoritative target state.

The stop-intent transaction records Cancelled or TimedOut as the eventual execution outcome, clears any active definition blocker, and prevents discovery from creating new forward work. It fences leased or queued forward attempts that have not committed start authorization and discards authorized repeatable discovery. Effectful attempts already authorized retain a settlement token derived from their operation ID and generation; this narrow token permits their result to commit without reopening forward scheduling, and the run remains Cancelling until every token resolves.

If no authorized effectful attempt requires settlement, the stop transaction immediately snapshots completed steps. It creates the first compensation item for a nonempty unwind or transitions directly to Terminal with NotRequired(EmptyUnwind). In particular, cancellation or timeout after run creation but before first discovery authorization executes no callback and terminalizes atomically with an empty unwind.

The result transaction normal-reads the settlement record, attempt, and run phase. A valid late success records the completed step but schedules no rediscovery. A structured failure records the reported failure. Lease expiry or lost transport after authorization records Unknown. A late result may replace Unknown only while its settlement token remains unresolved; an operator result and late worker result use the common resolution CAS, so exactly one commits. The operator may supply success or declare failure with the risk described in RFD 1, but cannot redispatch forward work after stop intent commits.

A repeated stop intent with the same outcome is idempotent while the run is Cancelling: the first one already committed the outcome and forbade new forward work, so a second request appends no history and mutates no run or state record. A different stop outcome cannot replace the one that committed, and the run timer stays consumed. A stop that retains a settlement token records the same replayable receipt a terminalizing stop does, so a redelivered timer replays the durable response instead of failing on work the first transition consumed.

A transport that proves an authorized effect never began — a connection that failed before the endpoint answered, or a correlated pre-start rejection — is the one case where an authorized effectful lease may leave the unresolved index without settling. The engine reconciles it in the run’s own transaction: while the run can still dispatch the attempt the same physical attempt is requeued without consuming an attempt ordinal, and under stop intent the attempt is fenced, the run releases ownership of it, and the settlement gate runs. The attempt provably never executed, so it is not part of the unwind. Requeueing it instead would leave the run Cancelling holding work no dispatcher can prepare, because preparing a forward target requires a running run.

Compensation order is snapshotted only after every effectful settlement token resolves, thereby including successful work that completed after cancellation was requested. While any token is Unknown, the run remains Cancelling with durable intervention work and no compensation item.

A stop request that retains no settlement token settles the run in the same transaction, whether or not completed steps exist. It fences the work the run currently owns and then either snapshots a nonempty unwind with its first compensation item or terminalizes an empty one as NotRequired(EmptyUnwind). A stop therefore never refuses work it could have compensated, and the snapshot is the same one a settlement-token stop reaches once its last token resolves.

Entering compensation snapshots every completed step in reverse completion order from durable history, regardless of its declaration-time compensation bit. The engine processes one item at a time. Before each target, current code is asked to execute its compensator; target_not_found records a durable skip and continues. A provided compensator uses independent attempts and retry limits. An accepted success, terminal failure, or skip is not reopened by later code changes.

The snapshot transaction never creates a Compensating run without durable next work. An empty plan appends a CompensationNotRequired history record with reason EmptyUnwind and transitions directly to Terminal. Successful execution uses NotRequired with reason ExecutionSucceeded without creating a snapshot.

After the last target, any durable skip produces PartiallyCompensated, even when every target skipped. With no skips, at least one executed compensator and all successes produce Compensated. A permanent failure produces CompensationFailed and stops the unwind.

Compensation work uses a separate work kind so later flow-control implementations can reserve capacity and avoid deadlocking recovery behind forward work.

Lease or transport loss after compensation start authorization becomes Unknown and stops the unwind without consuming the retry policy. A late result or operator resolution settles that attempt before the engine may retry a confirmed failure or advance past a confirmed success.

RFD 3 owns the common harness, topology metadata, and storage metrics. This RFD defines the engine workloads it must drive. Local and CI profiles use pure engine components and SQLite to catch algorithmic or revision-relative regressions; optional local FDB verifies conflict semantics but does not establish production capacity.

The workload matrix includes:

Every run records acknowledgement-to-fan-out completion and ready-to-lease p50, p95, and p99 latency; accepted events, materialized runs, and leased work per second; transaction attempts and conflicts by transition; transaction bytes; lease scan amplification; shard load distribution; backlog depth; and recovery time. Results separate offered load from accepted throughput and report errors or timeouts rather than dropping them from latency distributions.

The structural performance requirements are:

The production-like FDB profile runs the same matrix from external load generators against the named multi-machine topology in RFD 3. Its explicit throughput, tail-latency, recovery, and regression gates must be recorded before the hosted queue is called production ready. A benchmark on an unspecified machine, single-process SQLite, or single-machine FDB is never substituted for that gate.

After any process crash:

These invariants are tested by injecting a stop after every transaction commit and reopening the store.

The initial queue does not implement multi-tenant fairness, concurrency limits, throttling, debounce, batching, CEL waits (a durable event wait’s correlation map is the whole filter language; expressions and predicates are not part of it), cron, distributed dispatch, or FDB sharding. Those features must later preserve the transactional transition model.

State records, transitions, dispatcher, and crash tests are tracked in RFD 4 implementation checklist.