Durability Guarantees

A workflow may run for milliseconds, hours, or days. During that time, worker processes can crash, APIs can time out, requests can be delivered twice, and users can disappear.

Invariant's durability model is built around seven execution invariants. These invariants define the contractual guarantees the Runtime preserves regardless of how execution is interrupted.

---

The 7 Core Invariants

1. Deterministic Transition Evaluation

Problem: If state transitions depend on volatile runtime conditions or unobserved side effects, crash recovery cannot compute identical application state.

Guarantee: Given the exact same compiled workflow graph, initial state, and sequence of events, transition evaluation computes the exact same state update every time.

Mechanism: The execution kernel (ExecutionEngine.transition()) is a pure function. Side effects and model calls are prohibited inside state machine evaluations and must cross explicit capability or reasoning boundaries.

---

2. Monotonic Event Ordering

Problem: Out-of-order event delivery in asynchronous or concurrent environments can corrupt state machine transitions.

Guarantee: Events targeting a workflow execution (runId) are applied in a strictly increasing, gapless sequence ($1, 2, 3, \dots$).

Mechanism: The storage layer assigns sequence numbers (seq) within transactional locks, enforcing strict event-sourcing consistency per execution.

---

3. Atomic Transition Persistence

Problem: A crash occurring after updating application state but before recording emitted events or outbox commands leaves the system in a corrupted, half-committed state.

Guarantee: State updates, emitted events, and generated capability outbox intents commit together atomically, or not at all.

$$\text{State Update} + \text{Emitted Events} + \text{Outbox Intents} \implies \text{Single Storage Transaction}$$

Mechanism: RuntimeTransaction commits state mutations and outbox commands within a single database transaction. If the process crashes mid-commit, the entire transaction rolls back cleanly.

---

4. Recoverable Execution Ownership

Problem: A worker process executing a workflow can crash, freeze, or lose network connectivity mid-execution.

Guarantee: A workflow execution is never permanently stranded because the worker processing it died.

Mechanism: Workers claim executions using expiring leases (worker_id, lease_expires_at). If a worker dies, its lease expires. The Reclaimer Worker safely reclaims ownership from durable storage. While duplicate delivery attempts can occur under network partitions, state transitions and capability idempotency prevent invalid duplicate execution.

Worker A claims Run 101  ──►  Worker A dies 💀  ──►  Lease expires  ──►  Worker B reclaims Run 101 from Postgres

---

5. At-Least-Once Effect Dispatch & Idempotency

Problem: External API calls (like charging a credit card or sending an email) can fail due to network timeouts, leaving uncertainty about whether the external side effect occurred.

Guarantee: External capabilities are dispatched with at-least-once delivery guarantees without executing duplicate side effects on retries.

Mechanism: Capabilities are queued through a transactional Outbox and executed with stable idempotency key formulas (charge:${execution.runId}). External providers receive identical idempotency keys on retries.

capability("issue-refund", {
  idempotencyKey: ({ execution }) => `refund:${execution.runId}`,
  handler: async ({ input }) => {
    return await stripe.refunds.create({ ... }, { idempotencyKey: `refund:${execution.runId}` });
  },
});

---

6. Durable Reasoning Results

Problem: Re-invoking LLM prompts during crash recovery introduces unexpected latency, high token cost, and non-deterministic response drift.

Guarantee: Once a .reason() node completes and its result is committed, recovery never asks the model to re-evaluate the prompt.

Mechanism: Reasoning results are recorded as durable REASON_COMPLETED events in the event log. During recovery or replay, the kernel reads the committed result directly from storage.

---

7. Durable Event Re-entry Boundary

Problem: External signals (webhooks, user inputs, timer completions) can be lost if they rely solely on in-memory message channels or transient WebSocket connections.

Guarantee: No external signal or model output mutates workflow state without passing through a durable storage event boundary.

Mechanism: Transient channels (like WebSockets or SSE) are delivery mechanisms for fast-path notifications. Truth always resides in storage. If a connection drops, the event remains queued in storage and resumes execution seamlessly upon reconnection.

---

The Contractual Guarantee Matrix

Invariant	Failure Mode Covered	Preservation Guarantee
1. Deterministic Transitions	Volatile state drift	Pure, reproducible state machine logic
2. Monotonic Event Ordering	Race conditions & out-of-order events	Strict sequence number ordering per runId
3. Atomic Persistence	Mid-commit process crash	Complete commit or total rollback
4. Recoverable Ownership	Worker process death (kill -9)	Expiring leases & automatic worker reclamation
5. At-Least-Once Effects	Network timeout & external API outages	Transactional outbox + stable idempotency keys
6. Durable Reasoning Results	Recovery prompt re-evaluation	Re-reasoning is skipped; cached event result used
7. Durable Re-entry Boundary	Connection drop / SSE disconnect	Storage-backed event queues as sole source of truth

---

Go Deeper

• Execution Engine Kernel — Inspect ExecutionEngine.transition(), locking, and state materialization.
• PostgreSQL Adapter — See how PostgreSQL implements atomicTransitionCommit and lease reclamation.