Crash Recovery Simulation

Test how Invariant automatically recovers long-running workflows when a Node.js process crashes mid-execution.

---

1. Build It

Define a workflow with a capability step that simulates a process crash:

import { app } from "../runtime";

export const crashDemo = app.workflow("crash-demo")
  .step("prepare-order", async () => ({
    orderId: "ord_9912",
    amount: 150,
  }))

  .capability("simulate-crash", {
    idempotencyKey: ({ execution }) => `crash:${execution.id}`,
    handler: async () => {
      console.log("💥 SIMULATING PROCESS CRASH (SIGKILL)...");
      process.exit(1); // Process dies instantly here
    },
  })

  .step("finalize-order", async ({ state }) => ({
    status: "COMPLETED",
    orderId: state.orderId,
  }));

---

2. Run & Break It

Run the workflow script in your terminal:

npx tsx src/run-crash-demo.ts

Output:

🚀 Started workflow run: run_crash_8812
✅ Completed step: prepare-order
💥 SIMULATING PROCESS CRASH (SIGKILL)...
[Process exited with code 1]

The process dies mid-execution. step-1 was already committed to PostgreSQL.

---

3. Watch Invariant Recover

Restart your application process:

npx tsx src/run-crash-demo.ts

Output:

🔄 Re-acquiring runnable executions from database...
FOUND: Unclaimed run_crash_8812 (lease expired)
⏩ Step 'prepare-order' already completed (read from event log).
⏩ Step 'simulate-crash' marked completed.
✅ Executing step: finalize-order
🎉 Workflow completed successfully! Result: { status: 'COMPLETED', orderId: 'ord_9912' }

Invariant reclaims the expired lease, skips already-completed steps, and finishes the workflow cleanly.

---

4. Why That Worked

• Monotonic Event Log: prepare-order was already committed as a durable event in PostgreSQL before the crash.
• Worker Leases: When the process died, its lease expired. The background recovery scanner discovered the runnable execution and picked it up.
• Deterministic Replay: Invariant read past event facts from storage without re-evaluating completed steps.

---

Go Deeper

• [Runtime & Reliability](file:///Users/josevazquez/ai-infra/ai-agents/invariant-docs/src/runtime/execution.md) — Inspect the low-level lease recovery and transactional outbox implementation.