Advanced~55 minHazelJS

Implement Durable Execution and Recovery

Persist AgentRuns with createDurableRunStore / SQL stores, checkpoints, worker leases, and safe resume after crashes.

Authors: editorial-team
Published: Published 2026-08-13
Last reviewed: Reviewed 2026-08-13

Before you start

HITL guide or human-approval-gate pattern
Idempotent write tools

You will leave with

Configure file or SQL durable run stores on AgentRuntime
Explain checkpoints, leases, and workerId reclaim
Write a crash-resume test that asserts single side effects

On this page

Goal

Survive crashes and deploys mid-run: checkpoints persist, workers reclaim expired leases, resumes do not duplicate side effects.

Pattern: /patterns/durable-agent-run.

Step 1Choose a store

Environment	API
Local / single node	`createDurableRunStore(dir)`
Multi-instance / prod	`createSqlDurableRunStore(prisma)` (where documented in Agent OS / agent package)

Pass runRepository, checkpointService, and (for HITL) humanTaskService into AgentRuntime. Enable durableSuspend: true if you need approval waits.

Set workerId and lease TTL (runLeaseTtlMs / RepositoryAgentRunLeaseService.reclaimExpired as documented) so a dead worker’s run can be claimed safely. Without leases, two workers can advance the same AgentRun.

Step 3Idempotency + reconciliation

Durable kernels are typically at-least-once. Before re-executing a write tool after timeout:

Lookup by idempotency key / natural key
Only then mutate
If compensation is required, use /patterns/compensating-action

Step 4Resume paths

After crash: load executionId, continue via runtime resume APIs documented for your version
After HITL: approveAndResume
After user input waits: runtime.resume(executionId, input) where applicable

Always log stop reason and timeline for ops (/learn/agentic-development/observability-signals-for-agent-decisions).