ACAgentic Craft

Advanced~20 minHazelJS

Durable Agent Run

Persist AgentRun checkpoints and leases with createDurableRunStore / createSqlDurableRunStore so HazelJS agents survive crashes without double-applying side effects.

Authors
editorial-team
Published
Last reviewed

Problem

Long or failure-prone agent runs lose in-memory state; naive retries duplicate external side effects.

Context

Any HazelJS agent that performs multi-step tools, waits on HITL (durableSuspend), or must meet operational SLAs under process restarts and deploys.

Forces and constraints

  • Infrastructure fails mid-run
  • External APIs are at-least-once in practice
  • Approvals and human delays require async persistence
  • Exactly-once is hard; idempotency + reconciliation are achievable

Recommended design

Model work as a durable AgentRun: queued → running → suspended (HITL) → completed/failed/cancelled. Use createDurableRunStore (file) or createSqlDurableRunStore (Prisma SQL). Enable durableSuspend for approval waits. Checkpoint after steps; use leases for workers. Key every write. On resume (including approveAndResume), skip completed steps; reconcile uncertain ones.

Minimal pseudocode

import {
  AgentRuntime, createDurableRunStore, createSqlDurableRunStore,
} from '@hazeljs/agent';

// Local/dev
const fileStore = createDurableRunStore('./.hazel/runs');
// Production SQL
// const sqlStore = createSqlDurableRunStore(prisma);

const runtime = new AgentRuntime({
  runRepository: fileStore.runRepository,
  durableSuspend: true,
  defaultBudget: { maxSteps: 20, maxTokens: 200_000 },
});

const run = await runtime.execute('OrderResolver', input);
// crash-safe: new process can load run + approveAndResume or continue

Failure modes

  • Lost wakeups after approval
  • Lease expiration causing dual workers
  • Checkpoint after side effect but before record → confusion on resume
  • Non-idempotent tools making resume unsafe
  • Using in-memory repository in production

Security considerations

  • Authorize resume/cancel APIs
  • Encrypt durable payloads at rest if they hold sensitive observations
  • Prevent lease theft across tenants
  • Tenant-scoped run repositories

Observability signals

  • Inspector getTimeline({ executionId }) for steps, retries, lease owner
  • Alerts on stuck runs (lease held too long, approval timeout)
  • Reconciliation mismatch counters via @hazeljs/observability

Evaluation approach

Chaos tests: kill worker mid-tool; verify no duplicate side effects on resume; approval delay/resume tests across two AgentRuntime instances sharing the same store.

Trade-offs

  • Operational reliability vs implementation complexity
  • Durable waits improve UX for HITL but require storage and wakeups

Sources

Related patterns