ACAgentic Craft

Intermediate~16 minHazelJS

Idempotent Tool Execution

Make AgentRuntime write tools safe under retries and durable resume using idempotency keys and dedupe semantics in adapters.

Authors
editorial-team
Published
Last reviewed

Problem

At-least-once AgentRuntime resumes, approveAndResume retries, and worker crashes re-enter write tools and duplicate side effects (charges, tickets, emails).

Context

Any @Tool with sideEffect=write used from AgentRuntime or invoked inside @hazeljs/flow nodes. Complements durable agent runs.

Forces and constraints

  • Durable kernels prefer at-least-once delivery over silent loss
  • External APIs often support idempotency keys or natural-key upserts
  • Blind re-execution after timeout is unsafe
  • Reconciliation is required when outcome is unknown

Recommended design

Treat every write @Tool as an idempotent adapter: accept a stable idempotency key (or derive from executionId + tool + business key), persist or query dedupe before mutating, return the prior result on replay. Align keys with durable AgentRun step identity. Prefer upserts and natural keys when vendors lack idempotency headers. Pair uncertain timeouts with reconciliation before retry.

Minimal pseudocode

@Tool({ name: 'create_refund', requiresApproval: true })
async createRefund(input: {
  chargeId: string;
  amount: number;
  idempotencyKey: string; // e.g. `${executionId}:refund:${chargeId}`
}) {
  const existing = await payments.findByIdempotencyKey(input.idempotencyKey);
  if (existing) return existing; // safe replay

  return payments.refund({
    chargeId: input.chargeId,
    amount: input.amount,
    idempotencyKey: input.idempotencyKey,
  });
}

// Flow nodes can mirror the same idea:
// @Node('charge', { idempotencyKey: (ctx) => `order:${ctx.input.orderId}:charge` })

Failure modes

  • Key derived only from wall-clock → never dedupes
  • Key too coarse → distinct intents collide
  • Timeout after success without recording → retry without lookup doubles charge
  • In-memory dedupe lost across workers

Security considerations

  • Do not put secrets in idempotency keys
  • Authorize each attempt; idempotency is not auth
  • Audit first-success vs replay outcomes

Observability signals

  • Metric: tool_replay_hit_rate
  • Log executionId + idempotencyKey hash on write tools
  • Alert on duplicate-without-hit (possible key bug)

Evaluation approach

Crash/resume tests that assert a single external side effect; approveAndResume replay fixtures; chaos timeout after success.

Trade-offs

  • Dedupe stores add infra vs unsafe retries
  • Strict keys block legitimate repeats—document escape hatches

Sources

Related patterns