From Demo to Production Checklist

Direct answer

A demo proves the model sometimes completes a task. Production on HazelJS means runs are bounded, observable, recoverable, authorized, evaluated, and owned.

You already built the pieces across this track. This lesson is the go/no-go review.

Companion: /guides/from-agent-demo-to-production. DNA: /guides/design-an-agent-manifest. Docs: Agent OS.

How to use this checklist

For each item: done, accepted risk with expiry, or blocker. Do not ship with silent blockers.

1. Problem and success metrics

• Goal and done condition documented (Ops Desk one-pager)
• Offline golden set (≥10 tasks) with pass/fail
• Online metrics: success rate, human takeover rate, cost per success, p95 latency
• Kill switch / feature flag to disable the agent

2. Identity and tenancy

• Least-privilege tools for the agent identity
• Per-tenant isolation for memory, logs, credentials
• Secrets never in prompts or tool-arg logs (mask policies)

3. Tool safety and HITL

• Every tool has parameters + side-effect class
• Irreversible tools: requiresApproval: true
• durableSuspend: true for approve-required waits
• Resume via approveAndResume(executionId, { approved, approvedBy })
• Idempotent write adapters

const store = createDurableRunStore('.hazel/runs');
const runtime = new AgentRuntime({
  llmProvider,
  durableSuspend: true,
  runRepository: store.runRepository,
  checkpointService: store.checkpointService,
  humanTaskService: store.humanTaskService,
});

4. Budgets and policy

• maxSteps / loop.maxIterations enforced
• PolicyEngine for deny / mask / require_approval
• Clear stop reasons on every run

5. Durability

• File or SQL durable run store
• Crash resume does not double side effects
• Lease/reclaim story if you run multiple workers

6. Context and memory

• Observation caps documented
• RAG/memory off unless required
• Retention / deletion owner named

7. Evaluation and change control

• CI runs describeAgent on prompt/tool/DNA changes
• Versioned Agent DNA reviewed in PRs
• Rollback path for prompt / model / DNA

8. Observability and ops

• Timeline store + Inspector exercised
• Dashboards: success, cost, step depth, tool errors, approval latency
• Runbook: pause agent, revoke tool auth, compensate, export timeline

9. Ownership

• Named eng + product owners
• Security glance at OWASP LLM Top 10
• Cost center + monthly ceiling

How demos fail in production (re-read before launch)

1. Retry storms duplicate refunds
2. Context stuffing leaks PII
3. Unbounded loops burn budget
4. Over-broad tools (shell/SQL) with prod credentials
5. No golden suite — a prompt tweak tanks quality
6. No owner — weekend pager has no kill switch

Capstone

Ship Ops Desk read-only to a small internal audience first. Keep refunds behind approval until goldens and timelines are boring. Then expand tools one at a time.

What to do next

• Flagship lab (Meridian): /guides/design-a-production-agent-runtime
• Deepen Skillgate / MCP: /learn/tools-skills-mcp
• Context engineering track: /learn/context-engineering
• Agent OS hub: /agent-os