Design a Production Agent Runtime

Direct answer

A production Agent Runtime on HazelJS is not a chat widget with tools bolted on. It is AgentRuntime living in the same TypeScript backend as your HTTP APIs — with durable runs, policies, HITL, timelines — while DNA / Store / platform sit beside the process to version contracts and declare desired state.

The canonical lab is Meridian Ops Platform (hazeljs-meridian-ops/): the shipped Agent OS flagship that implements features F1–F22. This guide walks Meridian as a starter path so you can design the same shape in your product.

Docs: Agent OS, Agent package. Pattern summary: /patterns/production-agent-runtime-blueprint. Product plan: docs/agent-os/21-flagship-project-plan.md in the monorepo.

Tagline (Meridian): Ship AI workers as part of your backend — versioned like packages, governed like APIs, declared like infrastructure.

Why Meridian (and not only a thin starter)

If you only need a first @Agent / @Tool, start with /guides/build-your-first-tool-using-agent. Come here when you need the production shape.

Journeys (how to read this guide)

Narrative spine = journeys. F1–F22 is the appendix checklist, not the chapter list.

Prerequisites

• Node 18+, TypeScript comfort, curl/jq
• /learn/agent-foundations vocabulary (AgentRuntime, tools, stops)
• /learn/agentic-development strongly recommended (Ops Desk path)
• Meridian checked out from the HazelJS monorepo as hazeljs-meridian-ops/

Time: ~90–150 minutes with Meridian running.
Pin for this guide: Meridian depends on @hazeljs/*@2.0.1 (see its package.json).

Honesty up front (read once)

1. Meridian commerce/risk data is in-memory — demo DB, not production SQL.
2. HITL is off by default (AGENT_OS_HITL=0 → auto-approve). You will turn it on for Journey B.
3. Platform manifests use apiVersion: agent.hazeljs.dev/v1alpha1.
4. hazel agent run / CLI DNA smoke uses stub tools. Product truth is POST /api/chat (and specialist routes).
5. Fraud freeze in the tour may be partial — don’t oversell.
6. Out of scope here: marketplace commerce, Studio product UI, full Flow saga UI, CRDs/operator, K8s as default path.

Part 1 — Boot the flagship (Journey A)

Step 1 — Env and install

cd hazeljs-meridian-ops
cp -n .env.example .env   # leave OPENAI_API_KEY empty → DemoLLM
npm install

Useful defaults from .env.example:

Step 2 — Store sync (DNA → packages)

npm run store:sync

Publishes Meridian’s marketplace DNA into a local registry, materializes under .hazel/agents/, and writes lock.json.

Teach: Store answers how do we version and share the agent contract?

Step 3 — Platform sync (desired state)

npm run platform:sync

Applies platform/*.yaml Definitions/Deployments into .hazel/platform/resources.json (+ events).

Teach: Platform answers what should this environment run?
Critical: apply / platform:sync does not start or restart your Node process. It is not kubectl apply for the app.

Step 4 — Tour map

npm run tour

Prints the F1–F22 map and curl hints (same spirit as TOUR.md).

Step 5 — Dev server

npm run dev
# http://localhost:3060
# Inspector: http://localhost:3060/__hazel

Boot log should show a Skillgate report and lines like DNA overlay: ….

Checkpoint A

• store:sync and platform:sync succeed
• Boot shows Skillgate + DNA overlay
• /__hazel loads

Part 2 — Mental model (must stick)

Five questions

DNA ≠ implementation

DNA lists tool names, prompts, and policies. Side effects stay in TypeScript.

Overlay (AGENT_OS_DNA_OVERLAY=1) applies prompt / model / policies onto already registered agents. It must never replace live @Tool handlers with empty stubs (safeOverlayDna in src/platform/dna-overlay.ts).

Architecture (one picture)

Customer / Operator
        │
        ▼
 POST /api/chat  or  /api/support|ops|fraud/chat
        │
        ▼
┌─────────────────────────────────────────────┐
│  Meridian (Hazel app)                       │
│  AgentRuntime + durable runs + Inspector    │
│  optional: local platform (apply/reconcile) │
│                                             │
│  ops-router ──► support-desk                │
│             ├─► safe-desk (fallback/twin)   │
│             ├─► api-concierge (Skillgate)   │
│             └─► fraud-triage (HITL freeze)  │
└─────────────────────────────────────────────┘
        ▲
 .hazel/agents/* + lock.json
 .hazel/platform/resources.json

Checkpoint B

Explain in one sentence why editing DNA alone cannot add a new cancelOrder tool.

Part 3 — Runtime core (execute, tools, timelines)

Agents as backend jobs (F1)

Product entrypoints call AgentRuntime.execute — see src/chat/chat.service.ts — not a free-floating chatbot process.

Lab — track an order (F1–F2)

curl -s localhost:3060/api/support/chat \
  -H 'content-type: application/json' \
  -d '{"message":"Where is my package for ORD-1001?"}' | jq .

Expect a real lookupOrder / tracking path against the in-memory commerce store.

DemoLLM vs OpenAI

• Empty OPENAI_API_KEY → DemoLLM (fine for wiring labs)
• Set a key when you want realistic tool selection

Inspector + timeline (F15)

Open http://localhost:3060/__hazel. Correlate executionId with .hazel/timeline.jsonl (or configured timeline file). Lesson: /learn/agentic-development/observability-signals-for-agent-decisions.

Eval smoke (F16)

Meridian runs Jest / agent tests in-repo. Craft path: /learn/agentic-development/golden-tests-with-describe-agent and /guides/build-an-evaluation-harness.

Checkpoint C

Timeline (or response metadata) shows a real tool name from support-desk — not a CLI stub.

Part 4 — Safety: HITL, policies, contracts (Journey B)

Write tools need approval (F7–F8)

Meridian marks irreversible tools with requiresApproval: true (e.g. processRefund, freezeAccount) and runs with durableSuspend so the worker can pause without holding the HTTP request open forever.

Deep-dive: /guides/add-human-approval-without-breaking-the-run, /patterns/human-approval-gate.

Lab — refund with HITL off, then on

Default (auto-approve):

# AGENT_OS_HITL=0 (default)
curl -s localhost:3060/api/support/chat \
  -H 'content-type: application/json' \
  -d '{"message":"I want a refund for ORD-1002"}' | jq .

Real HITL:

1. Set AGENT_OS_HITL=1 in .env, restart npm run dev.
2. Send the refund curl again — expect a pending approval / suspended run.
3. Approve:

curl -s -X POST localhost:3060/api/approvals/<requestId>/approve | jq .

Policies and contracts (F8–F9)

PolicyEngine encodes require_approval / deny beyond prompt hope (src/agents/agents.module.ts). Support paths can fall back to safe-desk under contract/recovery options when the primary path fails — degrade safely.

Digital twin / canary (F10) exists for shadow compare before cutover — overview only here; use when promoting risky prompt/DNA changes.

Anti-confusion (repeat until boring)

npm run demo:smoke-cli   # F6 — see the contrast explicitly

Checkpoint D

You can explain why CLI smoke must never be used to “prove” a refund worked.

Part 5 — DNA, Store, and promotion

Five DNA packages (F3, F11)

Lockfile pins all five under .hazel/agents/.

Change prompt without rewriting tools

1. Edit dna/support-desk.marketplace.json (systemPrompt) — or nested DNA in platform YAML.
2. npm run store:sync
3. npm run platform:sync
4. Restart npm run dev (overlay on).
5. Chat again — same @Tool code, new governed prompt.

Disable overlay with AGENT_OS_DNA_OVERLAY=0 (decorator metadata only).

Related: /guides/design-an-agent-manifest.

On-ramp from thin starters (F17)

Scaffold with agent-os-starter / hazel agent new, then grow toward Meridian patterns (modules, DNA, Store, HITL, Skillgate). Do not pretend the thin starter is the flagship.

Checkpoint E

You can describe a staging → prod story: same packageRef / lock, different env platform overlays, restart to apply prompt/policy.

Part 6 — Control plane beside the runtime (F19–F21)

What apply does

Writes desired AgentDefinition / AgentDeployment state under .hazel/platform/ (see platform/README.md).

What apply does not do

• Does not deploy Meridian
• Does not replace AgentRuntime.execute
• Does not require Kubernetes (runtimeClassName: local is the default tour)

Nested DNA vs packageRef (Model B)

Prefer one canonical DNA in the Store; manifests reference it via packageRef so environments don’t fork copies. Nested DNA in YAML is fine for teaching — still overlay-safe.

Reconcile + events

# if hazel CLI available:
# hazel agent events --limit 20 --project .
tail -5 .hazel/platform/events.jsonl

Ops question: “What config is this env supposed to run?” → platform resources/events.
Live behavior → chat logs + Inspector timelines.

Checkpoint F

Say in one breath: apply ≠ run ≠ kubectl apply.

Part 7 — Multi-agent and Skillgate (Journey C)

Router door (F12)

curl -s localhost:3060/api/chat \
  -H 'content-type: application/json' \
  -d '{"message":"Track ORD-1001"}' | jq '{agent, steps, response}'

ops-router delegates to specialists (@Delegate / graph patterns). Why multiple DNA? Independent versioning + role separation.

Fraud path (partial in tour):

curl -s localhost:3060/api/fraud/chat \
  -H 'content-type: application/json' \
  -d '{"message":"Freeze ACC-RISK — high fraud risk"}' | jq .

Skillgate curation (F13)

REST controllers + @AgentSkill → OpenAPI → Skillgate.fromOpenApi → tools on api-concierge (src/skillgate/*). Include tags, deny /internal/, gate destructive/admin behind env flags.

curl -s 'localhost:3060/api/skillgate/report?json=1' | jq '{included: .included|length, denied: .denied|length}'
curl -s localhost:3060/api/ops/chat \
  -H 'content-type: application/json' \
  -d '{"message":"List recent orders"}' | jq .

Guide: /guides/skillgate-from-openapi-to-tool-registry. Comparison: /compare/mcp-vs-skillgate-vs-direct-api.

MCP export (F14)

# API must be up
npm run mcp

Same curated skills for IDE agents — transport, not a substitute for Agent OS governance.

Checkpoint G

When does Skillgate beat “register every CRUD route as a tool”? (Hint: curation, deny lists, write approval, reportability.)

Part 8 — Production design: copy vs leave

Absorb checklist

Minimum production bar

Before real traffic, complete /learn/agentic-development/from-demo-to-production-checklist and /guides/from-agent-demo-to-production: budgets, goldens, stop reasons, owners, kill switch.

Known gaps / later (do not invent)

• Prisma / SQL durable run profile
• Real commerce DB
• RAG helpdesk DNA pack
• Full Flow saga for long refunds
• Live remote-registry demo (client exists; local never requires Cloud)
• K8s runtimeClassName: nested as optional appendix only

OSS vs Cloud (F18, Journey D appendix)

Local registry is enough for the tour. Optional:

# HAZEL_REGISTRY_URL=…
# HAZEL_REGISTRY_TOKEN=…

Point at a hosted registry only when your team needs shared packages across machines — not required to learn Agent OS.

Capstone — Absorb Meridian memo

Fill this for your product (15 minutes):

Appendix A — F1–F22 → Meridian pointers

Full matrix: docs/agent-os/21-flagship-project-plan.md.

Appendix B — Env cheat sheet

See Meridian .env.example: PORT, OPENAI_API_KEY, AGENT_OS_HITL, AGENT_OS_DNA_OVERLAY, AGENT_OS_TIMELINE_FILE, Skillgate flags, optional HAZEL_REGISTRY_URL.

Appendix C — HTTP surface (lab)

Appendix D — Related Craft pages

• /agent-os hub
• /patterns/production-agent-runtime-blueprint
• /guides/from-agent-demo-to-production
• /guides/implement-durable-execution-and-recovery
• /guides/skillgate-from-openapi-to-tool-registry

Appendix E — Troubleshooting

What to do next

1. Finish the Absorb Meridian memo for your product.
2. Port one specialist (usually support-desk patterns) into your HazelJS DI app.
3. Add goldens + HITL before any irreversible tool.
4. Re-read /agent-os with Meridian’s five-layer table in mind.