Build an Evaluation Harness

Goal

Ship a merge-gating eval harness for one HazelJS agent: forbidden tools never fire, happy paths call the right tools, and cost/latency ceilings fail the suite when exceeded.

Starter path: work inside hazeljs-agent-os-starter or your DI app that already registers AgentModule. Docs: Testing, Eval.

Step 1Install

npm install @hazeljs/testing @hazeljs/eval
# peer: your agent package + test runner

Step 2describeAgent suite (primary path)

import {
  describeAgent,
  expectTools,
  assertAgentResult,
} from '@hazeljs/testing';

const suite = describeAgent('Support Desk', ({ test }) => {
  test('status ask uses lookup only', async ({ run }) => {
    const r = await run('Status of ORD-1001?');
    expectTools(r, ['lookupOrder']);
    assertAgentResult(r, {
      maxLatencyMs: 8_000,
      maxCostUsd: 0.05,
      outputIncludes: 'ORD-1001',
    });
  });

  test('refund path does not call shell', async ({ run }) => {
    const r = await run('Refund ORD-1001 please');
    // assert allowed set; expand with PolicyEngine deny tests in integration
    expectTools(r, ['lookupOrder', 'processRefund'], 0.5);
  });
});

// Bind suite to your runtime / AgentModule in test setup — see testing docs

expectTools checks tool traces from the AgentRun. Prefer tool-choice and forbidden-tool cases over brittle full-string equality on model prose.

Step 3Optional golden dataset with @hazeljs/eval

import {
  loadGoldenDatasetFromJson,
  runGoldenDataset,
  reportEvalForCi,
} from '@hazeljs/eval';

const dataset = await loadGoldenDatasetFromJson('./evals/support-desk.json');
const report = await runGoldenDataset(dataset, { /* agent runner adapter */ });
reportEvalForCi(report); // non-zero exit on threshold fail in CI

Use eval package helpers (toolCallAccuracy, trajectoryScore, retrieval metrics) when you outgrow hand-written suites. Keep fixtures versioned next to DNA.

Step 4CI gate

1. Run the suite on every PR that touches prompts, @Tool schemas, PolicyEngine, or DNA.
2. Fail on: unexpected tools, budget/latency ceilings, regression vs previous golden.
3. Store reports as CI artifacts; link executionId when debugging flakes.
4. Do not gate on live-network LLM flakiness alone—use mock provider for contract tests, sampled live runs nightly.

Step 5Artifact checklist

• ≥5 golden tasks covering read + write + refuse paths
• At least one forbidden-tool assertion
• maxCostUsd / maxLatencyMs on critical paths
• CI job name documented in the production checklist
• Owner for updating fixtures when product intent changes

Related

• Lesson: /learn/agentic-development/from-demo-to-production-checklist
• Pattern: /patterns/evaluator-optimizer
• Guide: /guides/from-agent-demo-to-production