ACAgentic Craft

Intermediate~24 minHazelJS

Eval Datasets and CI Gates

Scale beyond hand tests with @hazeljs/eval golden datasets and CI reporters—with a wiring sketch.

Authors
editorial-team
Published
Last reviewed
Progress is stored locally in this browser.

Direct answer

Use loadGoldenDatasetFromJson, runGoldenDataset, and reportEvalForCi when suites grow. Gate PRs that touch prompts, tools, DNA, or PolicyEngine.

Docs: Eval.

Worked example

typescript
import {
  loadGoldenDatasetFromJson,
  runGoldenDataset,
  reportEvalForCi,
} from '@hazeljs/eval';

const dataset = await loadGoldenDatasetFromJson('./evals/support-desk.json');

const report = await runGoldenDataset(dataset, {
  // adapter: map each case → runtime.execute / mock provider
  runCase: async (input) => {
    const out = await runtime.execute('support-desk', input, { maxSteps: 8 });
    return { output: out.response, tools: out.tools };
  },
});

reportEvalForCi(report); // non-zero exit when thresholds fail

CI practice

  1. Contract tests with mock LLM on every PR (fast).
  2. Sampled live-provider nightly job.
  3. Store reports as artifacts; link executionId when debugging flakes.

Artifact: CI job that fails on eval threshold breach

Sources

Related