ACAgentic Craft

Intermediate~14 minHazelJS

Evaluator–Optimizer

Iterate generation with HazelJS AgentRuntime loop stages (plan/execute/critique/validate) until quality bars or budgets are met.

Authors
editorial-team
Published
Last reviewed

Problem

Single-pass generation is inconsistent for tasks with clear quality criteria, but a full tool-using investigation agent is unnecessary.

Context

Drafting, rewriting, structured extraction, code suggestions—domains with rubrics or automated checks, run through AgentRuntime's outer confidence loop.

Forces and constraints

  • Quality bars can be stated as critiques or tests
  • Extra model calls cost latency and money
  • Evaluator bias can reinforce errors if coupled poorly
  • Deterministic checks beat LLM-only judges when available

Recommended design

Use AgentRuntime.execute with loop: { stages: ['plan','execute','critique','validate'], maxIterations, successScore }. Prefer deterministic gates first (contracts, linters, unit tests) inside validate; use critique for rubric feedback. Bound iterations with loop.maxIterations and RunBudget.

Minimal pseudocode

const result = await runtime.execute('DraftOptimizer', { task }, {
  loop: {
    stages: ['plan', 'execute', 'critique', 'validate'],
    maxIterations: 4,
    successScore: 90, // stop when critique/validate >= 90
  },
  maxSteps: 3,        // inner tool steps per outer iteration
  contract: draftContract,
  defaultBudget: { maxTokens: 80_000 },
});

return result.output;

Failure modes

  • LLM-as-judge shares blind spots with generator
  • Unbounded revise loops (missing maxIterations / successScore)
  • Rubric too vague to improve outcomes
  • Optimizing for the judge, not the user

Security considerations

  • Do not let critique prompts exfiltrate hidden system data
  • Keep untrusted content clearly delimited in both generator and critic roles
  • Apply @hazeljs/guardrails on final output before publish

Observability signals

  • Track outer-loop iteration count and critique scores in traces
  • Inspector getTimeline shows plan/execute/critique/validate stages
  • Store critique summaries for debugging (privacy-reviewed)

Evaluation approach

Meta-evaluate the judge against human labels; measure user preference or task success vs single-pass baseline; watch cost/quality curves under RunBudget.

Trade-offs

  • Higher median quality vs multiplied inference cost
  • Strong rubrics help; weak rubrics waste spend

Sources

Related patterns