Guardrails

For when you wire the checks yourself: where each one runs, what blocks, and how to count hits and misses. The rules and the queue are yours.

1. Input: before the main AI model. Parallel only while nothing can act.
2. Tool in: blocking, next to each tool. No rule, no call.
3. Approval: act tools wait in your Permissions queue after their check passes. No answer means no.
4. Tool out: labelled untrusted before the model reads it (Prompt injection).
5. Output: redact, cite and escape before the reply leaves.

A check that errors or times out holds the run for a person. That's failing closed.

A sketch, not a library. Call it where the Agent loop notes run a tool.

// Sketch. RULES, tools and queue are yours.
export async function guardedCall(call, user, tools, key) {
  const tool = tools[call.tool];
  const v = await checkIn(call, user);  // blocking
  if (v.verdict === "block")
    return { ok: false, error: v.why };  // loop fixes
  if (v.verdict === "hold")
    return queue.hold(call, v.why);  // pause, hand off
  if (tool.acts && !(await queue.ask(call, user)))
    return { ok: false, error: "not approved" };
  return checkOut(await tool.run(call.args, { key }));
}

export async function checkIn(call, user) {
  const rule = RULES[call.tool];
  if (!rule) return { verdict: "block", why: "no rule" };
  try {
    const why = await within(rule(call.args, user), 300);
    return why ? { verdict: "block", why }
      : { verdict: "pass" };
  } catch { return { verdict: "hold", why: "check down" }; }
}

function checkOut(res) {
  if (!res.ok) return res;  // errors stay errors
  const text = JSON.stringify(res.data ?? "").toLowerCase();
  const flags = FLAGS.filter((f) => text.includes(f));
  return { ...res, untrusted: true, flags };  // data
}

const RULES = {
  search_docs: async () => null,  // reads: still a rule
  create_ticket: async (args, user) =>
    args.account === user.account ? null
      : "account doesn't match",
};
const FLAGS = ["ignore previous", "system prompt"];
const within = (p, ms) => Promise.race([p,
  new Promise((_, no) => setTimeout(no, ms, "timeout"))]);

• Blocking: checkIn() returns before the tool runs.
• Fail closed: no rule blocks, and a slow or broken check holds. A hold stops your loop. It isn't a failure to retry.
• A block is an error: the loop sees ok: false and can fix the call.
• Flags are labels: flagged text stays data, never orders (Failure modes). A phrase list misses rewordings.

Trade-offs. A stricter rule misses less and blocks more good calls. A false block stalls a task; a miss lets a bad act run. Lean strict on act tools. Each blocking check adds its time to every call: keep rules fast, and time-limit any check that calls a model.

Label cases by hand: good calls that should pass, bad ones that should stop. Per tool, count hits, misses, false blocks and the slowest time. Add each later miss as a case.

const by = {};
for (const c of cases) {
  const t0 = Date.now();
  const v = await checkIn(c.call, c.user);
  const stop = v.verdict !== "pass";
  const r = (by[c.call.tool] ??= { hit: 0, miss: 0,
    falseBlock: 0, pass: 0, maxMs: 0 });
  if (c.bad) stop ? r.hit++ : r.miss++;
  else stop ? r.falseBlock++ : r.pass++;
  r.maxMs = Math.max(r.maxMs, Date.now() - t0);
}

Agent evaluation covers graded sets and repeat runs →

Your account check times out on a create_ticket call. What should happen, and how does your eval count it?