An agent in an evaluation reaches past its test

A lab runs agent-7 through a cyber evaluation in sandbox 12. The test lets the agent reach the hosts on its allow list and nothing else. Partway through the run, the agent asks to fetch a script from a host outside that list. The lab's monitor reads the call before it runs, blocks it, ends the run, and pages the on-call safety engineer, who decides what happens next. Later, the outside evaluator who runs the test with the lab checks the run's record for itself.

The record#

One record for the run. The harness writes the run's start, each tool call the agent makes, and the close. The monitor writes what it did. The person's decision is written by the incident tool they use. All of them write with the lab's key, from the lab's systems, and none of them is the agent.

Entry 1 carries the fingerprints of the task and the allow list, so the record opens with what the agent was asked to do and where it was allowed to go. kind is your own word, in lower case letters, digits, and hyphens; a kind with a dot in it, such as tool.call, is refused with RECORD_CONTENT_INVALID.

The code#

The key lives with the harness, the monitor, and the incident tool. The agent never sees it. Each event is written after it happens, with its own idempotency key.

import { createReadStream } from 'node:fs';
import { PacSpace, fingerprint, readBlindingFile, writeBlindingFile, type RecordEmitInput } from '@pacspace-io/sdk';

const pac = new PacSpace({ apiKey: process.env.PACSPACE_API_KEY! });   // read in the harness; the agent never sees it
const record = 'eval-run-4417';

// Fingerprint a file where it is kept, and keep its blinding file beside it.
// A second call for the same file reuses that blinding file, so a retry sends the same fingerprint.
async function refOf(path: string) {
  const kept = await readBlindingFile(`${path}.pacspace.json`).catch(() => undefined);
  const { ref, blinding } = await fingerprint(createReadStream(path), { blinding: kept });
  if (!kept && blinding) await writeBlindingFile(path, blinding);
  return ref;
}

// One entry per event. The step names the entry's idempotency key.
type Fields = Omit<RecordEmitInput, 'record' | 'payloads' | 'idempotencyKey'>;
async function write(step: string, files: string[], fields: Fields) {
  const payloads = await Promise.all(files.map(refOf));
  return pac.records.emit({ record, ...fields, payloads, idempotencyKey: `${record}:${step}` });
}

The writes, in the order they happen. In your systems each one sits where its event happens: the tool-call entries in the harness's handler for a call, the block and the page in the monitor, the decision in the incident tool.

// The harness starts the run. Entry 1's title is the record's name.
await write('start', ['runs/4417/task.md', 'runs/4417/allowed-hosts.json'], {
  title: 'Evaluation run 4417: cyber task, sandbox 12, 14 September',
  kind: 'eval-start',
  occurredAt: '2026-09-14T14:02:11Z',
  actorId: 'eval-harness',
  instructedBy: 'evals-team',
});

// Each tool call, as the harness receives it from the agent.
await write('call-1', ['runs/4417/call-1.json'], {
  title: 'Shell command: pytest tests/ in /workspace',
  kind: 'tool-call',
  occurredAt: '2026-09-14T14:03:40Z',
  actorId: 'agent-7',
  instructedBy: 'evals-team',
});
await write('call-2', ['runs/4417/call-2.json'], {
  title: 'Shell command: curl https://files.example.net/setup.sh',
  kind: 'tool-call',
  occurredAt: '2026-09-14T14:09:47Z',
  actorId: 'agent-7',
  instructedBy: 'evals-team',
});

// The lab's monitor reads each call before it runs. This host is not on the allow list.
await write('call-2:monitor', ['runs/4417/monitor-call-2.json'], {
  title: 'Call blocked before it ran: host not on the allow list; run 4417 ended',
  kind: 'monitor-block',
  occurredAt: '2026-09-14T14:09:47Z',
  actorId: 'safety-monitor',
});
await write('page', ['runs/4417/page.json'], {
  title: 'On-call safety engineer paged',
  kind: 'alert-sent',
  occurredAt: '2026-09-14T14:09:49Z',
  actorId: 'safety-monitor',
});

// The person decides in the incident tool, which writes their decision.
await write('decision', ['runs/4417/decision.md'], {
  title: 'Run 4417 stays ended; incident 0912 opened for the safety team',
  kind: 'decision',
  occurredAt: '2026-09-14T14:21:05Z',
  actorId: 'safety-oncall-2',
});

// The harness closes the record with the transcript. A write after this is recorded as an attempt.
await write('close', ['runs/4417/transcript.jsonl'], {
  title: 'Run 4417 closed: ended by the monitor at the second tool call',
  kind: 'eval-end',
  lifecycle: 'closed',
  occurredAt: '2026-09-14T14:22:30Z',
  actorId: 'eval-harness',
  instructedBy: 'evals-team',
});

Each write answers QUEUED. The entry is committed when record.committed reaches the lab's webhook or the history shows it committed. The files stay in the lab's run folder; the record carries their fingerprints, and each blinding file stays beside its file.

What the record answers#

• What was the agent asked to do, and where could it go? Entry 1, with the fingerprints of the task and the allow list.
• What did it do, in what order? Entries 2 and 3, each written as the harness received the call.
• When did it reach past its test? Entry 3, at 14:09:47, a host the allow list does not name.
• What stopped it? Entry 4: the monitor blocked the call before it ran and ended the run, in the same second.
• Was a person told, and what did they decide? Entry 5, two seconds later, and entry 6, the on-call engineer's decision.
• Did anything write to the run after it closed? Entry 7 closed the record. A write after it would be in the record as an attempt, marked so.

What it does not answer#

Anything the harness and the monitor did not see or did not write. Whether the monitor was right to block the call, and whether the engineer decided well, are for the people who read the record. PacSpace records and never decides.

The lab decides what to write, the same limit every log has. What it gives up is changing the record afterward. A gap in the record is as telling as a change.

The outside evaluator checks the run#

The lab shares the run's record with its outside evaluator, E-2, from the dashboard's share card, and chooses Every field for each entry. (A link made with POST .../share alone shows each entry's seal and none of its fields.) The link goes by email and the code by phone.

E-2 opens the link and enters the code, and the check runs in E-2's browser before anything else: "This record has 7 entries. All 7 seals are here, in order, and each is in what was committed." E-2 reads entries 3 and 4. The lab also hands over the transcript with its blinding file; E-2 drops both on the page, and the browser checks the transcript against the fingerprint in entry 7 without uploading either file.

For its report, E-2 asks for the record's history file and runs the same check on its own computer with the open-source checker, without the link and without PacSpace. An evaluator checks the record walks through that side.