Safety & Idempotency

This page covers the three safety rules an agent has to get right: idempotency on every write, bounded retries on the right errors, and a cadence that does not flood PacSpace or hide real failures.

Use When#

Use this page before an agent writes a production emit loop, checkpoint pass, or webhook handler. The patterns here apply to every write call and every retry path.

Inputs#

• A deterministic referenceId strategy the agent can reproduce from its own state.
• A retry policy with a hard ceiling on attempts and total elapsed time.
• A cadence plan for writes and reads, decoupled from event arrivals.

The Three Safety Rules#

1. Every write carries an Idempotency-Key, and the key is regenerable from the operation that produced it.
2. Retries are bounded, exponential, and only attempted on transient errors.
3. Writes and reads run on cadences the agent controls, not on every incoming event.

Every pattern below is an application of one of these rules.

Idempotency#

How It Works#

All write endpoints (POST /api/v1/balance/delta, POST /api/v1/balance/delta/batch, POST /api/v1/balance/checkpoint) accept an Idempotency-Key header. The SDK forwards the idempotencyKey option in the same header. The API treats the key as a uniqueness guarantee across retries:

• If the key is new, the request is processed.
• If the key matches a request still being processed, the API returns 202 Accepted.
• If the key matches a completed request with an identical body, the API returns the original response.
• If the key matches a completed request with a different body, the API returns 409 Conflict.

This means the agent can retry an emit as many times as necessary without ever double-recording — as long as the same inputs produce the same key.

Deterministic Key Construction#

The safest pattern is to reuse the agent's own referenceId as the idempotency key:

const referenceId = `${vendor}:${operation}:${bucket}:${uuid}`;

await pac.balance.emit(customerId, -1, reason, {
  referenceId,
  idempotencyKey: referenceId,
  metadata,
});

Three constraints on the key:

1. It must be regenerable from inputs the agent already has — never from Date.now() alone, and never from a value the agent cannot recompute on retry.
2. It must be unique per logical operation. Two different operations must never share a key.
3. It must be stable across retries. A retry that changes the key becomes a new operation.

Request Body Stability#

If the agent retries with the same key but a different body, the API responds 409 Conflict. In practice this means:

• Serialize request bodies deterministically. Do not rely on map ordering, random property ordering, or timestamps that change between attempts.
• If the agent genuinely needs to submit a different body, compute a new key. Do not silently mutate a retry.

Receipts and Deltas are Idempotent Through referenceId#

Even without an idempotency key, a deterministic referenceId gives the agent a second line of defense. If a delta with the same referenceId is resubmitted, the agent can reconcile against prior responses and skip downstream work.

Retry Policy#

What to Retry#

The SDK applies these rules automatically with a default of two retries and exponential backoff with jitter.

Bounded Backoff#

Every retry loop the agent writes should set a hard ceiling — both on attempts and on elapsed time — that prevents a single failed operation from running for hours. Two rules:

1. Pick a maximum number of attempts (2 is a good default; 5 is the upper end) and a maximum total elapsed time (30–60 seconds is reasonable for a live request).
2. Stop when either ceiling is reached. Dead-letter the operation and alert. Never retry forever.

const pac = new PacSpace({
  apiKey: process.env.PACSPACE_API_KEY,
  maxRetries: 2,
  timeout: 30_000,
});

Retry-After#

The API returns Retry-After on 429. The SDK honors it automatically. When writing a custom client, sleep at least Retry-After seconds before the next attempt.

Do Not Retry 4xx#

4xx responses (except 408 and 429) mean the request is wrong, not that the API is struggling. Retrying will not change the outcome and will only increase the rate of failed calls. Log the error, surface it to the operator, and stop.

Safe Cadence#

Derive on Demand, Not on Every Event#

The derive endpoint replays verified deltas. Running it on every incoming event inflates call volume and adds latency to paths that do not need fresh totals. Derive when:

• A user or downstream system asks for the current balance.
• The agent runs a scheduled reconciliation pass.
• The agent is about to issue a receipt or invoice.

Checkpoint Once Per Period#

Lock a period at its close. Running additional checkpoints inside a period is safe (checkpoint creation is idempotent for the same scope), but it is rarely useful and creates extra artifacts to reason about.

Webhook Handlers Stay Short#

Return 200 OK as soon as the signature is verified. Run any heavy processing — database writes, external calls, notifications — asynchronously. Slow handlers trigger retries, which become duplicate deliveries, which stress the agent's own state store.

Dedupe Webhook Events#

Every webhook delivery carries an X-Event-ID. Persist it. Skip any event whose ID the agent has already processed.

const eventId = req.headers['x-event-id'];

if (await isEventProcessed(eventId)) {
  return res.status(200).json({ received: true, duplicate: true });
}

await processEvent(req.body);
await markEventProcessed(eventId);

Cross-Environment Safety#

An agent running in both Sandbox and Production must keep the two environments fully separated:

• Use pk_test_* keys only in Sandbox code paths and pk_live_* keys only in Production paths.
• Do not share idempotency keys across environments. A key used in Sandbox should never be reused in Production.
• Do not share referenceId values across environments. A collision in one environment should never surface in the other.

The easiest way to enforce this is to prefix the idempotency key with the environment: live:${referenceId} or test:${referenceId}.

Failure Modes#

• Reusing the same idempotency key across different logical operations, producing 409 conflicts.
• Retrying 400 or 401 responses, multiplying failures without fixing them.
• Unbounded retries that continue for hours and mask the underlying error.
• Building referenceId values from wall-clock time alone, so retries produce duplicates.
• Letting a slow webhook handler run synchronously, triggering retries and duplicate deliveries.
• Calling derive on every event instead of on demand.

Related Pages#

• Emit — idempotency rules on the primary write endpoint.
• Integration Patterns — where idempotency fits in event-driven and periodic loops.
• Authentication — key routing and auth error handling.
• Signature Verification — webhook authentication and timestamp tolerance.