Failure behavior

What happens when the TruLayer ingest API is unreachable, and how to opt in to blocking semantics.

The TypeScript SDK is designed so that a TruLayer ingest outage never becomes an application outage. This page documents the default behavior and the one opt-in knob for teams who deliberately want the opposite tradeoff.

Default — drop and warn

When the ingest API is unreachable (network error) or returns a transient status (5xx), the SDK:

1. Retries the batch up to 3× with exponential backoff (500 ms, 1 s, 2 s).
2. On the third failure, drops the batch in-memory and emits a single console.warn.
3. Suppresses warnings for subsequent failures within a 60-second window to avoid log flooding — a fresh warning is emitted once the window rolls over.

User code never blocks on network I/O and never sees a batch failure surface as an exception. Trace capture runs in-process on the critical path; transport runs on a background flush loop that owns the retry logic.

This is the right default for almost every production service. A dead ingest endpoint should degrade observability, not customer-facing behavior.

Opt-in — TRULAYER_FAIL_MODE=block

Set TRULAYER_FAIL_MODE=block to make client.shutdown() (and the flush that runs during shutdown) raise a typed TruLayerFlushError when a batch exhausts its retries.

import { TruLayer, TruLayerFlushError } from "@trulayer/sdk";

process.env.TRULAYER_FAIL_MODE = "block";

const tl = new TruLayer({
  apiKey: process.env.TRULAYER_API_KEY!,
  projectName: "critical-eval-pipeline",
});

try {
  await tl.trace("nightly-eval", async (trace) => {
    /* ... */
  });
  await tl.shutdown();
} catch (err) {
  if (err instanceof TruLayerFlushError) {
    // Alert, mark the run as failed, or abort the job deliberately.
    console.error(
      `ingest failed: ${err.message} (batch size ${err.batchSize})`,
      err.cause,
    );
    process.exit(1);
  }
  throw err;
}

TruLayerFlushError exposes two fields:

• batchSize: number — number of traces in the failed batch.
• cause: unknown — the underlying network or HTTP error.

When to use block mode

Block mode is a niche tool. Reach for it only when:

• The workload is a batch job whose entire value depends on TruLayer receiving the output (eval pipelines, backfills, scheduled quality runs).
• Silently losing traces is materially worse than surfacing an error to the operator.
• The caller is prepared to handle TruLayerFlushError — typically by failing the job and retrying the whole run.

Do not use block mode for:

• User-facing request handlers (HTTP routes, gRPC servers). A transient ingest outage will cascade into customer-visible failures.
• Background services that must survive observability outages (payment processors, auth flows, webhooks).

Zero-network — TRULAYER_MODE=local

For CI and offline development, set TRULAYER_MODE=local. The SDK swaps the HTTP sender for an in-memory LocalBatchSender that stores every trace for inspection, never touches the network, and never warns.

TRULAYER_MODE=local pnpm test

Combine with the @trulayer/sdk/testing helpers for assertions on captured traces.

Replay — TRULAYER_MODE=replay

Set TRULAYER_MODE=replay together with TRULAYER_REPLAY_FILE=<path> to load a previously captured JSONL file on init(). Useful for golden-file regression tests and reproducing a production trace locally.

TRULAYER_MODE=replay \
TRULAYER_REPLAY_FILE=fixtures/golden.jsonl \
  pnpm vitest run

TRULAYER_MODE=replay implies local — replayed traces never escape to the live API, because they were produced by a previous capture and would double-count in the dashboard. Malformed JSONL lines are skipped with a warning.

Decision guide

Scenario	Recommended mode
Production HTTP service	Default (drop + warn)
Background worker with SLO on ingest	Default (drop + warn)
Nightly eval / backfill job	TRULAYER_FAIL_MODE=block
CI unit tests	TRULAYER_MODE=local
CI integration tests against a golden capture	TRULAYER_MODE=replay + TRULAYER_REPLAY_FILE
Local development without an API key	TRULAYER_MODE=local

Archived project

HTTP 403 responses with code: "error.project.archived" are treated differently from other errors. They indicate that the project associated with your API key has been archived — a deliberate configuration change, not a transient failure.

When the SDK receives this response:

1. It logs an ERROR-level message via console.error:
   [TruLayer] ERROR: Ingest permanently disabled — the project associated with this API key has been archived.
   Unarchive the project at https://app.trulayer.ai/projects to resume, then restart the process or create a new client.
   
2. The exporter is permanently disabled for that client instance. Subsequent flush attempts are no-ops and produce no further log output.
3. Your application continues running normally — only TruLayer observability is suspended.

Why the exporter does not retry

A 403 is an authoritative refusal. Retrying would produce noise without any possibility of success. The SDK treats this the same way a browser treats an HTTP 403: stop, log, and do not retry.

Resuming after unarchiving

Unarchiving the project (from Projects settings at app.trulayer.ai/projects) restores ingest immediately — no key rotation needed.

However, any already-running client instance that received the 403 will not automatically resume. You must either:

• Restart the process — the new process starts a fresh client that will send normally.
• Create a new client — call new TruLayer(...) again with the same API key. The new client is independent of the disabled one.

Detecting the error programmatically

In block mode (TRULAYER_FAIL_MODE=block), the TruLayerFlushError thrown on permanent 403s carries a statusCode property you can inspect:

import { TruLayerFlushError } from "@trulayer/sdk";

try {
  await tl.shutdown();
} catch (err) {
  if (err instanceof TruLayerFlushError && err.statusCode === 403) {
    console.error("Project may be archived — check app.trulayer.ai/projects");
  }
  throw err;
}

See Project lifecycle for full details on archiving, unarchiving, and the one-active-project constraint.

See also

• Testing helpers — in-memory assertions against captured traces.
• TypeScript SDK reference — full type signatures for TruLayerFlushError and the error hierarchy.
• Project lifecycle — archiving, unarchiving, and the effect on active API keys.