Session replay fixtures

Session replay stands in for the model with recorded traffic: it rebuilds each model call from a recorded session log, so the real Oda agent runs its loops with no provider key and the same output every run. Use it for automated scenarios that need real loop behaviour with fixed model output; to choose between the test tiers, see Testing tiers.

Set up a replay scenario

Replay mounts in place of a real model route:

• With a provider list, it registers a replay-only adapter for exactly those routes, so a scenario that reads the model catalogue still finds one.
• With no provider list, it intercepts every model call instead.
• Nothing reaches a provider either way. Every request is answered from the fixture. For scripted provider failures instead of recorded streams, see The mock LLM server.
• Disposing the test removes replay, so hot reload falls through to a real model route again.

The settings a scenario passes:

A run with no fixture path at all is refused: a fixture path is required.

What the fixture is

• The fixture is a projection of one recorded session: the header and every event payload, with the per-row sequence and time envelopes left out. Replay supplies contiguous sequences and fixed timestamps as it reads, so a run is deterministic.
• Replay never writes the fixture. It is never rewritten or renamed, and a historical fixture is migrated in memory as it loads; recording and refreshing fixtures belong to Recorded-session snapshots.
• The snapshot suite supplies the highest generation of each recorded session and checks that the file name and the header agree; a fixture that fails validation is refused, never repaired.

How live sessions bind to recorded scripts

• One script per recorded session, in recorded call order. When a live session makes its first model call, it claims the next unclaimed script. The parent streams first, so it takes the parent script, and each new child session takes the next child script, in recorded creation order. Every session then advances its own cursor, so calls within one session are independent of the other sessions.
• Model calls that carry no session id are treated as one anonymous session, bound to the primary script.
• More live sessions than recorded scripts fails loud: a model call arrived from an unrecorded session (#<n>); the scenario recorded only <m> session(s), and it tells you to re-record the scenario.

Overrides for what a log cannot record

Two failure shapes cannot be rebuilt from a session log alone: a call that threw before producing any chunk, and a call that hung or was cancelled. A scenario that needs them points overrideFile at a replay.override.json sidecar in the scenario folder.

• Two forms. A bare list of entries replaces the derived script. { "patches": [{ "at", "entry" }] } keeps it and swaps only the calls you name; at counts calls from 0, and at equal to the number of derived calls appends a call, for example the retry after an injected failure.
• Three entry kinds. chunks replays its chunks. throw replays its prefix chunks (possibly none), then fails the call with its message and code; a throw with prefix chunks counts as accepted, while a throw with no chunks counts as never accepted unless you set accepted: true. hang streams a short prefix, optionally writes a readyFile marker, then waits for the test to cancel.

{ "patches": [{ "at": 0, "entry": { "kind": "throw", "chunks": [], "message": "<error text>", "code": "<code>" } }] }

• Validation is strict. A malformed sidecar is refused with invalid override <file>: <location> <detail>, including an unknown entry kind and a non-boolean accepted. A patch index past the end is override patch index <n> out of range (derived script has <n> call(s); == length appends), and naming an index twice is duplicate override patch index <n>.
• Overrides touch the primary session only. Child scripts always derive from their own logs.

Placeholders in scripted entries

• A scripted string may embed {{fromRequest:<pattern>}}. At stream time, replay matches the pattern against the live request's text and substitutes the last match's first capture group, or the whole match when the pattern has no group. Use it for values no fixture can hold, such as an id minted during the run that the model must echo back.
• Resolution fails loud: an invalid pattern (fromRequest has an invalid pattern <pattern>), a pattern that matches nothing (fromRequest pattern <pattern> matched nothing in the request), or an unterminated placeholder (fromRequest placeholder is unterminated in <text>).
• Script-derived entries resolve placeholders the same way as sidecar entries.
• Fixtures also carry the snapshot suite's session tokens. {{session:N}} becomes the live id of the nth session that bound; using it before that session binds fails with session token {{session:N}} was used before that recorded session bound, and the full vocabulary is on Recorded-session snapshots.

When replay stops or refuses

• A version skew at load. A build whose format catalogue and session format version disagree does not load replay: format catalog v<n> does not match Session v<n>.
• A fixture that does not parse. For example session snapshot line <n> contains invalid JSON, session snapshot must start with a session header, or session snapshot line <n> cannot mix projected and complete body rows.
• A call with no finish. When derivation meets a stream that ended without a finish chunk, the message names the call and the fix: model call <turn>/<step> ended without a finish chunk (a thrown stream); this scenario needs a replay.override.json sidecar.
• A missing file. fixture not found: <file> tells you to run pnpm run test:snapshot:record first, and child fixture not found: <file> tells you to re-record the scenario.
• More calls than recorded. script exhausted: a session asked for a call its script does not have. Re-record the scenario.
• Fewer calls than recorded. The consumption check (assertConsumed) fails teardown with fixture not fully consumed, naming scripts that never bound to a live session and sessions that consumed only part of their script, so a silent underrun becomes a diagnostic.
• Bad settings. paceMs must be a non-negative integer, got <value>; a provider catalogue with a bad value is refused at load, for example provider "<id>" model "<id>" systemPromptUpdate must be "in-history" when present.

Limits

• Binding assumes sequential delegation. A scenario whose sibling sessions run at the same time cannot be bound deterministically, so this tier suits recordings whose delegation order is fixed.
• Only ordinary streams and marked compaction outputs derive from a log. A pure throw, a hang, and anything unmarked need the sidecar.
• Overrides apply to the primary session only; child scripts always derive from their own logs.
• The pace is realism only. paceMs changes delivery timing, and correctness must never depend on it; cancelling during a pace wait cancels the stream like any other cancel.
• The consumption check is the safety net. Call it at teardown: it turns fewer calls than recorded, or a script that never bound, into a failure instead of a passing test.

The tier that records the fixtures replay runs on, with its record, replay and refresh modes, is Recorded-session snapshots.

Questions, answered

Why does my scenario need a replay.override.json sidecar?

Because two failure shapes cannot be rebuilt from a session log: a call that threw before producing any chunk, and a call that hung or was cancelled. The sidecar supplies them, and the message for a stream that ended without a finish chunk points at it.

How does replay decide which live session gets which recorded script?

By first-call order. The first live session to make a model call takes the parent script, and the next new session takes the next child script, each advancing its own cursor. More live sessions than recorded scripts fails loud rather than guessing.

What happens if my scenario makes more model calls than the fixture recorded?

script exhausted, naming the call the session wanted and the calls the script has. Re-record the scenario so the script covers the run.

What happens if a session appears that the recording does not have?

The call fails with the unrecorded-session message: more distinct live sessions made calls than the scenario recorded. Re-record the scenario.

How do I slow a replayed stream down?

Set paceMs to delay each chunk. It exists for genuinely incremental delivery; never let correctness depend on it.

Does replay send anything to a model provider?

No. Every call is answered from the fixture, and the provider catalogue replay can expose is replay-only, so a keyless run is also a harmless one.

How do I check the scenario used the whole fixture?

Run the consumption check at teardown. It fails unless every recorded script bound to a live session and every bound session consumed its whole script, so an underrun cannot pass silently.

Why did a {{fromRequest:...}} placeholder fail?

The resolution names its reason: an invalid pattern, a pattern that matched nothing in the request, or an unterminated placeholder. Remember that the last match wins and its first capture group substitutes.

Can two subagents that ran at the same time be replayed?

Not deterministically. Binding follows first-call order and assumes delegation ran one session at a time, so a recording whose sibling sessions interleave cannot be bound reliably.

Where does the fixture path come from?

From the scenario's own settings, or from the snapshot suite, which supplies the selected fixture per scenario. With neither, replay refuses to run: a fixture path is required.

Does replay change the fixture file?

No. Replay never rewrites or renames a fixture, and a historical fixture is migrated in memory as it reads it. Recording is the snapshot suite's job.