Markdown

Detached agent sessions and supervision¶

Scope: the process and protocol design that separates an agent's user interface from its execution, so that closing the terminal detaches a client instead of killing the work. This page covers the supervisor and per-session worker topology, transcript leases that stop two processes writing one session, the event-cursor and snapshot rules that make reattachment correct across a worker restart, command journaling for idempotent mutations, schedule-tick claiming, and the recovery behaviour each of those buys. It is the process-lifetime layer beneath agent harness architecture, it is what makes the durable state governed in always-on agents actually survive, and it is distinct from the policy-and-budget concerns of the orchestration control plane, which decides what an agent may do rather than which process it runs in.

What is and is not reproduced here. No agent, model, or production supervisor was run here. The Python block is executed on system python3 3.12.3 with the standard library only, and the pasted output is byte-identical to the run. It models three protocol rules as small deterministic functions and checks each against the naive alternative; it is not a running supervisor and does not measure throughput, latency, or memory. Timing constants, chunk sizes, and protocol-version details attributed below to a published implementation are quoted from its design documents at commit a18809e (2026-08-07), cited under References, and are that implementation's choices rather than universal values.

What it is¶

In the simplest agent CLI, one process is the terminal UI, the agent loop, the tool executor, and the transcript writer. Closing the terminal ends all four. That is fine for a one-minute task and wrong for everything else: an agent that runs for an hour, that is triggered by a schedule, that another agent messages, or that needs to survive an SSH drop cannot have its lifetime bound to a terminal.

The pattern splits three responsibilities across three kinds of process.

The client owns rendering, keyboard input, themes, and local preferences. It owns no execution. Its rule is simple: an action that changes agent execution or persisted session state goes over the connection, and an action that changes only presentation stays local.

The supervisor owns discovery, routing, client attachments, worker health, cross-session message delivery, and coordinated upgrades. It does not call model providers, run tools, compact context, or scan transcripts. Keeping it out of the data path is what lets it stay responsive while a worker is saturated.

One worker per session tree owns everything that executes: the session, its prompt queue, its interpreter, its scheduler, and every child session below it. A worker crash therefore costs exactly one session tree.

There are two worker lifecycles, and conflating them is a common design error.

Lifecycle	Started by	Survives client exit	Appears in default listings
Resident	interactive sessions	yes	yes
Client-owned	one-shot, piped, and RPC callers	no	no, unless explicitly addressed

A client-owned worker uses the same runtime but is deliberately tied to its caller: normal completion removes it without archiving, unexpected client loss starts a bounded cleanup grace period, and reconnecting with the same stable client identity cancels that cleanup. Without this distinction, every scripted invocation leaks a resident agent.

Why use it¶

Detach is not a kill. The single most visible consequence is that closing the UI leaves the work running, and reattaching later shows it mid-flight rather than from the beginning.

Failure containment is per session. With one process per session tree, a native-library segfault or an out-of-memory kill in one agent's interpreter does not touch the other twelve.

Background triggers become possible at all. A schedule that fires at 09:00, a heartbeat that checks a deployment every ten minutes, a message from a peer agent, a goal continuation, and a prompt typed by a human all enter the same session queue. Once execution is decoupled from an attached client, "who triggered this turn" stops being an architectural question and becomes a field on the event.

Multiple observers, one execution. Several clients can attach to one session, and the supervisor serialises each event once and forwards the same buffer to every eligible attachment rather than re-encoding per client.

Upgrades stop being destructive. Because the supervisor knows every live worker, an upgrade can be two-phase: every worker prepares a non-destructive checkpoint, the aggregate manifest is validated and persisted, and only then are workers stopped. If any prepare fails, everything is released and all sessions keep running.

When to use it (and when not)¶

Use it when sessions are long, when work must continue unattended, when agents message each other, or when more than one client may look at the same session.

Do not use it for a stateless request-response service. If every invocation is short and independent, a supervisor plus per-session worker adds process management, a wire protocol, versioning, leases, and a recovery journal to buy detachment nobody needs. Run the agent in process.

Do not mistake this for a security architecture. Workers are isolated for lifecycle and failure containment. In the usual local deployment they run as the same operating-system user as the client, with the same permissions, so process separation buys nothing against a malicious prompt. Confidentiality and least privilege come from sandboxing and isolation, identity and access, and the policy engine.

Also be clear that a local supervision protocol is not a hosted control plane. A multi-tenant service additionally needs authentication, authorization, sandbox identity, artifact transfer, stable public data contracts, and multi-client ownership rules. Local designs that leak filesystem paths into their command shapes do not extend to a network boundary without a redesign of those shapes.

Architecture¶

flowchart TD
    clients["Clients: interactive, one-shot, RPC"]
    supervisor["Supervisor<br/>routing, attachments, health, journal"]
    catalog["Catalog process<br/>saved-session scans"]

    subgraph workerA["Worker: session tree A"]
        queue["Prompt queue"]
        session["Session"]
        sched["Scheduler"]
        kernel["Interpreter"]
        kids["Child sessions"]
    end

    workerB["Worker: session tree B"]
    store["Transcripts and session artifacts"]

    clients <-->|"attach, detach, commands"| supervisor
    supervisor --> catalog
    supervisor --> queue
    supervisor --> workerB
    queue --> session
    sched --> queue
    session --> kernel
    session <--> kids
    session --> store
    store -. "restore after restart" .-> session

Saved-session scanning runs in its own process for a specific reason: a corrupt or enormous transcript file can fail a catalog request without disturbing any running worker.

How to use it¶

The operational surface is a small set of lifecycle verbs over addressable agents: list active agents, attach to one, give it a stable readable name, stop one, inspect service status, diagnose or repair service state, and shut everything down. Naming matters more than it looks, because every other surface (messaging, scheduling, listing) addresses agents by that name.

Everything that can start a turn funnels into one queue, which is what makes the design tractable:

Trigger	Owned by	Typical use
Client prompt	human	interactive work
Peer or child message	another agent	fan-in of delegated results
Recurring heartbeat	human or agent	poll a deployment, check a long build
One-time or cron schedule	human or automation	run at 09:00 on weekdays
Goal continuation	the harness	keep pursuing a durable objective
Autonomous continuation	the harness	continue until quality gates pass or a limit is hit

Delivery mode is the parameter worth understanding. A message can steer a busy target by injecting into its active work, or wait until the current turn finishes, or pick automatically based on whether the target is idle. Steering an agent mid-turn is powerful and dangerous in the same way an interrupt is: it changes the context the model is reasoning over, halfway through.

Goals and autonomous continuation are complementary, not alternatives. The goal stores the objective and its progress; autonomous mode decides whether to inject another continuation given evidence, gate results, and limits on continuations, turns, tokens, and wall-clock time. A gate is a command that must pass before the session may finish, and its failing output is returned to the agent for another attempt. Re-running a gate that failed against an unchanged workspace is wasted money, so suppress it.

How to develop with it¶

Three protocol rules carry nearly all of the recovery correctness. Each has a naive version that works until the first restart.

Event cursors must carry a generation. A client that resumes from a bare sequence number is assuming a single monotonic stream. A worker restart resets that counter, so post-restart events arrive with numbers the client believes it has already seen, and they are dropped in silence.

Missed schedule ticks must coalesce. Advancing the schedule past every boundary that elapsed during an outage is correct. Delivering a prompt for each of them is not.

Mutations must be journaled before dispatch. A command recorded but not resolved is not safe to replay, because its side effect may already have landed.

"""session_recovery.py -- three recovery rules a detached agent session needs,
each shown against the naive version that looks correct until a restart.

  1. Event cursors must be (generation, sequence). A bare sequence silently
     accepts post-restart events as if they continued the old stream.
  2. Missed schedule ticks must coalesce, not accumulate.
  3. Mutating commands must be journaled before dispatch, and a command with no
     durable result must be reported uncertain rather than replayed.

Standard library only.
"""

from dataclasses import dataclass, field


# --------------------------------------------------------------------------
# 1. Generation-aware event cursors
# --------------------------------------------------------------------------
@dataclass(frozen=True)
class Event:
    generation: int
    sequence: int
    text: str


def apply_bare_sequence(events: list[Event], last_seq: int) -> list[str]:
    """The naive client: resume from the last sequence number it saw."""
    return [e.text for e in events if e.sequence > last_seq]


def apply_generation_aware(events: list[Event], cursor: tuple[int, int]) -> list[str]:
    """Resume only inside the generation the cursor belongs to. Anything from a
    newer generation means the worker restarted: the sequence space reset, so
    incremental replay is not comparable and a fresh snapshot is required."""
    gen, seq = cursor
    kept = []
    for e in events:
        if e.generation < gen:
            continue                      # retired generation
        if e.generation > gen:
            return kept + ["<<resync from snapshot>>"]
        if e.sequence > seq:
            kept.append(e.text)
    return kept


# --------------------------------------------------------------------------
# 2. Schedule ticks: claim before delivery, coalesce what was missed
# --------------------------------------------------------------------------
def ticks_naive(last_fired: int, now: int, interval: int) -> int:
    """Fire once per interval boundary that elapsed while we were down."""
    return (now - last_fired) // interval


def ticks_coalesced(last_fired: int, now: int, interval: int) -> tuple[int, int]:
    """Advance the schedule past every missed boundary but deliver at most one
    prompt. Returns (prompts_delivered, new_last_fired)."""
    missed = (now - last_fired) // interval
    if missed == 0:
        return 0, last_fired
    return 1, last_fired + missed * interval


# --------------------------------------------------------------------------
# 3. Command journal
# --------------------------------------------------------------------------
@dataclass
class Journal:
    records: dict[tuple[str, str], str | None] = field(default_factory=dict)

    def submit(self, client: str, command: str, run) -> str:
        key = (client, command)
        if key in self.records:
            stored = self.records[key]
            # Received before, but we never wrote a durable result. Replaying a
            # mutation whose side effect may already have landed is the bug.
            return stored if stored is not None else "uncertain"
        self.records[key] = None          # recorded BEFORE dispatch
        result = run()
        self.records[key] = result
        return result

    def crash_during(self, client: str, command: str) -> None:
        """Simulate: the command was recorded, then the worker died mid-flight."""
        self.records[(client, command)] = None


if __name__ == "__main__":
    # --- 1. cursors -------------------------------------------------------
    # Generation 4 emitted 3 events, the worker crashed, generation 5 restarted
    # its sequence counter at 1.
    stream = [
        Event(4, 1, "g4-e1"), Event(4, 2, "g4-e2"), Event(4, 3, "g4-e3"),
        Event(5, 1, "g5-e1"), Event(5, 2, "g5-e2"),
    ]
    bare = apply_bare_sequence(stream, last_seq=2)
    aware = apply_generation_aware(stream, cursor=(4, 2))

    print("client resuming after a worker restart")
    print("  bare sequence cursor      :", bare)
    print("  (generation, sequence)    :", aware)

    # --- 2. ticks ---------------------------------------------------------
    # A 10-minute heartbeat; the worker was down for 3 hours.
    interval, last, now = 600, 0, 3 * 3600
    naive_n = ticks_naive(last, now, interval)
    coal_n, coal_last = ticks_coalesced(last, now, interval)

    print("\n10-minute heartbeat after a 3-hour outage")
    print("  prompts if every tick replays :", naive_n)
    print("  prompts if ticks coalesce     :", coal_n)
    print("  schedule advanced to (s)      :", coal_last)

    # --- 3. journal -------------------------------------------------------
    calls = {"n": 0}

    def mutate() -> str:
        calls["n"] += 1
        return "compacted"

    j = Journal()
    first = j.submit("cli-1", "cmd-a", mutate)
    repeat = j.submit("cli-1", "cmd-a", mutate)     # client retried after reconnect

    j.crash_during("cli-1", "cmd-b")
    after_crash = j.submit("cli-1", "cmd-b", mutate)

    print("\ncommand journal")
    print("  first submit            :", first)
    print("  same command id again   :", repeat)
    print("  times the mutation ran  :", calls["n"])
    print("  resubmit after a crash  :", after_crash)

    # --- assertions -------------------------------------------------------
    # A bare sequence cursor drops g5-e1 and g5-e2 as "already seen" because
    # their numbers are <= the cursor. The client goes silently stale.
    assert bare == ["g4-e3"], bare
    assert "g5-e1" not in bare and "g5-e2" not in bare

    # The generation-aware cursor detects the reset and asks for a snapshot
    # rather than guessing which events it missed.
    assert aware == ["g4-e3", "<<resync from snapshot>>"], aware

    # Replaying every missed boundary queues 18 prompts for one 10-minute
    # heartbeat; coalescing delivers one and still advances the schedule so the
    # next tick is on the real boundary, not 3 hours behind.
    assert naive_n == 18 and coal_n == 1
    assert coal_last == now and (coal_last - last) % interval == 0

    # An idempotent retry returns the stored result without re-running the
    # mutation. The mutation ran exactly once across two submits.
    assert first == "compacted" and repeat == "compacted" and calls["n"] == 1

    # A command recorded but never resolved is reported uncertain. It is NOT
    # re-run: the side effect may already have landed before the crash.
    assert after_crash == "uncertain" and calls["n"] == 1

    print("\nall assertions passed")

Executed output:

client resuming after a worker restart
  bare sequence cursor      : ['g4-e3']
  (generation, sequence)    : ['g4-e3', '<<resync from snapshot>>']

10-minute heartbeat after a 3-hour outage
  prompts if every tick replays : 18
  prompts if ticks coalesce     : 1
  schedule advanced to (s)      : 10800

command journal
  first submit            : compacted
  same command id again   : compacted
  times the mutation ran  : 1
  resubmit after a crash  : uncertain

all assertions passed

The cursor result is the one to internalise. The bare-sequence client does not error, does not warn, and does not appear broken. It simply never shows the two events that arrived after the restart, because their sequence numbers are below a cursor from a stream that no longer exists. The generation-aware client detects that its cursor belongs to a retired generation and falls back to a snapshot.

That gives the rule the rest of the protocol hangs off: incremental replay is an optimisation over an interval the server can still cover, and a coherent snapshot is the durable recovery baseline. A protocol that promises full historical replay has committed to unbounded retention; one that treats replay as best effort and always offers a snapshot has not.

Practical corollaries worth writing down when you build this:

A client presents its cursor on attach, and the server answers whether the requested interval is complete, partial, or unavailable. Partial is a normal answer, not an error.
Snapshots for large transcripts are streamed as opaque chunks through a bounded cache. One published design targets 512 KiB chunks and moves transcripts above 4 MiB to a file-backed cache, so the supervisor never materialises a history-sized object in memory.
Backpressure must be attachment-local. A blocked client stops receiving incremental events; other clients and the worker continue; the supervisor keeps no unbounded per-client queue; after the drain, the slow client catches up from its cursor or takes a fresh snapshot.

How to maintain it¶

Lease every transcript. A persisted session is protected by a process-safe lease keyed on the canonical transcript path. A worker takes the lease before opening the session, replacing a runtime takes the new lease before releasing the old, a concurrent open returns a structured "already active" error naming the owner, and concurrent creates for the same path converge on a single worker launch. Without this, a scripted one-shot invocation and a resident worker will interleave writes into the same append-only file, and the corruption is not detectable from either side.

Fence the supervisor generation. Workers watch the supervisor socket, and if it disappears one of them takes an atomic launch lease and starts a replacement, which then adopts the live workers. That adoption is only safe if private worker connections are authenticated per worker and fenced to the current supervisor generation, otherwise a superseded supervisor can keep issuing commands to a worker that a newer one now owns.

Bound crash recovery. Retry restarting a failed root a small fixed number of times with increasing delay, then mark it failed rather than looping. One published design uses 250 ms, 1 s, and 5 s before giving up. Recovery should reap the dead process group and any detached subprocess trees it tracked, append a visible recovery marker into the transcript so the agent and the human both know a discontinuity happened, restore the root under the same public session id, and never replay uncertain side effects.

Version the protocol and the schema separately. A compatible addition can be capability-gated or bump a schema revision; only an incompatible wire change bumps the protocol version. Keep coverage in both directions, old client against new server and new client against old server, because in a detached architecture those combinations happen every upgrade by construction.

Persist schedules per session, not in one global file. A shared cron file is a single point of contention and of corruption across otherwise-independent sessions. Claim and advance a due tick before delivering its prompt, so a crash between claim and delivery cannot replay an uncertain prompt.

How to run it in production¶

Write worker descriptors, authentication tokens, session paths, and recovery journals with owner-only permissions. They contain enough to command an agent that holds the user's credentials.

Monitor four things. Worker count and restart rate per root tells you whether one workload is crash-looping. Attachment count and per-attachment lag tells you whether a slow client is being correctly isolated by backpressure rather than stalling the supervisor. Journal size, which should stay small if clients acknowledge durable results so old entries can be compacted, tells you whether acknowledgement is actually happening. Schedule claim age tells you whether ticks are firing or silently stuck behind a busy session.

Capacity planning is mostly memory. Each resident worker holds a session, an interpreter, and its children, so the ceiling is set by interpreter footprint rather than by any protocol limit; published designs of this shape impose no fixed session, worker, or client cap in the supervision layer itself. Establish your own ceiling by measurement, and prefer stress-testing many resident roots with schedules advancing concurrently, because that exercises the paths that only fail under fan-out.

Give agents stable names and require them for anything scheduled. An automation that addresses an agent by a generated id will break the first time that session is replaced by a fork or an import, whereas the public active-session id survives those operations by design.

Failure modes¶

Failure	Symptom	What to do
Bare sequence cursor across a restart	Client silently misses every post-restart event	Cursors carry a generation; a newer generation forces a snapshot
Replaying every missed schedule tick	Prompt flood after an outage, unbounded backlog	Advance the schedule fully, deliver at most one prompt
Mutation replayed after a crash	Duplicate side effect, double-charged operation	Journal before dispatch; report unresolved commands as uncertain
No transcript lease	Two processes append to one session file, corrupt history	Process-safe lease keyed on the canonical path
Superseded supervisor still commanding workers	Conflicting commands, split-brain routing	Fence worker connections to the current supervisor generation
One-shot invocations leaving resident agents	Agent list fills with stale sessions	Client-owned worker lifecycle with a bounded cleanup grace period
Unbounded per-client event queues	Supervisor memory grows with the slowest client	Attachment-local backpressure, catch up from cursor or snapshot
Treating process isolation as a sandbox	Prompt injection reaches the user's files and credentials	Isolate the whole worker externally
Upgrade stops workers before validating	Sessions lost when a prepare step fails	Two-phase prepare, validate the aggregate manifest, then commit

References¶

Supervisor and worker architecture, leases, protocol versioning, snapshot streaming, backpressure, journaling, and coordinated updates: https://github.com/PrimeIntellect-ai/prime-agent/blob/main/packages/coding-agent/docs/daemon.md
Client and runtime connection boundary, generation-aware cursors, reconnect and replay, session replacement: https://github.com/PrimeIntellect-ai/prime-agent/blob/main/packages/coding-agent/docs/agent-connection.md
Long-running and background agents: detached workers, agent-to-agent messaging, heartbeats, schedules, goals, autonomous continuation: https://github.com/PrimeIntellect-ai/prime-agent/blob/main/packages/coding-agent/docs/long-running-agents.md
System topology and end-to-end prompt flow across client, supervisor, worker, session, provider, and storage: https://github.com/PrimeIntellect-ai/prime-agent/blob/main/packages/coding-agent/docs/architecture.md
Repository the above documents are published in, read at commit a18809e (2026-08-07): https://github.com/PrimeIntellect-ai/prime-agent