Deep dive

Subagent Failure Deep Dive

Prepared 2026-06-06 for Jeremy review.

Executive summary

The failures were not one single bug. They came from a mixed orchestration stack: earlier successful work used OpenClaw-native subagent completion, later attempts used the ChatGPT/Codex spawn_agent/wait_agent tools available in this runtime, while OpenClaw CLI session spawning was blocked by sandbox filesystem permissions. I then compounded that by treating returned agent ids as reliable active work without durable liveness evidence.

Bottom line: subagents are not safe to rely on here until we standardize on one mechanism, make it observable, and add a durable supervision layer. For now, bounded main-session work blocks with clean worktrees are more reliable.

Observed symptoms

  • PM and Dev spawned agent ids later returned not_found.
  • openclaw agent via CLI failed with sandbox EPERM while trying to write identity/session files.
  • OpenClaw status showed no Tasks and no active PM/Dev sessions, despite my earlier “active” wording.
  • Manual main-session recovery succeeded: Hub Agent Overview was implemented, deployed, verified, and issue #21 closed.

Impact

  • Status reliability dropped: I reported work as active without enough proof.
  • PM handoff work stalled until Jeremy checked.
  • The architecture needs explicit liveness checks and recovery, not trust in a returned id.

Evidence timeline

Likely causes

1. Mixed subagent mechanisms

OpenClaw docs describe sessions_spawn, sessions_yield, and subagents as the native mechanism. The current runtime exposed separate spawn_agent/wait_agent tools, not the OpenClaw sessions_spawn stack. That means OpenClaw status, cron/task views, and session lists did not reliably reflect these tool-spawned agents.

2. Missing or unavailable OpenClaw session tools in this context

The runtime context indicated local embedded mode, with gateway-dependent tools unavailable. OpenClaw docs say sessions_yield is the correct waiting primitive for spawned subagents, and that sessions_spawn availability depends on tool policy. In this session, the proper OpenClaw session tools were not available as first-class tools.

3. Sandbox EPERM blocks CLI-based Gateway agent runs

Attempting to use openclaw agent from the sandbox hit filesystem writes outside the writable roots:

EPERM: operation not permitted, open '/Users/obot/.openclaw/identity/.fs-safe-...tmp'
EPERM: operation not permitted, open '/Users/obot/.openclaw/agents/main/sessions/.fs-safe-replace...tmp'

So CLI fallback is not a reliable way to start isolated PM/Dev sessions from this sandbox.

4. No durable work-session ledger

There was no durable record of spawned work-session id, expected checkpoint, liveness method, and recovery plan. Without that, “still going?” required ad hoc reconstruction.

5. Auto-archive/stale-run semantics can hide old children

OpenClaw docs note that subagent sessions can auto-archive and stale unended runs stop counting as active. That is good hygiene, but it means status must be checked through the right owner/session and cannot be inferred from memory.

Recommended mitigations

  1. Standardize on one runtime: prefer OpenClaw-native sessions_spawn + sessions_yield when available. Do not mix with separate spawn_agent unless we explicitly mark it experimental.
  2. Expose needed tools: configure the main Telegram session with OpenClaw sessions_spawn, sessions_yield, and subagents if policy allows.
  3. Do not use CLI agent fallback from sandbox: it currently cannot write required identity/session files. Use first-class tools or unsandboxed scheduled jobs.
  4. Create a work-session ledger: record session id, task, issue/PR, start time, checkpoint, liveness method, and recovery plan in a local JSON/Markdown file and linked issue comments.
  5. Add liveness gate: before saying a cycle is active, verify reachable session status or concrete evidence: recent diff, commit, check run, issue comment, report, or process.
  6. Use TaskFlow/cron for long work: for work expected to survive turns, use durable TaskFlow/cron-style jobs with explicit announce delivery rather than ephemeral subagents.
  7. Keep main-session recovery path: for bounded Hub/PM fixes, main can use clean worktrees and direct evidence when subagents fail.

Concrete changes already made

Proposed implementation plan

  1. Issue #25: implement a work-session ledger and status helper.
  2. Config audit: inspect tool policy for main Telegram sessions and decide whether to expose OpenClaw-native session tools.
  3. TaskFlow option: if session tools cannot be safely exposed, implement PM/Dev cycles as durable TaskFlow/cron jobs with explicit completion announcements.
  4. PM cycle update: every PM handoff must include monitoring plan and recovery action.
  5. Status command/report: add a small report that lists planned, active, stalled, completed, pushed/deployed, and reviewed work.

Open questions