roadmap audit · decision artifact · 2026-08-15

The roadmap audit audits itself

A full audit of jwildfire/obot.roadmap ran tonight under the new standing grant to auto-accept high-confidence findings. 32 changes were applied without asking. But the sharper instruction was the aside: “generally speaking, if it needs my attention it probably isn’t a good audit rule.” Applied as a test to all 22 rules, it fails five of them — and 18 of the 24 findings that landed in your lap came from just two rules. One broken rule was fixed and shipped tonight; the rest are the decisions below.

Decisions

@jwildfire · 2026-08-15 · in chat

“I reviewed the roadmap artifact. It’s good overall. R4 is the only place we disagree. I do want requirements tied to a single release. If a requirement is too big for that, split it into multiple requirements. It’s fine to defer sub tasks, just make a note on the original requirement, file a new requirement and then transfer the deferred tasks. Other than that note, your recommendations are approved.”

Six of the seven calls are adopted as recommended — the four verified-shipped closes plus the standing grant to close future ones the same way, re-homing the three orphaned pieces of work, retiring the stale checkbox list, parking long-untouched work automatically instead of asking why it stalled, catching a missing design when work enters development rather than after it ships, and taking goals off the delivery board.

The seventh is rejected and replaced. The proposal was to teach the audit that some requirements stay open on purpose because a later phase is coming. He does not want requirements that stay open on purpose. A requirement is tied to exactly one release; if the scope is bigger than one release, it is more than one requirement. Deferring is still allowed — it just has a procedure, in this order: note the deferral on the original (what is deferred and why), file a new requirement for the deferred scope with its own milestone, transfer the deferred sub-tasks to it, and the original then closes with its release. “Phase two is coming” is never a reason to hold a requirement open, because phase two is a different requirement.

Carried out the same morning. The four approved closes are closed. Both known violations of the new rule were split by the procedure above: the kidney-explorer requirement closed with the release that carried its population screen, and its unbuilt patient-profile half moved to a requirement of its own; the liver-explorer follow-up closed with the six enhancements it delivered, and its three unshipped items were transferred to a new requirement rather than re-filed, so their scoping and history survived the move. A sweep then found five more requirements in the same shape — they are in §5. The rule itself was written into the requirement-authoring path, where requirements are created, rather than into the audit that catches them afterwards.

0. What was done about all this, the same morning

Everything below was carried out on 2026-08-15 by an unattended session, under the decision above. Nothing was deleted.

0.5  A second pass, later the same day

A follow-up run once all of the above had landed found three things the first pass could not have seen, because they only exist after the changes were made.

Two rules started fighting over the same two cards

Making the stalled-work rule park things automatically is what took it from thirteen findings that all needed you to none. But it also made it executable — and it now collided with the rule that promotes a requirement once every task under it is closed. On the abnormal-baseline liver tools and the open.csr submission outputs, both rules fired the same night and proposed opposite moves: park it back to Backlog, versus promote it to Review or close it. Two automatic rules disagreeing about one card means whichever runs last decides where the card lands — which is the unpredictability the audit exists to remove, and nothing caught it because until this morning the two were never both automatic at once.

Fixed: a requirement whose every task is closed is finished, not stalled, so the promote rule owns it and the park rule stands aside. That is the same “one rule owns one situation” pattern the released-and-open rule already uses. Verified against live state afterwards: no card anywhere carries two automatic proposals. (4bddb57, tests written first.)

An eighth requirement in violation of your rule

The morning sweep found seven. The open.csr submission outputs requirement was an eighth and was missed, for an understandable reason: its delivered scope reached the app’s development tier on the same day the release went out rather than being described in release terms, so it did not read as spanning two releases. It did. RTF tables, the values store and the sidebar all shipped in the editing release on 27 July; the part that decides how editing reaches the reading view shipped in nothing, and nineteen days later had still not moved — for exactly the reason your rule names, that the requirement carrying it already read as done.

Split by the procedure: a deferral note on the original, and the deferred part filed as its own requirement under the open.csr goal with a milestone, carrying its four options and their recommendation across verbatim so the decision loses nothing in the move. No tasks to transfer — the original’s only task covered the shipped parts and is closed. The limit worth remembering is in the sweep, not in the rule: “spans two releases” is easy to see when a requirement says so and easy to miss when it does not, so the audit catching it on the next pass is the safety net.

The audit was flagging its own question queue

The four decisions rescued from the retired checkbox list were filed as ordinary issues — and were promptly reported as defects for being tracked by nothing, which was true and was also the entire point of them. Rather than teach the audit to ignore them, they were put on the board, along with the archive watch item. Hiding them would have recreated precisely the failure that let the checkbox list rot for three weeks: a decision queue that no surface tracks is a decision queue nobody reads.

1. What was applied without asking

Every change below is reversible and none of it deleted anything. Issue closes were deliberately not auto-applied — they are a state change with consequences, and agent closes are hook-blocked in this workspace by design. They are batched into the first decision below: closing four issues that already shipped C1.

Live findings after these changes: 45 → 28. Of the 28 remaining, most are the decisions below; the rest are covered by the rule changes proposed in §3.

2. The decisions — seven, all in one place

These are the findings that survived auto-application. Each is here because doing it silently would have cost something irreversible, not because the rule was clever.

Close four issues that are provably shipped C1

Four issues are open, sit at Released on the board, have zero open sub-issues, and carry in-thread evidence of shipping written by the sessions that shipped them. This exact set was already written out for you, as an unchecked box, on the review checklist handed to you on 2026-07-25 — the one that was never worked to the bottom (#114). So this has been sitting in your queue for three weeks.

• #3 obot GitHub App — the design §6 acceptance criterion was verified in-thread on 2026-07-04 by app-smoke-test.yml posting as obotclaw[bot]; the App has been the standing agent identity ever since.
• #21 safety.viz docs site — all four sub-issues closed; all three publishing tiers live and verified in safety.viz v0.1.0.
• #2 Safety Histogram pilot — both sub-issues closed; done-gate satisfied on the released site. (The remaining fork clean-up survives as its own issue, #81, now milestoned and sitting under the keep-adding-charts goal.)
• gsm.safety#44 branch model — “implemented and verified 2026-07-23” in its own comment; every PR since has landed under that model.

Re-home three pieces of work stranded under closed parents O1

Each of these is an open piece of work whose parent requirement was closed as completed above it — so the work survives, but it hangs off something finished and is invisible from any goal page. All three now have a backlog milestone so they are pickup-ready. Moving them means unlinking them from a closed parent, which is a structural change I did not make unasked.

Retire the three-week-old checkbox list of items waiting on you H1

On 2026-07-25 a session handed you a single long issue — a morning review-and-release guide — in which every line was a checkbox, and ticking one counted as your approval for the agent to act (#114). Nine boxes were ticked; eleven never were. It is quietly the single largest source of “waiting on @jwildfire” anywhere in the hub — and most of what is left has been overtaken by events. Here is every unticked box, checked against live state tonight.

Seven are dead letters. Each asks you to decide something that has since decided itself:

• Keep or kill the dashboard chat feature. Settled on 2026-07-26: the prototype was merged, but the feature was parked and never armed, and the requirement behind it is closed (#77). (boxes D1a / D1b)
• Approve the participant-profile v2 design. Overtaken — you built and shipped it yourself in safety.viz v1.5.0 the same week. (box D2)
• Approve merging the status-line footer link. Already merged. (box D6)
• Work through a bundle of audit findings. Superseded by this page, which is that pass. (box A2)
• Close the four issues that shipped but were never closed. That is the first decision on this page — tonight’s audit reached the same four independently. (box X2)
• A bundle of things only your own clicks can do — installing the app on two more repos, an avatar image, a reply on a discussion, a Reminders allowlist, three unreviewed diary entries — plus one real question buried inside it: whether goals belong on the delivery board. That question is now the sixth decision on this page. (box X3)

Four are genuinely still live. These are the ones the recommendation below re-files as ordinary issues, so read them as what you would be agreeing to keep:

• Whether the public roadmap keeps publishing what this program costs. The roadmap page currently shows total spend, total tokens, the number of agents run, and a per-agent breakdown, all visible to anyone. The box asks you to confirm that stays public; reverting it is one commit. (box D3)
• The Goal Atlas — a finished analysis of how every issue connects to a goal, waiting to be acted on. It offers six chart-related re-links ready to apply immediately, and behind them three proposed new goals, nine further re-links, and thirty-two candidate requirements it found lying around unfiled. The analysis is done; only the accept-or-reject is outstanding. (box D4)
• The platform gap analysis — where this program stands against competing tools. A capability matrix plus twelve ranked build proposals; the headline finding is that we have none of the ten review-workflow capabilities the competition has, against strong chart coverage. The open question is which proposals to actually file. (box D5)
• Close the ideas-triage requirement, now that it has demonstrably worked. The second-generation triage has promoted real ideas into real issues, which was its acceptance criterion (#58). (box X1)

3. The rule set, judged by your own test

You wrote: “generally speaking, if it needs my attention it probably isn’t a good audit rule.” That is a sharper claim than it looks. It says the measure of an audit rule is not whether it detects something true — all 22 of these detect true things — but whether the thing it detects can be resolved by the system that detected it. A rule that ends in a question is a rule that has outsourced its own job.

Here is every rule that fired tonight, scored that way.

Stop asking “why has this stalled?” — park it instead R1

Thirteen findings — 29% of the entire audit, and every one of them yours. Its detection is fine (board stage is Development or Review, no activity for 14 days, no open PR references it). Its proposal is “Establish what is actually blocking it and either move the stage back with a reason, or record the blocker.” That is a request for an explanation, and no amount of GitHub data can supply one. So it re-fires every single night, for the same 13 issues, forever.

Two deeper problems. First, stalled is not a defect — you run a wide program and park things deliberately; the rule treats normal behaviour as a fault. Second, it double-counts: the DILI tools requirement (#43) fired here and under the all-sub-issues-closed check, where the very same situation produced a stage move the audit could apply by itself. When two rules see one fact and only one of them proposes a change, the other is noise.

Stop demanding design write-ups for work that is already built R2

Five findings, all landing on you, all proposing “Write the Design section from the work as it actually stands.” But every one of the five is already in Development: the remaining-renderers release (#29), the hepExplorer follow-ups (#88), the open.csr change-request loop (#130), the roadmap-hierarchy work (#132), and the demo-301 build (#134). Retro-writing a design document for work that is already built is documentation theatre: it cannot inform a decision that was made weeks ago.

This one is also mis-classified. It is not a cleanup finding at all — it is a work order, and work orders belong in the backlog where they get a milestone and a stage, not in a findings list that re-renders them nightly.

Settle whether goals belong on the delivery board R3

One finding, and it is a genuine inconsistency: the keynote goal (#72) and the autonomy goal (#73) sit on the board, while the charts goal (#78), the build-the-app goal (#79) and the open.csr goal (#112) do not. The rule’s proposal is “Settle the convention” — which is a question, not a change. It has re-fired every night since 2026-07-25, and the same question is also sitting unanswered inside the three-week-old checklist above. Twenty nights of asking you the same thing is the clearest possible illustration of your point.

R4  Superseded One requirement, one release

Verdict on the rule: sound now, but it needs re-specifying — in the opposite direction

You asked whether OPEN-IN-RELEASED is a good rule under your own test now that the new rule outlaws its blind spot. It is — and it becomes one of the better ones — but it does not get there by being left alone. Three changes, and one of them is the reverse of what R4-a proposed.

1. Delete the escape hatch instead of adding one. The rule's whole ambiguity was that a Released-and-open issue might be a defect or might be intentional, and it had no way to tell. Your rule removes the second category entirely: intentional no longer exists. So the rule must have no marker, label or comment that silences it — any such hatch would restore exactly the ambiguity that made it marginal. Released-and-open is now always a defect, and the rule's job is to say which of the two defects it is.
2. Two outcomes, both applicable without asking. If the issue has open sub-issues, unresolved options in its Design section, or recent activity, the board stage is wrong — move it back, do not close. Otherwise the work is done and unclosed — close it. Neither branch needs a human. This is the surviving half of R4-a, and it is what caught the open.csr stage error correctly last night.
3. Add the split as a third outcome, and keep this one a proposal. A Released requirement with open sub-issues is now a rule violation in its own right, not merely a stage error — it is a requirement spanning more than one release. The audit can detect it precisely and can prepare the whole split: draft the new requirement, list the sub-tasks to transfer, name the release the original delivered. It should not file it unattended, because choosing the new requirement's milestone is a scheduling judgement, and because a wrongly-split requirement is expensive to unpick. So: auto-apply the close and the stage fix, propose the split.

Worth saying plainly: this stops being an audit rule at the moment it matters most. The audit catches a requirement spanning two releases weeks after the split became necessary, when the deferred half's rationale has already faded. The rule belongs where requirements are written, not where they are inspected — which is why it went into the requirement-authoring and task-decomposition guidance rather than only into the rule set. The audit is the backstop for requirements written before the rule existed, and there are seven of those.

3.5  What happened when the fixes were applied

Applying 27 changes produced 6 brand-new findings — a useful accident, because it exposed two rule interactions that only show up when the audit is actually acted on rather than read.

• Linking a sub-issue silently adds it to the board with no Status. Giving #83, #94, #152 and #153 a home under UNTRACKED-TASK immediately made all four fire UNSTAGED-BOARD-ITEM. Both rules are correct; neither knows the other exists. Fix: UNTRACKED-TASK’s proposal should set a Status in the same operation, so accepting one finding does not manufacture another. All four were then staged to Backlog in this session.
• SUBS-DONE-PARENT-OPEN treats sub-issue completeness as scope completeness. Correcting #129’s false Released stage made it fire — its one sub-issue is closed, so the rule now proposes Review. But Part C is real outstanding scope that simply has no sub-issue yet. The rule is reasoning about the tracker, not the work. Fix: require the Design section to be free of open decisions before proposing Review, or treat “zero sub-issues remaining and no unresolved options in the body” as the bar.
• The ladder works as designed. Moving #43 from Development to Review immediately escalated it to a close proposal — correct behaviour, and it means one more close is waiting behind C1.

The general lesson: a rule set is only testable by applying it. These three interactions had been latent since the audit shipped on 2026-07-24 because findings were being read, not executed in bulk. Worth re-running the audit immediately after any apply batch and treating a rule that reliably creates work for another rule as a defect in the first one.

4. What this does to the numbers

Tonight: 45 findings, 32 changes applied unattended (27 from the audit, 5 cleaning up after it — §3.5), 24 findings needing you — 53%. Live findings closed the night at 28.

With six recommendations adopted and the seventh replaced by your rule, the same roadmap state produces roughly 6 findings needing you, about 13%. The six that remain are genuine one-off scope judgements, like deciding where a stranded piece of work should now live, rather than the same question re-asked every night. Your replacement of the seventh does better than the forecast rather than worse: the rejected version would have silenced a class of finding, while the rule you gave instead eliminates the condition that produced it — and the sweep in §5 has already cleared every existing instance, so that rule starts tomorrow night from zero.

The one structural addition worth naming: three of the five failing rules fail for the same missing capability — the audit has no memory of stage transitions. It sees only tonight’s snapshot, so it can detect “this is in Development with no design” but not “this entered Development without one”. Adding a stage-history field to the ledger is what turns DESIGN-MISSING, a sharpened STALLED-IN-FLIGHT and part of OPEN-IN-RELEASED from interrogative into imperative. That is the single highest-leverage change to the audit, and it belongs inside the existing nightly-roadmap-audit requirement that owns these rules (#92) rather than in a new one.

5. The retroactive sweep — seven requirements, all closed

Your rule is not only forward-looking: any requirement already open across more than one release is in violation of it today. So the roadmap was swept for that shape — an open requirement whose delivered scope has shipped in a release while other scope remains — and every case found was split by the procedure and closed the same morning. Seven in total: the two known ones, and five the sweep turned up.

The pattern in all seven is the same and it is worth naming, because it is what the rule is actually for. None of these was neglected work. Each delivered its headline scope, shipped it, and then kept a small unshipped tail alive indefinitely — and the tail did not move, in one case for a month past its own release, precisely because the requirement carrying it already read as substantially done. An open requirement is the worst possible home for deferred scope: it is finished enough that nobody re-reads it, and open enough that nothing prompts anyone to.

Two of the seven needed no new requirement at all, and that distinction is worth carrying into the rule: the split is for deferred scope, not for every open thread. A defect found after release is an ordinary issue against shipped work. Scope that already has its own requirement just needs re-homing. Filing a new requirement for either would be paperwork, not tracking.

Net effect on the roadmap: eight requirements closed (the seven here plus the fourth verified-shipped close in the batch you approved), five new requirements filed, all carrying milestones, and eleven sub-tasks either transferred or re-homed with their history intact. Nothing was deleted. Every closed requirement now names the release it delivered, and there is no longer a single requirement in the hub that is open with its board stage at Released.

6. Already surfaced elsewhere — not re-litigated here

• #129 open.csr Part C — the board claimed Released; it is now back at Development. The decision itself (Options A–D for how editing reaches the reading view, with C recommended) is already written into the issue’s Design section and is yours to pick. Not duplicated here.
• #145 Lex Jansen archive — the watch-phase build decision was due at the ~2026-08-10 checkpoint and is five days overdue. Milestoned backlog tonight; it has its own workstream and deserves its own artifact, not a line item in this one.
• #153 demo-301 pipeline failing since 2026-08-03 — given a home under #134 and a 2026q3 milestone tonight, so it is now visible to goal #79 and the board.