Design spike · safety.viz · 9 October 2026

The 22-minute check can run in about 6

Every pull request to safety.viz waits on one check, “Build, format, and test”. On 9 October it took a median of 22.6 minutes. I timed five other ways of running the same tests on the same runners. Nothing is merged yet.

Decisions

9 October 2026, in the session, after reading this page: “Sounds good! Build out the objective. Run a security assessment in parallel and fold in any concerns from that before we start the build.” This settles the question at the end of the page: both steps are built, and the evidence refresh run is in the same objective.
9 October 2026, in the session’s prompt, asked whether the work should wait for release 1.12: “Now, in v1.11.0”. The change lands on dev beside the release 1.11 work and ships in that release.
9 October 2026, in the session, after the tree was filed: “objective approved. you sign as me. draft a handoff prompt and i’ll kickoff build later.” The session did not sign for him, for the reason given under Starting the build, where the sign-off command and the prompt are.
Follow-through: filed on the hub as the objective a fast check, with two requirements, the check itself and the evidence refresh run, and four tasks, three in safety.viz and one on the hub. The build starts once your sign-off comment is on the objective.

22.6 min

Today. The check runs both test suites, then the evidence guard runs both again.

12.1 min

Run the suites once and let the evidence guard read their results. No test changes.

6.3 min

Also run the browser suite on two runners at once, with the eight real-R tests on their own. No test changes.

All three figures are measured on GitHub’s runners, not estimated. The lower two come from a scratch branch that ran the full suite: 2,392 unit tests and 406 browser tests, all passing, and the evidence guard matching all 14 evidence files.
My recommendation is the third line, built as one pull request to dev. It keeps every test, every screenshot comparison and the evidence guard, and it keeps the check’s name, so the ruleset on dev does not change.
You decided the same day: build it now, in release 1.11. The record is in Decisions, above.

Where the 22 minutes go

One recent run on dev, 22 minutes 57 seconds, drawn to scale. The hatched part is work the same run had already done.

Setup and static checks: install, formatting, build, six vendored-source checks, docs site, browser install0.9 min
Unit tests, 2,392 of them1.2 min
Browser tests, 406 of them9.8 min
Evidence freshness guard: starts the unit tests and the browser tests again from scratch, then compares the outcome with the committed evidence files11.0 min

The guard’s own work, comparing test names and pass or fail against the 14 evidence files, takes under a second. The other 11 minutes are the repeat.
The repeat is by construction: npm run evidence:check always runs both suites itself, and the workflow calls it after it has already run them as its own steps.
Everything that is not a test adds up to under a minute. There is nothing to gain from caching installs or trimming the static checks.

Source: CI run on dev, 9 October 04:04 UTC; ci.yml; scripts/evidence.mjs.

Why it doubled in two days

Median length of a successful check, by day, over the last 100 successful runs.

3 Oct 36 runs

11.7 min

4 Oct 9 runs

12.8 min

6 Oct 2 runs

11.3 min

7 Oct 26 runs

11.2 min

8 Oct 18 runs

16.1 min

9 Oct 9 runs

22.6 min

The RBQM tab arrived on 7 and 8 October with tests that start real R in the browser, install gsm’s packages and run the pipeline. There are eight of them now.
Each one pays about 7 seconds to start R and about 41 seconds to attach some forty packages before it tests anything. That figure is from the measurements the tests themselves commit.
Together the eight take 8.4 minutes of test time. Because the check runs the suite twice, each minute added to the suite adds two to the check.
These tests are doing their job: they hold R in the browser to desktop R to eight decimal places. This spike does not propose dropping or weakening any of them.

Inside the browser suite

Test time by spec file, from the same run. Purple is the eight real-R tests; blue is the other 398.

Demo app spec 89 tests, 6 with real R

9.6 min

RBQM pipeline spec 1 test, real R

1.9 min

Docs site spec 35 tests, 1 with real R

1.6 min

Hep explorer spec 46 tests

0.6 min

The other 16 spec files 235 tests

2.1 min

The suite holds 15.7 minutes of test time and runs on two workers, so it could in principle finish in about 8. It takes 9.7.
The reason is the demo app’s spec file. Playwright runs the tests inside one file one after another, and that file alone is 9.6 minutes. The second worker finishes every other file in about 6 minutes and then waits.
So the suite cannot finish sooner than its longest file, however many workers it has. I checked this on the runner: four workers instead of two took 10.8 minutes against 10.4.
Eight tests out of 406 are 53% of the test time. Taking them out of the queue the other 398 stand in is what the recommendation does.

What I measured

A temporary workflow on a scratch branch ran each layout on the same kind of runner the check uses, on the same commit of dev. Times are from the first job starting to the last job finishing.

Layout	Wall clock	Result	What it showed
Today: one job, suites run twice	22.6 min	passes	Median of the 9 successful checks on 9 October.
One job, suites run once, guard reads the results	12.1 min	passes	Three runs: 12.0 and 12.1 minutes, and 22.0 when the browser install stalled for 9.5 minutes (see the note further down). The guard itself took under a second.
The same, with four browser workers	13.1 min	passes	No gain. The browser step took 10.8 minutes, held by the one long file.
The same, four workers, tests inside a file side by side	10.6 min	1 test fails	The local demo spec starts a server; two workers each start one on the same port and one test cannot connect. Would need test changes.
Four shards, tests inside a file side by side, then a gate	8.4 min	passes	Shards are cut by test count, and the first one gets the whole demo app spec: 7.3 minutes against 0.5 for the second.
Two browser jobs split by real R, a static job, then a gate	6.3 min	passes	Three runs: 5.3, 6.3 and 6.5 minutes. The spread is the real-R job, whose start-up time varies from run to run.

In the last layout the real-R job is the longest: 5.0 to 6.2 minutes for its 8 tests, against 4.2 to 5.3 for the job with the other 398. Static checks and unit tests ran beside them in 2 minutes. The gate took 20 seconds.
The gate merges the two browser reports and hands them, with the unit results, to the evidence guard. The guard reported all 14 evidence files fresh, with the same record counts as the guard in today’s check.
I also ran the failure: given a browser report holding 6 of the 406 tests, the guard exits with an error and names every missing test in all 14 files. This is not the whole story: the security assessment below found failures the guard alone does not catch.
A side test: the larger browser job passed, screenshot comparisons included, with the browser installed without its system packages. The install took 8 seconds instead of 22 and never touches the package mirror. One run only.

Sources: round 1, round 2, the temporary workflow, everything the scratch branch changes.

Recommendation

Run the suites once

22.6 → about 12 minutes
- The unit and browser steps each write a results file; the evidence guard reads the two files and no longer starts the suites itself.
- About 30 lines in the evidence script and three changed lines in the workflow. The script change is written and exercised on the scratch branch.
- Run on its own machine, npm run evidence:check behaves as it does today.
- No test changes, no new jobs, same check name.
Give the browser suite two runners and a gate

about 12 → about 6 minutes
- Four jobs in place of one: static checks with unit tests; browser tests without real R; the eight real-R tests; and a gate that merges the reports and runs the evidence guard.
- The gate takes the name “Build, format, and test”, so the required check on dev is unchanged and no ruleset edit is needed.
- The real-R tests are picked by a tag written on each test, so a new one joins its job by carrying the tag. The two browser jobs use one pattern, one of them inverted, so every test runs in exactly one job.
- The public repository’s runners are free, and the four jobs use about 13 runner-minutes against today’s 22.
Give the evidence refresh run the same split, later

7 to 11 minutes today; not measured
- The run that rewrites screenshots and evidence files before a check is a single pass over both suites, so it has no repeat to remove. It ran 52 times in the first nine days of October.
- Splitting it means collecting rewritten screenshots from two runners before the commit. That is more work than the check’s split and I did not try it.
- With steps 1 and 2 done, a change that needs new screenshots waits about 16 minutes in all, refresh then check, against about 33 today.

Three things the build has to get right

GitHub counts a required job that was skipped as passed. The gate has to run even when a job before it fails, and fail unless all three succeeded. The scratch branch’s gate always ran but did not check the three results; see the security assessment.
The evidence files key on each test’s name, so the tag must not change it. I checked on a throwaway test: a tagged test reports the same name, and the two patterns pick it up or leave it out as intended. The scratch branch itself still picks the eight by their requirement IDs.
Today’s check uploads one browser report for debugging a failure. The gate should upload the merged one.

Security assessment of the new layout

Added later on 9 October. A second agent, given the scratch branch and the plan but not this page’s conclusions, looked for ways the new check could say green when it should not. It ran the evidence guard against hand-edited results from a real run of the 2,392 unit tests and the real list of 406 browser tests. It pushed nothing, so what it says about GitHub’s own behaviour is read from documentation, not run.

A correction to this page

Above, I wrote that a job that drops out cannot pass silently. That holds for a missing test and no further. The gate on the scratch branch ran whenever the jobs before it finished and let the guard alone decide, and the guard looks only at test names and pass or fail.
The assessment showed the guard exits clean on a test that failed and then passed on a retry, on unit results marked unsuccessful because a test file failed to load, and on a browser report that carries errors. In today’s single job the test steps’ own exit codes catch all of these.
The spike’s timings stand. Its gate was not safe to require as written, and the requirements now say what makes it safe.

Finding	Severity	From this change?	What closes it
The gate can report success while a job before it failed	High	Yes	The gate’s first step fails unless each of the three jobs ended in success. Nothing in the workflow may continue on error.
GitHub counts a skipped required job as passed	High	Yes, as a hazard; the spike had this part right	The gate always runs. A unit test reads the workflow file and fails if the gate’s shape changes.
The guard trusts the results files more than exit codes did	Medium	Made worse	Handed files, the guard also fails on a run marked unsuccessful, on errors, on unexpected or flaky results, and on a test name that appears twice. One test for each.
The ruleset accepts a check of this name from any source	Medium	No, it was already so	A ruleset edit that ties the check to GitHub Actions, which is yours alone. It also means changing the program’s ruleset standard, so it is raised with the sign-off and left out of the build.
Picking tests by tag: a pattern that matches nothing, or a typo	Low	Yes	One variable feeds both browser jobs. The guard already names any test that ran in neither.
Installing the browser without its system packages could shift fonts inside the 2% screenshot limit	Low	Only if adopted	Not adopted here. Both workflows keep the packages and name the runner image.
Artifacts downloaded into the checkout could pick up a committed file	Low	Yes	Download to the runner’s temporary directory, fixed names, and an error when an upload finds no file.
Pull requests from forks	For information	No new exposure	The check stays on the read-only pull request trigger and never gains write permission.

Found sound, each by running it: the guard fails on a test that is absent, failed, skipped, timed out or never run; on an empty, missing or cut-off results file; and the flags are refused outside the check. Merging reports fails on an empty folder and on a damaged file, and rejects a path that climbs out of its folder.
Not verified by anyone: what the gate sees when only a failed job is re-run. It is in the list of deliberately broken runs the build has to show.

Looked at and set aside

Skipping the browser suite when only documents change. 10 of the last 60 merged pull requests touched no code, tests, scripts or site files. A skip needs a second path through a required check, which is where a silent pass comes from. At five minutes a check it is not worth that.
Running the real-R tests only when R-related files change, or only on dev. After the split their job is the longest of the three by about a minute, so leaving it out would save about a minute and take the comparison with desktop R out of the check.
Letting tests inside a file run side by side. It is the way to get well under five minutes, but the specs build the app and start servers in shared places, and the one attempt failed. It means changing tests, with the evidence files following.
Caching what R downloads. Not tried. The real-R tests check exactly which requests leave the browser, so a cache between the page and R’s hosts would sit inside the thing they test. I also did not measure how much of each test’s 48 seconds of start-up is download.
Speeding up the unit tests. They take 75 seconds, and after the split they run beside the browser jobs, off the slow path.

One thing I saw and did not chase

The step that installs the browser also installs system fonts through the runner’s package mirror. It normally takes 21 to 25 seconds. In one of my runs the mirror slowed to 39 kB a second and the step took 9.5 minutes; one of the last 70 real checks shows the same thing at 3.3 minutes.
It is rare and it clears on a re-run. The side test above suggests the install can skip the mirror altogether. It is not adopted in this work; the security assessment below says why, and it is kept as a follow-up.

The decision

Decided on 9 October; see Decisions at the top. As first put:

Build steps 1 and 2 together. This was the recommendation, and what was chosen, with step 3 added as a second requirement. They land as two pull requests, the single pass first.
Build step 1 alone. The smallest change, one file and three workflow lines, for about 12 minutes.
Leave the check as it is.

Starting the build

The tree is signed off by a comment from your own account on the objective. A session cannot write it for you: under the issue contract a comment a session wrote is not a sign-off, and the workspace’s guard refuses a write in your name.
Run the first block once. Then open a new session in the workspace and paste the second.

gh issue comment 412 -R jwildfire/obot.roadmap --body "Tree signed off: the two requirements (#410 and #411) and their tasks, as filed on 2026-10-09."

Build the faster required check for safety.viz, in release 1.11. The objective is "A fast check" (jwildfire/obot.roadmap#412). Its tree is filed, and my sign-off comment is on the objective; if you cannot find one from my account, stop and ask.

Run /requirement-session 410. When it closes with its proof, tell me, and ask before starting requirement 411: it is first on the cut line, and whether it runs depends on where release 1.11 stands that day.

Order inside requirement 410:

1. safety.viz#291 first, alone, and merged before anything else starts: the evidence guard reads the results it is handed, and the check runs each suite once. It takes the check from about 22 minutes to about 12 for every other session, so it is worth having on dev today.
2. safety.viz#292: the four jobs behind a gate named "Build, format, and test".
3. obot.roadmap#413 last: the developer guidelines, written from the workflow as merged. Show me the diff.

Read before building:
- The objective's Boundaries, which hold the constraints, the cut line and what is left out.
- The spike and its security assessment: https://jwildfire.github.io/obot.roadmap/reports/sv-ci-speed-spike-2026-10-09/ . Its source is in the obot.roadmap clone at reports/sv-ci-speed-spike-2026-10-09/.
- Requirement 410's Design section. It is the specification, and the part headed "What the security assessment of this layout adds" is not optional.
- The prototype: the branch spike/ci-speed in safety.viz, checked out at safety.viz/.claude/worktrees/spike-ci-speed. Read .github/workflows/spike-ci-speed.yml and its change to scripts/evidence.mjs, and start from the two flags if that helps. Never merge it. Its gate is wrong as written: it always runs, but it does not check the results of the jobs before it.

What matters most:
- No test is removed, skipped, renamed or loosened, and the screenshot limit stays at 2%. The speed comes from running each test once, on more runners, and from nothing else. If something can only be made to pass by checking less, stop and ask me.
- The check must never say green when a test did not run and pass. Prove it by breaking it: every deliberately broken run the tasks list is run for real on a scratch branch and linked, red at the gate, before the pull request opens. A failure you reasoned about and did not run does not count.
- The ruleset on dev is not touched and the check keeps its name. If you come to think a ruleset must change, write me the command and mark the task blocked.
- Release 1.11's build comes first. Another session, "safety.viz v1.11.0 demo app", is landing pull requests on dev through this same check, and it has been told this work is coming. Message it before each merge and after it, saying what changed and what it may need to do: after safety.viz#291 the evidence files on its open branches will conflict, and after safety.viz#292 a new test that starts real R carries the tag. If the new check gives a wrong answer once it has landed, a red on good code or a green on bad, revert the workflow change first and diagnose after.

Known from the spike, so you do not find them again:
- A unit test outside a chart's directory lands in all 14 evidence files. The new tests in safety.viz#291 need the evidence refresh run on that branch before its pull request opens.
- The bot cannot start a workflow_dispatch run, and a refresh it starts by repository_dispatch uses the workflow file on dev. Rehearse a changed workflow with a temporary push trigger on a scratch branch, as spike-ci-speed.yml does.
- The step that installs the browser sometimes stalls for minutes on a slow package mirror. Re-run it. Do not fix it by dropping --with-deps; the security assessment says why.
- The two browser jobs came out at 4 to 6 minutes each in three runs; the real-R job is the longer one and its length varies with R's start-up.

Not verified by anyone; check each before building on it, and tell me if one fails:
- What the gate sees when only a failed job is re-run.
- Whether a unit test can read ci.yml with what the project already depends on, or needs a YAML parser added. Say which you chose and why.
- For requirement 411 only: whether a refresh pattern and the tag combine in one --grep.

Housekeeping:
- Name any scratch branch you push spike/..., open no pull request from it, and list every one at the end for me to approve removing. Leave spike/ci-speed and its worktree as they are.
- When requirement 410 closes, give me one message: the lengths of the five most recent checks, the links to the broken runs, and whether you would run requirement 411 now or cut it.

What this spike left behind

A branch on GitHub, spike/ci-speed, two commits ahead of dev: the temporary workflow and the evidence script change. No pull request. The workflow runs only on pushes to that branch.
Two workflow runs under “Spike - CI speed (temporary)” in the Actions tab.
A local worktree for the branch, inside the safety.viz clone.
Nothing on dev or main changed. Removing the branch and the worktree is yours to approve.