Options · 30 October 2026

qcthat evidence extensions

Where qcthat stands today

I tried it. I added a Kind column and three non-test rows to the real safety.viz results. Unmodified qcthat carried them through the matrix and a JSON round trip, so the plumbing is open. The meaning is not: the report called all of them tests, and an issue whose only evidence was a screenshot printed "All issues have at least one test". Any option past A has to fix that.

The options

The mockups are drawn by hand from real safety.viz records for the nR ratio work in release 1.11.2. None of these has been built.

A. A new door for tests

Too narrow

qcthat learns to read test results from any language. CompileTestResults() becomes a generic with a new method for CTRF files. The report stays as it is, and qcthat-js only writes the file.

Adds
Vitest, Playwright and any other tool with a CTRF reporter
Leaves out
Everything that is not a test, which is the part you asked about
Cost
Small: a few functions in qcthat, nothing breaks

B. An evidence framework

Recommended

qcthat gains an evidence record: which requirement, what kind, what result, who, when and a link. Test results become one kind. Readers exist for each kind, the report groups by kind, and coverage counts only the kinds a project says should count.

└─█─Milestone: v1.11.2 (1 issue: 6 tests, 1 screenshot, 1 change, 1 acceptance)
  └─[x]─Issue 334: Hep Explorer: add the new R ratio (nR) to Point Size…
    ├─Tests
    │ ├─[v]─hep-explorer configure HEP-CTRL-007: Point Size offers Uniform…
    │ └─[v]─… 5 more
    ├─Screenshots
    │ └─[v]─HEP-DISPLAY-008-nr-ratio-point-size.png
    ├─Changes
    │ └─[i]─Pull request 335: merged by obotclaw[bot] on 6 checks, no human review
    └─Acceptance
      └─[v]─Release candidate 1.11.2: approved by @jwildfire, 15 October
Adds
Tests in any language, screenshots, reference comparisons, change reviews, human acceptance and AI review, with room for more
Leaves out
Requirements stay GitHub issues for now, with matrix rows carried as detail on the evidence
Cost
Medium: new columns and readers in qcthat, and a report rewrite for grouping. Test and Disposition stay, so today's reports and matrix files keep working

C. A full traceability framework

Later, if needed

Requirements become pluggable too. GitHub issues are one source, and matrix rows, hub requirements and other trackers could be others. Requirements form a tree, and each level can show its own coverage.

└─█─Milestone: v1.11.2
  └─[x]─Issue 334: Hep Explorer: add the new R ratio (nR) to Point Size…
    ├─[v]─Row HEP-CTRL-007: A Point Size select offers Uniform, rRatio and nrRatio…
    │ └─[v]─hep-explorer configure HEP-CTRL-007: Point Size offers Uniform…
    ├─[v]─Row HEP-DISPLAY-008: The participant nR-Ratio is the higher of the peak…
    │ ├─[v]─hep-explorer structureData HEP-DISPLAY-008: computeNRRatio is the…
    │ └─[v]─HEP-DISPLAY-008-nr-ratio-point-size.png
    └─[v]─Row HEP-CTRL-014: When Point Size is set to R Ratio or nR Ratio…
Adds
Everything in B, plus row-level coverage in the same report and support for teams not on GitHub Issues
Leaves out
Nothing in principle, but qcthat stops being the lightweight package its description promises
Cost
Large: a 2.0. The issue number runs through 15 R files, GitHub through 36, and matrix files attached to releases would change shape

D. A companion package

Fallback

A new package, say qcthat.evidence, adds the same features as B from outside and leaves qcthat alone.

Adds
The same as B, with no need to wait for anyone
Leaves out
Being part of the framework. The meaning of evidence would live in a second package that R users of qcthat never see
Cost
Medium, and it stays. qcthat's report functions are not exported, so changing the report from outside meant reaching into its namespace in my test. That breaks when qcthat changes

What could count as evidence

All of these exist in the safety stack today. The last column is my proposed default for whether a kind satisfies "this issue has evidence".

KindReal exampleCounts by default?
Automated test2,528 unit and 454 browser tests in safety.viz, testthat in gsm.safetyYes
Reference comparisonscripts/rbqm-reference.R, and the double programming box in qcthat's issue templatesYes, when a test runs it
Visual baseline228 screenshots in the safety.viz evidence foldersNo, it supports a test
Change reviewPull requests. Increment pull requests here merge on green checks with no human reviewNo, shown as a fact
Human acceptanceExpectUserAccepts(), and your approving review of each release candidateOnly where a project requires it
AI reviewThe matrices' AI Review column and the release review passesNo, and never as human acceptance

Requirements come in fewer kinds: GitHub issues (189 in safety.viz), hub requirements in another repository (39 with safety.viz tasks) and matrix rows (1,442). Trackers such as Jira are not used here, though other teams adopting qcthat may ask.

How an extension plugs in

This is a separate choice from the scope. There are two ways, and they can coexist.

Here is a proposed record. It borrows its shape from in-toto attestations, where each statement names a subject and a typed payload, and new types are allowed.

{
  "requirement": "jwildfire/safety.viz#334",
  "details": ["HEP-CTRL-007"],
  "kind": "acceptance",
  "label": "Release candidate 1.11.2 approved",
  "result": "pass",
  "by": "jwildfire",
  "at": "2026-10-15",
  "url": "https://github.com/jwildfire/safety.viz/pull/338",
  "source": "github-review"
}

Decisions for you

How far should the extension go?

Recommendation: Option B, shaped so Option C stays possible. It delivers "lots of kinds of evidence" now. The only second requirement source you have today is matrix rows, and B carries those as detail, so a full requirement framework would be built for one user.

How do extensions plug in?

Recommendation: the file, with a generic underneath it. The point of the work is reaching beyond R, so the extension point should not require R.

What counts as evidence for an issue?

Recommendation: tests count by default, every other kind is shown, and a project can require more on a given issue. Otherwise one screenshot could turn an issue green, as my test showed.

Next steps

  1. You make the three decisions above.
  2. I draft the proposal for the qcthat maintainers, and you post it, since qcthat sits outside your own account. You are its listed maintainer, but most recent work is by others, so I would treat this as a proposal to them. small
  3. The safety.viz cleanup goes ahead whichever option you pick: the parser fix, the link cleanup and the check for new tests. medium
  4. safety.viz writes evidence records for tests and screenshots, read by a short R script. This tests the format on real data before asking anyone to change qcthat. medium
  5. If the maintainers agree, the qcthat changes land in order: the generic, the file reader, the new columns, the coverage rule, then report grouping. medium
  6. The shared code moves to qcthat-js, and bio.viz becomes its second user. medium
  7. Readers follow for change reviews and acceptance, and ExpectUserAccepts() moves onto the same records. medium
  8. Option C gets a fresh look only if a real second source of requirements appears.

How far to trust this