Options · 30 October 2026
qcthat evidence extensions
- I recommend Option B, an evidence framework inside qcthat. Every piece of proof gets a kind, and test results from any language become one kind among several.
- Build it with requirement references as text. That leaves the door open to other kinds of requirement later, without a second rebuild.
- qcthat already leans this way. Its issue templates ask for double programming, unit tests and user tests, but its matrix can only see testthat results.
Where qcthat stands today
- There is one way in.
CompileTestResults()accepts only a testthat result, and it is a plain function, so another package cannot add a method to it. - There is one kind of requirement. 36 of qcthat's 58 R files call GitHub, and the matrix joins on an issue number.
- Human acceptance already works as evidence, but only by posing as a test.
ExpectUserAccepts()opens a sub-issue and passes or fails inside a testthat test. - Reviews as evidence are already wanted. An open requirement asks for it and says the team is not yet sure how it should work.
- A maintainer has already suggested making the core functions "less package-specific".
I tried it. I added a Kind column and three non-test rows to the real safety.viz results. Unmodified qcthat carried them through the matrix and a JSON round trip, so the plumbing is open. The meaning is not: the report called all of them tests, and an issue whose only evidence was a screenshot printed "All issues have at least one test". Any option past A has to fix that.
The options
The mockups are drawn by hand from real safety.viz records for the nR ratio work in release 1.11.2. None of these has been built.
A. A new door for tests
Too narrowqcthat learns to read test results from any language. CompileTestResults() becomes a generic with a new method for CTRF files. The report stays as it is, and qcthat-js only writes the file.
- Adds
- Vitest, Playwright and any other tool with a CTRF reporter
- Leaves out
- Everything that is not a test, which is the part you asked about
- Cost
- Small: a few functions in qcthat, nothing breaks
B. An evidence framework
Recommendedqcthat gains an evidence record: which requirement, what kind, what result, who, when and a link. Test results become one kind. Readers exist for each kind, the report groups by kind, and coverage counts only the kinds a project says should count.
└─█─Milestone: v1.11.2 (1 issue: 6 tests, 1 screenshot, 1 change, 1 acceptance)
└─[x]─Issue 334: Hep Explorer: add the new R ratio (nR) to Point Size…
├─Tests
│ ├─[v]─hep-explorer configure HEP-CTRL-007: Point Size offers Uniform…
│ └─[v]─… 5 more
├─Screenshots
│ └─[v]─HEP-DISPLAY-008-nr-ratio-point-size.png
├─Changes
│ └─[i]─Pull request 335: merged by obotclaw[bot] on 6 checks, no human review
└─Acceptance
└─[v]─Release candidate 1.11.2: approved by @jwildfire, 15 October
- Adds
- Tests in any language, screenshots, reference comparisons, change reviews, human acceptance and AI review, with room for more
- Leaves out
- Requirements stay GitHub issues for now, with matrix rows carried as detail on the evidence
- Cost
- Medium: new columns and readers in qcthat, and a report rewrite for grouping.
TestandDispositionstay, so today's reports and matrix files keep working
C. A full traceability framework
Later, if neededRequirements become pluggable too. GitHub issues are one source, and matrix rows, hub requirements and other trackers could be others. Requirements form a tree, and each level can show its own coverage.
└─█─Milestone: v1.11.2
└─[x]─Issue 334: Hep Explorer: add the new R ratio (nR) to Point Size…
├─[v]─Row HEP-CTRL-007: A Point Size select offers Uniform, rRatio and nrRatio…
│ └─[v]─hep-explorer configure HEP-CTRL-007: Point Size offers Uniform…
├─[v]─Row HEP-DISPLAY-008: The participant nR-Ratio is the higher of the peak…
│ ├─[v]─hep-explorer structureData HEP-DISPLAY-008: computeNRRatio is the…
│ └─[v]─HEP-DISPLAY-008-nr-ratio-point-size.png
└─[v]─Row HEP-CTRL-014: When Point Size is set to R Ratio or nR Ratio…
- Adds
- Everything in B, plus row-level coverage in the same report and support for teams not on GitHub Issues
- Leaves out
- Nothing in principle, but qcthat stops being the lightweight package its description promises
- Cost
- Large: a 2.0. The issue number runs through 15 R files, GitHub through 36, and matrix files attached to releases would change shape
D. A companion package
FallbackA new package, say qcthat.evidence, adds the same features as B from outside and leaves qcthat alone.
- Adds
- The same as B, with no need to wait for anyone
- Leaves out
- Being part of the framework. The meaning of evidence would live in a second package that R users of qcthat never see
- Cost
- Medium, and it stays. qcthat's report functions are not exported, so changing the report from outside meant reaching into its namespace in my test. That breaks when qcthat changes
What could count as evidence
All of these exist in the safety stack today. The last column is my proposed default for whether a kind satisfies "this issue has evidence".
| Kind | Real example | Counts by default? |
|---|---|---|
| Automated test | 2,528 unit and 454 browser tests in safety.viz, testthat in gsm.safety | Yes |
| Reference comparison | scripts/rbqm-reference.R, and the double programming box in qcthat's issue templates | Yes, when a test runs it |
| Visual baseline | 228 screenshots in the safety.viz evidence folders | No, it supports a test |
| Change review | Pull requests. Increment pull requests here merge on green checks with no human review | No, shown as a fact |
| Human acceptance | ExpectUserAccepts(), and your approving review of each release candidate | Only where a project requires it |
| AI review | The matrices' AI Review column and the release review passes | No, and never as human acceptance |
Requirements come in fewer kinds: GitHub issues (189 in safety.viz), hub requirements in another repository (39 with safety.viz tasks) and matrix rows (1,442). Trackers such as Jira are not used here, though other teams adopting qcthat may ask.
How an extension plugs in
This is a separate choice from the scope. There are two ways, and they can coexist.
- A file. An extension is anything that writes evidence records in an agreed format, and qcthat reads a folder of them. JavaScript, Python, a GitHub Action or a small script all qualify, and none needs R.
- An R package. An extension registers a method on a qcthat generic. This suits R users, but a JavaScript author would have to write R to contribute.
Here is a proposed record. It borrows its shape from in-toto attestations, where each statement names a subject and a typed payload, and new types are allowed.
{
"requirement": "jwildfire/safety.viz#334",
"details": ["HEP-CTRL-007"],
"kind": "acceptance",
"label": "Release candidate 1.11.2 approved",
"result": "pass",
"by": "jwildfire",
"at": "2026-10-15",
"url": "https://github.com/jwildfire/safety.viz/pull/338",
"source": "github-review"
}
- The requirement is text, with the repository in it. That fixes cross-repository references, and it means a matrix row or tracker item can fill the same field if Option C ever happens.
detailscarries the matrix row IDs under Option B, so they are not lost.qcthat-jsbecomes the first formal extension: an npm package that writes these records from Vitest and Playwright results and from the screenshot folders.
Decisions for you
How far should the extension go?
Recommendation: Option B, shaped so Option C stays possible. It delivers "lots of kinds of evidence" now. The only second requirement source you have today is matrix rows, and B carries those as detail, so a full requirement framework would be built for one user.
How do extensions plug in?
Recommendation: the file, with a generic underneath it. The point of the work is reaching beyond R, so the extension point should not require R.
What counts as evidence for an issue?
Recommendation: tests count by default, every other kind is shown, and a project can require more on a given issue. Otherwise one screenshot could turn an issue green, as my test showed.
Next steps
- You make the three decisions above.
- I draft the proposal for the qcthat maintainers, and you post it, since qcthat sits outside your own account. You are its listed maintainer, but most recent work is by others, so I would treat this as a proposal to them. small
- The safety.viz cleanup goes ahead whichever option you pick: the parser fix, the link cleanup and the check for new tests. medium
- safety.viz writes evidence records for tests and screenshots, read by a short R script. This tests the format on real data before asking anyone to change qcthat. medium
- If the maintainers agree, the qcthat changes land in order: the generic, the file reader, the new columns, the coverage rule, then report grouping. medium
- The shared code moves to
qcthat-js, and bio.viz becomes its second user. medium - Readers follow for change reviews and acceptance, and
ExpectUserAccepts()moves onto the same records. medium - Option C gets a fresh look only if a real second source of requirements appears.
How far to trust this
- Only the plumbing was tested: three made-up rows on unmodified qcthat 1.1.3, against the real issues and tests. The file reader, report grouping and coverage rule do not exist yet.
- The sizes are my judgement. The file counts come from a text search of qcthat's
devbranch, so they show how widely a concept spreads and not how many lines would change. - The record format is a first draft, and the maintainers may well change it.
- I looked at ReqIF too. It moves requirements between tools and does not replace a matrix, so it does not help here. The sources for it were mostly vendor pages.