open.gismo · data loading · direction: convention over configuration · 2026-08-27

Put the files where they go. The product tells you the rest.

One of four competing directions for how a person gets their own study into open.gismo. This one builds no interface at all. The mapping is a study document — text, in the study's own git history, reviewed as a diff — and the entire user surface is a generator that writes that document for you and a diagnostic that names the one line you have to change.

The commitment

A mapping screen produces a mapping. A mapping screen does not produce a record of a mapping — something a second statistician can review, a reviewer can diff against last month's, and CI can replay two years later during an inspection. In this domain the record is the deliverable, so the artifact should be the primary thing and any screen a secondary view of it.

So: no screen. Four text files the user may touch, two commands, and one rule — og_run() refuses to start while the diagnostic is red. Everything the product can work out, it writes into those files itself. Everything it cannot, it asks for as one named line at one named path, with the cost of not writing it printed beside the ask.

126
columns a full study must satisfy, across 14 raw domains
34
of those that any metric workflow actually names
73
settled with no human input on a measured CRO delivery
8
metrics that silently report zero when the guesser is confidently wrong

Measured on 2026-08-27 against demo-301/workflows/1_mappings/ (the fourteen Raw_* domains a study with both the RBQM and Safety lenses reads), demo-301/workflows/2_metrics/ (30 workflows), and a hand-authored ADaM/SDTM CRO delivery. Method and scripts on page 2. og_init()'s smaller default set is 12 domains and 90 columns.

What is actually broken today

open.gismo v0.2.0's entire load-your-own-data surface is one line of README — og_init("~/my-study", example = TRUE), “or drop your own Raw_*.csv files in ~/my-study/input/” — plus og_validate(), which reports and then stops. Three specific things, each read from source and confirmed by running it:

  1. The readiness check and the engine disagree about the same project. gsm.mapping::ApplySpec() reads exactly two keys per column, type and source_col, and source_col already emits "THEIR_NAME" AS "gsm_name" in the generated SQL. The string source_col does not appear anywhere in open.gismo's R/ or inst/. So og_validate() compares target names only: for any delivery not already carrying gsm's internal names, that check can never go green even when the pipeline would run perfectly. This was established by a parallel session the same night and is a prerequisite for everything below.
  2. Nothing gates the run. og_run() never calls og_validate(). The Shiny lane (og_app()) does; the headless lane does not. A missing domain is a warning() and the pipeline continues.
  3. Nothing checks that the identifiers join. Both LB.yaml and AE.yaml end in a dplyr::inner_join on subjid against Mapped_SUBJ. og_validate() reads one file at a time and never compares two.
The framing correction that matters

The gsm raw layer is not CDISC. Findings carry lower-case SDTM-adjacent names; subject, site and enrolment carry operational-EDC names (invid, agerep, subject_nsv, mincreated_dts). A sponsor with immaculate ADaM still has to map. This surface is the front door for everybody, not a concession to messy data — which is why hiding it inside an optional app would be the wrong instinct.

The design: four files, two commands

Everything a user touches is a file that git already tracks. Nothing is stored anywhere else, and there is no state that is not on disk.

exists today

input/

Put the delivery here under whatever names it arrived with. The naming convention Raw_{DOMAIN}.csv stays as the zero-config path, but it stops being the only one.

exists today · gains one block

config/data-config.yaml

Domain → file. Gains a decline: block, where a user records — with a reason, in prose — that a domain or a column will never arrive.

exists today · gains generated lines

workflows/1_mappings/*.yaml

Already the study's own editable copies of gsm.mapping's specs, already snapshotted into the project by og_init(). They gain source_col: lines, written by the generator and marked as guesses.

new

config/vocab/*.csv

The one thing that does not exist anywhere in the ecosystem: a value-level seam. A two- or three-column CSV per column name, applied before the mapping SQL runs. This is what makes a liver panel from three labs plottable.

The two commands

CommandWhat it isWhat it never does
og scan A generator. Reads every file in input/ — headers, types, cardinality, the distinct values of any column a module pins to literals — and writes candidate source_col: lines, candidate vocabulary tables, and the participant join step into the project's own files. Idempotent. Never overwrites a line a human has touched, never reformats a file, never reads a full data file when the header and a sample will do.
og doctor The whole interface. Replaces og_validate(). Every finding names the file, the line, the literal text to write, and what it costs in metrics and charts to not write it. Ordered by cost, not by domain. Never emits a finding it cannot price and cannot give an edit for. Never returns a report — it returns work.

The loop, in full

● ● ● the entire product
$ og init ~/hep-201 && cp ~/deliveries/2026-08-26/* ~/hep-201/input/
$ og scan ~/hep-201            # it writes the mapping
$ git -C ~/hep-201 diff         # you read what it guessed
$ $EDITOR ~/hep-201/workflows/1_mappings/SUBJ.yaml
$ og doctor ~/hep-201 && og run ~/hep-201

Five lines, three of which are things an R user types every day anyway. The design work is entirely in what og scan writes and what og doctor says — which is why page 1 is a terminal, not a screen. For a direction with no graphical interface, mocking up the output is mocking up the product.

What the product does automatically, and what it asks

The measure of this direction is the ratio between these two lists. Convention over configuration does not mean the user configures nothing; it means the number of decisions a user must express equals the number of places their data genuinely disagrees with a name the product already guessed, and not one more.

automatic · seven things
  • Domain routing. File → Raw_* domain by header fingerprint, not by filename. adlb.sas7bdat is recognised as Raw_LB because it has a value, a unit and a reference range per row.
  • Column renaming wherever an exact match or the alias table resolves it — 73 of 126 on the measured delivery.
  • Type coercion, which the spec's type: key already does.
  • Vocabulary candidates. For every column a module YAML pins to literal strings, the generator writes a table of the study's actual distinct values against the literals the module demands.
  • The participant join. When a domain lacks invid but Raw_SUBJ has it, the generator writes the same inner join demo-301 hand-wrote, into the domain's own YAML, as visible steps.
  • Key-overlap measurement between every pair the mapping steps join, on a sample, before the run.
  • Pricing. Every unresolved thing is reported with the metrics and charts it costs, resolved through the metric workflows' own specs.
asked · four things, each one line of text
  • A column the guesser could not settle → one source_col: line, or a decline: entry with a reason in prose.
  • A value vocabulary → fill the to column of a generated CSV. Blank means pass through unchanged, so the file is complete and inert until edited, and awk -F, '$2==""' lists your open items.
  • A unit reconciliation → a factor column in the same CSV. µkat/L,U/L,60. The factor is applied to the paired numeric columns the convention already knows about.
  • An identifier affix → one line in a .rules.csv: ^0301-,. The diagnostic prints the key overlap before and after, so the fix is verified rather than asserted.

The moment of highest friction — and it is not the renaming

Renaming is where a guesser is strongest and where every competing direction will also do well. The friction that decides whether this product is trusted is somewhere else, and it was measured rather than imagined.

✖ Measured, not hypothesised

Against a conformant ADaM delivery, the matcher's single most confident hit is wrong. Raw_SUBJ.subjid matches SUBJID on an exact case-insensitive name comparison — the highest-confidence tier there is. But ADaM's SUBJID is the site-level subject number, while adae and adlb carry only USUBJID, so those two domains resolve their subjid to a different identifier entirely.

Both columns are present and correctly typed. Today's og_validate() returns all-green. og_run() completes without an error. The two inner joins produce zero rows, 8 of the 29 specified metric workflows report zero at every sitecou0001, cou0002, cou0005, kri0001, kri0002, kri0005, saf0001, saf0003 — and srs0001, which has no spec at all, quietly reweights its site risk score across the 21 that did run.

The study site renders. Enrolment KRIs look healthy. Adverse-event and lab rates read zero everywhere. A bad mapping does not crash; it flatters.

This direction's answer is the cheapest thing on the build list and the most valuable: the key-overlap check runs before the mapping phase, on a sample of identifiers per domain, and og doctor refuses on it with the two identifier shapes printed side by side and the affix it detected. About sixty lines of code. Page 1 walks it.

The second-highest friction: the column that cannot exist

Raw_LB.toxgrg_nsv is a CTCAE toxicity grade. It has no source variable in SDTM and none in ADaM unless somebody derived one, and today the only thing a user is told is “missing column: toxgrg_nsv”. They cannot tell whether that is fatal, cosmetic, or fixable.

Convention over configuration's answer is that absence is a decision the user records, and the product prices it. The diagnostic prints what goes dark, and offers the line:

config/data-config.yaml
decline:
  Raw_LB.toxgrg_nsv: "not collected — our ADaM has ATOXGR only on AE, not LB"
  Raw_STUDY: "no CTMS extract; study metadata typed into study-config.yaml"

The run then proceeds with kri0005 and cou0005 declined rather than absent, and the decline — with the sentence the user wrote — is stamped into the snapshot's provenance record, which open.gismo already renders as the provenance chip. A screen-based direction gets a checkbox here. This one gets a sentence in version control that a reviewer can read two years later. That is the differentiator, and it is not available to a UI for free.

And it closes a known bug by construction

srs0001 carries no spec, so it weights whichever kri* workflows happened to run — meaning a partial mapping changes the site risk score's denominator instead of failing. Making a decline explicit lets the run record the denominator it actually used, which is the only honest way to publish that score from an incomplete study.

What it costs

PieceWhereRough sizeNotes
Fix og_validate() to resolve source_colopen.gismo~20 linesPrerequisite. Without it the readiness check contradicts the engine.
og_scan() — reader, profiler, matcheropen.gismo~250 linesHeaders and a sample only; never reads a whole file.
og_scan() — the YAML line writeropen.gismo~150 linesThe hard part. See the risk below.
og_doctor()open.gismo~300 linesRewrite of og_validate(): adds source resolution, key overlap, vocabulary coverage, decline pricing, cost ordering.
Vocabulary + unit applicationopen.gismo~60 linesApplied in .og_load_input() after read.csv, before anything else sees the frame.
Decline handling and provenance stampopen.gismo~80 linesPlus og_run() refusing while doctor is red.
inst/aliases.csvopen.gismo~170 rows52 of them lift straight out of safetyCharts. The rest are hand-written. This is a data asset with a maintenance cost, not a one-off.

What it leans on, unchanged

✔ The structural advantage

This ships with no change to any Gilead-BioStats repository. Every seam it needs already exists or lives inside open.gismo. Of the four kinds of mapping, the only one that would normally require an upstream recode: proposal — value-level — is handled here as a pre-read transform in open.gismo's own loader, so it costs a conversation with nobody. Upstreaming it later remains open and becomes easier once a real study has exercised the file format.

What could go wrong

Where this direction is weakest

Five, ordered by how likely each is to be the reason this direction should lose.

W1 — the strategic one

It concedes the person who usually owns the mapping

Across the commercial platforms surveyed, the mapping is not done by the person reading the charts. It is done by a vendor services team, a CRO data manager, or a study programmer. This direction makes open.gismo meaningfully better for a statistician who lives in a terminal and worse than useless for a data manager who does not open one. If the target user for “bring your own study” is that data manager, this direction is simply the wrong answer, and no amount of good terminal output rescues it. That is a question about who the product is for, and it should be settled before the code is written.

W2 — the design one

A right guess and a wrong guess look identical

A screen can render a low-confidence match in amber and force a glance. A line of YAML cannot. The mitigation here — every generated line carries a trailing comment naming how it was matched and what the runner-up was — is genuinely weaker, because comments are the first thing a reader's eye skips. The measured case makes this concrete and uncomfortable: the wrong guess arrived through the highest-confidence tier, so an honest confidence annotation would have marked it green.

W3

There is no look-at-my-data moment

No distinct-value browser, no cross-tab, no “show me twenty rows of PARAM.” A statistician will do that in R in another pane, so nothing is blocked — but the product learns nothing from it, and the first contact a user has with a brand-new delivery is a wall of problems rather than a look at their study.

W4

The edit volume is only tolerable if the ranking is right

The measured run leaves 53 columns unsettled across 11 files. Only 12 of those are named by any metric — that ratio is the entire argument that this is a small job, and it depends completely on the diagnostic ordering findings by what they cost. Get the cost model wrong and the user edits 53 lines in 11 files and never comes back.

W5

The proof loop is slow

og doctor reads headers and a sample, so it is effectively instant. But the thing a user actually wants to see is the chart, and that costs a full og run — minutes on demo-301-sized inputs. There is no live preview and this direction does not offer one.

Where it is strongest, in one paragraph

The clinical world's own answer to “what is in this dataset” is define.xml — a machine-readable document under version control, not a screen. Of the platforms surveyed on page 2, not one produces a mapping that a second person can review as a diff, that CI can replay, or that arrives in an inspection folder as evidence. open.gismo already snapshots the study's workflows into the study's own folder; this direction is the one that finishes that thought. It is also the only direction that ships without asking another organisation for anything.

Decisions this direction needs from @jwildfire

C1

Who is the person bringing their own study?

A statistician or study programmer who works in R and git, or a data manager who works in a browser. Everything else follows from this, and it is the one question this direction cannot answer for itself — see W1.

Recommendation Commit to the statistician for v1 and say so in the documentation, because that is who forks demo-301 today. Revisit only when a second kind of user has actually asked.
C2

Does the generator write into the study's files, or into a file of its own?

Writing source_col: lines into workflows/1_mappings/*.yaml puts the mapping where the engine reads it and where a reviewer expects it, but means a tool edits files a human also edits. A separate config/mapping.yaml that is merged at load time keeps them apart, at the cost of a second place to look and a merge rule.

Recommendation Write into the study's own files. One place, no merge semantics, and the diff is the review. Accept the line-writer verification cost.
C3

Should og_run() refuse while the diagnostic is red?

Today it runs regardless and warns. Refusing is what makes the diagnostic the interface rather than an optional report — but it will block somebody mid-experiment.

Recommendation Refuse, with og_run(force = TRUE) available and every forced run stamped as forced in its own provenance record. Never silently.
C4

What formats does the door accept?

Every reader in open.gismo today is read.csv — no .xpt, no .sas7bdat, no parquet, no database. A CRO delivery is transport files. Adding haven is a dependency and a documentation surface; refusing it means the user's first step is a conversion script the product does not help with.

Recommendation Accept .xpt and .sas7bdat via haven, because a transport file is what actually arrives. Leave parquet and databases out of v1 and say so.
C5

Where does the value-level seam live — open.gismo, or upstream in gsm.mapping?

A recode: key in the spec would be the right home architecturally and would serve every gsm consumer. It also means a proposal to a repository this programme cannot write to, and a wait.

Recommendation Build it in open.gismo's loader now as config/vocab/. Draft the upstream recode: proposal after one real study has used the file format, and hand it to @jwildfire to post rather than filing it.
C6

Does the alias table get its own home and owner?

52 rows lift straight out of safetyCharts (MIT, and @jwildfire is its author). The remainder is hand-written CDISC and EDC-vendor knowledge that will drift, and the measurement already found a miss caused by a single-valued default.

Recommendation Ship it as a versioned data file inside open.gismo with a regression test per standard, and treat the safetyCharts import as a seed rather than a dependency.