open.gismo · design session · 2026-08-27
Where "load your own data" is already solved by gsm.mapping, where it is genuinely missing, and the two things that turned out to be true only because the packages were run rather than read.
@jwildfire asked for one design session on loading your own data and two agents took it up the same evening. The companion artifact, Loading your own data into open.gismo, carries the competitor landscape, four screens and seven decisions, and it found a failure mode this page did not: identifiers that are present and correctly typed but do not actually join, producing an empty mapped domain in silence.
This page is the narrower half and does not repeat that work. It carries what was established by execution, including one correction to the companion's account of how a user fixes a column name today. The two were written independently and neither has been merged into the other; that is a call for @jwildfire or the Navigator, not for either session.
Read out of the installed packages on 2026-08-27, and exercised rather than assumed.
gsm.mapping::ApplySpec() renames and types a source frame by compiling the
spec into a DuckDB SELECT, with a typed CREATE TABLE behind it.type and source_col.
That is the entire contract, and it is enough.Ingest() walks the domains; CombineSpecs() unions specs across
workflows and warns on type conflicts.gsm.mapping/inst/workflow/1_mappings/.og_init() snapshots 44 workflow files into the project, so the mapping specs
are the user's own editable files rather than package internals.config/data-config.yaml already routes a domain to a file path.A subject file delivered with USUBJID, SITEID and
STUDYID instead of subjid, invid and studyid
maps cleanly once the spec says so: 14 of 14 columns produced, 1,000 rows,
subjid and invid populated. Any design that adds a second mapping layer
beside this is solving a problem that does not exist.
This is the correction. It changes what "fixing it means opening the workflow files
and adding source_col: lines" is worth.
og_validate() collects the columns each mapping spec declares and checks the user's
CSV for them. It reads the target names and the type key. It never reads
source_col — the one key whose entire purpose is to say "my column is called
something else".
The consequence is not cosmetic. A user who does exactly the right thing sees no change at all:
Validation prints "Fix these before og_run()" for a project that is ready to run.
Its status is error, which is the strongest thing it can say, on data the pipeline
maps perfectly. For a delivery whose columns are named anything other than the gsm internal
names, this screen can never reach green, no matter what the user does.
.og_input_specs() in
open.gismo/R/og_validate.R keeps names(cols) — the target names
— and compares them with setdiff(required, info$cols). There is no reference
to source_col anywhere in the file.source_col lines across 12 of the 13 mapping files, then
re-running. The report came back byte-identical.gsm.mapping::Ingest() returned all 14 columns correctly populated.The fix is small and it is not a mapping surface: teach .og_input_specs() to read
source_col and check for that name instead. It is worth doing on its own, before any
screen is designed, because every mockup on the companion page assumes a readiness check that
tells the truth.
Of the 90 column declarations across the 12 input domains, 23 are named by a metric. The other 67 feed reporting, labels and site metadata.
Each metric workflow names the columns it reads. Tracing those back through the mapping workflows to the raw domains gives the exact list a user must supply to get the full KRI set, and it is much shorter than the spec suggests. It also prices every individual dropdown on a mapping screen:
| Raw column | Metrics that need it | What it is |
|---|---|---|
| Raw_SUBJ$subjid | 23 | the subject key — every metric |
| Raw_SUBJ$invid | 12 | the site key — every site-level KRI |
| Raw_SUBJ$country | 11 | every country-level metric |
| Raw_SUBJ$timeonstudy | 9 | the exposure denominator |
| Raw_AE$subjid | 4 | joins adverse events to subjects |
| Raw_PD$deemedimportant | 4 | splits important from non-important deviations |
A mapping surface that opens on 90 rows reads as a data-management project. The same surface filtered to the 23 that carry every metric reads as an afternoon, with the rest available behind a toggle for the reporting layer. It also lets the screen answer the only question the user actually has, which is not "which columns are missing" but "what do I get if I stop here".
kri* analysis that
ran. A partial mapping therefore changes its denominator rather than failing, which is the
already-known silent-divisor problem arriving through the data-loading door.Raw_ENROLL names the subject key subjectid while every other domain
uses subjid. Any auto-matcher has to tolerate the ecosystem's own inconsistencies,
not just the user's.| Step in "load my own data" | Status | Where it lives |
|---|---|---|
| Rename a source column to a standard one | ✔solved | source_col |
| Coerce and check types | ✔solved | type, RunQuery schema |
| Know which columns are needed | ✔solved | metric workflow specs |
| Route a domain to a file | ✔solved | data-config.yaml |
| Report readiness before a run | ⚠partial | og_validate(), blind to source_col |
| Show the user what their file actually contains | ✖missing | — |
| Propose a match for an unrecognised column | ✖missing | no standards table exists |
| Rank a gap by what it costs | ✖missing | derivable, never derived |
| Let a person settle a mapping without a text editor | ✖missing | — |
| Write the decision back consistently | ✖missing | one domain spans several files |
| Read anything but CSV | ✖missing | every loader calls read.csv |
| Add a domain or variable | ✖missing | file an issue upstream and wait |
The missing rows are one surface, not eleven features, and the whole of its output is
source_col values plus file paths. That is what makes it buildable without touching
the engine, and what keeps it from becoming a second mapping layer.
Everything above is reachable except proposing a match, and that needs a table nobody in the gsm ecosystem has.
safetyCharts::meta_* carries a
standard_adam and a standard_sdtm column per target field, so
detectStandard() can score a data frame against every known standard, label the
best one, and pre-fill the mapping screen with it.type and source_col and nothing that says the ADaM
name for subjid is USUBJID.The mockup runs a seeded version of exactly this table over a hand-authored CRO delivery, so the suggestion quality is visible rather than promised. It scores the delivery as Partial ADaM, 19 of 42 standard names present, and gets 27 of 81 columns right without help.
Numbered from D-MAP so they do not collide with the companion artifact's D1–D7.
Teaching og_validate() to read source_col is small, has no
interface, and every screen depends on it. Recommendation: yes, and independently of any
design decision below.
A sidecar in open.gismo ships now; a new key in the gsm.mapping specs is the better long-run home but needs an upstream conversation. Recommendation: sidecar, designed so it can move.
Open the screen on the 23 metric-critical columns with the other 67 behind a toggle, or show all 90. Recommendation: required-only, because it changes the perceived size of the job.
The surface writes the mapping files itself, or it prints the lines and the user pastes them. Recommendation: the surface writes them, because one domain spans several files and hand edits will drift.
SAS transport is how study data actually arrives, and both open-source competitors read it. Convert once at import, or read natively. Recommendation: convert at import, so the project folder stays diffable.
Two sessions produced two pages for one ask. They are complementary rather than contradictory, apart from the correction on this page. Recommendation: fold this page's two findings into the companion and retire this one.