open.gismo · data loading · 2026-08-27

How does anyone get their own study in?

open.gismo v0.2.0 runs the whole pipeline against a plain project folder. The forkable demo proves it. What neither answers is the step before all of it: a person has an EDC export, and between that export and a running study sits work this product currently makes invisible.

Requested by @jwildfire, 2026-08-26: “Maybe a ultradesign session for open.gismo? re-visit the research comparing other sites and do some mockups on how users will load their own data?” · This is a design session — nothing was built, filed or changed in any repository.

The short version

The engine for this already exists and is not the problem. gsm.mapping maps raw data into the standard domains, and its spec already accepts a source_col: key that renames a source column onto a target one — an existing, exercised feature that no user of open.gismo would ever discover. What is missing is a surface, and three specific things nobody has named.

Reading it closely also changes the shape of the problem. The gsm raw layer is not CDISC: findings and events use lower-case SDTM-adjacent names, while subject, site and enrolment use operational-EDC names such as invid, agerep and subject_nsv. A sponsor with immaculate SDTM still has to map. The mapping surface is not a concession to messy data; it is the front door for everybody.

raw domains a study must satisfy

126

required columns across them

keys the mapping spec offers per column — type and source_col

checks that the identifiers actually join

Counted on 2026-08-27 from demo-301/workflows/1_mappings/ — the fourteen Raw_* domains those specs read and the distinct columns they require. The two spec keys are what gsm.mapping::ApplySpec() reads; the third number is the finding below.

Three problems, in the order they will bite

Each was established by reading the code and the shipped demo study, not inferred from the shape of the product.

✗ One — a study can map perfectly and produce nothing

The mapping workflows join every findings and events domain to the subject spine with an inner join: LB.yaml and AE.yaml both end in dplyr::inner_join on subjid. That join is deliberate and documented as the data-cleaning step. But nothing anywhere checks that the two sides share any identifiers.

A lab vendor that writes "0301-S036" where the demographics extract writes "S036" passes today’s validation completely — the column is present, the type is right, no values are missing — and then produces an empty mapped domain. In demo-301 that would silently remove six of the nine safety charts, the Hy’s Law candidate metric and the Grade 3+ Lab Abnormality Rate. og_validate() reads one file at a time and never compares keys across files, so there is no moment at which anyone is told.

◐ Two — the spec renames columns and cannot touch what is inside them

A spec column accepts exactly two keys, type and source_col, and ApplySpec() turns them into a SELECT ... AS .... That covers renaming and nothing else. Every genuinely hard case in a real export is a value problem rather than a name problem: an identifier with a site prefix, a baseline flag spelled BASELINE instead of Y, a lab result in different units, a toxicity grade written as Grade 3 rather than 3.

Those can be expressed today only by hand-writing a steps: entry — arbitrary R function calls in YAML — which is a far higher bar than the mapping it accompanies. This is the one place where a surface alone is not enough and the spec needs a decision.

◐ Three — going without a domain is a warning nobody reads

When an input file is absent, og_run() raises an R warning and continues (R/og_run.R:397). The pipeline runs, the site builds, and the metrics that needed that domain are quietly not there. Most studies will legitimately lack several of the fourteen: Raw_DATACHG, Raw_DATAENT and Raw_QUERY come from an EDC audit trail that many sponsors will never extract.

Going without should be a choice a user makes with the cost in front of them, recorded in the study config, and printed on the site wherever the missing metric would have appeared — not a warning in a console that a scheduled run throws away.

What already exists, and what has to be built

Three of the four seams are already in the engine. Nothing here proposes a second mapping layer beside gsm.mapping — the question is what surface a person touches.

The seam	Where it lives today	Status	What is missing
Which filedomain → path	config/data-config.yaml	✓Exists	A surface that writes it, and formats other than CSV at the door
Which columntarget ← source	source_col: in each mapping spec	✓Exists, exercisedgsm.mapping’s own workflows use it for Rave names	Everything user-facing. No detection, no proposal, no confirmation, no writer
Which typecoercion	type: in each mapping spec	✓Existsog_validate checks numeric coercion on a 5,000-row sample	Reported to the user before a run rather than after
Which valuerecode, reformat, derive	Nowhere declarative — only a hand-written steps: entry	○No seam	A spec key, and an upstream conversation with gsm.mapping
Do the keys joincross-file	Nowhere. Every check reads one file in isolation	○No check	An overlap measurement, and a run that refuses rather than warns
Going withoutdeclining a domain	An R warning, then the run continues	○Not a concept	A recorded decision the site can print

The design

Four screens, clickable, built against real column names and real downstream dependencies from demo-301. One of them is live: change a mapping and the readiness line, the preview and the config diff all move.

Screen 1

Point at your data

Drop a folder of CSV, SAS transport, SAS datasets or Excel. Everything is converted once, at import, into plain CSV under input/, so the project folder stays the diffable, forkable thing the product rests on. Each file is placed against a domain with a stated reason.

Screen 2

Fit report

The successor to og_validate(), with the two columns it does not have: whether the identifiers join, and what each absent domain costs in named metrics and named charts. Six ready, three needing a decision, five going without.

Screen 3

Map a domain

A grid, not a wizard. Required column, its type and its downstream consumer on the left; your column on the right with a proposed match and the reason for it. Three rows carry the argument: an identifier that maps cleanly and fails the join, a flag that needs its values recoded, and a column nobody supplied.

Screen 4

What it writes

A diff against the study’s own configuration, to be committed and reviewed. Renames use the existing source_col: key and are real today. Value transforms are drawn in a different colour because they are a proposal, and that is the decision below.

Open the four screens →

What the competitor research says, on this question specifically

Thirteen platforms, re-read against a narrower question than the gap analysis asked: not “is there a mapping surface” but “what does a person actually do”. The full survey is Part 1.

A mapping grid is not the open-source norm — it is our own ancestor’s idea. Of five open-source tools surveyed, exactly one gives an end user a column-mapping surface: safetyGraphics, with a Mapping tab, automatic fill from a detected standard, and a match reported in words (Partial SDTM (4/9 cols/fields matched)). The other four are conformance-or-nothing, or they are R code you write yourself.
Everywhere the mapping is good, it is captive. elluminate’s drag-and-drop Mapper, Medidata’s low-code transforms, Veeva’s ingestion engine and Empirica’s data configurations are all real, and all live inside the platform behind a role — administrator, data manager, vendor analyst. safetyGraphics is the one exception that hands the mapping back as exportable code. Nobody surveyed makes the mapping a reviewed file in the study’s own repository, which is precisely the move this portfolio already makes with workflows, thresholds and package pins.
The serious platforms all solve values, not just names. JMP ships a Recoder that merges, splits and renames values, keeps the original column and writes a formula recording the transformation. Medidata ships a unit-conversion library. Veeva checks types, ranges and formats on arrival. A design that maps names and stops has done the half that was never hard.

A correction to the anchor report

The gap analysis scored row D1 as seen on five platforms: safetyGraphics, JReview, Spotfire clinical, tidyCDISC and elluminate. On this closer reading the count survives and the membership does not. tidyCDISC has no mapping surface of any kind — a missing required variable rejects the upload outright — and should not have been credited. Spotfire clinical could not be established either way from public documentation. JReview’s mapping is real but lives in an administrator-configured data dictionary rather than anywhere a reviewer touches. And three platforms the row did not credit do have one: Medidata, Veeva and Empirica. Five of thirteen cells in that row were wrong in one direction or the other, which is roughly what a survey scored at sixty-three-row grain should be expected to get wrong, and is why this one asks a narrower question.

Decisions

Seven, each with a recommendation rather than a menu. D4 is the only one that reaches outside this portfolio, and it is the one everything else waits on.

Where the surface lives

A separate setup wizard, or part of the study site that already exists?

RecommendationA Setup › Data area inside og_app(), beside the existing Config views, plus a scriptable twin so the whole thing can run headless. The app is already the thing people open, it already reads the project folder, and a second binary would be a second thing to keep alive.

Considered: a standalone initialiser in the shape of safetyGraphicsInit(), which is our ancestor’s answer and works, but splits the product in two.

Does the surface write back to the repository?

A diff the user commits, or state the app keeps for itself?

RecommendationAlways a diff, against config/ and workflows/, in the study’s own repository. This is the differentiating decision on the whole page: it is the one thing no surveyed platform does, it costs nothing extra to build, and it means a mapping can be reviewed, blamed, reverted and reproduced a year later like every other decision in the study.

Considered: a sidecar mapping file the app owns. Simpler, and it quietly turns the study repository into something that is no longer the whole record.

What comes in the front door

CSV only, as today, or the formats people actually have?

RecommendationAccept CSV, .sas7bdat, .xpt and Excel; convert once at import into plain CSV under input/; never touch the originals. This keeps the promise that a project is a folder of readable text files while accepting what an export actually is. safetyGraphics takes anything rio reads; tidyCDISC takes .sas7bdat only; DaVinci prefers .rds for speed.

Considered: leave conversion to the user. It is one more undocumented step at the exact moment someone decides whether this product is worth the afternoon.

The transform seam — the one that needs somebody else

Value recoding, identifier reformatting and derivation have nowhere declarative to live. Where should they go?

RecommendationPropose two new spec column keys to gsm.mapping — recode: for a value map, and transform: drawing from a closed, named set (strip a prefix, pad, trim, upper-case, scale) rather than arbitrary expressions. A closed set is auditable and a free-text expression is not, which matters more here than expressiveness. Until that lands, the surface writes a steps: entry and labels it as the workaround it is.

Considered: steps: forever. It works, and it means the generated mapping is a program rather than a declaration, which is the wrong artifact to ask a reviewer to approve. gsm.mapping is a Gilead-BioStats repository, so this is a draft handed over rather than an issue filed.

Going without a domain

Should declining be a recorded decision or stay a warning?

RecommendationA declined: block in study-config.yaml naming the domains and columns the study is going without, printed on the site wherever the affected metric would have appeared. Absence that was not declared should stop the run rather than warn it. A quietly missing KRI is indistinguishable from a KRI with nothing to report, and the second is a finding.

Considered: keep the warning. It is the current behaviour and it is how a study ships a hole nobody can explain six months later.

How much should it guess, and does define.xml count?

Auto-detection is either a great convenience or a great way to be confidently wrong.

RecommendationPropose, never apply. Match on name after case-folding, then on type, then on value shape, and show the reason for every proposal in the row it belongs to — safetyGraphics’ detectStandard() is the model, right down to reporting the match as a fraction in words. Read define.xml later, for labels and code lists, and never as the authority for what a column is.

Considered: no detection at all. Fourteen domains and 126 columns is a long afternoon of dropdowns if nothing is pre-filled.

Two files, one domain

Central labs and local labs. One Raw_LB.

RecommendationFollow clinDataReview: accept a list of files for a domain, row-bind them, and add a column recording which file each row came from. It is a small change to data-config.yaml and the loader, and the provenance column is what makes it defensible.

Considered: one file per domain, and the user concatenates. Which they will do in Excel, unversioned, and nobody will ever know which vendor a row came from.

What I would build first

In the order that removes the most risk per day of work. The first two are not the mockups at all.

The spine check. Add a key-overlap column to og_validate(): for every domain that joins to another, measure how many of its identifiers actually match. No new interface, no new concepts, and it closes the silent failure entirely. This is the smallest useful thing on the page.
Make the run refuse. A zero-overlap join or an undeclared missing domain should stop og_run(), not warn it. Depends on D5 for what “declared” means.
The fit report as a view. Screen 2, rendering what the previous two steps now know. Read-only, and already worth opening.
The mapping grid, renames only. Screen 3 and Screen 4 restricted to what source_col: can express today. No upstream dependency, and it covers the majority of columns in a typical export.
The transform seam. Screen 3’s highlighted rows, once D4 is settled and whatever it needs has landed.

Steps one and two together are the difference between a study that fails loudly and one that publishes a site full of empty charts. They are worth doing whether or not any of the rest is ever built.

Method, and what it cannot tell you

Our side is verified. Domain and column counts were extracted from demo-301/workflows/1_mappings/; downstream dependencies from the thirty metric workflows and twelve module workflows in the same repository; the inner joins from AE.yaml and LB.yaml; the spec keys from gsm.mapping::ApplySpec(); the warn-and-continue behaviour from open.gismo/R/og_run.R.
Nothing was run. The claim that a zero-overlap join yields an empty mapped domain follows from the inner join in the workflow, not from executing the pipeline against broken data. The consequence for what the site then renders is stated as a consequence, not as an observation.
Competitor research is documentation only. Public product documentation, package reference indexes, user guides and PhUSE papers. No demos, no trials, no vendor contact, no accounts. Seen on means documented, and vendor documentation flatters. Rows that could not be established say so rather than being filled in.
The mockup’s export is invented. Every target column, type, domain and downstream consumer in the mockups is read from demo-301. The incoming lab-vendor file, its column names and its sample values are constructed to make the argument, and are labelled as such on the page.
Nothing was filed, changed or merged. No issue, no branch, no commit in open.gismo, and no writes to any repository outside jwildfire.