R/Pharma 2026 · Keynote

What can we actually do with AI agents right now?

Jeremy Wildfire · 21 October 2026 · jwildfire.github.io/keynote

Orange, an orange tabby cat, wearing a blue and gold cap

First, a key collaborator

Introducing Orange

2008–2022

He was a very good cat.

A quick disclaimer

Jeremy works for the Biostat Innovation Group at Gilead, but this is all just his opinions.

The question

What can we actually do with AI agents right now?
SafetyGraphics2016–2021 OpenRBQM2021–2025 Opus 4.5 GPT-5.1 Nov 2025 Experiments on the AI Frontiersummer 2026 AI in the Clinical Trial PipelineQ4 2025 spare time day job What we learnednow

How I tried to answer it

By working in public

Every post, and the linksjwildfire.github.io/keynote
1 10 JunThe question 2 12 JunIntroducing Obot 3 16 JunsafetyGraphics, again 4 2 JulsafetyGraphics ❤ gsm 5 12 JulIntroducing safety.viz 6 13 JulA billion tokens 7 17 JulPapers → prototypes 8 20 AugObot, interviewed again 9 6 SepNot a capabilitiesproblem
Orange asleep with his head on a laptop while Jeremy works the trackpad around him

Orange, at work

Open Source SafetyGraphics

Such a helpful coworker

2012

Why can't I do this for work?

The New York Times interactive 512 Paths to the White House, 2012, in use: states are picked for one candidate and the tree of paths and the count of ways to win change
Interactive graphics in the New York Times
A wall of thumbnails of interactive graphics built at Rho: bar charts, timelines, scatter plots, tables, line charts

We can!

👍

Page 1 of 18

A standard adverse events table

A standard adverse events table: preferred terms down the side, counts, percentages and p-values by group across

The Rho Graphics Group

A handful of webpages beats a massive table

The Adverse Event Explorer, an interactive table of adverse events by treatment arm
The Adverse Event Explorer · open source, in the browser

ASA–DIA Safety Working Group

Open Source Safety. Interactive Reports. Shiny Modules.

The SafetyGraphics hex sticker: a heart with a pulse line through it
AE Explorer
AE Explorer
Safety Outlier Explorer
Outlier Explorer
AE Timelines
AE Timelines
Paneled Outlier Explorer
Paneled Outlier Explorer
Safety Histogram
Histogram
Safety Results Over Time
Results Over Time
Safety Shift Plot
Shift Plot
Web Codebook
Web Codebook

The Safety Explorer Suite · Therapeutic Innovation & Regulatory Science, 2018

At R/Pharma in 2020: SafetyGraphics v2.0, the talk

The best thing we built

The hepatic explorer, and a clinical workflow to go with it

The eDISH hepatic safety explorer: a scatter plot of liver test results with participant drill-down
The explorer · eDISH, the standard plot for drug-induced liver injury
Step 1 of the clinical workflow: a decision diagram that starts from potential Hy's Law cases and branches on population confounders
The workflow · step 1 of 9, as it reads in safety.viz today

Introducing Inspector Orange

The first question, in every talk

What evidence do you have?

Written requirements and a manual test script for every chart.
Automated tests on the R package, run on every pull request.

JW

But is it validated?

Not really. It’s exploratory.

JW
Orange, seen from behind, watching a colour-coded spreadsheet on a laptop

Orange, reviewing

OpenRBQM

Open-source Risk Based Quality Management.
Quality by design.

OpenRBQM PHUSE Working Group

Open Source Monitoring. Interactive Reports. Flexible Data Pipelines.

An OpenRBQM site report on example data · quality built in from the start

OpenRBQM, a PHUSE working group

How the software gets made

A standard development lifecycle

a personautomated

Design

Gather requirements
Write the work item
Draft the technical plan

Build

Write the tests
Review the tests
Implement
Draft the change summary
Check code quality
Approve

Publish

Automated gates
Quality documentation
Draft the documentation updates
Finalize and release
Publish

OpenRBQM · openrbqm.github.io

An ecosystem of packages

core
gsm.coreThe engine

Builds and scores every metric
kri
gsm.kriMetrics

Key risk indicators, with their charts and reports
app
gsm.appApps

Shiny applications for OpenRBQM assessments
qtl
gsm.qtlLimits

Quality tolerance limits, and their reports
simaerep
gsm.simaerepFrom IMPALA

Finds sites that under-report adverse events
workr
workrWorkflows

Runs every step from a YAML workflow
viz
gsm.vizGraphics

Interactive charts in JavaScript, for the reports
qcthatQuality

Quality control for clinical trial packages

Quality · qcthat

gilead-public.github.io/qcthat

Continuous Qualification is Embedded in the Lifecycle

A qcthat report posted on a pull request: milestone v1.2.0, three issues, nine tests, every test passed, every issue has at least one test

The issue number goes in the test nametest_that("Flags sites above the upper limit (#19)", { ... })

Quality · qcthat

Quality scales across full package ecosystem

76

qualified releases across 9 packages

Every release, its qualification, and the date of each, in one registry

PackageVersionReleasedQualified
gsm.appv2.5.32025-08-202025-08-22
gsm.appv2.6.12025-09-242025-10-01
gsm.corev1.0.02025-03-142025-04-21
gsm.corev1.1.02025-04-112025-04-21
gsm.kriv1.0.02025-03-142025-03-14
gsm.kriv1.1.02025-04-142025-04-21
gsm.mappingv1.0.02025-03-142025-03-14
gsm.mappingv1.0.12025-04-112025-04-21
gsm.reportingv1.0.02025-03-142025-03-14
gsm.reportingv1.0.12025-04-152025-04-21

Sample rows from qualified-releases.csv

Now that's evidence.

In use on 20+ trials

Orange lying across a laptop with one paw on the trackpad, seen from above

Orange, in the pipeline

AI in the Clinical Trial Pipeline

Carefully adding agents to our package development process.

AI in the pipeline

What can we actually do with AI agentsat workright now?

We use AI heavily, but it's not autonomous. Humans own the process and all deliverables.
— developer diary #1, 10 June 2026

What are agents good at now?

Writing code!

as of Q4 2025

👍

Levels of AI usage

How much do you hand over?

5
The dark software factoryNobody reads the code at all
4
The engineering teamYou write the specs and plans; agents do the work
3
The developerIt writes most of the code; you review all of it← the day job
2
The junior developerYou pair with it and still read every line← the day job
1
The coding internIt writes the boilerplate; you check everything
0
Spicy autocompleteIt finishes your line

Levels: Dan Shapiro, The Five Levels · via Simon Willison's write-up · January 2026

  1. Levels of AI usage
  2. 5The dark software factory
  3. 4The engineering team
  4. 3The developer
  5. 2The junior developer
  6. 1The coding intern
  7. 0Spicy autocomplete

What AI in the pipeline looks like

Same lifecycle; Directly Responsible Humans

Design

Gather requirements
Write the work item
Draft the technical plan
Review the plan, then prompt

Build

Write the tests
Review the tests
Write code
Draft the change summary
Check code quality
Approve

Publish

Automated gates
Quality documentation
Draft the documentation updates
Finalize and release
Publish

a persona person and the agentthe agent, with human reviewautomated, as before

What are agents good at now?

The standard SDLC

as of Q2 2026

👍

What evidence do you have?

The same tests,
and a human owner

Where we put the effort

Faster code; planning and operations have to catch up

Upstreamour scaffold

Planning

Requirements
Designs
Phase gates

The standard development lifecyclevendor coding agents, plus our shared context and conventions

Design

Work item
Technical plan

Build

Tests
Code
Review

Publish

Quality gates
Documentation
Release

Downstreamour scaffold

Operations

Study set-up
Snapshot schedules
Monitoring plan checks

Build roadmap

Make R packages

Run packages on studies

  1. Levels of AI usage
  2. 5The dark software factory
  3. 4The engineering team
  4. 3The developer
  5. 2The junior developer
  6. 1The coding intern
  7. 0Spicy autocomplete

Where that leaves us

We're having great success at level 2.5!

Orange, seen from behind on Jeremy's lap, with one paw stretched onto the keyboard of a laptop showing GitHub

Orange, on GitHub

Experiments on the AI Frontier

A summer of public experiments

Enter Hacker Orange

What can we actually do withfrontierAI agents right now?

Only one way to find out.

Oh jeez. I'll check in later.

The goal

Modernize SafetyGraphics, using the OpenRBQM framework

The original Safety Histogram, as shown in its README2020
Safety Histogram
The original Outlier Explorer, as shown in its README2021
Outlier Explorer
The original Results Over Time, as shown in its README2019
Results Over Time
The original Shift Plot, as shown in its README2019
Shift Plot
The original Delta-Delta, as shown in its README2019
Delta-Delta
The original Paneled Outlier Explorer, as shown in its README2019
Paneled Outlier Explorer
The original AE Explorer, as shown in its README2020
AE Explorer
The original AE Timelines, as shown in its README2019
AE Timelines
The original Web Codebook, as shown in its README2021
Web Codebook
The original Hepatic Explorer, as shown in its README2022
Hepatic Explorer

10 original charts · last commit, top right · 25,400 lines of JavaScript · all on D3 v3

Open Source ftw

What's next?

A summer down the AI rabbit hole, with obot

Orange, drawn as a hex stickerOrangethe original
obot v1 as a robot cat holding a clipboardv1a personal assistant
obot v2 as a robot cat with a paper planev2OpenClaw
obot v3 as a robot cat with a terminal promptv3Fable 5
obot v4 as a robot cat in a crown and tiev4the organisation
obot v5 as a robot cat with a steering wheelv5goals, on Opus 5 → 5.5
  1. Levels of AI usage
  2. 5The dark software factory
  3. 4The engineering team
  4. 3The developer
  5. 2The junior developer
  6. 1The coding intern
  7. 0Spicy autocomplete
obot v1: a robot cat holding a clipboard

obot v1 · early May

A personal assistant

What is it?

A personal assistant running locally on my own laptop

Where it startedThe first commit is signed Orange.

  1. Levels of AI usage
  2. 5The dark software factory
  3. 4The engineering team
  4. 3The developer
  5. 2The junior developer
  6. 1The coding intern
  7. 0Spicy autocomplete
obot v2: a robot cat with a paper plane

obot v2 · May to June

OpenClaw

What is it?

OpenClaw running on a laptop with custom config files and a few Agent.md

What changedIt got its own machine, its own account and a job.

ugh. so many config files.
can't we just build something?

What worked

Real work, by text

👍

obot v2 · what I built

SafetyGraphics, inside an OpenRBQM workflow

meta:
  Type: Report
  ID: safety_histogram
  Name: Safety Histogram Report
  domains:
    labs: Mapped_LB
  widgetName: safetyHistogram
  widgetSettings:
    id_col: subjid
    measure_col: measure
    value_col: value
    group_by: sex
  # … the data spec …
steps:
  # …
  - output: Report
    name: gsm.safety::RenderSafetyChartsWidget
The workflow · gsm.safety v0.1.0
A gsm.safety example page dated May 15, 2026, titled Safety Histogram Workflow: the 2020 Safety Histogram, with its controls and small multiples by sex, rendered from the workflow file
What it rendered · 15 May
  1. Levels of AI usage
  2. 5The dark software factory
  3. 4The engineering team
  4. 3The developer
  5. 2The junior developer
  6. 1The coding intern
  7. 0Spicy autocomplete
obot v3: a robot cat with a terminal prompt

obot v3 · July

Claude Code, on Fable 5

What is it?

Claude Code on Fable 5, with a public roadmap, a set of skills and a bot account

What changedFable 5 came out, and I moved everything into Claude Code.

obot v3 · what I built · 11–12 July

The first renderer took a few weeks; the next six took a weekend.

1.0 billion tokens · $1,273 at API prices · $200 a month actually paid · about 10 hours of me

The new Safety Histogram: a blue histogram of a lab measure0.1 · Sat 8 am
Safety Histogram
The new Outlier Explorer: one line per participant over visits1.0 · Sat 10:30 pm
Outlier Explorer
The new Results Over Time: box plots by visit and treatment group1.0 · Sat 10:30 pm
Results Over Time
The new Shift Plot: baseline against comparison values1.0 · Sat 10:30 pm
Shift Plot
The new Delta-Delta plot: change in one measure against change in another1.0 · Sat 10:30 pm
Delta-Delta
The new AE Timelines: one row of adverse events per participant1.0 · Sat 10:30 pm
AE Timelines
The new Hepatic Explorer: the eDISH scatter with its quadrants1.2 · Sun 10 pm
Hepatic Explorer
The Hepatic Safety Explorer clinical guide page1.2 · Sun 10 pm
Its clinical guide

thats more like it

What worked

Porting old code

👍

obot v3 · what I built next · 17 July

Papers → prompts → prototypes

The hepatic explorer's composite view: two eDISH scatter plots, a four-panel shift plot and a migration table
Hepatic composite view · from a 2024 paper
The QT Safety Explorer: a central tendency view, an outlier scatter and a categorical table
QT explorer · from a clinical workflow

What worked

Translating research into software

👍

🚀🔥💪

  1. Levels of AI usage
  2. 5The dark software factory
  3. 4The engineering team
  4. 3The developer
  5. 2The junior developer
  6. 1The coding intern
  7. 0Spicy autocomplete
obot v4: a robot cat in a crown and tie

obot v4 · August to 10 September

The organisation

What is it?

Claude Code on Opus 5, with four standing roles and a timer script, built to run without me

What changedI asked how far it could get without me in the loop at all.

obot v4 · what I built

Four roles, and a timer script

Jeremysets strategy, decides Conciergeanswers, never builds Operating officerasks become requirements Workersone job each, then gone Timer scriptevery five minutes Fleet managerwakes, clears, exits talks todirectsdispatches watches clears launches Not an agent. The only part with a clean record.

whaaa? can the concierge orchestrate
a fleet of treats for me?
I thought we were building stuff ...

obot v4 · what it built

The machinery spiraled, then we cut it back

Files in obot.agent, June to October

0100200300400500 JunJulAugSepOct v1–v2v3v4v5 0 charts11 charts2 charts7 charts 472 files 80,000 lines 10 September Cut to 7

obot.agent the playbook

v4472 files: 27 home-made tools, 115 drafts, 26 skills, 39 scriptsA dashboard, a morning digest, car audio briefings, a spend guard

v57 files: the instructions, one skill, a daily standup

obot.roadmap the hub

v4Scripts went from 4 to 79Audits, decision logs, status pages

v5Scripts trimmed to 61; reports keep growing, 36 to 810

thank
goodness

  1. Levels of AI usage
  2. 5The dark software factory
  3. 4The engineering teamback, with me driving
  4. 3The developer
  5. 2The junior developer
  6. 1The coding intern
  7. 0Spicy autocomplete
obot v5: a robot cat with a steering wheel

obot v5 · September to October

Back to building

What is it?

Claude Code on Opus 5, then 5.5, with a roadmap of GitHub issues: I set the goal, and agents do the rest

What changedOn 10 September I shut v4 down. 468 files of scaffolding went; seven stayed.

  1. Levels of AI usage
  2. 5The dark software factory
  3. 4The engineering team
  4. 3The developer
  5. 2The junior developer
  6. 1The coding intern
  7. 0Spicy autocomplete

obot v5 · how it works

Same workflow, but I just set goals, answer questions and review releases

meagentsautomated, as before

Review artifacts and answer questions

Design

Set the goal
Gather requirements
Write the work item
Draft the technical plan
Review the plan, then prompt

Build

Write the tests
Review the tests
Implement
Draft the change summary
Check code quality
Approve

Publish

Automated gates
Quality documentation
Draft the documentation updates
Finalize and release
Publish

obot v5 · the roadmap

Objectives → Requirements → Tasks

jwildfire.github.io/obot.roadmap

Requirements1716 of 17 done

Tasks6160 of 61 done

R in the browser, measured
gsm.bio's statistics
Group comparison, the picture
Group comparison, the tests
Association scatter, correlation matrix
safety.viz's shared parts, opened up
Biomarker screen
The demo app hosts the charts
Cross-tabulation and the cut rule
Stratified survival
Results out of the browser
Results out of R
bio.viz's site, styled as safety.viz's
Group comparison over time
An R app for the biomarker charts
The biomarker app as a web app
The difference grid, a second view
Releases · bio.viz and gsm.biov0.1.0v0.2.0v0.3.0v0.4.0
Fri 2Sat 3Sun 4Mon 5Tue 6Wed 7Thu 8Fri 9
One weekend, 2–4 October

Done · releasedIn reviewReadyOpen · backlogReleaseobot.roadmap · GitHub releases · 18 Oct 2026

obot v5 · what I built · 2–4 October

One orchestrator, one brief, one weekend

In: one brief of about 800 words · about 22 messages from me · Claude Code, as shipped
Out: 52 agents, 40 of them reviewers · 3.9 billion tokens, $1,640 at API rates (80% of weekly allotment) · 6 releases

Group comparison: box plots of two biomarkers by arm at five visitsbio.viz
Group comparison
Association scatter: one biomarker against another with a fitted line and its bandbio.viz
Association scatter
Biomarker screen: one row per biomarker with an estimate, its interval and p-valuesbio.viz
Biomarker screen
Correlation matrix: a grid of marks below the diagonal and coefficients above itbio.viz
Correlation matrix
Cross-tabulation: stacked bars of responders by armbio.viz
Cross-tabulation
Stratified survival: two Kaplan-Meier curvesbio.viz
Stratified survival
A static figure from R: box plots with points, by arm, at two visitsgsm.bio
The same charts, from R
The roadmap tracker: the Biomarker charts objective with 12 of 12 requirements releasedthe hub
12 of 12 requirements filed that weekend

What worked

Reviewing its own work

👍

The payoff · in numbers

What we built

May to October 2026 · 15 repositories · counted 13 October

Clinical Trial Tools

20

Interactive charts

14 for safety, 6 for biomarkers

7

Clinical guides

About 19,000 words on how to read a chart

3

Apps

The safety.viz demo, the gsm.bio app, open.csr

Technical

599

Pull requests merged

341 on charts and apps, 253 on the scaffold

43

Releases

31 of charts and apps, 17 of them safety.viz

345,000

Lines of source

43% of them are tests

5,500

Tests

2,963 of them in safety.viz

The payoff · what I built

The safety.viz demo: safetyGraphics ❤️ OpenRBQM

Try it
yourself

The live app · Orange's notes added for the talk

jwildfire.github.io/safety.viz/demo

Costs

At least $11,750 at API prices. I paid $200 a month.

$6,000 $4,000 $2,000 obot v3, July: $3,329.60 on 14 days with usage $3,330 obot v4, 1 to 19 August: $3,830.50, recorded in full obot v4, 20 August to 10 September: at least $2,189.76. A floor. $6,020 or more obot v5, 1 to 6 October: $1,954.96, recorded obot v5, 11 to 30 September: at least $451.72, on three days. A floor. $2,410 or more v3July v4Aug to 10 Sep v511 Sep to 6 Oct A floor: most of those session logs were deleted before they were read What I paid $200 a month One Claude Max plan, with a weekly limit Per million tokens $0.97 → $0.41 July against October, mostly the price cut Two long weekends July: 1.0B tokens for $1,273 October: 3.9B tokens for $1,639

A floor: no June, most of September missing, and the July weekend about $1,100 short.

SDLC · quality

A lot of tests, and still an experiment

Agents test

2,963

tests in safety.viz, up from 343 in July.

Agents review

40 of 52

agents on the October weekend were reviewers. One caught a bad p-value.

A person signs

Not yet

Nothing here has had a formal review.

Tests and reviews are cheap now. The signature is not.

What evidence do you have?

Lots of tests. No sign-off.

Orange looking up at the camera from a chair, in front of an open laptop

Orange, with a question

What we learned

What can we do today?
What's next?
Is it a good thing?

The question, one last time

What can we actually do with AI agents right now?

Real work across the development pipeline.

Agents write code.

Packages, tests, config

Humans own the decisions.

Strategy, architecture, judgement

The gap

Don't judge AI by what you can do on a trial.

Illustration, not data Frontier capabilities Clinical trial tools What you see at work What you're not seeing A year ago Today

The rules at work are right. They also show you a fraction of what these tools can do.

How fast it is moving

Frontier is moving extremely fast

NovDecJanFeb MarAprMayJun JulAugSepOct 20252026 24 Nov 2025Opus 4.5 JanOpenClaw 7 AprMythos 9 JunFable 5 9 JulGPT-5.6 24 JulOpus 5 1 SepFable 5.1 22 SepOpus 5.5 · GPT-6 Sol/Luna 21 Octtoday v1early Mayassistant v2May–JunOpenClaw v3JulFable 5 v4Aug–SepOpus 5 v5Sep–OctOpus 5 → 5.5 Did not try MCP tool protocol, since Nov 2024 Open-weight models Qwen 3.6 Apr, 3.8 Aug Muse Meta’s personal agent, 8 Sep Jev TypeSafe’s decision model, 15 Sep Dots OpenAI’s always-on agents, 29 Sep an obot version ran on it another release

How fast it is moving

New ways of working, written up in public.

6 May · US FDA

Elsa 4.0, with custom agents

…the agency’s internal AI tool available to all FDA staff, from scientific reviewers to investigators. FDA press release · Elsa 4.0 and a consolidated data platform

23 Sep · Rails World keynote

Done coding by hand

At 37signals, a couple of weeks ago, we made the decision that it clearly means we’re done writing code by hand. David Heinemeier Hansson, creator of Ruby on Rails · as transcribed in The Pragmatic Engineer, 1 Oct

15 Oct · Claude Code team

Review, moved to machines

We didn’t remove review. But we did move most of the reading to machines… Addy Osmani, Morgan Westlee Lunt and Qing Wang · How the Claude Code team does code review

16 Oct · Anthropic research

At Anthropic, agent hours outnumber human hours

Agent hours worked outnumber human hours 6 to 1. How employment is changing at Anthropic

From my reading list, and the FDA

The gap, a year from now

We're at the beginning.

Illustration, not data · after today, three guesses Slow Steady Fast Steady Fast Exponential Frontier capabilities Clinical trial tools A year ago Today A year from now

Nobody can tell you what these tools will do next year. They are not going away.

We can think bigger

“I think that most people are underestimating just how radical the upside of AI could be, just as I think most people are underestimating how bad the risks could be.”
Dario Amodei · Machines of Loving Grace · October 2024

What's next

More Medicines Faster

Illustration, not data Frontier capabilitiesClinical trial tools A year ago Today A year from now Fast Slow Fast Some day?

Time to close the gap

Closing the gap

Try it on something that's yours.

  1. 1Pick an agentClaude or Codex, $20 for a month. Others are fine too.
  2. 2Set up GitHubA free account is fine.
  3. 3Pick a development processFor R: Hadley Wickham's R Packages book, free at r-pkgs.org.
  4. 4Write a prompt

    Make me an Open Source R package that does X. Include a working example and user guide in GitHub pages. Follow all of the best practices in Hadley Wickham's R Packages book.

  5. 5Ask questions!

    Explain how you did X

What are agents good at now?

Teaching me things!

👍

Closing the gap

Or do something fun!

I make games with my kids, using AI. Here are our favorites.

Angry Dudes: a Roman legionary at the slingshot faces a Greek fort with Archimedes' Claw on the wallAngry DudesA slingshot siege through history: 10 armies, 37 real battles Magic Friends title screen: a mermaid in the sea and a unicorn on a cloud under a sunriseMagic Friends, Up We Go!A climbing game for little kids: 60 levels to bring back the sunrise Timed Out title screen: an hourglass and a choice of Easy, Medium or HardTimed OutPut events from Wikipedia in order on a timeline
jwildfire.github.io/projects

Mixed feelings

Coding feels completely different than it did a year ago.

It's kind of fun, but I'm certainly not working less.

But, what's next?

More medicines, faster!

And no more powerpoint!

Data Centers. Rogue Agents.

So. Much. Change.

100 years of innovation!
Trials on a chip!
Robots folding my clothes!

Acknowledgements

Thanks to my amazing co-workers, patient bosses, very patient family and all of my open source collaborators!

jwildfire.github.io/keynote
← → · O overview · N notes · P presenter · F fullscreen