Chapter 9: Keep records someone else could read¶

In this chapter you will learn to:

Keep raw values untouched and record every change.
Define groupings before anyone uses them, and version the definitions.
Explain why reproducibility starts in the registry.
Handle registry data as a covered entity would.

Case study: The Code That Could Not Reproduce the Paper¶

I joined a study team after its main paper had been published. As in many academic groups, the analysis had passed through several students and trainees, and they had written the code behind the paper. When another group asked for our script so they could validate the same survey instrument in their own data, the study team turned to me, because by then I was the only person on the team who could run code. The principal investigator had no way to look at the code and judge whether it was correct.

I ran the shared code against the data set I had, and it did not reproduce the published results. The instrument used Likert-scale items, some of which had to be reverse coded before scoring. Instead of creating new reverse-coded variables, someone had reverse coded the original variables in place, a common choice for an analyst without formal training in data management. The data set I received may have had items flipped more than once, and nothing recorded which items had been reversed, when, or by whom. I could not tell which values were right.

The only way to a trustworthy answer was to go back to the raw export from the survey platform, find the original values, and rebuild the scoring from the beginning. A version of this story appears in Ten Habits of Great Data Analysts, told from the analyst's side; here the point is what the study team needed to have in place before anyone outside the team asked.

Lessons for registry leaders¶

A request for the code behind a published result is reasonable, and it often arrives years after the people who wrote the code have moved on. A registry leader does not have to read code to prevent this outcome. What prevents it is a rule that raw values are never overwritten, a dated copy of every original export, and code that is kept, commented, and run again by someone other than its author before a result is published.

Reproducibility starts in the registry¶

Observational research has a reproducibility problem, and the OHDSI (Observational Health Data Sciences and Informatics) community discusses it often. In one study, nine independent research teams tried to reproduce the same patient cohort from a published free-text description, using the same clinical codes and tools. Their cohorts ranged from about a third to ten times the size of the original, with a median agreement of 9.4 percent, and the authors recommend sharing analytic code built on a common data model so that a study can be reproduced unambiguously (Ostropolets et al., 2023). When another group attempted to reproduce 150 published real world evidence studies using the same databases as the original investigators, most results came out close to the originals, but a subset did not, and the authors attributed those differences to incomplete reporting and updated data (Wang et al., 2022).

The FAIR principles (findable, accessible, interoperable, reusable) can look like box-checking, something a funder or a journal asks for and a team does to show good intentions (Wilkinson et al., 2016). Followed downstream, they are the practical means by which the next analyst, reviewer, or regulator arrives at the same number you did. Open standards, shared code, and versioned definitions are how a registry's results outlast its first analyst.

Spin-off data sets¶

Ten Habits of Great Data Analysts describes an academic registry where several research fellows each drew a data set of their own for a poster or abstract, recoded the disease groups by their own criteria, and left no record of the decisions. In my experience this is common practice, not an exception. In a cohort of fewer than 100 people divided into several subgroups, moving two or three people between groups, or collapsing two groups into one, can change the finding. When those decisions are undocumented, the same registry tells different stories in different abstracts, and no one can say which one is right.

Handle the data as if you were a covered entity¶

Many patient advocacy groups are not covered entities under HIPAA (the Health Insurance Portability and Accountability Act). I recommend handling registry data as if you were one: identified data stay in approved, access-controlled storage, never on a personal desktop, and identifiers are kept apart from research data. Academic researchers complete training in research ethics, the protection of research participants, and data security as a condition of their work, and seeking out the same training helps a registry team: it protects participants, and it explains the questions researchers, IRBs, and journals ask about how you handle data, and why data use agreements are so detailed. Researchers are reluctant to receive data from, or share data with, a partner who might disclose it by accident, especially when the data describe their own patients.

The rules¶

Keep source documents organized by participant so that another person could audit any row.
Never recode in place. Keep raw values exactly as they arrived, and create new, documented variables for every recode, reverse score, and grouping. A variable recoded in place, perhaps more than once, can only be rebuilt from the original export.
Keep a dated copy of every original export from the survey platform or vendor, and never edit it; curate in a copy and keep a curation log.
Define every grouping before anyone uses it: what it takes to belong to a type, a subgroup, or a cohort, how edge cases are handled, and whether variants of uncertain significance are in or out. Take a consensus from the clinicians, record it with a date and a version, and apply it the same way for every request.
Derive every analysis data set from the shared, versioned definitions rather than letting each project spin off its own recodes. For each one, record the inclusion rules, the definition versions, the extract date, and the participant identifiers it contains. In a cohort under 100, moving two or three people between groups can change the finding.
Record the assay, the lab, the units, and the reference range with every lab value. Two assays for the same marker can both be valid and move together over time while reporting on different scales, one reading 3 where the other reads 10; without the assay, the values cannot be combined or compared.
Comment and version-control every script, keep the code that produced each poster, abstract, and paper with the data set it ran on, and have someone other than its author rerun it before publication. Treat all code as if it will be part of a regulatory submission: many journals ask for it, and regulators can ask for the programs behind a submission.
Record every changed value as it happens: who changed it, when, from what to what, and why.
Handle the data as a HIPAA covered entity would, whether or not you are one. Identified data stay in approved, access-controlled storage, never on a personal desktop, and identifiers are kept apart from research data.
Seek out training in research ethics, the protection of research participants, and data security for everyone who touches the data.
Name a keeper of the definitions and the history, and a second one, because registries outlast staff.
Be able to share a participant's documentation with a clinician or an external reviewer under the consent you have, with a documented process for doing it.
Store documents where they survive a change of platform, a change of staff, or the founder stepping back.

What AI can do, and what only a person can do¶

What AI can do¶

Draft log templates; comment code and summarize a change history for review; suggest a file naming and versioning scheme; check a script against an agreed specification.

What only a person can do¶

Decide what a definition means and get the clinicians to agree on it; rerun the code and compare it with the published number; be the one whose name is on the change; decide who may see identified data.

Take-home¶

The results a registry publishes are only as reproducible as the trail behind each value and each cohort: raw values kept untouched, definitions and code versioned and rerun, identified data protected, and a historian, preferably two, who knows why each decision was made.

Practice¶

Exercise¶

Work through the interactive lab: find the planted problems in the synthetic Lantern Registry export, then check your answers.
Participant LR-009 has a weight of 45 kg recorded at about age 4. Record this in the curation and definitions log as you would in a real registry.
Two research fellows want to report the number of participants with "severe" disease. Write the definition of severe you would agree with the clinicians before either fellow draws a data set, and give it a version and a date.

Check your work: the curation log entry

Date	Record	Field	Raw value	Issue	Action	Corrected value	By
2044-03-18	LR-009	`weight_kg`	45	Implausible for age; likely entered in pounds (about 20.4 kg)	Query sent to family; raw value kept	Pending confirmation	Data manager

The raw value stays in the raw export. A correction goes into the curated copy only after the family confirms it, with the log entry pointing to the confirmation.

Your workbook page¶

The Curation and definitions log page of the Slow Registry Workbook applies this chapter to your own registry. Fill it in online or print it and work by hand. The AI prompt library has a prompt for this chapter's topic, paired with the step only a person can do.

Check your understanding¶

Two fellows report different counts of severe cases from the same registry. What do you check first?

Whether the definition of severe was agreed and versioned, and which version, inclusion rules, extract date, and participant list each fellow used.

Why handle registry data as a HIPAA covered entity would, even if your organization is not one?

It protects participants, and it matches what researchers, IRBs, and journals expect of anyone they share data with.

Up next: The hardest trail to keep is the one that leads from a person to a tube of blood or to the same person in another study. Continue to Chapter 10.