Chapter 4: Decide what to capture before you go looking¶

In this chapter you will learn to:

Choose a primary focus for a registry, decide which goals to combine with it, and match each goal to the data sources it needs.
Describe what electronic health record data reliably hold and what they do not.
Tell a validated instrument from a common data element.
Plan the granularity of a field before collection.

Case study: What the Record Did Not Show¶

My son Charlie sees his neurologist about every six months. Before most visits the clinic sends me a survey asking about his seizure frequency and what has changed, and I fill it out. When I download his medical records, there is no guarantee that any of it is there. The note might say seizures increased or seizures decreased. The messages I exchanged with the clinic between visits, about raising a dose or a problem we were having, do not come through at all. A lot happens in six months in a condition as variable as epilepsy, and if the record is all you have, you cannot extract that variability, because no one recorded it in a form that travels.

I learned the same lesson from the other direction in ALS research. The ALS Functional Rating Scale Revised is asked at most visits. It is the most commonly used measure in the disease, it produces a score, and it has a LOINC code, so we assumed it would come out of the record as structured data. Out of roughly a hundred patients, I would see it recorded in a structured way perhaps once. Everything else had to be pulled out of the clinical note, where it was written, spelled, and abbreviated in many different ways. If a measure is that important to your registry, you need to know before you design anything how likely you are to find it in the source you are counting on.

A registry or natural history study can pursue several goals, and most combine more than one under the same roof. There is no separate "risk factor registry" or "quality of life registry"; these are goals within one registry. The planning question is which goal is the primary focus and which others to combine with it, because each goal needs different data from different sources. Staying in contact means holding who people are and how to reach them. Studying risk factors means asking what exposures or triggers could have brought the condition on, a common goal in neurodegenerative conditions and cancers where the causes are poorly characterized. Describing natural history means characterizing variants and progression, often so that clinical trials can be stratified: if some children have many seizures and others few, or some have more motor involvement than others, a trial meant to show an effect on motor symptoms needs to enroll the people in whom that effect could show, and in ALS a drug expected to affect speech needs participants chosen by their speech involvement. Describing what it is like to live with the condition means capturing quality of life and burden, such as sleep, feeding, and adaptive and challenging behaviors. I have heard families describe these on FDA panels, and they can support a trial even when they are not its primary outcome, but most groups collect them to tell the world the condition is hard rather than with that strategy in mind, and it is the least standardized area in the field. The case for qualitative data is made in Ten Habits of Great Data Analysts and belongs here by cross-reference. A head-to-toe check-in every few months is natural history, and it can seed stratification and computable phenotypes from basic facts like variant and onset. A typical week in a family's life will not come out of an electronic health record at all.

Source	What it reliably gives you	What it will not give you without extra work
Electronic health record, specialist clinic	Diagnoses the specialist made; medications ordered; procedures; labs and lab values; anything carried in FHIR resources, mostly current; the thorough history taken at diagnosis, in the notes	Records from primary care, gastroenterology, or any clinic outside the system you pulled; portal messages between visits; whether a medication is still being taken; devices, communication devices, and therapies recorded elsewhere, even in the same building; scored instruments in structured form; anything that happened between visits
Patient-directed record download	What the system chooses to release, usually the notes and the structured data above	Pre-visit surveys the clinic sent; messages; a reconciled medication list
Your own surveys	The granularity and frequency you choose; quality of life and burden; the supplement questions the record cannot answer	Comparability to other studies unless instruments are administered as intended; the participant's time
Diaries and apps	Seizure frequency and similar events over weeks and months	Adherence to the diary; verification
Wearables and sensors	Continuous measures between visits	Interpretation; device differences; the sensor data a trial would accept
Biosamples	Omics and biomarkers	Chain of custody and linkage (Chapter 10)
Interviews and open-ended questions	What it is like, and which questions to ask next	Scale; time to analyze

Lessons for registry leaders¶

A record holds what a clinic chose to document, in the form the clinic chose. A pre-visit survey, a portal message, or a scored instrument can exist inside the clinic and still never reach the data a registry receives. Registry leaders who look at representative records before designing anything, and who ask families directly for what the record does not hold, avoid building a registry around data that will not arrive.

The rules¶

Decide the registry's primary focus and which other goals to combine with it, because the goals decide the sources. Natural history and stratification lean on records plus a scheduled check-in; risk factors and quality of life come almost entirely from asking. These are goals within one registry, not separate kinds of registry, and most registries combine several.
Do not assume the electronic health record holds everything. Just because it happened, and you told somebody in the healthcare setting, does not mean it will be in the data you are allowed to receive.
Before you design a single field, look at real records. Download your own (Registry Forge, Patient Edition walks a parent through downloading the structured export from an Epic MyChart portal, and through requesting the fuller chart from a records office) or pull a representative participant's with consent, and see what is there and what is not. No profiling tool and no AI can do this for you, because the question is what your clinics record.
Get records from every source, not only the specialty center. The children's hospital record will not include the gastroenterologist across town, and may not include the wheelchair, the communication device, or the speech therapy even when they are in the same complex.
Expect structured data for medications ordered, diagnoses, procedures, and labs, mostly current. Do not expect portal messages, and do not expect to know whether a person is still taking a medication; you will see rows of orders going up and down with no one to reconcile them. Confirm with a sample what your vendor or your transfer delivers before you rely on it.
Expect scored instruments in the notes, not in structured fields, even when a code exists for them. If a measure is essential, plan to abstract it, and budget the time.
The visit schedule bounds what the record can tell you. If a specialist is seen every six months, there is almost no chance of recovering seizure frequency over two or three months from the record. For anything crucial, collect it yourself at the frequency you establish, or pre-analyze the record and ask the participant to reconcile it.
Talk to families before you build. Ask what their doctor asks them, what their downloaded records look like, and whether they can see seizure frequency in them. If they cannot, your registry will not see it either, because a vendor sending you the same records does not add what was never documented.
Plan the granularity of every field, and ask for the continuous value rather than a pre-collapsed category wherever you can. You can always group later; you can never ungroup. The exception is a question where participants prefer a range, such as income group rather than salary. Researchers once came to me for help with a prenatal study that had asked weeks of gestation as categories, and one bucket ran from about 32 to 37 weeks, straddling the prematurity threshold. They could no longer tell who had been premature, and the study could not answer its own question. Nothing in a list of questions handed to an AI would have flagged that, because the tool does not know why that threshold is the one you care about.
If the goal is regulatory acceptance, work with the people who develop and approve drugs: FDA and industry, through pre-competitive consortia such as the Critical Path Institute's workspaces, on what to collect and how.
Know the difference between a validated instrument and a common data element. If it produces a score or works as a screener, it is probably a validated instrument: get permission, use the current version, administer it as intended, and do not break it apart or change the questions, or you lose the ability to compare your results to anyone else's. A common data element is a well-written question from a bank: ask it exactly as written. Adding a category (a group you care about, in a race or ethnicity question) does no harm; deleting or collapsing categories (the income brackets) means no one can compare to you.
A modified instrument is not worthless, but it is not comparable. Revalidating an instrument for a new population is extensive work that most academic researchers will not take on without funding. Modify when your purpose is a signal that attracts a researcher who can do it properly, and say so; administer it as intended when you are the one who has to defend the number.
Think of the registry as something that attracts other researchers. A well-characterized population with its participants in one place is intriguing to an early career investigator, or to someone who knows how to run a proper exposure study, and you may not be the last person to query these families. Collecting a signal well is a legitimate purpose.

What AI can do, and what only a person can do¶

What AI can do¶

Suggest outcomes, validated instruments, and common data elements to consider; draft form text; propose ontology terms for review; help abstract a scored instrument from notes for a person to check.

What only a person can do¶

Look at representative records and know what to expect; ask a participant whether they are still taking the medication; ask families what their clinic writes down; decide which questions to ask and how often.

Take-home¶

Just because it happened, and you told somebody in the healthcare setting, does not mean it will be in the data you are allowed to receive; anything crucial is collected by asking, at the frequency you set, and no tool will guess at a question you did not ask.

Practice¶

Exercise¶

Open the data source plan and the synthetic Lantern Registry export.

For each of these fields in the export, record where the information would best come from (health record, survey, diary, clinician, lab report), how often, and at what granularity: sz_per_month, meds, onset_age, dev_delay, seizure_types.
Find the fields where the export has already lost information through the way it was collected.
The registry wants seizure frequency for every month. Participants see their neurologist about every six months. Propose a source and a schedule.

Check your work

Field	A better source	What the export shows
`sz_per_month`	A seizure diary or app, completed by the family, summarized monthly	Free text is accepted, so the field holds 999, -1, and "daily" alongside counts
`meds`	A medication list reconciled with the family at each contact, coded to RxNorm, with dose in separate fields	Brand names, abbreviations, and doses mixed in one field
`onset_age`	A survey item asking age at first seizure in months, with a unit fixed by the form	Months, weeks, years, unlabeled numbers, and "birth" in one column
`dev_delay`	A survey item with yes, no, and unknown, or a clinician assessment	Yes, yes, 1, TRUE, No, and N/A
`seizure_types`	A list of standard seizure types, chosen by the family or abstracted from a clinical note	Free text, including an event count typed into a type field

For monthly seizure frequency, the health record will almost never have it at that granularity when visits are six months apart. A diary or app the family completes, with a monthly summary the coordinator reviews, is the realistic source.

Your workbook page¶

The Data source plan page of the Slow Registry Workbook applies this chapter to your own registry. Fill it in online or print it and work by hand. The AI prompt library has a prompt for this chapter's topic, paired with the step only a person can do.

Check your understanding¶

A new registry leader says, "We are building a risk factor registry." How would you reframe that?

Risk factors are a goal, not a kind of registry. Registries and natural history studies usually combine several goals, such as risk factors, natural history, and quality of life, under one roof. The decision is which goal is the primary focus and which others to combine with it, since each goal draws on different sources.

You plan to compare your registry with published studies that use a scored questionnaire, but you want to drop two items that do not fit your population. What happens?

Your results are no longer comparable with studies that administered the questionnaire as intended. The modified version may still be informative, so say plainly that it was modified and why.

A common data element for race lists categories, and you want to add a group your community cares about. Is that a problem?

Adding a category generally does no harm. Deleting or collapsing categories makes comparison with other studies impossible.

The neurologist asks about seizure frequency at every visit. Will the registry receive that answer from the health record?

Not necessarily. Pre-visit surveys, portal messages, and scored instruments often never reach the data a registry receives, and when they do they are usually in the notes rather than in structured fields. Look at representative records before you rely on it.

Up next: Forms do not fill themselves in, and the person who makes that happen is the role most groups never name. Continue to Chapter 5.