Chapter 16: Have the conversations only you can have¶

In this chapter you will learn to:

Know what regulators and the consortia that advise them ask of registry data.
Plan corroboration before collecting a measure you hope to qualify.
Prepare a briefing pack for a regulatory or industry conversation.

Case study: A Measure Built Without Its End User¶

This case study is a composite of situations I have seen across many registries and committees, with the details made generic.

A registry wanted to show that a digital measure could track a symptom its community cares about better than the existing instruments. Every month, participants completed a standardized instrument for that symptom, which the registry had modified to suit its population without consulting the instrument's developers and without checking whether it was designed to be reported by patients or by clinicians. At the same time, a vendor's app collected continuous data from participants' wearable devices, which the registry was entitled to access. The hope was a significant finding: if the device measure tracked the symptom more accurately than self-report or clinician report, it might eventually be qualified as a biomarker or used as an outcome measure, and help future clinical trials succeed.

After substantial collection, the registry brought its data to a consortium that makes recommendations to the FDA. The questions were ones the registry had not planned for. Is there clinician-confirmed evidence for what participants report? Can the electronic health record, or a clinician's abstraction from the clinical notes, corroborate the finding? Has anyone confirmed that each participant has the underlying condition, when enrollment was self-reported online? If the measure tracks functional decline over time, does the clinical note show that decline? Is the clinician administering the same instrument in clinic, and can those results be reported? In what format would the health record data need to arrive? And because the instrument had been modified and was not validated in this population, its results could not be compared with other studies that used it.

The data were not poor. They were valuable, and the effort behind them was considerable. They were collected without reference to the standards and metrics their end user has to apply, and a regulator cannot set those standards aside. In situations like this, data that could have supported a qualification are treated as preliminary or anecdotal, and the next step is a larger study designed from the start to answer those questions.

Lessons for registry leaders¶

Fit for purpose is the thread through this whole book, and this is where it comes full circle. The regulator, and the consortia that advise it, are end users with their own standards, and those standards decide whether the collection can be used. Learning them early takes conversations with the people who apply them. An AI tool will not anticipate them, and it may set the burden of provenance too high for one purpose and ignore it for another.

Could you defend it to the FDA?¶

A common piece of advice is never to put anything in an email you would not want read aloud in court. The registry version is never to put anything in a registry you would not be prepared to defend to the FDA. That is not a call for anxiety about every field. Survey, qualitative, and quality of life data captured ethically are valuable however they are collected. It is a call to know, for each kind of data, who might eventually rely on it and what they would ask. The FDA's own term for this is context of use, "the manner and purpose of use" for a drug development tool, and its qualification programs for biomarkers and clinical outcome assessments reach "a conclusion that within the stated context of use, the DDT can be relied upon" (FDA, Drug Development Tool Qualification Programs).

The rules¶

Assume a regulator may one day look at your data, and know for each kind of data who might rely on it and what they would ask.
Learn what regulators ask of real world data before you need them to: fitness for use, provenance, how the diagnosis was established, and how follow-up was kept.
If you hope a measure will become a biomarker or a clinical outcome assessment, talk to the people who qualify them before you start collecting: the consortia that advise the FDA, the instrument's developers, and the FDA's qualification programs. Learn the intended context of use.
Know who is supposed to report each instrument (the patient, a clinician, an observer such as a parent, or a performance task), and collect it that way.
Plan corroboration from the start: a clinician-confirmed diagnosis, health record data or clinician abstraction from notes that can confirm what participants report, and the same instrument administered in clinic where possible. Ask in what format the health record data must arrive.
Prepare for a company conversation by knowing exactly what you can share, under which consent, and through which governance step.
Bring the enrolled, confirmed, and evaluable counts and the tier breakdown to every such meeting.
Label anything that rests on a press release or a company statement as such, and never present it as a scientific result.
Keep minutes of every regulatory or industry conversation and of what was committed.
Keep a patient or family partner in the loop for these conversations; they decide what the community is willing to share.

What AI can do, and what only a person can do¶

What AI can do¶

Summarize guidance documents and qualification program requirements; draft a briefing pack from your own documentation; help prepare questions.

What only a person can do¶

Learn what a regulator or consortium will ask by asking them; sit in the meeting; answer a follow-up question; negotiate what a company will receive and what the community gets in return.

Take-home¶

Never put anything in a registry you would not be prepared to defend to the FDA; learn your end user's standards before you collect, because no tool can have that conversation for you, and what you can say in that room is decided by the habits in every earlier chapter.

Practice¶

Exercise¶

The Lantern Registry has been invited to present its seizure frequency data to a consortium that makes recommendations to the FDA. Using your work from Chapters 8, 9, and 12, prepare the briefing pack: the enrolled, confirmed, and evaluable counts, the tier breakdown, the corroboration available, and your answer to the question "Could you defend it to the FDA?" for the seizure count field.

Check your work

Counts: 32 records for 31 children; 25 with a qualifying genetic report on file, none yet confirmed by a qualified reviewer; 15 evaluable for monthly seizure frequency (a qualifying report on file, a seizure count without an open query, and a valid follow-up date after a valid enrollment date).
Corroboration: none yet. The registry has no health record data, no clinician-administered instrument, and no reviewer named for the genetic reports.
The seizure count field: it accepts free text, holds placeholders, has no recorded source or recall period, and the follow-up schedule is not defined. Read naively, the mean is 47.3 seizures a month; read with care, it is 14.5 among the 15 evaluable participants. A consortium would ask how the count was collected and whether a clinician or diary corroborates it.
What to say: describe the data as they are, name what would make them fit for the purpose under discussion, and ask what the consortium needs.

Your workbook page¶

The Briefing pack and meeting minutes page of the Slow Registry Workbook applies this chapter to your own registry. Fill it in online or print it and work by hand. The AI prompt library has a prompt for this chapter's topic, paired with the step only a person can do.

Check your understanding¶

What does the FDA mean by context of use?

In its drug development tool qualification programs, the FDA describes context of use as "the manner and purpose of use" for a tool. Qualification is a conclusion that within that stated context, the tool can be relied upon.

A registry modified a standard instrument without consulting its developers. What problem does that create later?

Its results cannot be compared with studies that used the instrument as intended, and a regulator or consortium will ask whether it was administered by the right reporter and validated in the population.

Up next: All of this work prepares the registry for AI, and the last part is about using it without losing the thread. Continue to Chapter 17.