Chapter 8: Verify the diagnosis before you count it¶
In this chapter you will learn to:
- Define tiers of diagnostic evidence fitted to the registry's purpose.
- Classify participants into tiers from the evidence on file.
- Decide whether to adjudicate now or later, and who adjudicates.
- Report enrolled, confirmed, and tier counts together.
- Ask the same evidence question of every field, including instruments collected by a method they were not validated for.
Case study: Who Reads the Genetic Report¶
I have worked in this field for years and I do a lot of work in omics, and I still could not tell you with certainty what I would pull from my son Charlie's genetic report, to the level a regulator, a clinician, or a researcher would need before they felt confident in what I had submitted. I am an experienced abstractor. I do not have genetics training or a medical degree. For some purposes that is fine. If my registry exists to know where our families are and how to reach them, so that we can send a newsletter, offer a respite care scholarship, or tell them about a new trial, then a parent telling me their child has the condition is enough, and I do not need anyone to upload anything.
For other purposes it is not fine, and it is nothing personal. A sponsor about to spend money on expensive analytics, or to give people a drug on the strength of what I shared, needs to know that somebody trained in reading these reports looked at them. It is not that they doubt I am capable. It is that they need the assessment to be standardized and the assessor to be credentialed, so that if the work leads to a discovery, it is irrefutable. The way I have come to think of it is that they are not refusing to trust me; they want to back me up completely, so that people take the registry seriously. If you ran a coffee shop with a professional cleaning crew who knew exactly what every client wanted, you would not decide to cut corners and have the person pouring the coffee wash the windows instead. The roles are not interchangeable, and confirming a diagnosis is a role.
Lessons for registry leaders¶
The evidence a diagnosis needs depends on what the data will be used for. When others will build on it, they need a credentialed reader, and that requirement protects the registry's findings rather than questioning its leaders. Collecting the report now and arranging adjudication when a user needs it can be a sound use of limited funds.
The same question for every field¶
The diagnosis is the field where verification is easiest to see, because the evidence arrives as a document and the reader holds a credential. The question behind it applies to every field the registry collects: what evidence stands behind this value, and is that evidence enough for the use someone will put it to? A diagnosis, a seizure count, a medication list, a functional rating scale, a weight, and a date of last follow-up each rest on a source, and each source supports some uses and not others.
Functional rating scales are a clear case. Many scales were developed and validated for administration by a trained clinician at a visit. A registry may ask participants or caregivers to complete the same items themselves every month, because that is the only way to get the measure at that frequency and at that cost. The data that result are not the validated instrument; they are a self-administered survey built from its items. Whether that difference has consequences depends on the use. If the registry wants to describe how families report function over time, to see which domains change first, or to plan the visit schedule for a later study, the survey may serve well, and the publication states what was collected and how. If the registry wants to report scores on the instrument's scale, compare them with published clinician-administered values, or offer them to a sponsor as an endpoint, the absence of validation for self-report is a limitation that reviewers will raise and that may rule the use out. Some instruments have a validated self-report version or published work on self-administration, and others do not, so the registry can check before it chooses.
The same reasoning applies down the list. A medication list typed by a parent supports a description of what families report taking; a medication list abstracted from the chart supports a question about prescribing. A seizure count from a monthly recall supports a rough picture of burden; a count from a daily diary supports an analysis of change over time. A weight reported from a home scale is fine for a growth description and not for a dosing study. None of these is a wrong field to collect. The error is to collect at one standard of evidence and report at another without saying so.
The practical step is to record in the data dictionary what each field's source is and what level of evidence it represents, in the same way the diagnosis field records its tier: who supplied the value, by what method, and whether that method is the one the measure was validated for. When a use arrives, the registry can match the use to the evidence rather than discover the mismatch at peer review. Where a use needs a higher standard than the registry collects, the registry can say so, and collect to that standard for the participants who need it, as the diagnosis rules below suggest.
The rules¶
- Decide the purpose before you decide the evidence. A contact or support registry, one that sends the newsletter, offers services, and announces trials, needs only the family's word. Anything a researcher, sponsor, or regulator will build on needs more.
- Collect the actual document, not a summary of it, and be specific about which document: the clinical laboratory report that names the test performed and the classification of the variant. Most families have one. "Positive" in a text box is not evidence.
- Define tiers of diagnostic evidence in the protocol (laboratory report reviewed by a qualified person; report shows a variant of uncertain significance; clinician letter only; self-report only) and classify every participant into one.
- Decide who adjudicates, and whether to adjudicate now or later. The person who reads the report and records a conclusion is a qualified person, a medical advisor or a genetic counselor, and experience in the field does not substitute for the credential. Record who reviewed, when, and what they concluded.
- Check with your IRB before you ask anyone to upload a genetic report, and before you assume you can share it later. Write both the upload and the sharing into the protocol, and make sure you are allowed to abstract from it.
- Consider collecting now and adjudicating later. If you do not yet know who will need a confirmed diagnosis or when, it may be better to ask families to upload whatever they have, record what you have, and keep the consent to recontact, so that the sponsor or researcher who eventually needs confirmation can see what exists and arrange the adjudication to their own standard. Hiring a genetic counselor before you know who needs the adjudication, and whether that standard will still be the standard in ten years, can be money better spent on a thorough natural history description.
- Do not try to be all things to all people at once, before you know who those people are and when they will come. The timeline is one more thing AI cannot predict.
- Re-check classifications on a schedule, because variants get reclassified and self-reports get confirmed.
- Report the tier breakdown in every summary, and never pool tiers without saying so. Expect a sponsor recruiting for a trial to ask how you know, and have the answer in writing.
- Apply the diagnosis standard to every field. For each measure the registry collects, record its source and the method of collection, note whether that method is the one the measure was validated for, and match each use to that evidence before you publish, share, or offer the field as an endpoint.
What AI can do, and what only a person can do¶
What AI can do¶
Flag inconsistent diagnosis fields; extract the variant string from an uploaded report for a person to check; suggest the tier a record appears to meet.
What only a person can do¶
Read the report with clinical judgment; ask the family for the document; decide the case definition.
Take-home¶
How much verification you need depends on what the data are for; when they will be built on, the diagnosis is confirmed by a qualified person who read the actual report, and that requirement is the field backing you up, not doubting you. The same question, what evidence stands behind this value and is it enough for this use, applies to every field you collect.
Practice¶
Exercise¶
Classify every record in the synthetic Lantern Registry export using the verification tier form. Use these tiers:
- Tier 3: a genetic report showing a pathogenic or likely pathogenic variant in GENEX, reviewed by a qualified person, with the reviewer and the date recorded.
- Report on file, awaiting review: a report has been uploaded and the variant entered is pathogenic or likely pathogenic, and no qualified person has reviewed it yet.
- Tier 2: a genetic report is on file showing a variant of uncertain significance.
- Tier 1: a clinician letter only.
- Tier 0: self-report only, or a result still pending.
The export records that a report was uploaded and which variant was entered. It does not record that a qualified person reviewed any report. Some records fit none of these categories, so decide what to call them and why.
Check your work
| Classification | Records | Count |
|---|---|---|
| Tier 3 | None yet: no review is recorded for any report | 0 |
| Report on file, awaiting review | Every record not listed below | 25 |
| Tier 2 | LR-006 (variant of uncertain significance) | 1 |
| Tier 1 | LR-027 (clinician letter, no report) | 1 |
| Tier 0 | LR-003 ("positive," self-report), LR-011 (suspected, result pending) | 2 |
| Conflicting evidence | LR-017 (pathogenic variant in a different gene, GENEY) | 1 |
| Not a case | LR-026 (epilepsy of unknown cause, GENEX testing negative) | 1 |
| Duplicate | LR-012 (same child as LR-005: date of birth, sex, country, variant) | 1 |
The export has 32 records for 31 children. Of these, 25 have a report on file showing a pathogenic or likely pathogenic variant in GENEX and would reach Tier 3 once a qualified person reviews them; until then, none is confirmed. A report to a sponsor gives these numbers and the tier breakdown, and says plainly that review is pending.
Your workbook page¶
The Verification tier form page of the Slow Registry Workbook applies this chapter to your own registry. Fill it in online or print it and work by hand. The AI prompt library has a prompt for this chapter's topic, paired with the step only a person can do.
Check your understanding¶
A contact registry exists to send a newsletter and announce trials. Does it need confirmed diagnoses?
No. For that purpose the family's word is enough. The evidence a diagnosis needs rises with what the data will be used for.
Why might a registry collect genetic reports now but arrange adjudication later?
If it is not yet clear who will need confirmed diagnoses, or to what standard, the money may be better spent on a thorough natural history description. Collecting the reports now, with IRB approval to share them and consent to recontact, lets the eventual sponsor or researcher arrange adjudication to their own standard.
A registry collects a functional rating scale that was validated for clinician administration, as a monthly caregiver survey. Can it publish the results?
It depends on the claim. The registry can publish what caregivers reported, month by month, as long as the methods say that the items were self-administered and that the instrument has not been validated for that use. It cannot present the results as scores on the validated instrument, compare them with published clinician-administered values, or offer them as a trial endpoint without that validation, and a reviewer will ask. Recording the method of collection in the data dictionary, next to each field, is what lets the registry match each use to the evidence it holds.
Up next: Verification only counts if the evidence and the decision are recorded where someone else can find them. Continue to Chapter 9.