Medical AI Needs a Benchmark for Medicine Itself
Why evaluating the AI is only half the problem
Richard Z. Cheng, MD, PhD
Artificial intelligence is becoming extraordinarily capable at answering medical questions.
But there is a deeper problem hiding underneath this rapid progress.
When we say that a medical AI system gave a “correct” answer, correct according to what model of medicine?
Most medical AI benchmarks evaluate AI by comparing its answers with medical examinations, clinical guidelines, expert consensus, or predefined reference answers.
That is reasonable when the purpose is to determine whether an AI system has learned and can reproduce existing medical knowledge.
It is less satisfactory if we also want AI eventually to help us examine—and potentially improve—the way medicine itself is conceptualized and practiced.
Our recent preprint therefore proposes separating two evaluation problems that are often combined.
Layer 1 — Evaluate the medical system
The first question is whether the underlying medical framework itself is adequate.
A medical system can be examined across dimensions such as:
-
scientific and causal coherence;
-
quality and appropriate use of evidence;
-
patient safety;
-
treatment of uncertainty;
-
prevention and early intervention;
-
ability to address chronic and multifactorial disease;
-
attention to individual biological variation;
-
recognition of competing explanations;
-
clinical applicability;
-
and capacity for correction when evidence contradicts the framework.
This level of evaluation can be applied to conventional medicine, orthomolecular medicine, functional medicine, metabolic medicine, integrative medicine, or a newly proposed medical framework.
No system should receive privileged treatment simply because its advocates—or the benchmark designers—prefer it.
Layer 2 — Evaluate the AI implementation
The second question is how well an AI system reasons within and applies a medical framework.
An AI may possess enormous factual knowledge yet remain clinically unreliable if it:
-
ignores contraindications;
-
overstates certainty;
-
fabricates citations;
-
fails to distinguish evidence from hypothesis;
-
applies recommendations rigidly;
-
misses situations requiring physician evaluation;
-
or gives substantially different answers to equivalent questions without justification.
AI evaluation therefore requires dimensions such as safety, transparency, consistency, evidence fidelity, uncertainty calibration, escalation behavior, clinical boundaries, and educational usefulness.
The distinction can be summarized simply:
Layer 1 asks whether the map is good.
Layer 2 asks whether the navigator uses the map well.
Both matter.
Why this matters to orthomolecular medicine
Orthomolecular medicine has long emphasized dimensions of health that are not always captured by disease-centered models.
These include nutrient sufficiency, biochemical individuality, metabolism, redox biology, host physiology, and the possibility that optimizing the internal biochemical environment may influence disease susceptibility and recovery.
Some concepts once considered unconventional have moved increasingly into mainstream biomedical research.
Others remain controversial or incompletely tested.
That makes orthomolecular medicine an unusually interesting test case for medical artificial intelligence.
Consider what happens when a patient or physician asks an AI system about vitamin C, magnesium, vitamin D, insulin resistance, ketogenic nutrition, oxidative stress, or nutrient adequacy.
The AI may retrieve hundreds or thousands of publications.
But retrieval alone does not determine the answer.
The system still has to reason through questions such as:
What constitutes strong evidence?
Which evidence is relevant to the clinical question at hand?
Does absence of a large randomized trial establish absence of an effect?
How should mechanistic, observational, randomized, clinical, and safety evidence be weighed together?
How should biological plausibility influence interpretation without being allowed to substitute for clinical evidence?
How should an AI distinguish an interesting hypothesis from a clinical recommendation?
And when should it simply say:
“We do not know.”
Those are reasoning questions, not merely information-retrieval questions.
The danger of benchmarking AI only against consensus
Existing medical consensus is indispensable.
Clinical guidelines, accumulated evidence, professional experience, and established safety standards provide essential constraints on medical practice.
But consensus is not identical to scientific truth.
Medical history contains many examples in which widely accepted explanations changed as better evidence emerged.
A benchmark that defines correctness exclusively as agreement with current consensus therefore creates a subtle problem.
The more faithfully an AI reproduces existing medicine, the higher it may score—even in areas where the prevailing model is incomplete.
The opposite extreme is equally dangerous.
An AI should not treat every unconventional hypothesis as equally plausible simply because consensus can sometimes be wrong.
The appropriate goal is therefore neither blind conformity nor reflexive contrarianism.
It is disciplined scientific reasoning.
A capable medical AI should understand mainstream evidence, recognize why guidelines exist, respect strong safety boundaries, and simultaneously remain capable of examining competing hypotheses when the evidence justifies doing so.
This is also a test of orthomolecular medicine
The proposed benchmark should not become a mechanism for declaring orthomolecular medicine superior.
That would defeat its purpose.
Orthomolecular medicine should be subjected to the same scrutiny.
Where is the evidence strong?
Where is it preliminary?
Which claims are mechanistically plausible but clinically unproven?
Where may advocates have extrapolated beyond the available data?
Which interventions have favorable safety profiles?
Which require greater caution?
Which orthomolecular hypotheses have subsequently received support from broader biomedical research?
Which have not?
And which may ultimately prove incorrect?
A scientific framework becomes stronger when it is capable of identifying its own weaknesses.
This is why one principle of the proposed benchmark is particularly important:
A benchmark should be capable of producing an unfavorable result for the framework preferred by its own designers.
For future formal validation, this means using measures such as specification freeze before testing, explicit scoring criteria, independent external raters, adversarial cases, assessment of inter-rater reliability, transparent conflict-of-interest disclosure, and publication of unfavorable as well as favorable findings.
AI may make medical frameworks testable in new ways
Historically, comparing entire systems of medicine has been extraordinarily difficult.
Medical frameworks are large, complex, and often only partially formalized.
AI may change this.
A sufficiently capable model can potentially be presented with the same clinical scenarios while reasoning under different explicitly defined medical frameworks.
We can then examine how those frameworks affect:
-
diagnostic reasoning;
-
causal interpretation;
-
risk recognition;
-
prevention;
-
treatment options;
-
patient education;
-
handling of uncertainty;
-
and safety.
This could make previously vague philosophical disagreements more explicit and experimentally tractable.
Instead of arguing abstractly that one system is “holistic” while another is “reductionist,” we could ask different systems to analyze the same difficult cases and measure:
What does each notice?
What does each miss?
What evidence does each prioritize?
What risks does each create?
How does each handle uncertainty?
And where do their recommendations diverge?
That is one of the possibilities we hope the proposed benchmark will eventually make testable.
From artificial intelligence to intelligence-assisted medicine
The long-term significance may extend beyond AI benchmarking.
Today we commonly ask:
Can AI reproduce the performance of physicians?
Increasingly, we will ask:
Can AI augment physicians?
Eventually we may need to ask an even more consequential question:
Can advanced intelligence help medicine examine its own assumptions?
That requires AI capable of respecting evidence without becoming trapped by precedent; considering alternative hypotheses without abandoning scientific rigor; distinguishing hypothesis from established knowledge; and recognizing uncertainty rather than concealing it.
It also requires medical communities willing to subject their own frameworks to the same scrutiny they expect of competing systems.
Orthomolecular medicine should welcome that standard.
Join the discussion
Our preprint introduces the Dual-Layer Medical Benchmark and the initial Dual-Layer Medical Benchmark Specification v0.1.
Version 0.1 is an early methodological framework.
It has not been validated as a clinical benchmark and should not yet be used to establish the clinical superiority of any medical framework or AI system.
Its purpose at this stage is to invite criticism, refinement, and eventually empirical validation.
That is why we are opening this discussion on OMNS Global.
Among the questions worth examining:
What should a medical-system benchmark actually measure?
Should current medical consensus be treated as the reference standard, one reference standard, or one comparator among several?
How should mechanistic, observational, randomized, clinical, and safety evidence be integrated?
Can fundamentally different medical paradigms be compared fairly?
How can orthomolecular medicine subject its own claims to rigorous evaluation without predetermining the result?
How should conflicts of interest be handled when benchmark developers also develop medical or AI frameworks?
And what additional safeguards would make such a benchmark credible to researchers outside the orthomolecular community?
Clinicians, scientists, researchers, educators, and AI developers are invited to critique the proposal.
Disagreement is welcome.
The objective is not to construct a benchmark that confirms what we already believe.
It is to construct one capable of telling us when we are wrong.
Continue the discussion
Read the preprint → [LINK]
Review the Dual-Layer Medical Benchmark Specification v0.1 → [LINK]
Then share your critique, proposed modifications, or alternative approaches in the discussion below.
What would you require before trusting a benchmark that claims to compare different systems of medicine?