Explainable AI (XAI) Metrics in Clinical
Decision Support Systems
Recent work emphasizes that high-stakes healthcare AI – e.g. disease diagnosis and
personalized treatment – demands explanations that clinicians can trust and act on.
Evaluating these explanations requires careful metrics. Three key dimensions are fidelity
(faithfulness of the explanation to the model), interpretability (ease of human
understanding), and actionability (whether the explanation guides meaningful clinical
action). In practice, many quantitative XAI metrics have been proposed, but few are tailored
to clinical use or validated with human experts. This report surveys current approaches in
healthcare XAI evaluation, highlights gaps (especially the lack of ground truth and need for
human-in-loop validation), and suggests ways to make future research more impactful.
Fidelity Metrics
Fidelity measures how well an explanation (often a simpler “surrogate” model) reproduces
the black-box model’s behavior. In healthcare, high fidelity is critical: an explanation that
misrepresents the model could mislead clinicians. Common fidelity metrics include:




Prediction Agreement: The fraction of instances on which the surrogate’s prediction
matches the original model’s predictionsalfordrepository.worktribe.comresearchgate.net. For example, Qureshi et al. formally
measure both score fidelity (probability outputs similarity) and decision fidelity (label
agreement) between XAI methods and the original modelresearchgate.net.
Feature Attribution Alignment: Correlation or rank agreement between feature
importances given by the explainer and ground-truth or baseline importances. This
can involve metrics like Spearman’s ρ or Kendall’s τ. For instance, one checklist
suggests measuring correctness by comparing local explanations to any available
reference (ground truth) for feature
relevancepmc.ncbi.nlm.nih.govpmc.ncbi.nlm.nih.gov.
Perturbation Tests: Removing or perturbing top-ranked features to see if model
output changes as expected (akin to “removal/insertion” metrics). These are often
used in XAI toolkits (e.g. Quantus) to test if explanation highlights truly important
featuresjmlr.org.
Ensemble Robustness: Quantifying how sensitive explanations are to randomness
(model retraining or explainer hyperparameters) – e.g. variance of explanations across
runs. This aligns with the CLIX checklist’s consistency item, which recommends
measuring agreement of feature sign or rank under different XAI
designspmc.ncbi.nlm.nih.govpmc.ncbi.nlm.nih.gov.
In summary, fidelity is typically quantified by how closely explanations mimic the black-box.
Many toolkits (e.g. Quantusjmlr.org) implement dozens of such metrics (faithfulness,
localization, etc.), and recent studies have standardized measures like Qureshi et al.’s set of
six metrics (including fidelity, robustness, simplicity, etc.) for healthcare tabular
dataresearchgate.netresearchgate.net.
Interpretability Metrics
Interpretability refers to how easily a human (clinician) can understand the explanation.
This is inherently subjective, but some proxies are used:




Complexity / Simplicity: Number of features or rules in the explanation, depth of a
surrogate tree, length of textual explanation, etc. Lower complexity generally implies
higher interpretabilitysalford-repository.worktribe.comfrontiersin.org. For example,
Qureshi et al. measure simplicity as the average number of features used by each XAI
methodresearchgate.net. The Salford survey similarly notes using tree depth or
number of leaf nodes as a proxy for complexitysalford-repository.worktribe.com.
Cognitive Load: Though hard to measure directly, some works use user studies: e.g.
ask clinicians whether an explanation is “clear” or “too complex” (Likert scales).
CLIX suggests rating reasonableness and domain relevance of explanations on 4point scales as judged by expertspmc.ncbi.nlm.nih.gov.
Human Predictability Tests: Asking users to predict the model’s output on new
cases given the explanation – high accuracy suggests better understanding (this
bridges interpretability and fidelity).
Coverage/Breadth: How many features or cases the explanation addresses. Some
define broadness as part of interpretability (a very narrow or very broad explanation
might be harder to grasp).
In practice, no single numerical interpretability metric is standard in healthcare XAI. Instead,
frameworks emphasize human judgments. For instance, Lopes et al. (2022) acknowledge
interpretability is subjective but suggest computational proxies like explanation size or
complexity as stand-insfrontiersin.org. The lack of a ground-truth “correct” explanation
means interpretability often must be gauged via human studies, which we discuss below.
Actionability and Usability
Actionability concerns whether explanations enable clinicians to make safe, informed
decisions. Actionable explanations highlight modifiable risk factors or treatment levers.
Recent work stresses actionability as a key criterion in clinical AI. For example, Cabitza et al.
define actionability as the ability of an explanation to support downstream clinical decisionmaking – i.e. to guide contextually appropriate actionspmc.ncbi.nlm.nih.gov. In their
checklist, explanations are rated by clinicians on whether they are “informative and impactful
– guiding clinical workflows or suggesting actions”pmc.ncbi.nlm.nih.gov. Notably, they
point out that highlighting non-modifiable features (e.g. patient age, length of stay) may be
plausible but not actionable, since clinicians cannot change thempmc.ncbi.nlm.nih.gov.
Quantitatively, actionability is rarely captured by standard XAI metrics. One proposed
measure (“Pragmatism”) computes the cost for a patient to reach a counterfactual outcome:
features that cannot feasibly change (e.g. age) get infinite cost, rendering such counterfactuals
non-actionablearxiv.org. In practice, most evaluations default to user ratings: e.g. having
clinicians score whether the recommended changes are practical. CLIX formalizes this by
asking users to rate actionability on a Likert scalepmc.ncbi.nlm.nih.gov. In summary, current
actionability evaluation relies heavily on domain experts: either interactive rating (human-inloop) or rule-based checks (e.g. discarding immovable features). There is no widely used
standalone metric for actionability in healthcare XAI.
Quantitative Evaluation Frameworks
Several recent frameworks aggregate these metrics and guide evaluation:




Quantus (2023) is a comprehensive Python toolkit for XAI evaluationjmlr.org. It
implements 30+ automated metrics organized into categories (faithfulness,
robustness, localization, complexity, etc.). It highlights the need for standardized,
systematic evaluation, noting that many papers rely on one-off testsjmlr.org. While
Quantus is domain-agnostic, its faithfulness and simplicity metrics are directly
applicable to clinical models.
Qureshi et al.’s XAI_Tab_Health (2025) implements six “well-established” metrics
(fidelity, simplicity, consistency, robustness, precision, coverage) for tabular
healthcare dataresearchgate.net. They formalize these metrics and open-source them,
allowing benchmarking of methods like LIME, SHAP, etc. This framework confirms
known trade-offs (e.g. SHAP gives perfect fidelity for trees, LIME is simpler but
lower fidelity) and provides a reproducible pipeline for healthcare XAI
evaluationresearchgate.netresearchgate.net.
Nauta et al. (2022) provide a taxonomy (Co-12) of XAI evaluation properties and
discuss a variety of metrics. They classify fidelity, completeness, consistency,
continuity, contrastivity, compactness, confidence, etc., and enumerate automated
evaluation methods for eachfrontiersin.org. Notably, they define Pragmatism
(usability of counterfactuals) and Perceptual Realism (how “realistic” explanations
look to humans) as specialized metricsarxiv.org. This work underlines that many
quantitative metrics exist, but few are specifically validated with clinicians.
CLIX (Cabitza et al., 2025) is a clinician-informed checklist for XAI in
CDSSpmc.ncbi.nlm.nih.govpmc.ncbi.nlm.nih.gov. It defines 14 items spanning
clinical (domain relevance, plausibility, actionability) and technical attributes
(fidelity, confidence, consistency, bias). Each item comes with suggested evaluations:
e.g., domain relevance and reasonableness are to be rated by experts on a 4-point
scalepmc.ncbi.nlm.nih.gov; correctness (akin to fidelity) is to be computed “as for
model accuracy” against any available ground-truth
explanationpmc.ncbi.nlm.nih.govpmc.ncbi.nlm.nih.gov. CLIX is one of the first
structured efforts to standardize XAI reporting in healthcare, emphasizing mixed
(quantitative + qualitative) assessmentpmc.ncbi.nlm.nih.govpmc.ncbi.nlm.nih.gov.
These frameworks share common themes: fidelity and stability are quantified by comparing
explanations to model behavior (surrogate accuracy, variance under perturbation), whereas
interpretability and actionability are mostly assessed by expert judgment. In practice, many
studies use a battery of such metrics. For example, Vashishth et al. (2024) recently evaluated
5 XAI methods using fidelity, simplicity (features count), consistency, robustness and
coverageresearchgate.net. The lack of a single gold-standard means researchers often choose
metrics based on their priorities: e.g. emphasizing trust (fidelity, confidence) vs. ease-of-use
(simplicity, actionability).
Human-in-the-Loop Evaluation
Because healthcare explanations must serve clinicians, human-centered evaluation is
crucial. However, user studies remain rare due to cost and complexity. Surveys of XAI
research find only ~20–25% of papers include any human study, and fewer with domain
expertsarxiv.orgfrontiersin.org. When human evaluation is used, it typically involves
questionnaires on trust, satisfaction, or understanding.




Expert Ratings: As in CLIX, clinicians may rate explanations for realism, usefulness,
safety, etc. This captures reasonableness, coherence with medical knowledge, and
actionabilitypmc.ncbi.nlm.nih.govpmc.ncbi.nlm.nih.gov. Quantifying these ratings
(e.g. via Likert scales) is subjective but directly reflects clinical value.
Task-based Studies: Some works give clinicians tasks (e.g. diagnosis or risk
assessment) with and without explanations to see if performance improves. For
instance, a recent AAAI study let neurosurgeons interpret XAI outputs on multimodal
imagingfrontiersin.org. Such user studies assess whether explanations actually help
decision-making, addressing “human-grounded” metrics like decision accuracy or
time.
Simulated Users: Nauta et al. describe “simulated user studies” where one assumes
certain features are “untrustworthy” and checks if XAI methods steer models away
from themarxiv.org. These automate aspects of human preference but rely on arbitrary
definitions.
Human-aligned Metrics: In other domains (e.g. computer vision), efforts like
PASTA build datasets of human preference (e.g. annotated saliency maps) and train a
metric to mimic human judgmentarxiv.org. While PASTA focuses on visual
explanations, it demonstrates that automated metrics often poorly correlate with real
usersarxiv.org, suggesting similar approaches could benefit healthcare XAI.
Overall, the consensus is that human-in-the-loop evaluation is essential but
underdeveloped. High-level frameworks (Nauta 2022, Frontiers 2024) note that trust, mental
models, and satisfaction are often left unevaluated or inconsistently measuredfrontiersin.org
【33†】. There is a clear gap: few standardized protocols exist for clinician studies of XAI,
and findings (when collected) are hard to compare across papers.
Datasets and Benchmark Tasks
Explainability research in healthcare typically repurposes public clinical datasets used for
prediction tasks. Common examples include:



EHR/ICU Data: MIMIC-III/IV (critical care EHRs) and eICU-CRD (multi-center
ICU) are widely used. For instance, predicting ICU mortality or sepsis onset on
MIMIC-III is a common benchmark, after which methods like SHAP or attention are
applied. Li et al. (2023) built a multimodal mortality prediction using MIMIC-III
(clinical notes, vitals, events) and developed an explanation module over itarxiv.org.
These datasets allow evaluation of XAI on tabular/time-series data with clinical
context. Other EHR sources include large hospital registries and biobanks, though
most XAI studies cite MIMIC or eICU when disclosing their code.
Medical Imaging: Public imaging sets (e.g. MIMIC-CXR for chest X-rays, CheXpert,
NIH ChestX-ray) are used for disease classification (pneumonia, COVID, etc.) and
then for applying saliency map or localization explanations. For example, studies
often measure the overlap of a saliency map with annotated lesions (e.g. Intersectionover-Union) as a quantitative metricpmc.ncbi.nlm.nih.gov. Besides radiology,
dermatology (ISIC skin lesion) and pathology (CAMELYON for cancer metastasis)
datasets serve similar roles.
Omics and Genomics: While less common in classic CDSS, some works use gene
expression or proteomic datasets (e.g. TCGA cancer genomics) to explain disease
subtyping. These typically adapt XAI attributions to features (genes) and interpret
them via pathway analysis.
Benchmark setups usually follow this pipeline: (1) Model training on a predictive task (often
deep learning or ensemble models achieving high accuracy); (2) Apply XAI methods
(SHAP, LIME, attention, counterfactuals, etc.) to the trained model; (3) Evaluate
explanations either by computational metrics (as above) or by human review. For tabular
tasks, frameworks like Qureshi’s test multiple explainers on the same set of features across
MIMIC-derived tasksresearchgate.net. For imaging, cross-validation and saliency overlap
tests are common. Importantly, there is no fixed “explainability benchmark dataset” akin to
ImageNet: researchers often reuse whatever clinical data they have, making direct
comparison of XAI methods difficult.
Limitations and Gaps
Despite extensive work, critical gaps remain in XAI evaluation for healthcare:





No Ground Truth Explanations: By definition, we usually lack a “correct”
explanation. Unlike labeled images or symptoms, we cannot query an oracle for why
the model predicted cardiovascular risk for Patient A. This absence makes fidelity the
closest proxy: we check consistency with the model, not with reality. The ICML
position paper “Bridging the Gap in XAI” explicitly notes that “the absence of a
ground truth for explanations” is a central issuearxiv.org. Without ground truth,
metrics like correctness or completeness are only relative to the model. This leaves
actionability and clinical validity largely subjective or unmeasured.
Subjectivity of Interpretability and Actionability: Interpretability is inherently a
human judgment. Even if a surrogate is mathematically simple, it may not map to
medical reasoning. Thus, proxy metrics (tree depth, feature count) can mislead: e.g. a
short rule list might still violate domain logic. Similarly, an actionability score
(Pragmatism) depends on domain knowledge of what patients can changearxiv.org.
Current quantitative metrics rarely incorporate clinician input, except via separate
user studies. This is a gap: most evaluation is model-centric, not user-centric.
Lack of Standardization: As several reviews note, the XAI field lacks agreed
benchmarks or protocolsjmlr.orgfrontiersin.org. Different papers pick different
metrics or question formats, so results are hard to compare. For example, the Frontiers
survey shows many ad-hoc questionnaires in user studies, with little reuse of standard
scalesfrontiersin.org. Without consensus, researchers may unknowingly “cherry-pick”
metrics that favor their method.
Trade-offs and Gaming: Many evaluation methods can be gamed. High fidelity can
come at the cost of very complex explanations (low interpretability)researchgate.net.
Conversely, some metrics (like localization IoU) can be misleading if the “ground
truth” region is itself uncertain. The Quantus paper warns that one-sided, qualitative
tests have led to questionable conclusions and calls for cautionjmlr.org. Moreover,
certain metrics (e.g. saliency map confidence) may be unstable to noise, raising
robustness concerns.
Under-evaluated Properties: CLIX and other analyses highlight factors like
confidence of explanations, causal validity, and narrative coherence that are rarely
quantitatively measuredpmc.ncbi.nlm.nih.govpmc.ncbi.nlm.nih.gov. Fairness and
bias-checking via explanations are mentioned, but systematic metrics in healthcare are
nascent (CLIX Item 11 suggests using global explanations to spot

biasespmc.ncbi.nlm.nih.gov). In sum, many “softer” goals (safety, trust, adherence to
clinical guidelines) are acknowledged but lack clear metrics.
Data and Domain Gaps: Public datasets like MIMIC focus on acute ICU care. XAI
for chronic disease, outpatient settings, or underrepresented populations is less
studied. Few benchmarks for personalized treatment planning (e.g. oncology
regimens) exist. Datasets that include explicit causal mechanisms (for creating
synthetic “ground truth” explanations) are virtually absent in healthcare, unlike some
computer vision tasks.
Recommendations and Novel Directions
Given these gaps, a dissertation aiming for impact could explore several promising
directions:





Human-Grounded Metrics: Develop and validate metrics that incorporate clinician
feedback. For example, create a clinical explanation benchmark where experts
annotate a subset of cases with “key drivers” of outcome. One could then measure
how well model explanations align with these expert-identified features (e.g. rank
correlation). This injects some ground truth into fidelity/correctness. Alternatively,
adapt PASTA’s idea by crowdsourcing ratings of explanation pairs (e.g. which of two
saliency maps better matches physician intuition) to train a “human-aligned” scoring
function for healthcare.
Domain-Specific Pragmatism: Extend the “Pragmatism” concept for counterfactuals
into healthcare-specific actionability. For instance, assign costs to clinical
interventions (e.g. changing blood pressure via medication is “cheap”, altering age is
impossible) and quantify the effort a patient/doctor would need to enact an
explanation’s suggestion. This metric would penalize explanations that rely on nonmodifiable factors. Empirical studies could compare different approaches (e.g. SHAP
vs. counterfactuals) on pragmatic scores in tasks like blood glucose control or fall-risk
mitigation.
Composite Utility Scores: Propose an overall explainability score that weights
fidelity, interpretability, and actionability. For example, one might define “Clinical
Explainability” = α·(fidelity) + β·(simplicity) + γ·(actionability) (normalized). The
weights α,β,γ could be chosen based on clinician surveys about priorities. This kind of
composite metric could guide model selection (e.g. prefer slightly lower accuracy if it
gains much higher actionability). Such a metric would need user studies to calibrate
the weights, which would itself be a valuable contribution.
Interactive Explainability Tools: Build prototypes where clinicians can query the
model (human-in-loop) and the system learns from feedback. For instance, if a doctor
marks a proposed explanation as unreasonable, the system could adjust or present
alternatives. Evaluating such an interactive XAI (vs. static methods) on clinical
workflows could be novel. This addresses the need to close the loop between
automated explanations and human insight, which current metrics overlook.
Benchmark Suite Creation: Assemble a standardized suite of healthcare XAI tasks
and evaluation scripts (analogous to vision’s PASCAL or NLP GLUE). This could
include several public datasets (MIMIC mortality, CheXpert pneumonia, etc.) with
predefined tasks, models, and metrics. Publishing code and leaderboards would
encourage comparability. Including both computational metrics (via Quantus) and
protocols for small user studies (e.g. checklists like CLIX) would make it
comprehensive.


Causal/Counterfactual Explanations: Investigate metrics that capture causal
validity. For example, check if feature importances agree with known medical causal
graphs. One could use synthetic patient data where causal relationships are controlled,
to test XAI methods’ ability to identify true causes. Incorporating causal knowledge
into metrics (e.g. causal fidelity vs. correlational fidelity) is relatively unexplored in
healthcare XAI.
Contextual and Personalized Evaluation: Design experiments where XAI is
evaluated in situ. For instance, measure how explanations affect junior clinicians’
decisions on real patient cases (beyond synthetic tasks). Or personalize metrics by
patient subgroups: an explanation might be more or less useful depending on
comorbidities. Dissertation work could include pilot user studies in a specific domain
(e.g. ICU monitoring) to derive practical guidelines for explanation design.
In making a dissertation unique, emphasize a clinically-informed evaluation of XAI. While
many projects test new explainers, fewer rigorously define how to judge them in the real
world. Combining quantitative and qualitative methods will stand out. For example, one
could (a) implement a suite of existing metrics (fidelity, complexity, Deletion/Insertion,
Pragmatism) on a clinical task, (b) conduct clinician ratings for the same cases, and (c)
analyze how well the automated metrics correlate with expert judgments. Such a study would
highlight which metrics truly reflect clinical relevance (addressing the “no-ground-truth”
challenge) and may reveal blind spots.
Additionally, leveraging regulatory context (e.g. EU AI Act’s requirement for transparency)
could justify metrics like explanation completeness or documentation quality as practical
considerations. Aligning research with emerging standards (CLIX, XAI Manifestos) will
make it timely.
Finally, focusing on actionability and trust (often cited but rarely quantified) in a specific
decision support scenario could be impactful. For instance, in personalized treatment
planning (oncology or ICU), developing XAI that suggests adjustable factors (e.g. medication
dosages) and measuring how those suggestions help doctors might open new directions.
In summary, current XAI evaluation in healthcare uses a mix of fidelity-based metrics
(automated, model-centric) and ad-hoc human assessments. There is growing recognition of
the need for human-in-loop validation (CLIX, PASTA) and domain-aware metrics, but
standard solutions are lackingarxiv.orgpmc.ncbi.nlm.nih.gov. A dissertation that
systematically bridges this gap – by introducing validated, clinician-aligned metrics and
benchmarks – would make a novel and valuable contribution to making AI explanations truly
usable in clinical practice.
Sources: Recent literature on XAI in medicine and ML (surveys, toolkits, and conferences)
has been cited throughout. Key references include a 2024 healthcare XAI surveysalfordrepository.worktribe.comsalford-repository.worktribe.com, the CLIX clinical XAI
checklistpmc.ncbi.nlm.nih.govpmc.ncbi.nlm.nih.gov, the Quantus evaluation toolkitjmlr.org,
and other peer-reviewed studies and
preprintsresearchgate.netresearchgate.netarxiv.orgarxiv.org that inform these conclusions.