Project Report
On
Transformer Based Unsupervised Domain Adaptation in Medical
Images in a Federated Setup
Submitted in partial fulfillment of the requirements
for the award of the degree of
BACHELOR OF TECHNOLOGY
IN
INFORMATION TECHNOLOGY
Submitted by
ASRAR UL HAQ
2020BITE092
Project Supervisor
Dr Janib Ul Bashir
Department of Information Technology
National Institute of Technology Srinagar, J&K
CERTIFICATE
This is to certify that the Project titled Transformer Based Unsupervised Domain Adaptation in
Medical Images in a Federated Setup has been presented by Asrar Ul Haq (2020BITE092) in
partial fulfillment of the requirements for the award of the degree of Bachelor of Technology in
Information Technology.
Dr Janib Ul Bashir
Project Supervisor
Department of Information Technology
NIT Srinagar, J&K
Transformer Based Unsupervised Domain Adaptation in Medical Images in a Federated Setup - 2024
STUDENT DECLARATION
I, Asrar Ul Haq Enrollment No. 2020BITE092, hereby declare the work,
which is being presented in the Project Report, entitled “Transformer Based
Unsupervised Domain Adaptation in Medical Images in a Federated
Setup” in the partial fulfilment of the requirements for the award of the
degree of “Bachelor of Technology in Information Technology and
Engineering”, is an authentic record of my own work.
The matter embodied in this report has not been submitted by me for the
award of any other degree.
Date: 19th Feb,2024
Asrar Ul Haq
Enroll: 2020BITE092
B. Tech
Department of Information Technology
and Engineering
Department of Information Technology, NIT Srinagar
Page 1
Transformer Based Unsupervised Domain Adaptation in Medical Images in a Federated Setup - 2024
ACKNOWLEDGEMENT
We are filled with gratitude to all those who helped us in completion of the
report. The most pleasant point of presenting a report is the opportunity to
thank those who have contributed to it. The list of all those to whom we
extend our sincere thanks and acknowledgement is too long. Indeed, this
page of acknowledgement shall never be able to touch the horizons of
generosity of those who tendered their help to us.
Date: 19th Feb,2024
Asrar Ul Haq
Enroll: 2020BITE092
B. Tech
Department of Information Technology
and Engineering
Department of Information Technology, NIT Srinagar
Page 2
Transformer Based Unsupervised Domain Adaptation in Medical Images in a Federated Setup - 2024
ABSTRACT
Medical image analysis plays a crucial role in modern healthcare, aiding in diagnosis,
treatment planning, and monitoring of diseases. However, deploying machine learning
models across different medical institutions is challenging due to variations in imaging
protocols and equipment. Unsupervised domain adaptation (UDA) techniques aim to
mitigate this challenge by transferring knowledge from a labeled source domain to an
unlabeled target domain. In this study, we propose a novel approach leveraging
transformer-based architectures for UDA in medical images within a federated learning
framework.
In this project, we propose a novel approach for Transformer-Based Unsupervised
Domain Adaptation in Medical Images within a federated setup. Leveraging the power of
transformers, we address the challenge of adapting deep learning models to diverse
medical image datasets collected from different sources without compromising data
privacy. The federated setup ensures data security by distributing data across different
locations or organizations, allowing model training without centralizing all data in one
place. Our method aims to derive domain-dependent latent representations that capture
both domain-specific characteristics and globally shared features across medical image
datasets. By utilizing hierarchical Bayesian priors and advanced adaptation techniques,
we enable effective knowledge transfer and adaptation to new datasets while preserving
patient privacy. Through extensive experiments and evaluations, we demonstrate the
effectiveness and robustness of our proposed approach, highlighting its potential for
improving the generalizability and reliability of AI systems in medical imaging across
diverse clinical settings.
Department of Information Technology, NIT Srinagar
Page 3
Transformer Based Unsupervised Domain Adaptation in Medical Images in a Federated Setup - 2024
CONTENTS
Student Declaration……………………………………………………….1
Acknowledgement……………………………………………………...…2
Abstract……………………………………………………………………3
Introduction -----------------------------------------------------------------------5
Motivation -------------------------------------------------------------------------6
Problem Statement --------------------------------------------------------------7
Domain Adaptation in federated Learning ---------------------------------8
Types Of Domain Adaptation ------------------------------------------------10
 Based on labeling data
 Based on Domain Divergence
 Based on how Domain Adaptation is Achieved
Methodology ---------------------------------------------------------------------13
Future Work ---------------------------------------------------------------------14
References……………………………………………………………….15
Department of Information Technology, NIT Srinagar
Page 4
Transformer Based Unsupervised Domain Adaptation in Medical Images in a Federated Setup - 2024
1. INTRODUCTION
In recent years, the application of deep learning in medical image analysis has
shown great promise in revolutionizing healthcare diagnostics and treatment planning.
However, one of the significant challenges faced in this domain is the ability to
generalize models trained on one dataset to perform well on data from different
sources or domains. This challenge is exacerbated by concerns surrounding data privacy
and the need to comply with stringent regulations governing the handling of medical
data.
To address these challenges, this project focuses on the integration of TransformerBased Unsupervised Domain Adaptation within a federated setup. Before delving into
the specifics of our approach, let us first understand the key concepts involved: Domain
Adaptation (DA), transformers, and federated learning (FedTP).
Domain Adaptation (DA) refers to the process of adapting a machine learning model
trained on a source domain to perform well on a target domain, where the source and
target domains may exhibit differences in data distribution. In the context of medical
image analysis, DA becomes crucial when deploying models trained on data from one
healthcare institution to new datasets collected from different institutions, which may
have variations in imaging protocols, equipment, or patient demographics.
Transformers have emerged as a powerful architecture in the field of natural language
processing (NLP) for their ability to capture long-range dependencies and semantic
relationships within sequences of data. In recent years, transformers have been
successfully applied to various computer vision tasks, including medical image analysis,
due to their effectiveness in handling complex spatial relationships and capturing
contextual information from images.
Federated learning (FedTP) is a distributed machine learning approach that enables
model training across multiple decentralized data sources while keeping the raw data
localized. This decentralized training process preserves data privacy and confidentiality,
making it particularly suitable for sensitive domains such as healthcare. By federating
model training across different healthcare institutions, federated learning allows for
collaborative model development without the need to centralize patient data, thereby
addressing privacy concerns and regulatory requirements.
Department of Information Technology, NIT Srinagar
Page 5
Transformer Based Unsupervised Domain Adaptation in Medical Images in a Federated Setup - 2024
2. MOTIVATION
The motivation behind this project is influenced by both implicit and explicit processes,
with two major drivers guiding our efforts:
1. Privacy-Focused Federated Learning (FedTP):
In response to growing concerns regarding data privacy, our project embraces a
federated learning approach. By distributing model training across multiple decentralized
data sources, federated learning ensures that sensitive medical data remains localized and
secure. This approach addresses privacy concerns and regulatory requirements, providing
a robust framework for collaborative model development in healthcare without
compromising patient privacy.
2. Unannotated Data Domain Adaptation (DA):
- Limited availability of labeled medical data poses a significant challenge in model
training. To overcome this hurdle, our project leverages domain adaptation techniques to
effectively train models on unannotated data. By adapting models to new datasets without
the need for extensive labeled data, we enhance the scalability and generalizability of AI
systems in medical imaging. This approach empowers healthcare institutions to leverage
their existing data resources more efficiently, facilitating advancements in diagnostic
accuracy and patient care.
Department of Information Technology, NIT Srinagar
Page 6
Transformer Based Unsupervised Domain Adaptation in Medical Images in a Federated Setup - 2024
3. PROBLEM STATEMENT
Current medical image analysis models face challenges due to domain shift, impacting
their accuracy and fairness across diverse datasets. While federated learning offers
privacy-preserving solutions, data-sharing across institutions introduces new domain shift
challenges.
Objective:
To address these challenges, our objective is to develop a Transformer-based
Unsupervised Domain Adaptation (UDA) framework with built-in bias mitigation. This
framework aims to:
1. Extract robust features across diverse medical image domains.
2. Identify and adjust for potential biases in the data.
3. Improve accuracy, fairness, security, and generalizability of medical image analysis
models.
Department of Information Technology, NIT Srinagar
Page 7
Transformer Based Unsupervised Domain Adaptation in Medical Images in a Federated Setup - 2024
4. Domain Adaptation in Federated Learning
Domain adaptation in federated learning refers to the process of adjusting a machine
learning model trained on data from one source to perform well on data from another
source within a federated setup.
In simpler terms, imagine you have different hospitals, each with its own patient data.
With domain adaptation in federated learning, you can train a model on data from one
hospital and then adapt it to work well with data from other hospitals, even though the
data might be slightly different due to factors like patient demographics or imaging
equipment.
This adaptation process helps ensure that the model can effectively generalize to new
data sources while preserving patient privacy and confidentiality, which is crucial in
healthcare settings.
Figure 1: Domain Adaptation In Federated Learning
Department of Information Technology, NIT Srinagar
Page 8
Transformer Based Unsupervised Domain Adaptation in Medical Images in a Federated Setup - 2024
Before diving in, let us quickly go through some of the most important concepts regarding
domain adaptation. For this, let us use an example scenario: a classification model trained on
photos captured by a mobile phone, and this model is used to classify images on images
captured by a DSLR camera.
Source Domain: This is the data distribution on which the model is trained using labeled
examples. In the example above, the dataset created by the cellphone photos is the source
domain.
Target Domain: This is the data distribution on which a model pre-trained on a different
domain is used to perform a similar task. The target domain is the dataset generated using the
photos using the DSLR camera in the example above.
Domain Translation: Domain Translation is the problem of finding a meaningful
correspondence between two domains.
Domain Shift: A domain shift is a change in the statistical distribution of data between the
different domains (like the training, validation, and test sets) for a model.
Department of Information Technology, NIT Srinagar
Page 9
Transformer Based Unsupervised Domain Adaptation in Medical Images in a Federated Setup - 2024
5. Types of Domain Adaptation
5.1 Based up on Labeling data set:

Supervised: The target domain data is fully annotated in Supervised Domain
Adaptation (SDA). Unsupervised Domain Adaptation expects large amounts of target
data to be effective, and this is emphasized even more when using deep models.
However, SDA can function optimally even without such vast amounts of target
domain training data, labeling which is likely not very expensive.

Semi Supervised: In Semi-Supervised Domain Adaptation (SSDA), only a few data
samples in the target domain are labeled. Each class weight vector is an estimated
“prototype” that can be regarded as a representative point of that class.

Weakly Supervised: Weakly Supervised Domain Adaptation (WSDA) refers to a
problem setting wherein only “weak labels” are available in the target domain. For
example, in a semantic segmentation domain adaptation problem, that is, ground
truth masks are unavailable in the target domain, but the categories of the objects to
be segmented are available.

Unsupervised: In Unsupervised Domain Adaptation (UDA), any kind of labels
(weak/hard) for the target domain data are entirely missing. A model trained on
source domain data must adapt to the target domain independently.
Figure 2: RTN used in Unsupervised Approach
Department of Information Technology, NIT Srinagar
Page 10
Transformer Based Unsupervised Domain Adaptation in Medical Images in a Federated Setup - 2024
5.2 Based on Domain Divergence

Homogenous: Homogeneous DA refers to a problem where the feature spaces of
the source and target domains are identical with identical dimensionality, and the
difference lies in only the data distribution. Homogeneous DA considers that source
and target domain data are collected using the same type of features, that is, crossdomain data are observed in the same feature space but exhibit different
distributions. Thus, this is also called a “distribution-shift” type Domain Adaptation
problem.

Heterogenous: In Heterogeneous DA problems, the source and target domains are
non-equivalent and might have different feature space dimensionality. In
heterogeneous DA, cross-domain data are described by different types of features
and thus exhibit distinct distributions (for example, training and test image data with
different resolutions or encoded by different codebooks). It is thus also known as a
“feature space difference” type DA problem and is a much more challenging problem
than Homogeneous DA.
Figure 3: CDLS used in Heterogenous Domain
Department of Information Technology, NIT Srinagar
Page 11
Transformer Based Unsupervised Domain Adaptation in Medical Images in a Federated Setup - 2024
5.3 Based on Adaptation techniques

One Step DA and Multi Step DA
The final form of categorization of Domain Adaptation techniques is based on how the
domain adaptation is achieved: most DA settings assume that the source and target domains
are directly related; thus, transferring knowledge can be accomplished in one step. We call
them One-Step DA.
In reality, however, this assumption is occasionally unavailable. There is little overlap
between the two domains, and performing One-Step DA will not be effective. Fortunately,
there are some intermediate domains that are able to draw the source and target domains
closer than their original distance. Thus, we use a series of intermediate bridges to connect
two seemingly unrelated domains and then perform One-Step DA via this bridge, named
multi-step (or transitive) DA.
For example, face and vehicle images are dissimilar due to different shapes or other aspects,
and thus, one-step DA would fail. However, some intermediate images, such as “football
helmet,” can be introduced to be an intermediate domain and have a smooth knowledge
transfer.
Figure 4: One step DA and Multi Step DA
Department of Information Technology, NIT Srinagar
Page 12
Transformer Based Unsupervised Domain Adaptation in Medical Images in a Federated Setup - 2024
6. Methodology
In our proposed approach for federated domain adaptation, we aim to align the
representations learned among different nodes with the data distribution of the target
node. To achieve this, we extend adversarial adaptation techniques to suit the constraints
of the federated setting.
1. Adversarial Adaptation Extension:
We will extend existing adversarial adaptation techniques to adapt to the federated setup.
This involves modifying the adversarial training process to align the representations
learned by each node's model with the data distribution of the target node while
preserving privacy and data locality.
2. Dynamic Attention Mechanism:
We will devise a dynamic attention mechanism to enhance knowledge transfer in the
federated domain adaptation process. This mechanism will allow the model to
dynamically adjust its focus on different features or regions of the data during training,
thereby improving adaptation performance across diverse datasets.
3. Feature Disentanglement:
Leveraging feature disentanglement techniques, we aim to disentangle the underlying
factors of variation present in the data. By separating out domain-specific and domaininvariant features, we can enhance the transferability of learned representations across
different nodes in the federated setup.
4. Empirical Evaluation:
We will conduct extensive experiments on several image and text datasets to empirically
evaluate the effectiveness of our proposed methodology. This includes benchmarking our
approach against existing methods and assessing its performance in terms of accuracy,
fairness, and generalization across different domains.
Department of Information Technology, NIT Srinagar
Page 13
Transformer Based Unsupervised Domain Adaptation in Medical Images in a Federated Setup - 2024
7: Future Work:
1. Problem Identification:
Identify and address specific challenges related to domain shift and image variability in
medical imaging datasets. This involves understanding the sources of domain shift and
variability and developing targeted solutions to mitigate their impact on model
performance.
2. Model Optimization:
Focus on further optimizing transformer-based models for unsupervised domain
adaptation in medical imaging. This includes exploring novel architectures, fine-tuning
hyperparameters, and investigating pre-training strategies tailored to the unique
characteristics of medical image data.
3. Ablation Studies on Existing Approaches:
Conduct comprehensive ablation studies on existing techniques to identify the most
crucial components contributing to model performance. By systematically analyzing
different aspects of the model, we can gain insights into its behavior and identify areas
for improvement.
4. Diverse Data Acquisition:
Explore strategies for acquiring diverse medical datasets from various sources and patient
demographics. This involves collaborating with healthcare institutions to collect data
representing a wide range of medical conditions, imaging modalities, and patient
populations.
5. Ablation Studies:
Conduct detailed analysis to understand the impact of different model components on
performance and behavior. By systematically removing or modifying specific
components of the model, we can evaluate their individual contributions and assess their
importance in achieving desired outcomes.
6. Benchmarking against State-of-the-Art (SOTA):
Continuously benchmark against state-of-the-art methods to ensure competitiveness and
stay abreast of advancements in the field. By comparing our approach to existing
techniques on standardized datasets and evaluation metrics, we can validate its
effectiveness and identify areas for further improvement.
Department of Information Technology, NIT Srinagar
Page 14
Transformer Based Unsupervised Domain Adaptation in Medical Images in a Federated Setup - 2024
REFERENCES
[1] FedTP: Federated Learning by Transformer Personalization Hongxia Li, Zhongyi
Cai, Jingya Wang, Jiangnan Tang, Weiping Ding, Chin-Teng Lin, and Ye Shi
[2] FEDERATED UNSUPERVISED DOMAIN ADAPTATION FOR FACE
RECOGNITION Weiming Zhuang , Xin Gan , Xuesen Zhang , Yonggang Wen , Shuai
Zhang , Shuai Yi
[3] Transformers in Medical Imaging: A Survey Fahad Shamshad, Salman Khan, Syed
Waqas Zamir, Muhammad Haris Khan, Munawar Hayat, Fahad Shahbaz Khan, and
Huazhu Fu
[4] Federated Learning with Dynamic Transformer for Text to Speech Zhenhou Hong,
Jianzong Wang*, Xiaoyang Qu, Jie Liu, Chendong Zhao, Jing Xiao Ping An Technology
(Shenzhen) Co., Ltd.
[5] OnDev-LCT: On-Device Lightweight Convolutional Transformers towards federated
learning Chu Myaet Thwal a , Minh N.H. Nguyen b , Ye Lin Tun a , Seong Tae Kim a ,
My T. Thai c , Choong Seon Hong a,
[6] FEDERATED ADVERSARIAL DOMAIN ADAPTATION
[7] Structure-preserving image translation for multi-source medical image domain
adaptation
[8] AnoFed: Adaptive anomaly detection for digital health using transformer-based
federated learning and support vector data description
[9] Deep unsupervised domain adaptation: A review of recent advances and perspectives
[10] Federated model aggregation via self-supervised priors for highly imbalanced
medical image classification
Department of Information Technology, NIT Srinagar
Page 15