Assignment 1: Fine-tuning Vision Transformer (ViT)
with LoRA on CUB Dataset and Explainability Analysis
Deadline: August 30 2025, 23:59 IST
Objective
The goal of this assignment is to fine-tune a pre-trained Vision Transformer (ViT) on the CUB-200-2011 bird
dataset using parameter-efficient fine-tuning (LoRA). Specifically, you will:
1. Fine-tune only the final encoder layer of the ViT using LoRA.
2. Train a new classification head for the CUB dataset.
3. Evaluate the model on the test split.
4. Perform explainability analysis for selected samples using any one of:
• Attention Rollout
• Gradient Rollout
• ViT-CAM
Dataset
• CUB-200-2011 (Caltech-UCSD Birds 200 dataset)
• Contains 200 bird species with ∼6,000 training images and ∼5,800 test images.
• Dataset link: CUB-200-2011
Deliverables
You are required to submit the following:
1. Code implementation: Well-commented training and evaluation scripts.
2. Training curves: Plots of loss and accuracy vs. epochs.
3. Test accuracy report: Final model performance on the test split.
4. Explainability visualizations: For randomly selected test samples.
• Visualization code should be standalone.
• In simple terms, the code should be able to load the trained model and pick a random set of 5–6
examples and provide the visualization map.
5. Short discussion (1–2 pages):
• Comparison of LoRA fine-tuning with full fine-tuning (expected efficiency).
• Insights from the explainability maps.
1