A FRAMEWORK FOR COMPARING PERFORMANCE OF THREE MACHINE
LEARNING MODELS ON IMAGING MODALITIES TO DETECT PNEUMONIA
GROUP MEMBERS
S/N STUDENT NAME
REGISTRATION NUMBER
POGRAM
1
SAIDI ALLY ATHUMANI
T21-03-03113
BSC-HIS
2
DAVID DAUSON RUTALOMBA T21-03-12925
BSC-HIS
3
BARAKA GODFREY GISHA
T21-03-05608
BSC-HIS
4
PATRICK RICHARD
T21-03-13227
BSC-HIS
5
NEEMA JOSHUA
T21-03-10273
BSC-HIS
Table of Contents
1. Introduction ............................................................................................................................................... 1
1.1 Background of the Study .................................................................................................................... 1
1.2 Problem Statement .............................................................................................................................. 1
1.3 Objectives ........................................................................................................................................... 1
1.3.1 General Objective ........................................................................................................................ 1
1.3.2 Specific Objectives ...................................................................................................................... 1
...................................................................................................................................................................... 2
2. Data Collection ......................................................................................................................................... 2
2.1 Data Sources ....................................................................................................................................... 2
2.2 Data Collection Methods .................................................................................................................... 3
2.3 Data Preprocessing Steps .................................................................................................................... 3
3. Model Selection ........................................................................................................................................ 3
3.1 Criteria for Choosing Models ............................................................................................................. 3
3.2 Algorithms .......................................................................................................................................... 3
4. Evaluation Metrics .................................................................................................................................... 4
4.1 Performance Measurement(Hyperparameters) ................................................................................... 4
4.2 Benchmarks......................................................................................................................................... 5
5. Implementation Plan ................................................................................................................................. 5
5.1 Development Steps ............................................................................................................................. 5
5.1.1 Data Collection and Cleaning ...................................................................................................... 5
5.1.2 Model Development ..................................................................................................................... 5
5.1.3 Model Evaluation ......................................................................................................................... 9
5.2 Testing Steps ..................................................................................................................................... 10
5.3 Deployment Steps ............................................................................................................................. 12
ii
5.4 Timeline with Milestones.................................................................................................................. 12
6. Resources and Tools ............................................................................................................................... 13
6.1 Hardware ........................................................................................................................................... 13
6.2 Software ............................................................................................................................................ 13
6.3 Frameworks and Libraries ................................................................................................................ 13
7. Model Performance Summary ................................................................................................................ 14
Conclusion .................................................................................................................................................. 15
References ................................................................................................................................................... 16
iii
1. Introduction
1.1 Background of the Study
Pneumonia poses a significant health risk globally, particularly in regions like Tanzania where
timely and precise diagnosis is essential. Traditional diagnostic methods, reliant on manual
interpretation of imaging such as X-rays, MRI, and CT scans, are labor-intensive and prone to
human error, leading to potential delays and inaccuracies. Recent advances in machine learning
offer potential solutions to automate and enhance diagnostic processes, yet there is limited
research on the comparative effectiveness of different machine learning models for pneumonia
detection. This study aims to fill this gap by establishing a framework for evaluating and
comparing three machine learning models using various imaging modalities to improve
diagnostic accuracy and efficiency in the Tanzanian healthcare context.
1.2 Problem Statement
Accurate and timely diagnosis of pneumonia is crucial for effective treatment, particularly in
Tanzania. Current diagnostic methods suffer from delays and inaccuracies due to the manual
interpretation of medical images, which is resource-intensive and error-prone. Although machine
learning shows promise in enhancing diagnostic accuracy, there is a need for comparative studies
to determine the most effective machine learning models for pneumonia detection. This project
addresses this need by developing a framework to evaluate and compare the performance of
three machine learning models using X-ray, MRI, and CT scan images.
1.3 Objectives
1.3.1 General Objective
To develop a framework for comparing the performance of three machine learning models across
different imaging modalities to detect pneumonia.
1.3.2 Specific Objectives
 To create separate datasets from X-ray, MRI, and CT scan imaging modalities.
 To design and create a user-friendly interface to display the results.
 To develop three machine learning models for pneumonia detection.
 To compare the performance of the machine learning models based on accuracy and
efficiency.
 To develop a framework for the systematic evaluation of these models.
1
Figure 1: General procedures for model development and deployment
2. Data Collection
2.1 Data Sources

Kaggle for X-ray datasets.

Mendeley for CT scan datasets.

Figshare for MRI datasets
2
2.2 Data Collection Methods
 Acquiring de-identified patient data to ensure privacy and compliance with ethical
standards.
 Using automated scripts and tools to download and preprocess images from publicly
available medical databases.
2.3 Data Preprocessing Steps

Standardizing image formats and resolutions.

Performing data augmentation to increase dataset variability.

Splitting data into training, validation, and test sets.

Annotating images with diagnostic labels (pneumonia positive/negative).
3. Model Selection
3.1 Criteria for Choosing Models
 Proven effectiveness in image classification tasks.
 Compatibility with the types of medical imaging data.
 Ability to generalize across different imaging modalities.
3.2 Algorithms
Convolutional Neural Networks (CNNs) for image classification.
Transfer learning models such as, VGG, and Inception.
Ensemble methods to combine predictions from different models.
3
4. Evaluation Metrics
4.1 Performance Measurement(Hyperparameters)
Figure 2:Performance measurement parameters
 Accuracy: Proportion of correctly identified pneumonia cases.
 Precision: Ratio of true positive detections to total positive predictions.
4
 Recall (Sensitivity): Ratio of true positive detections to total actual positives.
 F1 Score: Harmonic mean of precision and recall.
 Area Under the Receiver Operating Characteristic Curve (AUC-ROC).
4.2 Benchmarks
Comparing model performance against standard diagnostic accuracy rates in clinical settings.
Establishing thresholds for acceptable performance based on medical guidelines.
5. Implementation Plan
5.1 Development Steps
5.1.1 Data Collection and Cleaning
1. Download datasets from Kaggle (X-rays), Mendeley (CT scans), and Figshare (MRI).
2. Standardize image formats and resolutions across all datasets.
3. Annotate images with diagnostic labels.
4. Perform data augmentation techniques such as rotation, zooming, and flipping to enhance
dataset variability.
5. Split the data into training, validation, and test sets (e.g., 70% training, 15% validation,
15% test).
5.1.2 Model Development
Utilize Google Colab as the development environment, leveraging its GPU resources for
efficient model training.
5
Figure 3: Import pytorch libraries
6
Figure 4: import seaborn libraries
For each imaging modality, design a specific CNN architecture optimized for that data type.
 Train the models on the respective datasets, employing techniques such as early stopping
(epochs) and learning rate adjustments to prevent overfitting.
Figure 5: Epochs performnce for xray model
7
Figure 6: MRI model performance epochs
8
 Use transfer learning with pre-trained models (e.g, VGG) to improve performance where
applicable.
5.1.3 Model Evaluation
Validate the models using the validation datasets.
Evaluate model performance on the test datasets, calculating accuracy, precision, recall, F1 score,
and AUC-ROC for each model.
9
Figure 7:Models performance visualization
5.2 Testing Steps
1. Perform cross-validation to ensure model robustness and reliability.
2. Conduct sensitivity analysis to determine the impact of different data augmentations and
preprocessing techniques on model performance.
10
Figure 8: confusion matrix for models
3. Compare model outputs with clinical diagnoses to assess real-world applicability.
11
Figure 9:Machine learning model development and model operations
5.3 Deployment Steps
1. Develop a user-friendly interface using Flask or Django to display model results and
facilitate interaction.
2. Integrate the models with existing medical diagnostic systems in Tanzanian healthcare
facilities.
3. Train healthcare professionals on how to use the system and interpret results.
4. Establish a feedback loop for continuous improvement based on user input and new data
5.4 Timeline with Milestones
 Month 1-2: Data collection and preprocessing.
 Month 3-4: Model development and training.
 Month 5: Model evaluation and tuning.
 Month 6: Interface development and integration.
 Month 7: Testing and validation.
 Month 8: Deployment and training.
12
6. Resources and Tools
6.1 Hardware
High-performance computing resources (GPUs/TPUs) for model training.
Storage systems for large datasets.
6.2 Software

Python programming language.

Deep learning frameworks such as TensorFlow, Keras, and PyTorch.

Data processing libraries such as NumPy and pandas.

Visualization tools such as Matplotlib and Seaborn.
6.3 Frameworks and Libraries

TensorFlow/Keras for model development.

Scikit-learn for preprocessing and evaluation metrics.

OpenCV for image processing.

Flask or Django for developing the user interface.
13
7. Model Performance Summary
14
Conclusion
This framework provides a comprehensive plan for organizing and executing the machine
learning project to compare the performance of different models in detecting pneumonia from
various imaging modalities such as X-ray, MRI and CT-SCAN
15
References
1. https://data.mendeley.com/datasets/p2r42nm2ty/1
2. https://www.kaggle.com/datasets/paultimothymooney/chest-xray-pneumonia
3. https://figshare.com/articles/media/High_framerate_reconstruction_of_a_lung_MRI_dataset/7453226/1
16