Lecture Notes in Networks and Systems 1017
Kohei Arai Editor
Intelligent
Computing
Proceedings of the 2024 Computing
Conference, Volume 2
Lecture Notes in Networks and Systems
1017
Series Editor
Janusz Kacprzyk , Systems Research Institute, Polish Academy of Sciences, Warsaw,
Poland
Advisory Editors
Fernando Gomide, Department of Computer Engineering and Automation—DCA,
School of Electrical and Computer Engineering—FEEC, University of Campinas—
UNICAMP, São Paulo, Brazil
Okyay Kaynak, Department of Electrical and Electronic Engineering, Bogazici
University, Istanbul, Türkiye
Derong Liu, Department of Electrical and Computer Engineering, University
of Illinois at Chicago, Chicago, USA
Institute of Automation, Chinese Academy of Sciences, Beijing, USA
Witold Pedrycz, Department of Electrical and Computer Engineering, University of
Alberta, Alberta, Canada
Systems Research Institute, Polish Academy of Sciences, Warsaw, Canada
Marios M. Polycarpou, Department of Electrical and Computer Engineering, KIOS
Research Center for Intelligent Systems and Networks, University of Cyprus, Nicosia,
Cyprus
Imre J. Rudas, Óbuda University, Budapest, Hungary
Jun Wang, Department of Computer Science, City University of Hong Kong, Kowloon,
Hong Kong
The series “Lecture Notes in Networks and Systems” publishes the latest developments
in Networks and Systems—quickly, informally and with high quality. Original research
reported in proceedings and post-proceedings represents the core of LNNS.
Volumes published in LNNS embrace all aspects and subfields of, as well as new
challenges in, Networks and Systems.
The series contains proceedings and edited volumes in systems and networks, spanning the areas of Cyber-Physical Systems, Autonomous Systems, Sensor Networks, Control Systems, Energy Systems, Automotive Systems, Biological Systems, Vehicular Networking and Connected Vehicles, Aerospace Systems,
Automation, Manufacturing, Smart Grids, Nonlinear Systems, Power Systems,
Robotics, Social Systems, Economic Systems and other. Of particular value to both
the contributors and the readership are the short publication timeframe and the worldwide distribution and exposure which enable both a wide and rapid dissemination of
research output.
The series covers the theory, applications, and perspectives on the state of the art
and future developments relevant to systems and networks, decision making, control,
complex processes and related areas, as embedded in the fields of interdisciplinary and
applied sciences, engineering, computer science, physics, economics, social, and life
sciences, as well as the paradigms and methodologies behind them.
Indexed by SCOPUS, INSPEC, WTI Frankfurt eG, zbMATH, SCImago.
All books published in the series are submitted for consideration in Web of Science.
For proposals from Asia please contact Aninda Bose (aninda.bose@springer.com).
Kohei Arai
Editor
Intelligent Computing
Proceedings of the 2024 Computing
Conference, Volume 2
Editor
Kohei Arai
Faculty of Science and Engineering
Saga University
Saga, Japan
ISSN 2367-3370
ISSN 2367-3389 (electronic)
Lecture Notes in Networks and Systems
ISBN 978-3-031-62276-2
ISBN 978-3-031-62277-9 (eBook)
https://doi.org/10.1007/978-3-031-62277-9
© The Editor(s) (if applicable) and The Author(s), under exclusive license
to Springer Nature Switzerland AG 2024, corrected publication 2024
This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether
the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of
illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission
or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar
methodology now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
The publisher, the authors and the editors are safe to assume that the advice and information in this book
are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the
editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors
or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in
published maps and institutional affiliations.
This Springer imprint is published by the registered company Springer Nature Switzerland AG
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
If disposing of this product, please recycle the paper.
Preface
It is with great pleasure that we introduce the proceedings of the Computing Conference
2024, held on July 11 and 12, 2024. This conference served as a platform for researchers
and professionals from around the globe to convene and exchange ideas at the forefront
of computing and its diverse applications. The enthusiasm and dedication displayed by
participants underscored the significance of this event in fostering collaboration and
advancing the field.
We received an overwhelming total of 457 contributions from esteemed scholars and
practitioners. These submissions underwent a rigorous double peer-review process, facilitated by experts in their respective domains. After careful evaluation and deliberation,
a total of 165 papers were selected for publication in these proceedings.
The diverse array of topics covered in these papers reflects the breadth and depth of
contemporary computing research, spanning areas such as artificial intelligence, machine
learning, cybersecurity, data science, and beyond. Each paper represents a valuable
contribution to the collective knowledge base of the computing community, offering
insights, innovations, and solutions to pressing challenges.
We extend our heartfelt gratitude to all authors, reviewers, organizers, and attendees
whose efforts and contributions made this conference a resounding success. It is our
hope that the insights shared and connections forged during this event will continue to
inspire and propel advancements in computing for years to come.
Regards,
Kohei Arai
Contents
Optimised Round Robin with Virtual Runtime for CPU Scheduling . . . . . . . . . . .
Jagriti Bhatia, Sanskriti Mathuria, Vandana M. Ladwani,
and Shobana Padmanabhan
1
Faster Lock-Free Atomic Shared Pointers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Jörg P. Schäfer
18
CICO2 e: A Compute Carbon Footprint Estimation Tool Based on Time
Series Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Christian Plewnia and Horst Lichter
39
Image Classification Method Based on Chaos Neural Network . . . . . . . . . . . . . . .
Kohei Arai
56
Human-Created and AI-Generated Text: What’s Left to Uncover? . . . . . . . . . . . .
Steven Salter, Phoey Lee Teh, and Richard Hebblewhite
74
AraXLM: New XLM-RoBERTa Based Method for Plagiarism Detection
in Arabic Text . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Mona Alshehri, Natalia Beloff, and Martin White
81
Boosting Customer Retention in Pharmaceutical Retail: A Predictive
Approach Based on Machine Learning Models . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Angel Espinoza-Vega and Henry N. Roa
97
Introducing Prediction Concept into Data Envelopment Analysis Using
Classifier in Economic Forecast . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
Guangzao Huang, Zijiang Yang, Grace Liu, and Guoli Ji
Investigating Machine Learning Techniques Used for the Detection
of Class Noise in Data: A Systematic Literature Review . . . . . . . . . . . . . . . . . . . . . 128
Cindy van den Berg and Sunet Eybers
The Role of Chatbots in Data Analytics: An Evaluation of Functional
Abilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
Preeti Patel and Sowgol Shooshtarian
An Analytical Investigation into the Impact of Product Color on the Price
of Retail Products and Purchasing Decisions of Consumers . . . . . . . . . . . . . . . . . . 169
Zaid M. Altukhi and Nasser F. Aljohani
viii
Contents
A Numerical Approach for the Fractional Laplacian via Deep Neural
Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
Nicolás Valenzuela
RBF-SC: A Fast Community Detection Technique Using Radial Basis
Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
Fang Hu, Jia Liu, Lina Wu, Xingang Fang, Mingfang Huang,
and Haotian Liu
Improving Medication Prescription Strategies for Discordant Chronic
Comorbidities Through Medical Data Bench-Marking and Recommender
Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
Tom Ongwere, Nimbalkar Rutuja, and Tam V. Nguyen
Sentiment Analysis for Predicting the Variation Trend of Stocks: A Case
Study of Vanke Co., Ltd. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
Qiuyi Jin and Jin Zheng
Suicide Ideation Prediction Through Deep Learning: An Integration
of CNN and Bidirectional LSTM with Word Embeddings . . . . . . . . . . . . . . . . . . . 271
Christianah T. Oyewale, Ayodeji O. J. Ibitoye, Joseph D. Akinyemi,
and Olufade F. W. Onifade
A Multi-clustering Unbiased Relative Prediction Recommendation
Scheme for Data with Hidden Multiple Overlaps . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
Avivit Levy, Michal Chalamish, B. Riva Shalom, Guy Sharir,
Opal Peltzman, and Sivan Salzmann
An Unsupervised Deep Learning Model for Aspect Retrieving Using
Transformer Encoder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
Atanu Dey, Mamata Jenamani, and Arijit De
Personalized Student Performance Prediction Modeling for Student
Digital Twins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
Sean Mondesire and Emmanuel Nsiye
A Social Profile-Based Recommendation Architecture for E-Learning
Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330
Xola Ntlangula and Wai Sze Leung
A Comparison of Student Engagement Across Three Teaching Modalities
in an Introductory Statistics Course . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344
Sher B. Chhetri, Mario Toussaint, Nonhle C. Mdziniso,
and Rebecca A. Hillman
Contents
ix
Impact Factor Game Scoring Model for Positive Emotion Game Design . . . . . . . 358
V. Sithira Vadivel
Cyber Safety Awareness Through a Massive Open Online Course
(MOOC): Community Engagement Knowledge Transfer . . . . . . . . . . . . . . . . . . . . 365
Elmarie Kritzinger
Immersive Environments at School: “Stop Cyberbullying by Proximity” . . . . . . . 377
Cristina Sánchez-Romero and Eva María Muñoz-Jiménez
Simulation and Analysis of Cyber-Attack on Modbus Protocol for Smart
Grids in Virtual Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 384
Shampa Banik, Rajesh Manicavasagam, Trapa Banik, and Shudipta Banik
Cybersecurity Governance in the Medical Ecosystem: An Orientation
Guide with Specific Reference to the Merging of IT and OT Devices . . . . . . . . . . 402
Sebastiaan Von Solms and Jaco du Toit
The Impact of Utilising the Amazon AWS Hybrid Deployment Model
on Assuring a Secure Migration of a Commercial Web Application
into the Cloud . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 418
Khalied Koorowlay and Rafid Al-Khannak
Mitigating Cache Pollution Attack Using Deep Learning in Named Data
Networking (NDN) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 432
Mohd Maizan Fishol Hamdi, Zhiyuan Chen, and Milena Radenkovic
A Community Security Operations Centre (ComSOC) Model for SMMEs
in Developing Countries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 443
Nombeko Ntingi, Jaco du Toit, and Sebastiaan von Solms
Optimizing Energy States in Mobile Embedded Systems: A SIPN-Based
Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 457
Robert Stojic, Daniel Peters, and Florian Thiel
Secure Content Protection Schemes for Industrial IoT with SRAM
PUF-Based One-Time Use Cryptographic Keys . . . . . . . . . . . . . . . . . . . . . . . . . . . . 478
Saloni Jain, Ashwija Reddy Korenda, Bertrand Cambou,
and Chris Lucero
Feasibility Study with Actual Space Rockets Towards Information
Theoretically Secure Radio Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 499
Sumio Morioka, Satoshi Obana, and Maki Yoshida
x
Contents
FlexHi: A Flexible Hierarchical Threshold Signature Scheme . . . . . . . . . . . . . . . . 509
Muhammed Ali Bingol, Sermin Kocaman, Ali Dogan,
and Sibel Kurt Toplu
Autoencoder-Based Solution for Intrusion Detection in Industrial Control
System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 530
Silvio Russo, Claudio Zanasi, Isabella Marasco, and Michele Colajanni
Determining the Segmentation Type Impact on an ID Card Fraud
Detection System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 544
Daniel Benalcazar, Pamela Zurita, Diego Pasmiño, and Rodrigo Lara
Transforming Medical Waste Management Through IoT and Machine
Learning: A Path Towards Sustainability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 556
Nahin Nasir, Mohammed Shahriar Hossain, Md. Saiful Islam,
Md. Arifiul Islam, and Md Manirul Islam
IntentRec: An Advanced Recommender System Leveraging User-Item
Intent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 576
Abhishrut Vaidya and Niladri Chatterjee
Process for the Identification of Vehicle Functions for Cloud Offloading . . . . . . . 596
Martin Sommer, Daniel Baumann, Tobias Rösch, Falk Dettinger,
Eric Sax, and Michael Weyrich
Smart Workplace Past Covid-19: Perceived Challenges and Potential
Smart Solutions for University Work Environment . . . . . . . . . . . . . . . . . . . . . . . . . 609
Rabail Tahir and John Krogstie
Need for Cultural Sensitivity in the Design and Development
of Technology to Aid in Dementia Care: A Review of Literature . . . . . . . . . . . . . 625
Arshia Khan, Sakina Rao, and Alfia Parvez
Sentiment Analysis of Post-COVID-19 Work-From-Home Culture:
A Literature Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 637
Joseph Kwame Adjei and Hannah Alhassan Suhuyini
Security Gaps in the Mobile Money System in Rwanda: Challenges, Risks
and Mitigation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 653
Catherine Njogu, Furaha Benedict, Susan Muthoni,
Marie Noelle Kanyamuneza, Evalyne Lwoba, Everlyn Musembi,
Yussuf Papy, and Edwin Kairu
Contents
xi
Comparative Analysis of Flow and Cardano Blockchains: Navigating
Adoption Challenges, Computing Techniques and Implications
for the Blockchain Landscape . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 665
Ayda Bransia, Bálint Molnár, and Simon Thompson
Correction to: Optimizing Energy States in Mobile Embedded Systems:
A SIPN-Based Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Robert Stojic, Daniel Peters, and Florian Thiel
C1
Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 671
Faster Lock-Free Atomic Shared Pointers
Jörg P. Schäfer(B)
German Aero Space Center, Institute of Transportation Systems, Berlin, Germany
joerg.schaefer@dlr.de
Abstract. CPU’s don’t increase in speed anymore, as Moore’s Law has
claimed for so long. Although, “the free lunch is over” (Herb Sutter),
parallel algorithms can gain more throughput and reduce latency, which
is crucial to complex real-time applications like audio and video processing, robotics, or real-time sensor data processing in embedded hardware.
Parallel algorithms, however, come with the price of concurrency and
synchronization. For example, the priority-inversion is a problem, where
low-priority threads can block high-priority threads due to locking data
structures used by both threads. Lock-free data structures, on the other
hand, use atomic CPU instructions to avoid these problems. They, however, are hard to implement and even harder to prove correct. Atomic
shared pointers have been proposed as a (part of a) solution to making
lock-free algorithms easier to write and verify. Since they are a fundamental tool in the toolbox of parallel algorithms, their run-time performance
has a huge trailing impact. So far, there is just a hand full of existing
implementations to atomic shared pointers. This work contributes an
improved implementation to atomic shared pointers, a formal proof of
its correctness, and an extensive performance evaluation in comparison
to other implementations showing that it outperforms others in most
use-cases.
Keywords: Concurrency · Algorithm · Atomic shared pointers ·
Lock-free · Real-time implementation
1
Introduction
Real-time applications such as in robotics or video and audio processing increase
their workload and processing capabilities by using multiple threads. In such
applications, there are often multiple threads running with different priorities,
e.g., at least one low-priority and one high-priority thread. Such threads usually communicate via common variables and data structures. To avoid raceconditions, most of these data structures ensure thread-safety using exclusive
locking variables (mutexes).
Using a mutex for synchronization comes with the priority inversion problem,
where low priority threads slow down high priority threads by locking such a common mutex: When the low priority thread is sent to sleep while holding the lock,
With thanks to the German Federal Ministry of Digital and Transport (BMDV) for
funding this work through project “EDDY” (Förderkennzeichen 19F2208B).
c The Author(s), under exclusive license to Springer Nature Switzerland AG 2024
K. Arai (Ed.): SAI 2024, LNNS 1017, pp. 18–38, 2024.
https://doi.org/10.1007/978-3-031-62277-9_2
Atomic Shared Pointers
19
the high priority thread needs to wait until the low priority thread is finished. A
second disadvantage of using mutexes comes with the CPU’s over subscription.
The problem is effectively the same as before, i.e., threads are more likely prone
to suspension while locking a mutex, which blocks other threads accessing the
same mutex. These situations are exactly, what real-time applications need to
avoid.
On the other hand, modern CPU’s provide atomic operations on single (or
double) words, e.g., exchange and compare-and-swap (CAS ) to name the most
important ones. Several simple data structures have already been implemented
lock-free, i.e., without using mutexes and instead relying on these atomic instructions. However, this area of research is an open field because it is particularly
hard to develop even simple data structures lock-free [7,11,15]. Several new problems arise, one of which is the ABA problem, where concurring threads might
change an atomic variable forth and back to the value last seen, making other
threads believe that nothing happened during their last visit of this atomic variable. This false assumption might lead to algorithms with undefined behaviour
in this situation.
Herb Sutter proposed to use lock-free atomic shared pointers [13,14], as they
solve several problems including the ABA problem. In his proposal, he could
show that a singly connected list is implementable in a few lines of (C++) code
without any further ado. At that time, however, no implementation for atomic
shared pointers existed.
A shared pointer is a concept in programming languages that usually leave
memory management to the language’s user. Here, shared pointers are smart
pointers that help managing the lifetime of dynamically allocated memory and
the object that resides in there. When a shared pointer is created to manage the
lifetime of a dynamically allocated object, it takes ownership of it, i.e., the last
living copy of the shared pointer is responsible for its proper destruction. From
here on, we focus on C++ (up to C++23) as programming language.
The ubiquitous way to implement shared pointers is by additionally allocating a control block for a particular object, which contains a reference counter.
Thus, shared pointers referencing the same object also share the same control
block including this shared reference counter. Copying a shared pointer increments this reference counter. On the other hand, when a shared pointer dereferences its object, e.g. by being destroyed or being made referencing another
object, it needs to decrement its reference counter. Upon reaching zero, the
object is destroyed and the memory deallocated by the last accessing thread.
Now, when a shared pointer dereferences an object, it needs to write to
at least two different memory addresses: the pointer to the control block and
the reference counter within the control block, which cannot be done atomically in most of the current CPU’s.1 One way to achieve atomic operations on
shared pointers is guarding them with an exclusive locking variable. At least
GNU-C++ 12.3.0 and Clang 14.0.0 use this approach for the implementation of
std::atomic<std::shared_ptr>.
1
TSX enables CPU-level transactions, but substantial reasons exist, to avoid it [9].
20
J. P. Schäfer
Now that most of the important terms have been placed and put into relation, Sect. 2 provides a brief overview of the history and related work regarding atomic shared pointers. Section 3 contributes a detailed description of a new
split-reference counting based algorithm for lock-free atomic shared pointers that
allows negative counters.2 Section 4 provides a formal proof for its correctness.
Section 5 provides an evaluation and comparison of the proposed implementation
against existing implementations regarding throughput in various scenarios.
2
Related Work
This millennium started with a “fundamental turn toward concurrency in software” [12]. A crucial aspect in writing such multi-threaded applications is the
synchronization of multiple threads. The problem that comes with it starts at a
very low level: Scott Meyers already mentioned lots of race conditions that can
occur, when the programming language is not designed to be used for multithreaded applications. In particular, he mentioned a series of problems that can
occur even in very simple C++ (up to version C++-03) code due to compiler
optimizations [10].
For C++-11, the ISO C++ committee [1] thoroughly introduced threads
in the language, including a memory model [2] to avoid these problems. This
standard also introduced the atomic class, that guarantees atomic access to
objects, though mostly with a hidden lock (mutex). In some cases (e.g. for up
to 64 Bit primitive types such as an integer), the atomic-implementation uses
atomic CPU instructions instead of locks, which enables implementing lock-free
algorithms and data structures.
As it turned out, not only is the design of lock-free data structures a very hard
problem for simple data structures already [7], even lots of reviewed and published proposals for lock-free data structures and algorithms have been proven
incorrect [11]. In 2014, Herb Sutter proposed atomic shared pointers [13,14] as
a basic building block for lock-free data structures, as they automatically would
solve several problems, including the well-known ABA-problem [4].
To that, Timur Doumler held a talk about shared pointers in 2022, in which
he compared three existing shared pointer implementations against each other
(and against the lock-based standard implementation) [5,6]. One of these atomic
shared pointer implementations was a proof of concept implementation provided
by Anthony Williams [16], who took a leading part in the C++ committee
enabling multi-threaded applications in the C++-11 standard [15]. Though, his
implementation uses 128 Bit atomic variables, probably hoping at the time of
writing that C++ would support atomic operations on double words. But, compiler builders decided against that, probably in favor of being backwards compatible to some AMD x86-64 CPU’s produced until 2008, which do not support
these instructions. Hence, Anthony’s implementation is technically not lock-free,
but provides an approach to build one. The most competitive implementation
regarding throughput comes with Facebook’s open-source library folly [8].
2
Source code available under: https://github.com/frygge/atomic shared ptr.
Atomic Shared Pointers
21
While all implementations above and the one proposed in this work are based
on a split reference counting technique, Daniel Anderson showed a new approach
based on hazard pointers and deferred reclamation [3] at the CppCon 2023,
though he also changed the API.
The implementation provided in this work not only improves the runtime
over existing implementations but also shows that carefully avoiding CAS-loops
is crucial for high throughput in over-subscription scenarios. Additionally to
the evaluation in Sect. 5, the source code repository also contains an extensive
comparison.
3
Algorithm
The proposed implementation of (atomic) shared pointers is very similar to the
ones that already exist: A shared pointer instance is a pointer to a control block,
which contains (a pointer to) the actual data as well as a reference counter
(aka usage counter ). Copying the shared pointer increments the usage counter
while destroying a shared pointer instance (or letting it point to something else)
decrements the usage counter. The instance decrementing the usage counter to
zero is responsible for the destruction of the control block and the actual object.
In our proposed implementation, each shared pointer instance also has a local
(temporary) counter. Furthermore, the control block has a usage counter and a
global (temporary) counter. Algorithm 1 provides near-C++ pseudo code for the
data structures.
Having a global temporary reference counter enables always having a correct
usage counter, which counts the amounts of atomic and non-atomic instances
referencing an object.
Incrementing the local counter by a thread before accessing the control block
provides a hint to concurrent threads that the control block is still in use, so
they will not destroy it before the former thread could increase the usage counter.
Henceforth, I assume that the reader is familiar with the basic implementation
details of shared pointers.
Our implementation needs to simultaneously read and update the local
counter and a pointer to the control block in atomic instances. To that, we
pack a pointer (which requires only 48 Bits in current systems) and a 16 Bit
counter in a 64 Bit word. This trick avoids the necessity of atomic operations on
double words (i.e., 128 Bit). Changing this detail would not affect the proof of
correctness in Sect. 4.
Analogously, our implementation needs to read and update a pair of counters
from the control block. Here, we use a similar trick, splitting a word in two 32
Bit counters.
It is worth mentioning, that the counter is placed into the packed pointer
such that integer overflows do not change the value of the pointer. In particular,
this allows the counter to be signed. For the same reason, the global temporary
counter in the paired counter can be signed. This implementation detail is crucial
to some of the optimizations over the other existing implementations.
22
J. P. Schäfer
Algorithm 1. Data Structures
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
struct paired_counter {
signed tmp
unsigned ref
}
struct ctrl_block<T> {
atomic<paired_counter> pc
T* data
}
struct counted_ptr<T> {
signed tmp_ref
ctrl_block<T>* pctrl
}
struct sptr<T> {
counted_ptr<T> cptr
}
struct atomic_sptr<T> {
atomic<counted_ptr<T>> cptr
}
Similar to the family of atomic arithmetic CPU instructions (e.g., fetch_add,
fetch_xor, etc.), such operations can also be performed on counted_ptr and
paired_counter, except, that the operation shall only be performed on the
counter of the counted_ptr and simultaneously on both counters of the
paired_counter using a pair of arguments. The details for the implementation of these structures are omitted here, but can be found in the source code.
For example, cptr.fetch_sub( 2 ) would subtract 2 from the counter of the
counted pointer cptr and then return the old value of the counted_ptr object.
Similarly, pc.fetch_add( 3, 5 ) would atomically add 3 to tmp and 5 to ref
of the paired_counter object pc.
3.1
Notation and Assumptions
This subsection declares some notation and defines the terms atomic instance,
non-atomic instance, temporary instance that are used throughout the rest of
this work.
Objects in memory that are referenced by shared pointers (also called shared
objects) are denoted via letters p, q, r, . . ..
Definition 1 (Shared Pointer). A (non-atomic) shared pointer is an instance
of a structure to access an object in memory and manage its lifetime. The shared
object referenced by multiple shared pointers will be destroyed immediately when
the last instance stops referencing it (i.e., by being destroyed itself or being
updated to not reference the shared object anymore).
Atomic Shared Pointers
23
Non-atomic shared pointers can be read concurrently. In particular, multiple threads can make copies of a shared pointer concurrently without external
synchronization.
Note, that a non-atomic shared pointer is not necessarily encapsulated into a
sptr<T> structure. For example, Line 4 of Algorithm 3 moves a non-atomic
instance (created in Line 3) into an atomic instance (after which, the non-atomic
instance does not exist anymore), simultaneously retrieving the old pointer of
the atomic instance, making it a non-atomic instance.
Definition 2 (Atomic Shared Pointer). An atomic shared pointer is an
instance of a structure that provides atomic access to a shared pointer via the
operations load, store, exchange, and cas (compare-and-swap). These instructions are similar in their semantic interpretation as the corresponding CPU
instructions in modern CPU’s, except that they work on shared pointers including
the lifetime management of the referenced objects.
Definition 3 (Temporary Shared Pointer). A temporary instance of a
shared pointer is a quasi-instance of a non-atomic shared pointer that only lives
temporarily during the execution of the operations of atomic shared pointers.
Atomic and non-atomic shared pointers referencing an object p are denoted
by ap and sp , respectively.
Being referenced by shared pointers, each shared object p has a global reference counter U p (or shortly usage counter )3 and a global temporary reference
counter T p (or shortly global counter ).
Each atomic and non-atomic instance ap and sp has a local temporary reference counter (or shortly local counter ) denoted by lap and lsp , respectively.
We denote the number of temporary instances referencing a shared object p
by J p and the total number of (atomic, non-atomic, and temporary) instances
referencing p by I p .
From here on, we assume correct usage of shared pointers. In particular,
this requires that non-atomic shared pointers are either read and written to by
exactly one thread or accessed read-only by multiple concurrent threads. In the
end, concurrent write access to shared pointers is the essence of atomic shared
pointers.
3.2
Operation: Load
The load-operation atomically reads the shared pointer from an atomic instance
and returns a non-atomic copy. To understand the wording used in the correctness proof in Sect. 4, this section describes the behaviour of the load-operation
in more detail.
3
As we will see in Lemma 1, U p always counts the global number of atomic and nonatomic shared pointer instances for this particular object p; hence the short name
usage counter.
24
J. P. Schäfer
Algorithm 2. Atomic Shared Pointer: load
1
2
3
4
5
6
load( atomic_sptr<T>& a ): sptr<T>
cptr = a.cptr.fetch_add( 1 )
if( cptr.pctrl == NIL )
return NIL
cptr.pctrl->pc.fetch_add( 1, 1 )
return { 0, cptr.pctrl }
The load-operation (Algorithm 2) creates a non-atomic copy sp of the atomic
instance ap stored in the atomic shared pointer a and returns it to the caller. To
that, it first creates a temporary instance in Line 2 indicated by incrementing
the local counter lap . If that instance is empty (aka NIL, i.e., it does not point to
any object), it is returned as such. Otherwise, Line 5 converts the just retrieved
temporary instance (stored in cptr) into a non-atomic instance sp by balancing
the temporary (local and global) counters lap and T p and simultaneously incrementing the usage counter U p . Then, Line 6 returns the non-atomic instance sp ,
stored in cptr to the user with a local counter lsp = 0.
Note, that this implementation executes at most two atomic instructions that
need CPU-level synchronization.
3.3
Operation: Store
Algorithm 3. Atomic Shared Pointer: store
store( atomic_sptr<T>& a, const sptr<T>& s )
if( s.cptr.pctrl != NIL )
s.cptr.pctrl->cptr.fetch_add( 0, 1 )
// 1., acquire globally
4
old_cptr = a.cp.exchange({ 0, s.cptr.pctrl })
// 2., exchange pointers
5
if( old_cptr.pctrl != NIL )
6
release( old_cp.pctrl, { old_cptr.tmp_ref, 1 })
// 3. release old pointer
1
2
3
The store-operation (Algorithm 3) creates a copy of the non-atomic instance
stored in its parameter s (Line 3) and converts the copy into an atomic instance
by putting into the atomic variable a (Line 4). Simultaneously, the exchange in
Line 4 converts the old atomic instance stored in a into a non-atomic instance
and stores it inside the variable old_ctpr. Since store is supposed to simply
overwrite the old value of the atomic variable, Line 6 releases the old instance,
Atomic Shared Pointers
25
which might be the last one, in which case the shared object and its control
block is being destroyed through the function release.
An R-value interface (C++ specific) would allow an even more efficient implementation and can be found in the source code.
3.4
Operation: Exchange
The exchange-operation (Algorithm 4) atomically swaps the atomic instance
ap stored in a with the non-atomic instance sq stored in the parameter s in
Line 4. To that, the implementation in Line 1 first needs to create a copy of sq
(incrementing the usage counter) in Line 3.
Similar to the store-operation, a more efficient R-value implementation
exists in the source code.
Algorithm 4. Atomic Shared Pointer: exchange
exchange( atomic_sptr<T>& a, const sptr<T>& s ): sptr
if( s.cptr.pctrl != NIL )
s.cptr.pctrl->pc.fetch_add( 0, 1 ) // acquire globally
old_cptr = a.cptr.exchange({ 1, s.cptr.pctrl })
// exchange pointers
5
return { old_cptr } // return the old pointer
1
2
3
4
3.5
Operation: Compare-and-Swap
The cas-operation (short for compare-and-swap) tests if the shared object p
referenced by the atomic instance ap in a is the same as the shared object q
referenced by sq in expected. If that is the case, it puts a copy of the instance
sr stored in desired into a (similar to store in Algorithm 3). If the comparison
failed, the cas-operation provides a non-atomic copy of the atomic instance
stored in a.
The cas-operation can be implemented in a weak and a strong fashion, but
the algorithmic implementation provided here basically remains the same.4
With the wording provided in the three operations load, store, and
exchange, the code of Algorithm 5 speaks for itself. However, there are some
interesting notes to make:
1) The algorithm initially jumps to a label inside the while-loop. With minor
changes, this jump can be avoided, which would make the algorithm to start
(optimistically) with an atomic cas of the packed pointer in Line 8. But it
4
Real weak implementations have not shown to perform better on the systems available for the evaluation.
26
J. P. Schäfer
Algorithm 5. Atomic Shared Pointer: compare and exchange (cas)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
28
29
30
31
32
33
34
cas( atomic_sptr<T>& a,
sptr<T>& expected,
const sptr<T>& desired ): bool
exp_cptr = { 0, expected.cptr.pctrl }
acquired = false
goto start // although not necessary, this turned out to be more
efficient
while( true ) {
if( a.cptr.cas( exp_cptr, desired.cptr )) {
if( exp_cptr.pctrl != NIL )
release( exp_cptr.pctrl, { exp_cptr.tmp_ref, 1 })
return true
}
// failed due to unexpected pointer or counter...
if( expected.cptr.pctrl != exp_cp.pctrl ) {
start:
old_cptr = a.cptr.fetch_add( 1 )
// enter: increase local ref counter
if( expected.cptr.pctrl != old_cptr.pctrl ) {
// if still unexpected
if( old_cptr.pctrl != NIL )
old_cptr.pctrl->pc.fetch_add( 1, 1 )
// balance and acq.
expected = { 0, old_cptr.pctrl }
if( acquired and desired.cptr.pctrl )
release( desired.cptr.pctrl, { 0, 1 })
return false
}
exp_cptr = { old_cptr.tmp_ref+1, old_cptr.pctrl }
expected.cptr.tmp_ref -= 1
// compensate ‘‘enter” in line 16
}
}
if( not acquired ) {
if( desired.cptr.pctrl )
desired.cptr.pctrl->pc.fetch_add( 0, 1 )
acquired = true
}
Atomic Shared Pointers
27
turned out, that first retrieving a copy of the atomic instance in form of a
temporary instance (Line 16) yields better run-time performance. This might
be different with a masked_cas (see Sect. 6) which could avoid a failing CAS
due to an unexpected local counter.
2) We have to put a copy of the desired instance sr into the atomic variable,
which is created in Line 30 before executing the cas in Line 8.5
3) Although unlikely, something close to the ABA-problem can happen: When
the cas fails in Line 8 due to an unexpected shared object being referenced
(i.e., the condition in Line 14 is true), Line 16 copies the atomic instance creating a temporary instance. Now, in the meantime, the atomic variable a could
have been changed (back) to referencing our expected object q (i.e., Line 17
is false). In this situation, we release the temporary instance referencing q by
decrementing the local counter of the non-atomic instance to q stored in the
variable expected. Here, we can avoid an instruction on an atomic variable
due to the ability of the local and global (temporary) reference counters to
be negative.
3.6
Helper Functions
There is only one helper function to explain: release. It reduces the global and
usage counter of a shared object by a specified amount. When that results in
both counters being 0, it destroys the control block and shared object.
Algorithm 6. Atomic Shared Pointer: release
1
2
3
4
5
6
7
5
release( ctrl_block<T>* ctrl,
paired_counter pc )
old = ctrl->pc.fetch_sub( pc )
if( old == pc ) { // pc − old = 0
delete ctrl->data
delete ctrl
}
If we would create a copy of the non-atomic instance sr after we put it into the
atomic variable, a (very rare and unlikely) situation could be constructed where the
global and usage counter T r and U r would be zero although there were still threads
holding onto r via temporary instances (Other situations with a wrong usage counter
could be created, too. I mention that only as an anecdote, since I had it implemented
this way and surprisingly just found the bug through trying to prove its correctness
(which apparently did not work). I rather expected seeing that bug to occur in the
tests or applications I built using this shared pointer implementation).
28
J. P. Schäfer
3.7
Diverging Temporary Reference Counters
The implementation above has a drawback: The reference counters can diverge,
i.e., local and global (temporary) reference counters can increase indefinitely.
At some point, this leads to integer overflows breaking the whole algorithm.
This problem can be solved by simultaneously reducing local and global reference counters by the same amount when they are above a certain threshold.
For example, in the load function, we can balance the counters after Line 5,
which yields Algorithm 7. The same strategy has been implemented in the casoperation, which also creates temporary instances potentially diverging the temporary counters. Details can be found in the source code.
Algorithm 7. Atomic Shared Pointer: load and balance
1
2
3
4
5
6
7
8
9
10
11
load( atomic_sptr<T> &a ): sptr<T>
cptr = a.cptr.fetch_add( 1, 0 )
if( cptr.pctrl == NIL )
return NIL
cptr.pctrl->pc.fetch_add( 1, 1 )
if( cptr.tmp_ref > THRESHOLD ) {
tmp = cptr + { 1, 0 }
if( a.cptr.cas( tmp_cptr, { 0, cptr.pctrl })
cptr.pctrl->pc.fetch_sub({ tmp_cptr.tmp_ref, 0 })
}
return { 0, cptr.pctrl }
4
Correctness
Though the core idea of the proposed implementation is similar to the other
existing implementations, i.e., using split reference counters, the proposed algorithm differs by allowing negative local reference counters. Since it is therefore
not trivial to see the correctness of the algorithm, this section proves it.
The goal of this section is the proof of Theorem 1, which claims that an object
is destroyed iff 6 the last instance releases it.
The following lemmata help formulate the argumentation of Theorem 1. They
prove essential invariants of the algorithm.
It is worth mentioning, that we only need to focus on the lines of the pseudo
code executing (atomic) operations on variables that are shared by multiple
threads, because only there, race conditions can occur due to concurrency.
6
This word is a common synonym for if and only if in mathematics and theoretical
computer science.
Atomic Shared Pointers
29
Lemma 1. For each shared object p, the global usage counter U p always equals
the amount of atomic and non-atomic instances referencing p.
Lemma 2. Let api and spj be the atomic and non-atomic instances referencing
a shared object p. Then,
lapi +
lspj − T p = J p
(1)
counts the number of temporary instances referencing p.
Lemma 3. Let api and spj be the atomic and non-atomic instances referencing
a shared object p. Then,
lapi +
lspj − T p +U p = I p .
(2)
=J p
counts the total number of instances referencing p.
Proof. Assuming that an invariant holds true before the execution of an atomic
instruction, we have to show, that the invariant still holds true right after it as
well. Following this approach, we will proof the invariants of Lemmas 1, 2, and 3
simultaneously.
Load. Line 2 retrieves a reference to the shared object p of the atomic instance
ap , simultaneously increasing the local counter lap . Hence, right after the execution of this instruction, we have a new temporary instance referencing p and
Lemma 2 still holds. Equation (2) of Lemma 3 remains intact as well. Line 5
increments U p , simultaneously incrementing T p , i.e., the temporary instance is
atomically converted to a non-atomic instance, which is then returned to the
caller, i.e., Lemma 1 still holds. Lemma 3 also still holds, since both, U p and T p
have been incremented, zeroing each other out. Lemma 2 remains intact because
the only change was the decrement of an lap on the left hand side of Eq. (1), which
corresponds to the release of a temporary instance.
Store. As this operation can read from the non-atomic instance sp via the constant parameter s, Line 3 increments the usage counter U p , which is the moment
of a new non-atomic instance sp2 being born, i.e., Lemmas 1 and 3 remains intact.
Understand, that it makes no difference, whether this new non-atomic instance
is stored in a separate variable owned only by this thread or not. More important
is the interpretation, that now there is a second non-atomic instance that references the same object p as the read-only variable s. Since neither a temporary
(local or global) counter has been changed nor was a temporary instance created
or released, Lemma 2 remains intact, trivially.
Line 4 converts this new non-atomic instance sp2 into an atomic instance
(by putting it into the atomic variable a), simultaneously converting the old
instance aq from the atomic variable into a non-atomic instance sq . Hence, both
30
J. P. Schäfer
invariants of Lemmas 2 and 3 still hold for both, p and q. Lemma 1 remains
intact, trivially. Now, the old instance sq from a is not required anymore, so it
is released in Line 6, simultaneously reducing the usage counter U q ; thus, all
invariants remain intact for q.
In the second implementation, the non-atomic instance sp from s is moved
into a, converting it into an atomic instance. Simultaneously, the atomic instance
aq from a is retrieved and converted into a non-atomic instance sq . For both
objects p and q, the usage counters did not change, as it was an atomic conversion
of both instances, simultaneously. Thus, all invariants hold after the execution
of the instruction in Line 9.
Exchange. Here, the proof of the invariants is similar to the proof of the invariants in the store-operation. The only difference is, that we don’t release the old
instance but rather return it to the caller. Though, in the second implementation, we have to reset the content of the parameter s, which is, however, not
affected by concurrency.
Cas. The interface of the cas-operation is more complex, as is its behaviour and
implementation. At the beginning of the operation, let ap be the atomic instance
stored in a, eq be the non-atomic instance stored in expected and dr be the
non-atomic instance stored in the parameter desired. Although, there is a gotostatement in Line 4 jumping to Line 15, we rather proof the invariant for this
operation in another order, as it makes the proof more concise. In this operation,
we have to discuss the atomic instructions in Lines 6, 8, 14, 17, 20, and 28.
Following the code flow, Line 6 can only be reached after Line 27 has been
executed. There and then in Line 28, a new non-atomic instance sr is virtually created increasing U r . Hence, Lemmas 3 and 1 remain intact. Further on,
in Line 6, we have to distinct two cases: If 1) the atomic cas-instruction is
successful, two important changes have been made. On the one hand, the old
atomic instance ap in a is no more. Remember though, that the instruction is
only successful, if a’s content equals exp_cptr, which implies that p and q are
the same object. By interpreting exp_cptr as a non-atomic instance from here
on, we can interpret the cas as a conversion of the atomic instance ap to a nonatomic instance sp . Since no temporary counter or temporary instance has been
changed, all invariants remain intact.
On the other hand, a now holds an atomic instance to r. We interpret this
situation, such that the non-atomic instance sr from Line 28 has been converted
into an atomic instance ar . Again, there are no temporary counter or instance
changes. Ergo, all invariants also remain intact for r. Line 8 releases the nonatomic instance sp that was previously stored in a decrementing U p and reducing
T p by lsp . Therefore, Lemma 1 and Eq. (2) of Lemma 3 still hold.
If 2) the atomic cas-instruction in Line 6 was not successful, no change has
been made so far. The next atomic instruction in Line 14 retrieves the atomic
instance ap (note, that the referenced object might have changed from p to p
since the last atomic operation), simultaneously creating a temporary instance
Atomic Shared Pointers
31
by incrementing the local reference counter lap . Lemma 2 still holds, since the
number of temporary instances increased by one as well as the left hand side of
Eq. (1). Equation (2) also still holds, since I p increased by one due to the new
temporary instance and the left hand side increased by one due to the increment
of lap . Lemma 1 remains intact since there is no change in the amount of atomic
or non-atomic instances.
After that, a second case distinction is necessary in Line 15: Either the referenced object is still not the expected object q, then Line 17 converts the
temporary instance into a non-atomic instance and Line 18 hands that nonatomic reference to the caller. Analogously to previous conversions of temporary
instances into non-atomic instances, all invariants remain intact. Before returning to the caller, the previously created non-atomic instance sr (if it was created)
will be released in Line 20. It is easy to see, that all invariants remain intact.
The second case in Line 15 is that the atomic shared pointer changed in
the meantime referencing the expected object, i.e. p and q are the same now.
In this case, Line 24 decrements the local counter leq , which compensates incre
menting the local counter lap = laq earlier, eventually destroying the temporary instance; thus, both sides of Eq. (2) of Lemma 3 decremented by one. Also,
Lemma 2 remains intact due to releasing the temporary object being reflected
in decrementing the right hand side of Eq. (1). Lemma 1 also trivially remains
intact. In both cases and branches, all invariants for all relevant objects still
hold. Considering the analysis above as the step of a proof by induction, the
following is the start of it:
In every application, each object p is first constructed and referenced by a
non-atomic shared pointer instance sp with U p = 1, T p = 0 and lsp = 0. So, all
invariants of Lemmas 1, 2, and 3 hold for freshly created and referenced objects.
From there on, the case distinction above shows, that each atomic operation
does not change either of the invariants, which concludes the proof of Lemmas 1, 2, and 3.
Theorem 1. A shared object p is destroyed immediately after there is no
(atomic, non-atomic, or temporary) instance left, referencing it.
Proof. Each shared object p is destroyed via the release-function (Algorithm 6).
There, an object p is destroyed if and only if T p = 0 and U p = 0. Now, we have
to show that
U p = 0 ∧ T p = 0 ⇐⇒ I p = 0
holds, i.e., this condition is true if and only if there are no instances referencing
p.
“⇒”: Since U p = 0, there are neither atomic nor non-atomic instances referencing p (Lemma 1). This implies, that there are no local counters lap or lsp left,
such that Eq. (1) reduces to J p = −T p . Since T p = 0, there are no temporary
instances left, either.
32
J. P. Schäfer
“⇐”: Since I p = 0, there are no temporary, atomic, or non-atomic instances
referencing p. In particular, there are no local counters lap or lsp . Therefore, U p = 0
and J p = −T p = 0 due to Lemmas 1 and 2, respectively.
Now, what remains to be seen is, that the last instance referencing p is always
a non-atomic instance. If that holds, the correct time of p’s destruction clearly
must be in release, because non-atomic instances are always released via this
function.
There are only two code-locations, where temporary instances are held. The
first location is in Algorithm 2. Here, however, their is no branch and the temporary instance (created in Line 2) will always be converted into a non-atomic
instance (in Line 5). In the meantime, creating the temporary instance prevents
release from destroying the shared object as it caused I p ≥ 1.
The second location is in Algorithm 5 in Line 14. Here, however, the temporary instance either references p also being referenced by the variable expected,
or it will be converted into a non-atomic instance in Line 17. In the first case,
the temporary instance will be destroyed in Line 24. However, this temporary
instance cannot be the last instance referencing p because the variable expected
is exclusively owned by the executing thread and also references p. In the second
cas, the object will not be destroyed in the meantime for the same reason as in
Algorithm 2.
5
Evaluation
This section compares Anthony William’s [16], Facebook’s (folly’s) [8], and the
standard’s (provided by the C++ STL) atomic shared pointer implementations
regarding the throughput of their atomic operations.
All experiments have been conducted on three different CPU’s: AMD Ryzen 7
3700X (16 hardware threads); AMD Ryzen 9 6900HS (16 hardware threads); and
AMD Ryzen 9 5950X (32 hardware threads). In our evaluation, we study different
situations regarding the number of (software) threads and the contention of
these threads. Since the CAS operation often occurs in a loop, we measured the
throughput of single CAS calls as well as CAS loops which count as finished
until an expected value could be replace by a new desired value.
Since the term of contention comes with no mathematical metric to measure
it, we can only apply heuristics to increase or decrease the contention of the
concurring threads. To do that, we measure the throughput of an operation by
running n threads in a loop performing this particular operation round robin on
a finite pool of m atomic variables. Here, decreasing the pool size m increases
the probability of two threads to conflict, i.e., the contention increases. Letting
all threads perform on exactly one atomic variable corresponds to maximum
contention. Although, two concurring threads might miss each other (e.g. because
one operates on the atomic shared pointer while the other thread just left it),
their memory access still causes the cache to bounce. The other extreme, no
contention, would correspond to an infinite number of atomic variables. Here,
the experiment provides individual atomic variables to each thread, such that
they work on completely distinguished cache lines.
Atomic Shared Pointers
33
In most of the experiments, each thread is associated with an additional
individual local shared pointer. To avoid side-effects due to the allocation routine,
all atomic and non-atomic shared pointers are initialized beforehand with their
own pre-allocated memory locations. Each of these shared pointers resides in
their own cache-line to avoid unnecessary cache-bouncing. Details about how
the pre-allocated memory is used follows in the operation’s subsections.
All experiments have the same initialization and measurement strategy: First,
all threads synchronize using a barrier. Then, the main thread (which is managing the experiment) sleeps for a short warm-up time (100 ms) while the worker
threads start doing their job (which is calling the operation in question in a
loop). After the warm-up time, the main thread collects the progress (in number
of finished calls) from worker-individual atomic counters and then sleeps again
for a longer measurement time (2 s). When the main thread wakes up, it reads
the progress of each worker thread again and uses the previous values to gain
the progress during the measurement phase.
5.1
Operation: Load
To measure the throughput of the load-operation, each thread simply loads all
atomic shared pointers round robin, immediately discarding the result.
Figure 1 shows that our implementation and folly’s lead the race, being on par
for low number of threads. However, when the number of threads grows (and
therefore the contention too), the speedup of our implementation over folly’s
increases drastically. Similar behaviour could be observed on other scenarios for
the load-operation.
Although high-contention scenarios should be avoided in the first place,
sometimes there is no way around, in particular for the load-operation. If that
is the case, the least is to hold the throughput on a constant level. As the
high-contention diagrams show, our proposed implementation holds up to this
demand. On the other hand, even folly’s implementation drops throughput
when oversubscribing the CPU, allegedly because of using a CAS loop in their
implementation of load. Those fail more often under high contention, thus the
Fig. 1. Throughput of the load operation on an AMD Ryzen 9 5950X 32-core machine
under high contention (left), and shrinking contention for maximum processor subscription (right)
34
J. P. Schäfer
expected number of turns increases until the CAS loop succeeds. The proposed
implementation, however, outperforms this approach because it only contains
two determined atomic calls (increasing the local and then the global and usage
counter).
5.2
Operation: Store
To measure the throughput of the store operation, each thread simply stores
their own shared pointer round robin in the pool’s atomic shared pointers.
In most scenarios, the proposed implementation is on par with folly’s
implementation. Figure 2, however, shows a general disadvantage of the standard’s implementation, which uses mutexes for synchronization: Its performance
drops drastically in over-subscription scenarios. This corresponds to the insight
mentioned in the introduction: Sleeping threads locking a mutex block other
threads trying to access it. Similarly, its performance drops drastically in maxsubscription scenarios when the contention grows.
Fig. 2. Throughput of the store operation on an AMD Ryzen 9 5950X 32-core machine
under low contention (left) and shrinking contention for maximum processor subscription (right)
Fig. 3. Throughput of the exchange operation on a Ryzen 7 3700X (left) and a Ryzen
9 5950X (right) machine under high contention
Atomic Shared Pointers
5.3
35
Operation: Exchange
To measure the throughput of the exchange operation, each thread loops over the
pool of atomic shared pointers round robin swapping their current own shared
pointer with the one in the pool’s atomic shared pointer.
Since our implementation of the exchange-operation is quite similar to the
store’s implementation, the throughput is quite similar as well. In almost all
situations, the throughput of our implementation is on par with folly’s. Sill, there
are situations where the qualitative behaviour of Anthony William’s implementation seems to differ (see Fig. 3) depending on the executing machine.
5.4
Operation: Weak and Strong CAS
Measuring the throughput of a single CAS call is done similar to measuring the
throughput of the store operation: Each thread tries to put its own shared pointer
into one of the pool’s atomic shared pointers. We set the expected argument
of the cas-operation to be empty (NIL). Since none of the atomic variables is
empty, this always leads to a fail of the CAS operation, i.e., we measure the
branch that provides us with the current shared pointer in the atomic variable,
which effectively ends up being a load operation with a previous comparison.
Fig. 4. Throughput of the (strong) CAS operation on an AMD Ryzen 9 5950X machine
under low contention (left) and high contention (right)
Figure 4 shows exemplary that the proposed implementation outperforms the
other implementations. In comparison to folly’s shared pointers, the reason might
be the same as in the load-operation: Instead of using a CAS loop to increase
the local reference counter, Algorithm 5 (cas) only uses a determined and finite
set of atomic operations to succeed (fetch_add).
It again seems, that avoiding CAS-loops when possible benefits throughput.
Even on low-contention situations, it decreases the number of CPU instructions
by a factor of two, as folly’s implementation first performs a load and then the
CAS loop with one turn; thus, even without contention, two atomic CPU instructions are necessary compared to one atomic CPU instruction in the proposed
implementation.
36
5.5
J. P. Schäfer
Operation: Weak and Strong CAS-Loops
Measuring the throughput of a CAS loop is done similar to measuring a single
CAS call, except that a CAS loop has to run until the CAS was successful.
Clearly, we use the retrieved value from a previous turn in the CAS loop for the
expected argument to the next CAS call.
Here, the no-contention and high-contention scenario shows roughly twice
the throughput. In all other test-scenarios, however, there is just a minor performance improvement. Figure 5 shows exemplary results.
5.6
Discussion
It’s no surprise that the standard’s implementation suffers under oversubscription, since it uses mutexes to synchronize concurrent threads. On the
other hand, Anthony William’s implementation is not lock-free either because
his implementation operates atomically on 128 Bit, which is not supported in
a lock-free fashion by most C++ compilers. Therefore, his implementation uses
mutexes internally as well. Still, it does not suffer under over-subscription as
much, as could be seen in all low- and high-contention scenarios in the experiments. I suspect that the reason for still performing well in over-subscription
use-cases is the fine-granularity of the mutex-based locks. On the other hand,
the lock-inversion problem would still occur, but has not been tested here.
Due to the arrangement of the packed pointer in the proposed implementation, adding or subtracting a value from its counter really just is an atomic
fetch_add or fetch_sub. In folly’s implementation, however, a CAS loop is
required to avoid an integer overflow when manipulating the packed pointer’s
counter. This CAS loop is prone to fail under heavy contention; and hence, the
performance drops for an operation that should yield at least constant throughput under increasing pressure of threads (see Fig. 1).
Fig. 5. Throughput of the weak CAS loop on an AMD Ryzen 9 5950X under high
contention
Atomic Shared Pointers
6
37
Conclusion and Outlook
This work provides a proposal for the implementation of atomic shared pointers, a crucial building brick for lock-free data structures. The implementation
extents the split-reference counting approach by allowing the local (temporary)
reference counters to be negative, which allows to reduce the number of atomic
instructions. Section 4 contributes a proof for the correctness of the proposed
algorithm, which is especially important for lock-free data structures, since they
are extra hard to debug. The evaluation in Sect. 5 shows that the proposed implementation outperforms other implementations up to an order of magnitude in
some high-contention scenarios and up to factor 2 in many other cases.
A lot of the performance is lost due to the necessity of cas loops where a
masked_cas as CPU instruction could avoid the problem: A shared_ptr implementation requires a CAS on a data type (e.g. 64 or 128 Bit) that combines
a (split) reference counter and an actual value (here a pointer). However, the
compare-part of the CAS shall only compare the pointer of the atomic variable
but ignore the counter in the comparison. If that pointer matches the expected
pointer, the CAS operation can exchange the whole atomic variable with a new
pointer and reference counter dealing with the old reference counter afterwards.
However, actual implementations have to rerun the CAS-loop due to simple
reference counter changes. With a masked_cas, these unnecessary but likely situations under high-contention, could be avoided. Here, the old value shall be
returned independently of the success of the masked_cas, so that the caller also
gets to know the old part that was masked out from the comparison, similar to a
fetch_add. Having such a CPU instruction could yield an increased performance
in both, the counted_pointer as well as the paired_counter. The generality
of using a bitmask for the comparison yields benefits in other algorithms as well.
References
1. ISO C++ committee (2011). https://isocpp.org/
2. Adve, S.V., Gharachorloo, K.: Shared memory consistency models: a tutorial. Computer 29(12), 66–76 (1996)
3. Anderson, D.: Lock-free atomic shared pointers without a split reference count?
In: CppCon 2023 (2023). https://isocpp.org/blog/2023/09
4. Dechev, D., Pirkelbauer, P., Stroustrup, B.: Understanding and effectively preventing the ABA problem in descriptor-based lock-free designs (2010). http://
www.stroustrup.com/isorc2010.pdf
5. Doumler, T.: April 2022. https://accu.digital-medium.co.uk/wp-content/uploads/
2022/04/talk.pdf
6. Doumler, T.: A lock-free atomic shared ptr. ACCU, April 2022. https://accu.org/
video/spring-2022-day-1/
7. Van Eerd, T.: Lock-free by Example. In: CppCon 2014 (2014). https://github.com/
CppCon/CppCon2014
8. Facebook: Facebook Open-Source Library “Folly” (2023). https://github.com/
facebook/folly
38
J. P. Schäfer
R Transactional Synchroniza9. Intel: Performance Monitoring Impact of Intel
tion Extension Memory Ordering Issue (2023). https://cdrdv2.intel.com/v1/dl/
getContent/604224
10. Meyers, S., Alexandrescu, A.: C++ and the perils of double-checked locking. Doctor Dobbs J. 29, 07 (2004)
11. Sutter. H.: Effective concurrency: lock-free code - a false sense of security. https://
herbsutter.com/2008/08/05/
12. Sutter, H.: The free lunch is over: a fundamental turn toward concurrency in software. Dr. Dobb’s J. 30(3), 202–210 (2005)
13. Sutter, H.: Atomic smart pointers, rev. 1. Standard C++ Foundation (2014).
https://isocpp.org/files/papers/N4162.pdf
14. Sutter, H.: Lock-free programming (or, juggling razor blades). In: CppCon 2014
(2014). https://github.com/CppCon/CppCon2014
15. Williams, A.: C++ Concurrency in Action: Practical Multithreading. Manning
Pubs Co., Manning (2012)
16. Williams, A.: Lock-free atomic shared pointers (2021). https://github.com/
anthonywilliams/atomic shared ptr
Optimised Round Robin with Virtual
Runtime for CPU Scheduling
Jagriti Bhatia1(B) , Sanskriti Mathuria1 , Vandana M. Ladwani1 ,
and Shobana Padmanabhan2
1
Department of Computer Science and Engineering, PES University, Bengaluru,
Karnataka, India
{pesug20cs145,pes1ug20cs386}@pesu.pes.edu, vandanamd@pes.edu
2
School of Computer Science and Engineering, RV University, Bengaluru,
Karnataka, India
shobanap@rvu.edu.in
Abstract. The Central Processing Unit (CPU) consists of the main
circuitry to perform all the operations in a computer. One of the most
important operations an Operating System performs is CPU Scheduling,
which is responsible for managing all tasks and allocating CPU time to
each task in an optimal order. This extends to applications in distributed
systems since scheduling is required to maximise server utilisation in
load-sharing mechanisms. One of the more commonly used scheduling
algorithms for the mentioned application is Round Robin (RR), which
is considered to be one of the most efficient. However, the efficiency of
RR entirely depends on the chosen Static Time Quantum. In this
paper, we have proposed an optimisation to the pre-existing RR algorithm, namely ‘Riti’, where Round Robin is implemented using Virtual
Runtime and by calculating Dynamic Time Quantum. ‘Riti’ has
achieved a 27.6% reduction in average Turnaround Time and 37.4%
reduction in average Waiting Time as compared to Traditional Round
Robin.
Keywords: CPU scheduling · Round Robin · Dynamic time
quantum · Virtual runtime · Turnaround time · Waiting time · Convoy
effect
1
Introduction
A process goes through multiple states from the time it is created. Figure 1
shows the primary states in the life cycle of a process and how a process may
pass through all of these states.
At any given time, it is possible only for one process to run on a single-core
CPU. Process scheduling is required for discarding a running/completed process
to which the CPU is allocated and choosing another process from the waiting
J. Bhatia, S. Mathuria, V. M. Ladwani, and S. Padmanabhan—Contributed equally.
c The Author(s), under exclusive license to Springer Nature Switzerland AG 2024
K. Arai (Ed.): SAI 2024, LNNS 1017, pp. 1–17, 2024.
https://doi.org/10.1007/978-3-031-62277-9_1
2
J. Bhatia et al.
Fig. 1. Process States
queue based on a particular strategy. This allows the OS to allocate CPU time
for each process, and hence keep the CPU busy at all times, enabling full use
of the CPU and trying to avoid the possibility of deadlock. There are a variety
of scheduling algorithms - Shortest Job First, First Come First Serve, Priority
Scheduling, Round Robin, etc.
The efficiency of a scheduling algorithm is measured through various criteria.
1. CPU Utilization - The aim is to keep the CPU as busy as possible so that no
CPU cycle goes to waste. Normal CPU utilization ranges from 50% to 100%.
2. Turnaround Time - The time a certain process takes, from being added to
the ready queue to completion, is called Turnaround time (or TAT) for that
process.
T AT = Completion T ime − Arrival T ime
(1)
3. Waiting time - The time spent by a process in the ready queue before it is
completed is known as waiting time.
W aiting T ime = T AT − Burst T ime
(2)
This paper focuses on Round Robin. Round Robin is a very frequently used
scheduling algorithm, however it has one major drawback - Convoy effect, due
to which the waiting time for the processes is very high. To tackle this issue, we
have proposed to optimise the Round Robin algorithm using ‘virtual runtime’.
The paper is structured in the following manner - Sect. 2 talks about the
relevant research that has been conducted on Round Robin variations for CPU
scheduling in the past. Section 3 provides a brief explanation of the basic concepts
of the ‘Riti’ algorithm, that is built on. Section 4 gives an in depth explanation
as to how the proposed algorithm works. Section 5 shows how ‘Riti’ performs
as compared to other Round Robin algorithms present currently. To conclude,
Sect. 6 highlights the benefits and limitations of the ‘Riti’ algorithm.
Optimised Round Robin
2
3
Related Work
The efficiency of a CPU depends on its ability to schedule processes and efficiently provide resources. Many different scheduling algorithms have been studied and worked upon to enhance the performance of CPU.
Alsulami et al. in their work talk about the necessity of a good scheduling
algorithm for a good operating system performance. The authors have studied
Round Robin with a dynamic time quantum. They state that the performandisplay shortest job first schedulingce of the dynamic time quantum depends on
the method used to calculate it. The authors calculated dynamic time quantum
using four different methods and have done a comparative study among them [4].
Various research papers use dynamic time quantums, such as Butangen et al. [6]
which make use of the dynamic mean of the burst times of the running processes.
Mishra and Mitawa too present an approach which sets the time quantum to
the average burst time [12].
Pathak et al. display shortest job first scheduling modification in traditional
Round Robin [16]. Even Srujana et al. [19] incorporate SJF in Round Robin.
‘Riti’ differs from these as it splits the processes into two queues and follows
different scheduling strategies for both queues, still giving preference to shorter
jobs by running the queue with shorter processes first.
Traditional Round Robin in the Cloud environment is not optimal as the
cloud environment is constantly subject to changes and this can cause the waiting
time of the processes to increase. Sanaj and Prathap propose that time slices
should be assigned on the basis of the mean burst time of all the tasks in the
waiting queue [18].
The idea of Recomputed Time Quantum [14] greatly influenced the calculation of dynamic time quantum. Aijaz et al. calculated dynamic time quantum
for each cycle by adding the total burst time to the previous time quantum and
dividing it by the number of processes [2]. We adopted this method in our proposed algorithm as well. However, Riti focuses on reducing the convoy effect and
hence gives importance to executing the shorter processes first. Riti also implements a fair-scheduling method for the shorter processes, hence differentiating
it from this paper.
Farooq et al. proposed that a suitable way of finding dynamic time quantum is
to use 0.8 ∗ max(burst times). The aim is to achieve the best results in efficiency
without having to sort the processing according to burst times in ascending order.
The author states that the goal is to lower the duration of an algorithm along
with efficiency constraints [7].
Dynamic time quantum is also implemented by Alaa et al., however, this
paper proposes to split the ready queue into two queues on the basis of the
medium burst time. After arranging the processes based on their burst time in
increasing order, they are split into two queues - light task queue and heavy task
queue, depending on the medium burst time. The light task queue is executed
before the heavy task queue and the time quantum for each queue is considered
to be the burst time of the medium processes respectively [3]. ‘Riti’ also places
processes into a light task queue or a heavy task queue. However, sorting based
4
J. Bhatia et al.
on burst time is not performed since the value based on which the processes
are split is not dependent on the burst times of the processes. Hence, avoiding
sorting reduces the time complexity.
H. B. Parekh and S. Chaudhari’s work [15] comes closest to ‘Riti’ in theory,
as they make use of SJF and priority scheduling. ‘Riti’ differs from this work as
it splits the ready queue into two based on the burst times, as compared to [15]
which uses a single queue.
3
Background
3.1
Round Robin Scheduler
Round Robin, the preemptive process scheduling algorithm, is cyclic in nature.
It is a widely used algorithm, mainly because it has a good average response
time. Despite being considered a very fair algorithm, Round Robin does have its
disadvantages. Varied burst times cause convoy effect and greatly degrade the
performance of the algorithm. Priorities of the processes are also not considered.
The convoy effect is a phenomenon that slows down the entire Operating
System due to the presence of a few large processes (CPU-intensive processes).
Essentially, when large processes arrive in the queue before the smaller processes,
they take a much larger time to execute and hence starve the smaller processes
of their fair share of resources.
The time quantum used for the execution of Round Robin is vital. It controls
the performance of the algorithm. If it is too small, there is a spike in the number
of context switches and the overhead will increase. On the other hand, if the time
quantum is too large, Round Robin will act as FCFS.
3.2
Completely Fair Scheduling (CFS)
The CFS scheduler attempts to implement fair-share scheduling but in an efficient manner. As seen in [8], CFS replaced O(1) scheduler in Linux as the O(1)
had low throughput for background jobs and poor interactive performance. If
the CPU following CFS switches too often, it means that the CPU is being
more fair but at the same time results in an increase of context switches. At the
same time, switching less frequently leads to less fairness. CFS uses two control
parameters to take care of this problem:
– Sched latency- This value is used for determining after how much time a
switch should be considered by the CPU. The time to be allocated to a single
sched latency
process is number
of processes ; hence the time allocated changes dynamically,
based on the workload on the CPU.
– Min granularity: This is the minimum amount of time a time slice can be; to
avoid many context switches it should not go lower than this value.
Optimised Round Robin
3.3
5
Virtual Runtime
This term comes up in the Completely Fair Scheduler (CFS). In this method, the
CPU uses a counting-based technique known as virtual runtime to fairly divide
itself among all competing processes. As each process runs, it accumulates this
virtual runtime. When the CPU has to execute a process, it chooses the one
with the least virtual runtime, thus ensuring equal distribution of CPU resources
among all the processes.
3.4
Niceness
CFS allows the users/admins to assign priority to processes in the form of a
‘nice’ value. This value ranges from −20 to 19. A positive value indicates that
the process is ‘nice’ and hence means the process does not need the CPU as
urgently. On the other hand, a negative value indicates that the process needs
the CPU more urgently and thus the process has higher priority. This nice value
can be used to calculate the weight of a process, which can be used in the
calculation of time slice. The formula and conversion from nice value to weight
are taken from [5].
The conversion from nice to static priority is defined by the NICE TO PRIO
macro in the Linux Kernel. It is defined in include/linux/sched/prio.h. The relation is prio = nice + 120. The conversion of nice value to priority to weight is
1024
roughly equivalent to 1.23
nice . This is defined in prio to weight in kernel/sched.c
weighti
time slicei = n−1
∗ sched latency
i=0 weighti
(3)
weight0
∗ runtime
weighti
(4)
vruntimei = vruntimei +
4
Proposed Methodology
As mentioned earlier, Round Robin does not take into account the priority of the
processes. In our proposed algorithm, ‘Riti’, we consider three things along with
the process ID - burst time, arrival time, and the nice value associated with the
process. Initially, the processes arrive and are placed into the queue. The ready
queue is then split into two smaller queues as shown in Fig. 2. If the burst time is
less than min granularity ∗ number of processes, then the process is assigned
to Queue-1. Else, the process is assigned to Queue-2. This results in Queue-1
containing all the processes with smaller burst times, and Queue-2 containing
all the processes with larger burst times. This is not a multi-scheduler algorithm,
the concept of splitting the processes into two queues has been adopted so that
the smaller processes are not starved of CPU time due to the larger processes.
The algorithm begins by running Queue-1 first so that the processes with
a smaller burst time can be executed before the processes with a greater burst
time, and are not starved. Queue-1 follows CFS concepts of scheduling, and uses
6
J. Bhatia et al.
nice value to calculate the virtual runtime in order to schedule the processes
based on their priority.
Once the execution of Queue-1 is completed, only then Queue-2 begins its
execution. Here, a dynamic time quantum formula is applied [2].
TQ =
LT Q + T BT
number of processes
(5)
TQ - Time Quantum LTQ - Last Time Quantum TBT - Total Burst Time
Here, Total Burst Time is the sum of the burst times of all the processes in
the queue. The time quantum will change every loop, based on the value of the
previous time quantum.
Finally, the values of average TAT and WT are calculated. These are compared to the average TAT and WT produced by running the same ready queue
using the traditional Round Robin algorithm. It is noticed that there is a significant difference between the two, as the optimized algorithm performed much
better.
Initially, to split the ready queue into Queue-1 and Queue-2, a time complexity of O(no of processes) is required. The time complexity for the execution of
Queue-1 is O(no of processes2 ), because the processes are looped through once
each cycle and a second loop is required to accumulate the virtual runtime.
The time complexity for the execution of Queue-2 is also O(no of processes2 ).
The dynamic time quantum must be calculated every cycle, hence resulting in
a O(no of processes2 ) complexity. As a result, the proposed algorithm’s overall
time complexity becomes O(no of processes2 ).
Fig. 2. Method to split the ready queue
Optimised Round Robin
7
Algorithm 1. Pseudo code for Riti
N ← number of processes
Q1 ← light load queue
Q2 ← heavy load queue
min gran ← 6
sched latency ← 48
for each process i in ready queue do
if Burst T imei ≤ min gran ∗ N then
Q1.append(process)
else
Q2.append(process)
end
end
// Q1 execution
for each process i in Q1 do
vruntimei ← 0 runtimei ← 0 weighti =
prio to weight[nice valuei + 20]time slicei =
weighti
n−1
∗ sched latency
weight
j=0
j
end
for each cycle in Q1 do
for each process i in Q1 do
runtimei = runtimei + time slicei if runtimei ≥ burst timei
then
Q1.remove(processi )
else
0
vruntimei = vruntimei + weight
weighti ∗ runtimei
end
end
sort Q1 according to vruntime in ascending order
end
// Q2 execution
BT
// TBT - total burst time
T Q ← TN
for each cycle in Q2 do
for each process i in Q2 do
runtimei ← T Q if runtimei ≥ burst timei then
Q2.remove(processi )
end
end
BT
new T Q ← LT Q+T
N
T Q ← new T Q
end
‘Riti’ works best when the ready queue processes have varied burst times. For
example, if there are large CPU-bound processes in the queue but an I/O process
8
J. Bhatia et al.
that has a short CPU burst time enters the queue, it is prioritised and finished
first. Even within multiple I/O processes with short CPU burst times, there is
fairness in the scheduling of the I/O processes. The main focus of ‘Riti’ is to
ensure the smaller processes get completed smoothly without getting disrupted
by the larger processes, making virtual runtime an important aspect.
5
Observation
The proposed methodology has been depicted with the help of two examples - (1) with common Arrival Time, and (2) with different Arrival Times.
The min granularity is considered to be 6 in the following examples, and the
sched latency is considered to be 48.
5.1
Example of Processes with Same Arrival Time
Here, the Arrival Time of all the processes is considered to be zero. It is assumed
that all the processes have already arrived before the scheduling begins. The
process details are given below in Table 1.
Table 1. Parameters and values used throughout the experiment
p id AT BT Nice Value
P1
0
35
3
P2
0
43
−12
P3
0
16
17
P4
0
41
−16
P5
0
14
0
In this example, the processes arrive at the same time but have randomly generated Burst Times and Nice Values.
Table 2. Optimized round robin
p id TAT WT
P1
65
30
P2
147
104
P3
30
14
P4
149
108
P5
15
1
Optimised Round Robin
9
When the scheduling of processes is according to the proposed algorithm, the
results observed are shown in Table 2, and the following values are observed as
well:
Avg TAT = 81.2
Avg WT = 51.4
P3 and P5 , being process with smaller burst times will join Queue-1. Shown
below is the Gantt chart for Queue-1.
P3
0
P5
1
P3
15
30
The process that will join Queue-2 are P1 , P2 and P4 , as they have larger burst
times. The following is the Gantt chart for Queue-2.
P1
30
P2
65
P4
104
P2
143
P4
147
Calculating TQ for traditional Round Robin: T Q =
Hence, T Q = f loor( 48
5 ) =9
149
Sched latency
number of processes
Table 3. Traditional Round Robin
p id TAT WT
P1
119
84
P2
144
101
P3
70
54
P4
149
108
P5
84
70
By applying the Traditional algorithm, the values in Table 3 are observed, along
with the following results: Avg TAT = 113.2 Avg WT = 83.4
The Gantt chart for traditional Round Robin is depicted below.
P1 P2 P3 P4 P5 P1 P2 P3 P4 P5 P1 P2 P4 P1 P2 P4 P2 P4
0
9
18 27 36 45 54 63 70 79 84 93 102 111 119 128 137 144 149
Figures 3 and 4 show the comparison of TAT and WT for this example.
10
J. Bhatia et al.
Fig. 3. Comparison of turnaround time (same arrival time)
Fig. 4. Comparison of waiting time (same arrival time)
From Figs. 3 and 4, in the case of P1 , P3 and P5 , a large change is noticed in
TAT and WT for Proposed and Traditional RR methodologies. This shows a
significant change in average TAT and average WT values.
5.2
Example of Processes with Different Arrival Time
In this example, the Arrival Times of the processes are not the same, they all
arrive at different times (see Table 4). The Burst Times and Nice Values for all
the processes have been randomly generated.
Table 4. Parameters and values used throughout the experiments
p id AT BT Nice Value
P1
0
17
−14
P2
1
39
15
P3
2
31
−5
P4
3
26
−8
P5
4
10
1
Optimised Round Robin
11
Table 5. Optimized round robin
p id TAT WT
P1
17
0
P2
122
83
P3
117
86
P4
41
15
P5
49
39
Using the proposed methodology, the following values are obtained from the
observed values in Table 5: Avg TAT = 69.2 Avg WT = 44.6 As shown in the
Gantt chart the lighter processes - P1 , P4 and P5 join Queue-1.
P1
0
P4
P5
17
26
P4
27
P5
44
P5
49
53
P2 and P3 , having larger burst times will join Queue-2 as shown below.
P2
53
P3
88
P2
119
123
As seen in the previous example, the TQ for traditional Round Robin will be 9.
Table 6. Traditional round robin
p id TAT WT
P1
53
36
P2
122
83
P3
118
87
P4
104
78
P5
77
67
With the Tradition Round Robin, the result is as follows, compiling the observed
values in Table 6: Avg TAT = 94.8 Avg WT = 70.2
The Gantt chart for traditional Round Robin is as follows.
P1 P2 P3 P4 P5 P1 P2 P3 P4 P5 P2 P3 P4 P2 P3 P2
0
9
18 27 36 45 53 62 71 80 81 90 99 107 116 120 123
Comparison of TAT and WT for example B (processes with different Arrival
Time) is depicted through Figs. 5 and 6.
12
J. Bhatia et al.
Fig. 5. Comparison of Turnaround Time (different Arrival Time)
Fig. 6. Comparison of Waiting Time (different Arrival Time)
A large difference is seen between TAT and WT respectively, for Proposed and
Traditional RR methodologies for processes P1 , P4 and P5 , as shown in Figs. 5
and 6.
5.3
Comparison with Other Round Robin Variations
To prove the effectiveness of the proposed methodology, ‘Riti’ has been compared with other Round Robin variations, namely Efficient Dynamic Round
Robin (EDRR) [7], New Median-Average Round Robin (NMARR) [17], Modified Round Robin CPU Scheduling Algorithm With Dynamic Time Quantum
(RRDTQ) [1], Efficient Round Robin Algorithm (ERRA) [2], An Improved
Round Robin CPU Scheduling Algorithm with Varying Time Quantum (IIRVQ)
[11], and Modified Median Round Robin Algorithm (MMRRA) [13]. Some of
these algorithms have also been compared in [9]. Three groups were created
based on the range of burst time of the processes – Group - 1: Burst time ranging between 5–100
– Group - 2: Burst time ranging between 5–150
– Group - 3: Burst time ranging between 5–250
Ten processes were randomly generated for each group. All the Round Robin
variations along with Traditional Round Robin, and ‘Riti’ were run against each
other for all the processes. It can be observed from the below graphs (see Figs. 7,
Optimised Round Robin
13
Fig. 7. Group - 1 Results
Table 7. Comparison of Riti in Group - 1
Algorithm TAT Comparison (in %) WT Comparison (in%)
ERRA
6.129
7.26
MMRRA
32.88
38.38
Fig. 8. Group - 2 Results
8 and 9) that ‘Riti’ had the least Waiting Time and Turn Around Time. A
comparison of the same is also depicted in Tables 7, 8 and 9.
From the graphs depicted above, it can be concluded that the proposed algorithm, ‘Riti’, leads to a significant difference in average Turnaround Time and
average Waiting Time as compared to the traditional algorithm.
The following tables show the performance of ‘Riti’ as compared to ERRA, which
was the best performing amongst the Round Robin Optimizations and MMRRA,
14
J. Bhatia et al.
Table 8. Comparison of Riti in Group - 2
Algorithm TAT Comparison (in %) WT Comparison (in%)
ERRA
8.46
10.6
MMRRA
30.17
35.753
Fig. 9. Group - 3 Results
Table 9. Comparison of Riti in Group - 3
Algorithm TAT Comparison (in %) WT Comparison (in%)
ERRA
3.67
4.69
MMRRA
26.8
32.14
which performed the most poorly. It can be observed that ‘Riti’ is significantly
better than both in terms of TAT and WT.
6
Conclusions
6.1
Benefits of ‘Riti’
As compared to the traditional Round Robin algorithm, ‘Riti’ shows a significant
reduction in both, average Turnaround Time as well as the average Waiting
Time, as shown in Figs. 10 and 11. It is also ensured that the smaller processes
are scheduled first by splitting them into a different queue. In this manner,
convoy effect is avoided.
From Table 10, it can be inferred that the average TAT for traditional RR
is 113.2, and for ‘Riti’ is 81.2, leading to a difference of a significant 32 units.
And the difference for WT between the two methodologies is 32 units. This large
difference can be visualised in the graph - Fig. 7.
Optimised Round Robin
15
Fig. 10. Comparison with traditional round robin (different arrival time)
Table 10. Traditional round robin vs optimized round robin (example a - processes
with same arrival time)
Algorithm
TAT WT
Traditional RR
113.2 83.4
Proposed RR (Riti) 81.2
51.4
The data in Table 11 indicates the difference between average TAT and WT
for both the proposed and traditional RR methodologies respectively. This result
can be visualised by the graph depicted in Fig. 8. Thus the proposed algorithm
‘Riti’ achieves significant performance improvement over the existing Round
Robin scheduling algorithm.
Fig. 11. Comparison with traditional round robin (same arrival time)
16
J. Bhatia et al.
Table 11. Traditional round robin vs optimized round robin (Example B - Processes
with different arrival time)
Algorithm
TAT WT
Traditional RR
94.8 70.2
Proposed RR (Riti) 69.2 44.6
6.2
Limitations of ‘Riti’
While comparing with other Round Robin variations, it was observed that ‘Riti’
tends to make more context switches while scheduling, which can result in the
reduction of the efficiency of the CPU with respect to scheduling processes. Also,
while this paper has used a simulation of processes to test out the proposed algorithm, an extensive study has not been carried out yet using real-time processes.
6.3
Future Scope
1. ‘Riti’ can be tested on more processes in a real-time scenario.
2. This algorithm has the scope to be applied as a load balancing algorithm
in distributed computing schedulers. Currently, Round Robin is being used a
static load balancing algorithm. The modifications made to traditional Round
Robin as shown in ‘Riti’ has the potential to be used as a dynamic load
balancing algorithm. It can predict which server the load can be assigned to
such that there is optimal usage of all servers.
3. Similar applications as the above, such as being used by scalable multiprocessors [10], or by network routers to allocate bandwidth to different data
packets. It can even be applied by database systems to distribute queries to
different database servers show that this algorithm has a lot of scope to be
used in real-life situations.
References
1. Sohrawordi: A modified Round Robin CPU scheduling algorithm with dynamic
time quantum. Int. J. Adv. Res. 7, 422–429 (2019)
2. Aijaz, M., Tariq, R., Ghori, M., Rizvi, S.W., Qazi, E.F.: Efficient round robin
algorithm (ERRA) using the average burst time. In: 2019 International Conference
on Information Science and Communication Technology (ICISCT), pp. 1–5 (2019)
3. Fiad, A., Zoulikha, M.M., Hayat, B.: Improved round robin scheduling algorithm
with varying time quantum. In: 2020 Second International Conference on Embedded and Distributed Systems (EDiS), pp. 33–37 (2020)
4. Alsulami, A.A., Al-Haija, Q.A., Thanoon, M.I., Mao, Q.: Performance evaluation
of dynamic round robin algorithms for CPU scheduling. In: 2019 SoutheastCon,
pp. 1–5 (2019)
5. Arpaci-Dusseau, R.H., Arpaci-Dusseau, A.C.: Operating Systems: Three Easy
Pieces, 1.00 edn. Arpaci-Dusseau Books, August 2018
Optimised Round Robin
17
6. Butangen, A.K.G., Velasco, C.E., Codmos, J.C.B., Bayani, E.F., Baquirin, R.B.:
Utilizing dynamic mean quantum time round robin to optimize the shortest job
first scheduling algorithm. In: Proceedings of 2020 6th International Conference
on Computing and Data Engineering, ICCDE 2020, pp. 14–18. Association for
Computing Machinery, New York (2020)
7. Farooq, M.U., Shakoor, A., Siddique, A.B.: An efficient dynamic round robin algorithm for CPU scheduling. In: 2017 International Conference on Communication,
Computing and Digital Systems (C-CODE), pp. 244–248 (2017)
8. Jose, J., Sujisha, O., Gilesh, M., Bindima, T.: On the fairness of Linux O(1) scheduler. In: 2014 5th International Conference on Intelligent Systems, Modelling and
Simulation, pp. 668–674 (2014)
9. Joshi, A., Goyal, S.B.: Comparison of various round robin scheduling algorithms.
In: 2019 8th International Conference System Modeling and Advancement in
Research Trends (SMART), pp. 18–21 (2019)
10. Li, T., Baumberger, D., Hahn, S.: Efficient and scalable multiprocessor fair scheduling using distributed weighted round-robin. In: Proceedings of the 14th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, PPoPP
2009, pp. 65–74. Association for Computing Machinery, New York (2009)
11. Mishra, M., Rashid, F.: An improved round robin CPU scheduling algorithm with
varying time quantum. Int. J. Comput. Sci. Eng. Appl. 4, 1–8 (2014)
12. Mishra, R., Mitawa, G.: Improved round robin algorithm for effective scheduling
process for CPU. In: 2021 Third International Conference on Intelligent Communication Technologies and Virtual Mobile Networks (ICICV), pp. 590–593 (2021)
13. Mora, H., Abdullahi, S.E., Junaidu, S.B.: Modified median round robin algorithm
(MMRRA). In: 2017 13th International Conference on Electronics, Computer and
Computation (ICECCO), pp. 1–7 (2017)
14. Oyam, N.A., Pidlaoan, L.J., Baquirin, R.B., Bayani, E.F., Fronda, R.J.: Refining
the round robin algorithm using a recomputed time quantum: a comparison. In:
Proceedings of 2020 6th International Conference on Computing and Data Engineering, ICCDE 2020, pp. 1–4. Association for Computing Machinery, New York
(2020)
15. Parekh, H.B., Chaudhari, S.: Improved round robin CPU scheduling algorithm:
round robin, shortest job first and priority algorithm coupled to increase throughput and decrease waiting time and turnaround time. In: 2016 International Conference on Global Trends in Signal Processing, Information Computing and Communication (ICGTSPICC), pp. 184–187 (2016)
16. Pathak, P., Kumar, P., Dubey, K., Rajpoot, P., Kumar, S.: Mean threshold shortest
job round robin CPU scheduling algorithm. In: 2019 International Conference on
Intelligent Sustainable Systems (ICISS), pp. 474–478 (2019)
17. Sakshi, C.S., Sharma, S., Kautish, S., Alsallami, S.A.M., Khalil, E.M., Mohamed,
A.W.: A new median-average round robin scheduling algorithm: an optimal approach for reducing turnaround and waiting time. Alexandria Eng. J. 61(12), 10527–
10538 (2022)
18. Sanaj, M.S., Prathap, P.M.J.: An enhanced round robin (ERR) algorithm for
effective and efficient task scheduling in cloud environment. In: 2020 Advanced
Computing and Communication Technologies for High Performance Applications
(ACCTHPA), pp. 107–110 (2020)
19. Srujana, R., Mohana Roopa, Y., Datta Sai Krishna Mohan, M.: Sorted round
robin algorithm. In: 2019 3rd International Conference on Trends in Electronics
and Informatics (ICOEI), pp. 968–971 (2019)
CICO2 e: A Compute Carbon Footprint
Estimation Tool Based on Time Series
Data
Christian Plewnia(B) and Horst Lichter
Research Group Software Construction, RWTH Aachen University, Aachen, Germany
{plewnia,lichter}@swc.rwth-aachen.de
Abstract. In recent years, multiple research papers and tools were published that addressed the interest of estimating the carbon footprint of
computational activities. Generally, the presented carbon footprint estimation methods calculate the product of the compute hardware’s energy
consumption and a factor expressing the emissions for the corresponding
energy produced in the region where the compute hardware is located.
However, there are three open issues. First, the methods for determining the energy consumption are inaccurate or lack an evaluation of the
accuracy. Second, most tools use as carbon intensity a static long-term
average, e.g., over a year, that the tool authors gathered once, but since
some regions have a carbon intensity varying each day and throughout
the year, the accuracy of using a static carbon intensity is unclear and
requires an evaluation. Third, most tools enable estimates for a single
computer or a homogeneous set of computers only, excluding the easy
carbon footprint estimation for scenarios with a heterogeneous set of
compute hardware. In this paper, we make three contributions. First, we
analyze the evaluation gap regarding methods for determining the compute hardware’s power consumption. Second, using example cases, we
show that estimating a carbon footprint using a static long-term average
carbon intensity compared to hourly carbon intensity time series data
can lead to large errors; in the worst case among the example, the error
was 325.8%. Third, we present a tool assisting with carbon footprint
estimates using time series carbon intensity data and also supporting
heterogeneous compute hardware.
Keywords: Carbon footprint · Energy consumption · Green
computing
1
Introduction
Carbon emissions are made responsible for the climate change [1], creating a
strong interest in understanding and reducing the carbon footprint of many
kinds of energy-consuming activities, e.g., industry production processes, energy
production, and traveling. Understanding an activity’s carbon footprint involves
quantifying the greenhouse gas (GHG) emissions attributed to the activity.
GHGs include carbon dioxide (CO2 ), methane (CH4 ), and nitrous oxide (N2 O).
c The Author(s), under exclusive license to Springer Nature Switzerland AG 2024
K. Arai (Ed.): SAI 2024, LNNS 1017, pp. 39–55, 2024.
https://doi.org/10.1007/978-3-031-62277-9_3
40
C. Plewnia and H. Lichter
To simplify the quantification, the individual emissions can be expressed as CO2 equivalent (CO2 e) emissions.
The interest in understanding an activity’s carbon footprint reached computer science. Contributions come from the area of machine learning [3–6,15]
and cloud computing [9–12]. The contributions include various tools for estimating the carbon footprint of compute activities [14–20]. Briefly summarized,
these tools first measure or estimate the energy consumption of a single computer or compute infrastructure and then estimate the carbon footprint caused
by the energy consumption by multiplying the energy consumption with a carbon intensity (CI) for a region chosen by the user. The CI expresses the mass of
CO2 e emissions per unit of energy consumption, e.g., as gram CO2 e per kilowatt
hour.
There are three issues with the existing tools for estimating the carbon footprint. First, these tools use methods for estimating the energy consumption that
are inaccurate or lack an evaluation of the accuracy. We explain this in the
related work in more detail.
Second, many tools use per selected region a static CI gathered once by the
tools’ authors, e.g., an annual average CI. Using a static CI is accurate if the user
knows the time interval covered by the average CI and makes an estimate for
this time interval. However, users generally do not intend to estimate the carbon
footprint for this predefined time interval only. If the real CI rarely deviates
from the static CI, this issue could be negligible, but in some regions, the mix of
energy production and the CI varies strongly over time; an example is Finland
in 2022 as shown in Fig. 1. Thus, the issue is that the accuracy of the estimated
carbon footprint depends on how well the static CI represents the time interval
chosen by the user.
Fig. 1. Finland’s energy production carbon intensity in 2022
Compute Carbon Footprint Estimation Based on Time Series Data
41
Third, most of the tools assume a compute infrastructure consisting of a single server or a homogeneous set of servers. However, there are scenarios like
the operation of web applications where the underlying compute infrastructure
can be heterogeneous, i.e., it consists of different kinds of hardware with individual power. Thus, the third issue is the currently limited applicability of the
estimation tools that excludes some computational activities.
In this paper, we make three contributions. First, in Sect. 2, we present an
overview of the existing tools and their carbon footprint estimation methods,
we describe the lack of evaluation, and we discuss the data gap that prevents
an evaluation. Second, in Sect. 3, we quantify the error of estimating a carbon
footprint with an annual average CI for exemplary cases; we compared estimations based on an annual average CI with estimations based on an hourly CI
time series for selected regions and time intervals in 2022. Third, in Sect. 4, we
present an estimation tool that fills the gap of relying on hourly time series CI
data; this tool enables estimates for single computers and a set of different computers, i.e., a heterogeneous compute infrastructure. Besides the contributions,
we provide some ideas for future work in Sect. 5 and summarize our findings in
Sect. 6.
2
Related Work
The interest in determining the carbon footprint of computational tasks is not
new. As shown in Table 1, there are multiple tools for assisting users in estimating
the carbon footprint of a computational activity:
– Reporting dashboards Some cloud providers give their customers access
to dashboards reporting the customer’s past cloud computing usage’s carbon
footprint [9–11]. Of all the tools discussed here, they are the only ones considering emissions from other activities (e.g., production and waste) besides
the data centers’ energy consumption. Those emissions are reported using the
Greenhouse Gas Protocol scopes as categorization [2]. While this provides a
much broader perspective on the emissions, the tools have only a rough documentation or none at all and the exact calculations are not made public.
An alternative integrating estimates for multiple cloud providers in one dashboard is the open source tool “Cloud Carbon Footprint” (as of September
2023, the supported providers are AWS, Azure, and GCP) [18]. For each
cloud provider considered, the tool obtains the user’s cloud usage using the
provider’s API, maps the usage of cloud resources to hardware (considering
CPUs, GPUs, memory, storage and network traffic), estimates the hardware’s
energy consumption, and, finally, estimates the carbon footprint.
– Instrumentation libraries There are multiple libraries that allow developers to instrument their code such that the code execution’s energy consumption is measured and the carbon footprint is estimated [3,6,7,15].
– Estimation assistants The web applications “Green Algorithms” and
“ML CO2 Impact” enable their users to estimate a compute activity’s carbon
footprint based on the activity’s runtime and the used hardware [4,5]. ML
42
C. Plewnia and H. Lichter
CO2 Impact considers either a CPU or GPU, the computation’s runtime and
the computation’s location, while Green Algorithms also considers memory
and allows to consider both types of processors at the same time.
In the remainder of this section, we provide an overview of the methods used by
the aforementioned tools since the methods’ details are crucial to the estimations’
accuracy. For this analysis, we studied the openly available documentations and
source codes for all but the cloud providers’ tools. The tools have in common that
they first determine a compute activity’s energy consumption and then multiply
the result with a carbon intensity to obtain the footprint. They differ in how
they determine energy consumption and what carbon intensity information they
use, which we explain subsequently.
2.1
Determination of the Energy Consumption
The tools compute the energy consumption by summing up the estimated or
measured power of selected hardware components as shown in Table 1.
Power measurements for CPUs, GPUs, and memory rely on processor features. The Running Average Power Limit (RAPL) interface initially developed
by Intel can provide the CPU’s and memory’s power consumption. The NVIDIA
Management Library (NVML) interface does the same for GPUs by NVIDIA.
Power estimations for CPUs and GPUs are based on either the processor’s
Thermal Design Power (TDP) (Green Algorithms, ML CO2 Impact) or on factors derived from datasets with power measurements (Cloud Carbon Footprint).
Processor power estimations using the TDP assume it to be an approximation
of the processor’s power. A processor’s TDP specifies the thermal energy that
the processor may dissipate, which serves as a reference for what the cooling
must be able to handle. Since most of a processor’s power ends up as heat
and processor manufacturers have an interest in sufficient cooling to ensure the
processor’s stable operation, we conclude that the TDP can be considered as an
upper bound to the processor’s power.
The processor power estimations by Cloud Carbon Footprint base on factors derived from the SPECpower dataset for CPUs and a dataset with power
measurements for GPUs created by the company Teads [13]. Each record in the
SPECpower database contains information on the total power consumption of
one or more servers when the CPU idles and when it is at max load. From this
data, the authors of Cloud Carbon Footprint derived an aggregated idle and
max load energy consumption per CPU thread (also referred to as vCPU) for
each CPU family, e.g., Intel Coffee Lake and Skylake. With a mapping of cloud
services to the combination of CPU families, a number of vCPUs, an assumed
load as well as hours of cloud service usage, the CPU energy consumption is
estimated. The approach for GPUs is similar.
To estimate the power of memory, storage, and network traffic factors are
used. For memory, the factor relies on statements made by memory manufacturers Crucial and Micron [21,22]. Cloud Carbon Footprint’s power estimations
for storage and network rely on estimated factors [13]. The factor for storage is
unknown
unknown
(dataset)
(TDP)
(TDP)
e
m
m
e
m
m
e
e
e
Azure Emissions Impact Dashboard1
Google Cloud Carbon Footprint1
Cloud Carbon Footprint
carbontracker [3]
code carbon2
Eco2AI [7]
Experiment Impact Tracker [6]
Tracarbon3
Green Algorithms [5]
ML CO2 Impact
CICO2 e (this paper)
(TDP)
(RAPL)
(RAPL)
(TDP)
(RAPL)
(RAPL)
unknown
CPU
AWS Customer Carbon Footprint Tool
Tool
(NVML)
(dataset)
e
e
e
(TDP)
(TDP)
(TDP)
m (NVML)
m (NVML)
m (NVML)
m (NVML)
m
e
unknown
unknown
unknown
GPU
e
e
e
m
m
e
e
m
e
e
(factor)
unknown
unknown
unknown
storage
(factor)
(factor)
(factor)
(RAPL)
(RAPL)
(factor)
(factor)
(factor)
not considered
e
unknown
unknown
unknown
network
not considered
static
static
static
static
static
static
time series
static
unknown
unknown
unknown
intensity
Carbon
not considered not considered time series
not considered not considered
not considered not considered
not considered not considered
not considered not considered
not considered
not considered not considered
(RAPL) not considered
(factor)
unknown
unknown
unknown
memory
Power [(m)easure/(e)stimate (method)]
(1) provides emissions for GHG protocol scopes 1, 2, and 3, i.e., covers more than emissions related to computational energy consumption (2) if
RAPL is not available, falls back to TDP-based estimate for CPU; if available, uses latest carbon intensity retrieved from API for private compute
infrastructures (3) row values only apply for local computers running linux; different method used for cloud servers
instrumentation
reporting
libraries
assistants
estimation
dashboards
Table 1. Software tools for determining a compute activity’s carbon footprint and their methods for determining the energy consumption
and carbon intensity
Compute Carbon Footprint Estimation Based on Time Series Data
43
44
C. Plewnia and H. Lichter
derived from projections for 2020 published in the 2016 U.S Data Center Usage
Report.
A current issue of all power estimation approaches is a missing validation.
The TDP-based estimation is likely inaccurate as it does not consider known
variances in the power depending on the processor’s load, but it provides an
upper bound for a processor’s power. Although the derivation of the factors
used for memory, storage, and network traffic appears plausible, their accuracy
is unknown; the authors of Cloud Carbon Footprint describe this transparently in
their documentation’s methodology section [13]. Thus, currently all estimation
tools are experimental. In our opinion, a thorough evaluation of the existing
estimation methods cannot be conducted without more comprehensive data on
the power consumption of the hardware components.
2.2
Determination of the Carbon Intensity
All considered tools estimate the carbon footprint using a CI reflecting the energy
mix of the server’s location, i.e., a CI for the data center or the region thereof.
Many tools use per location a static CI value gathered once by the authors.
We refer to such a CI that is constant over time as static. An exception is the
instrumentation library carbontracker [3], which recurrently requests an updated
CI during the execution of the instrumented code. This is an example of what
we refer to as time series CI, i.e., a series of CI values over time.
3
Carbon Footprint Estimation with Static vs. Time
Series Carbon Intensity
In this section, we study the accuracy of estimating the carbon footprint of a
compute activity with a static CI as it is practiced by many tools presented
in the related work. We quantify the error of estimating with a static CI by
comparing it to estimating with hourly time series CI data, i.e., as a baseline
we use the more accurate estimation based on the time series CI data. As Fig. 1
shows, a region’s CI can vary over the time of day and throughout the year,
indicating that estimating with a static CI may exhibit strong inaccuracies.
The error cannot be expressed as a single static number but depends on three
variables: the CI development (the time series CI data) of the region of the data
center we use to run a compute activity as well as the compute activity’s starting time and runtime. Thus, our study has to consider these three dimensions
when quantifying the error. Conducting a study across the full extent of these
dimensions is not feasible and also not necessary to provide first insights into the
accuracy of estimating with a static CI. Thus, we limit the considered regions
and time to exemplary cases.
3.1
Method
To quantify the error, we start by estimating the carbon footprint for compute
activities of varying runtimes using a static CI and time series CI data. As
Compute Carbon Footprint Estimation Based on Time Series Data
45
static CI, we do not use the values used by the tools of the related work as this
comparison would be convoluted and unfair for the tools: convoluted since not
all tools use the same static values and they do not all cover the same regions;
unfair since their static CI values are not all based on the same time frames.
Instead, we calculate each region’s average CI over a common time frame and
use this average CI as the region’s static CI.
We continue with estimating the carbon footprint for hypothetical compute
activities with runtimes from one hour to the whole year, e.g., for 2022 the runtimes are in the interval [1, 8760]. For each runtime, we calculate the carbon
footprint for every possible starting hour such that the compute activity completes within the time frame. For example, considering 2022 again, for a one-day
runtime, we calculate the carbon footprint for the job starting on the 1st of
January 2022 at 00:00, on the first of January at 01:00 until the latest possible
starting point on the 31st of December 2022 at 00:00; for a 1 year runtime, there
is only one possible start time to stay within 2022, which is January 1st at 00:00.
Similar to the tools of the related work, we assume the compute activity causes
a static power consumption. We calculate with a value of one kilowatt, but the
value is not important as we are only interested in how the carbon footprints
estimated with the static CI and the time series CI data relate to each other.
Then, we determine how much the carbon footprint estimated with the static
CI deviates from the carbon footprint estimated with the time series CI data, i.e.,
we determine the error in the estimation using the static CI. For the comparison,
we look at the relative error, i.e., for a geographical region g, a starting time s,
and a runtime r, the relative error is the difference in the carbon footprint
estimated using the static CI and the time series CI data in relation to the
carbon footprint estimated using the time series CI data:
relError (g, s, r) =
footprintstatic (g, s, r) − footprintts (g, s, r)
footprintts (g, s, r)
For the error analysis, we look at maxima as well as average values. In the first
case, we are interested in finding out what error to expect in the worst case.
For this, we proceed as follows: let Sr be the set of all starting times such that
a job with a runtime t completes within the considered time frame; then, for
every region and runtime, we search across the starting times Sr for where the
absolute value of the relative error is the highest:
maxError (g, r) = max abs (relError (g, s, r))
s∈Sr
In the second case, we are interested in finding out what error to expect when
we would start a job at a randomly selected time within the considered time
frame. To achieve this, we calculate for every region and runtime the average of
the relative errors’ absolute values across all starting times:
abs (relError (g, s, r))
avgError (g, r) = s∈Sr
|Sr |
46
3.2
C. Plewnia and H. Lichter
Dataset (Regions and Time Frame)
We applied the method to CI data provided by the company Electricity Maps
[24]. It gathers CI data for many regions in the world from various sources
and provides unified datasets. In June 2023, Electricity Maps published historic
datasets for 2021 and 2022 under an open-source license. The datasets contain
CI data for the direct emissions of the energy production as well as life cycle
analysis (LCA) emission data that includes not only direct emissions from the
energy production but also indirect emissions, e.g., “emissions from the extraction of resources required to build up installed capacity, emissions from direct
operations, and end-of-life related emissions” [23]. The datasets come in multiple
resolutions, with the most fine granular being hourly CI data.
In this study, we used as time series CI data the hourly LCA CI data by
Electricity Maps for 2022 for the regions Finland, France, Germany, the UK,
and Poland. We found those regions to be interesting as they exhibited different
mixes in their energy production. However, the choice of regions is not crucial as
we only intend to demonstrate a few exemplary cases of quantifying the error.
3.3
Results
In the presentation of the results, we focus on runtimes of up to 30 d as this
covers some of the largest batch jobs reported in machine learning in recent
years, e.g., Google’s Meena (30 d), T5 (20 d), Switch Transformer (27 d), and
OpenAI’s GPT-3 (about 15 d) [8]. Common compute activities are nightly batch
jobs or build jobs, which typically run no more than half a day. Therefore, we
decided to select runtimes of the extreme values (1 h and 8760 h), as well as a
few representative runtimes of nightly jobs (3, 6, and 12 h) and long-running jobs
(15 and 30 d).
Figure 2a and Table 2a depict the maximum errors when estimating with the
region’s annual CI instead of the region’s hourly CI. For example, for a compute
activity running in a data center in France for 15 d, the error in the carbon
footprint estimation will be at most 46.3%. We find a few aspects of the results
noteworthy:
1. With longer runtimes, the maximum errors tend to get smaller, although they
are not monotonically falling.
2. The runtimes representative for nightly and machine learning jobs are within
the lower range of runtimes where the maximum error is rather high.
3. The maximum errors are always the highest for compute activities running
for one hour. This coincides with extremes in the hourly CI data.
4. The maximum errors are always the lowest for compute activities running for
8760 h, i.e., jobs running for the whole year 2022. The error is always 0 for
8760 h since the job’s time frame is the same as the time frame over which
the annual CI was calculated.
One aspect not shown by the here presented results is the direction of the deviation. In some cases, the relative error identified as the maximum error is negative,
Compute Carbon Footprint Estimation Based on Time Series Data
47
Fig. 2. Carbon footprint estimation errors using an annual CI relative to the estimation
using hourly CIs for runtimes between 1 h and 1 year (8760 h) and selected regions in
the year 2022.
meaning the carbon footprint estimated with the annual CI is lower than the
one estimated with the hourly CI.
Figure 2b and Table 2b depict the average errors when estimating with the
region’s annual CI instead of the region’s hourly CI. Due to the average applied,
the errors shown here are lower than the maximum errors shown before. Similarly, as for the maximum errors, the average errors are the highest for a runtime
of one hour and, with an error of 0%, the lowest for a runtime of 8760 h.
48
C. Plewnia and H. Lichter
Table 2. Errors in estimating a compute activity’s carbon footprint using an annual CI
relative to estimating the footprint using hourly CIs for a few representative compute
activity runtimes for selected regions in the year 2022.
(a) Maximum errors
Runtime [h]
8760 (= 1 year)
720 (= 30 days)
360 (= 15 days)
12
6
3
1
Maximum error [%]
Finland France Germany
UK Poland
0.0
123.5
145.0
221.1
225.4
228.2
231.4
0.0
30.7
52.9
201.5
267.3
293.2
294.7
0.0
35.1
41.3
169.3
175.3
181.2
183.3
0.0
43.3
77.2
243.2
276.2
313.5
325.8
0.0
12.7
14.4
52.2
76.7
83.8
85.6
(b) Average errors
Runtime [h]
8760 (= 1 year)
720 (= 30 days)
360 (= 15 days)
12
6
3
1
Average error [%]
Finland France Germany UK Poland
0.0
39.4
40.0
43.9
44.3
44.5
44.6
0.0
12.0
14.3
25.7
26.7
27.3
27.6
0.0
7.4
10.5
23.8
26.2
27.3
27.8
0.0
10.6
14.1
34.4
36.6
37.9
38.5
0.0
3.6
4.6
9.3
10.4
10.9
11.1
The materials used to obtain the presented analysis results are provided in
the paper’s supplemental material. This covers the input datasets by Electricity
Maps, the source code for computing the errors, and the resulting datasets with
the maximum and average errors for all runtimes across the selected regions.
3.4
Discussion
The study allows us to draw three conclusions regarding the error of estimating
a compute activity’s carbon footprint with an annual CI instead of an hourly CI
for the considered time frame of 2022 and the regions Finland, France, Germany,
the UK, and Poland:
1. There can be large errors in the carbon footprint estimation. For instance, in
the worst case, we would have had a large error of 325.8% (maximum error
for the UK and a 1-hour runtime).
Compute Carbon Footprint Estimation Based on Time Series Data
49
2. On average, the expected errors are much lower than the maximum error. For
example, the average error for a 1-hour compute activity running in the UK
is 38.5%.
3. The errors highly depend on the considered compute activity, i.e., the region
where the compute activity is executed as well as the compute activity’s
starting time and runtime. For example, for compute activities running in the
considered regions for 1 h or 720 h, the maximum error ranges from 12.7% to
325.8% and the average error ranges from 3.6% to 44.6%.
The above conclusions are specific to the year and regions and thus may not be
applicable to other years and regions. In general, the study revealed that there
is a risk of large errors when estimating a compute activity’s carbon footprint
with a static CI instead of time series CI data. Further, in general, we have no
knowledge about the extent of the errors when using a static CI until we compare
the results to estimations with time series CI data.
The aforementioned problems lead to the question of whether to always prefer
the carbon footprint estimation with time series CI data. The tool we present
in the next section makes it possible to conduct carbon footprint estimations
using time series CI data with little additional effort compared to the tools
presented in the related work. The only additional input that a user must provide
is information about the compute activity’s starting time. The tool takes care
of the estimations as long as time series CI data is available – an effort that can
be taken care of by the tool developers. Thus, the only case where a simplified
estimation with a static CI is preferable is when an estimation with time series
CI data is impossible due to missing data.
3.5
Threats to Validity
This study can only consider a certain time frame. Future changes in the energy
mix, e.g., caused by building new renewable energy capacity, and varying weather
conditions can affect a region’s CI. Thus, the errors obtained by this study are
not valid beyond the considered time frame. However, we think it is unlikely that
many regions will have a (close to) constant CI in the foreseeable future, which
is why we think the study’s idea will remain relevant beyond the considered time
frame.
4
Compute Carbon Footprint Estimation Tool (CICO2 e)
The Compute Infrastructure CO2 e estimator (CICO2 e) is a web application
enabling its users to estimate the energy consumption and carbon footprint for
a configurable heterogeneous compute infrastructure using CI time series data.
The user can configure the composition of the compute infrastructure, which is
used to determine the energy consumption, and choose what CI time series data
to use, which is used together with the energy consumption to determine the
carbon footprint. Subsequently, we describe the tool’s usage.
50
C. Plewnia and H. Lichter
To obtain an estimate, the user must first configure a compute infrastructure.
The user can add data centers to the infrastructure and servers to each data
center. An example of an infrastructure with one data center and two types
of servers is shown in Fig. 3. Each server is defined by how much energy it
consumes and a usage plan specifying one or more rules for how many instances
of that server are running at a time, e.g., 1 server instance running in November
2022 or a weekly recurring pattern of six instances running on weekdays during
2022. Figure 4 shows an example of a usage plan and its rules configuration.
The flexible configuration of the usage rules allows for modeling different kinds
of compute activities, e.g., one-time scientific jobs, constantly running machine
learning clusters and flexibly scaling web application clusters. To assist the user
in estimating the energy consumption of a server, an estimate can be calculated
based on a specification of the CPU, the GPU and the memory. The energy
consumption is estimated based on the TDP (for the CPU and GPU) and on a
factor (for the memory).
Fig. 3. Screenshot of the compute infrastructure composition in CICO2 e
The servers only define the direct energy consumption of the compute hardware. Indirect energy consumption by supplemental hardware like cooling and
network devices is covered by the Power Usage Effectiveness (PUE) factor, which
a user can configure for the data center. It is up to the user to get the PUE from
the data center’s provider.
For the data center, the user must also configure the CI data matching the
data center’s energy mix. We added hourly CI datasets covering 12 regions for
Compute Carbon Footprint Estimation Based on Time Series Data
51
Fig. 4. Screenshot of the usage plan configuration in CICO2 e
the years 2021 and 2022, which were released by Electricity Maps [24] using the
Open Database License. In the example in Fig. 3, the data center uses the CI
data for Ireland.
Once the compute infrastructure is configured, the user can request the carbon footprint estimate. It is computed by a server application in the background.
The estimations use the processors’ TDP and the factor for the memory power
consumption that is also used by other tools. Since we use the TDP for the
processor power estimation, the result is not the real consumption but an upper
bound. Using the result from the server, the user interface shows the estimated
total energy consumption, the estimated total carbon footprint of the energy
consumed (GHG protocol scope 2 emissions), and a graph depicting a development of power consumption and carbon footprint over the time of the compute
infrastructure usage. For example, Fig. 5 shows an estimate for the previously
configured compute infrastructure.
The application was designed such that CI datasets can be added by adapting
the backend’s configuration. This allows developers to integrate updated hourly
CI data by Electricity Maps or other sources, including the latest Electricity
Maps data that is available for a fee.
52
C. Plewnia and H. Lichter
Fig. 5. Screenshot of the estimated carbon footprint in CICO2 e
5
Future Work
Future work should focus on the evaluation of compute hardware energy estimation methods, which requires addressing the data gap. For processors and
memory, we imagine that a benchmark similar to the SPECpower benchmark
could be developed, which does not rely on a complex setup for measuring the
AC input, but uses the RAPL and NVML interfaces discussed in the related
work; however, this goes with the assumption that these interfaces provide reliable power measurements.
Once the evaluation is done, the next step is to look into methods for optimizing the carbon footprint of compute activities based on estimations; we found
some work going in this direction, but we think this will not be fruitful until we
have a reliable estimation method.
While in our experience many consumers of energy pay a fixed price per
kilowatt hour, we can imagine that variable pricing can become more common
in the future. The more the energy production becomes renewable, the more the
availability of energy may vary – and with it the price. In such a case, being able
to estimate energy costs using time series pricing data could become interesting,
which could be achieved by replacing the CI time series data with pricing time
series data in our tool. Similarly, paying compensation for carbon emissions could
become more common in the future. If this happens, being able to estimate and
minimize the compensation expenses could become an interesting topic as well.
Compute Carbon Footprint Estimation Based on Time Series Data
53
We have also plans for the future of CICO2 e. Electricity Maps intends to
provide free access to their carbon intensity datasets for 2023 at some point;
as soon as they are available we want to integrate them into our tool. We also
consider adding assistance for users in composing cloud compute infrastructures
since hardware specifications of cloud servers are usually not easy to find; the
Cloud Carbon Footprint project already gathered information that could be used
to realize this assistance.
6
Conclusion
In this paper, we addressed three issues of carbon footprint estimation tools.
First, in Sect. 2 we show that many tools’ methods or estimating the energy
consumption of a computer lack an evaluation. Thus, for many tools the estimate’s accuracy is unknown. An exception are tools relying on the processor’s
TDP: while the TDP does not correspond to the energy consumption, it provides
an upper bound to it. The general issue with developing a reliable, more accurate method is the data gap with respect to the power consumption of compute
hardware. Closing this gap is a challenge left for future work.
The second issue we addressed is that most tools use a static carbon intensity
(CI) to estimate the carbon footprint. We showed in Sect. 3 that the use of a static
CI can lead to inaccurate carbon footprint estimates. Among the exemplary cases
we analyzed, the error of calculating the footprint with a static CI was up to
325.8% compared to calculating with an hourly CI time series. The highest error
occur for the compute jobs with short runtimes and the error tends to decrease
with longer compute job runtimes. To overcome these errors, we recommend
carbon footprint estimation tools to rely on time series CI data instead of a
static CI if possible or to clearly document what CI is used and what it means
for the estimate’s accuracy.
Third, there was a gap not covered by existing tools. They could create a
carbon footprint estimate for either a past cloud infrastructure that could be
composed of a heterogeneous set of servers or for a model of a past or planned
infrastructure that, however, could consist of a single computer or a homogeneous set of servers only. CICO2 e is an addition to the existing tools that fills
this gap as it can estimate the carbon footprint for a model of a past and planned
heterogeneous compute infrastructure alike. Following our own recommendation,
CICO2 e estimates carbon footprints based on time series CI data. We intend to
continue the development of CICO2 e (e.g., by supporting easier modelling of
compute infrastructures composed of public cloud servers) and welcome contributions.
Supplemental material
– Analysis
The datasets and the source code of the analysis in Sect. 3 can be found in
the paper’s supplemental material published at GitHub:
https://github.com/swc-rwth/CICO2e-paper-supplemental-material
54
C. Plewnia and H. Lichter
– CICO2 e application
The CICO2 e web application is online accessible at:
https://cico2e.swc.rwth-aachen.de
– CICO2 e source code
The source code of CICO2 e is published at GitHub:
https://github.com/swc-rwth/CICO2e
References
1. Intergovernmental Panel on Climate Change (IPCC) (Ed.): Climate Change 2022
- Mitigation of Climate Change: Working Group III Contribution to the Sixth
Assessment Report of the Intergovernmental Panel on Climate Change. Cambridge:
Cambridge University Press (2023). https://doi.org/10.1017/9781009157926
2. The Greenhouse Gas Protocol: A Corporate Accounting and Reporting Standard, revised ed. ed. o.O.: World Resources Institute and World Business Council
for Sustainable Development (2004). https://ghgprotocol.org/corporate-standard.
Accessed Nov 24 2023
3. Anthony, L.F.W., Kanding, B., Selvan, R.: Carbontracker: Tracking and predicting
the carbon footprint of training deep learning models (2020). https://doi.org/10.
48550/arXiv.2007.03051
4. Lacoste, A., Luccioni, A., Schmidt, V., et al.: Quantifying the carbon emissions of
machine learning (2019). https://doi.org/10.48550/arXiv.1910.09700
5. Lannelongue, L., Grealey, J., Inouye, M.: Green algorithms: quantifying the carbon
footprint of computation. Adv. Sci. 8(12), 1–10 (2021). https://doi.org/10.1002/
advs.202100707
6. Henderson, P., Hu, J., Romoff, J., et al.: Towards the systematic reporting of
the energy and carbon footprints of machine learning (2022). https://doi.org/10.
48550/arXiv.2002.05651
7. Budennyy, S.A., Lazarev, V.D., Zakharenko, N.N., et al.: eco2AI: carbon emissions tracking of machine learning models as the first step towards sustainable AI. Dokl. Math. 106(Suppl 1), S118–S128 (2022). https://doi.org/10.1134/
S1064562422060230
8. Patterson, D., Gonzalez, J., Le, Q., et al.: Carbon emissions and large neural
network training (2021). https://doi.org/10.48550/arXiv.2104.10350
9. Amazon: Amazon Web Services Customer Carbon Footprint Tool. https://
aws.amazon.com/aws-cost-management/aws-customer-carbon-footprint-tool/.
Accessed Nov 24 2023
10. Microsoft: Azure Customer Carbon Footprint Tool. https://www.microsoft.com/
en-us/sustainability/emissions-impact-dashboard. Accessed Nov 24 2023
11. Google: Google Cloud Platform Customer Carbon Footprint Tool. https://cloud.
google.com/carbon-footprint. Accessed Nov 24 2023
12. Thoughtworks, Inc.: Cloud Carbon Footprint. https://www.cloudcarbonfootprint.
org/. Accessed Nov 24 2023
13. Thoughtworks, Inc.: Cloud Carbon Footprint – Methodology documentation.
https://www.cloudcarbonfootprint.org/docs/methodology/. Accessed Nov 17 2023
14. Carbontracker. Software source code repository. https://github.com/lfwa/
carbontracker. Accessed Nov 24 2023
15. CodeCarbon. Software source code repository. https://github.com/mlco2/
codecarbon/. Accessed Nov 24 2023
Compute Carbon Footprint Estimation Based on Time Series Data
55
16. Eco2AI. Software source code repository. https://github.com/sb-ai-lab/Eco2AI.
Accessed Nov 24 2023
17. Experiment Impact Tracker. Software source code repository. https://github.com/
Breakend/experiment-impact-tracker. Accessed Nov 24 2023
18. Green Algorithms. Software source code repository; git commit: ef4d0a1. https://
github.com/GreenAlgorithms/green-algorithms-tool. Accessed Nov 24 2023
19. ML CO2 Impact. Software source code repository. https://github.com/mlco2/
impact. Accessed Nov 24 2023
20. Tracarbon. Software source code repository. https://github.com/fvaleye/
tracarbon. Accessed Nov 24 2023
21. Crucial: How Much Power Does Memory Use? https://www.crucial.com/support/
articles-faq-memory/how-much-power-does-memory-use. Accessed 17 Nov 2023
22. Micron: Calculating Memory Power for DDR4 SDRAM. Technical note, revision
B 8/18 EN (2017). https://www.micron.com/-/media/client/global/documents/
products/technical-note/dram/tn4007 ddr4 power calculation.pdf. Accessed Nov
24 2023
23. Electricity Maps ApS: Electricity Maps Methodology. https://www.
electricitymaps.com/methodology. Accessed Nov 24 2023
24. Electricity Maps ApS: Electricity Maps Data Portal. https://www.electricitymaps.
com/data-portal. Accessed Nov 24 2023
Image Classification Method Based on Chaos
Neural Network
Kohei Arai(B)
Saga University, Saga City 840-8502, Japan
arai@is.saga-u.ac.jp
Abstract. An image classification method with Chaos Neural Network (CHAOS
NN) is proposed. CHAOS NN allows to find closer to a globally optimum solution
than Back Propagation Neural Network (BPNN) due to the fact that behavior of
the solution from CHAOS NN is chaotic so that the solution may get out from
local minima. In order to determine the parameters required for CHAOS NN,
training samples are used. The required parameters for the simulation are calculated beforehand using the training samples. Experimental results with simulation
and actual satellite imagery data of Landsat TM show that the proposed CHAOS
NN based image classification performance are better than BPNN performance
by 18.2–54.3% for the simulation data while by 13.4% for Landsat-5 TM data,
respectively.
Keywords: Neural network · Chaos neural network · Back propagation learning
process · Image classification
1 Introduction
Image classification using a conventional multi-layer neural network (BPNN) with backpropagation learning reflects the error in the weighting factor of the previous layer so
as to minimize the squared error of the error in the output layer [1]. By doing so, the
steepest descent learning of the optimum weighting coefficient is performed. Therefore,
the optimized weighting factor depends on the given teacher, and there is no guarantee
that it will be included in the global optimal solution, so there is no guarantee that it will
be classified into the optimal class.
A chaos-neural network (CHAOS NN) has been proposed as a learning method that
increases the possibility of converging on this global optimal solution [2]. CHAOS NN
is used in addition to periodic responses when learning [3]. It also incorporates a chaotic
response that draws an unstable orbit. In other words, when a global optimal solution
cannot be found only by a periodic response under a given condition of CHAOS NN,
there is a possibility of finding a global optimal solution by using unstable solution
behavior and chaotic response [4]. However, since the number of weighting factors to
be optimized increases in CHAOS NN, this is calculated efficiently and effectively [5].
In this paper, it is first proposed a method for optimizing the weighting factors
required when applying CHAOS NN to supervised image classification. CHAOS NN
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2024
K. Arai (Ed.): SAI 2024, LNNS 1017, pp. 56–73, 2024.
https://doi.org/10.1007/978-3-031-62277-9_4
Image Classification Method Based on Chaos Neural Network
57
neurons are normal BPNN neurons with feedback from the output added. Therefore, in
addition to the weighting factor between the nodes of each layer of BPNN, it is necessary
to determine the weighting factor that determines the effect of this feedback. The former
is determined by applying the steepest descent learning as in BPNN, and the latter is an
index of ease of classification, which is the average of the ratios of interclass variance
and intraclass variance calculated from the training sample.
It is proposed a method to determine using). It is also proposed an image classification
method based on the learning method and applied this to simulation data and actual
earth observation satellite image (Landsat-5 TM: Thematic Mapper) data to show the
superiority of the proposed method using CHAOS NN. It is shown in comparison with
the image classification by BPNN.
The following section describes the related research works on neural network and
classification methods followed by the proposed method. Then the experiments, simulation study and the experimental study are described followed by conclusions with some
discussions.
2 Related Research Works
As for the neural network related research works, there are as follows.
A sea surface temperature estimation method for ocean areas and seasons using a
GIS as Neural Network: NN is proposed [6] together with Sea Surface Temperature: SST
estimation method for ocean areas and seasons using Geographic Information System as
a neural network [7]. Also, sea surface temperature estimation method for ocean areas
and seasons u sing a GIS as neural network is reviewed [8].
Maximum entropy method based blind separation by means of neural network with
feature enhancements by wavelet analysis is proposed [9]. Meanwhile, visualization of
learning process for back propagation neural network clustering is investigated [10].
On the other hand, convolutional neural network considering physical processes and its
application to disaster detection is proposed [11].
As for the classification relates research works are as follows,
Classification by re-estimating statistical parameters based on auto-regressive model
is conducted [12]. In the meantime, multi-temporal texture analysis in TM classification is discussed [13]. Also, Maximum Likelihood: MLH TM classification taking into
account pixel-to-pixel correlation is proposed [14].
A supervised TM classification with a purification of training samples is proposed
[15] together with TM classification using local spectral variability [16]. Also, classification method with spatial spectral variability is proposed [17]. Meanwhile, TM
classification using local spectral variability is discussed [18].
Application of inversion theory for image analysis and classification is proposed [19].
On the other hand, polarimetric SAR image classification with maximum curvature of
the trajectory in Eigen space domain on the polarization signature is proposed [20].
A hybrid supervised classification method for multi-dimensional images using color
and textural features is proposed [21]. On the other hand, polarimetric SAR image
classification with high frequency component derived from wavelet multi resolution
analysis: MRA is proposed [22].
58
K. Arai
Comparative study of polarimetric SAR classification methods including proposed
method with maximum curvature of trajectory of backscattering cross section in
ellipticity and orientation angle space is conducted [23].
Human gait gender classification using 2D discrete wavelet transforms energy is
proposed [24]. Together with Human gait gender classification using 2D discrete wavelet
transforms energy [24]. In the meantime, Human gait gender classification in spatial and
temporal reasoning is discussed [25].
Comparative study on discrimination methods for identifying dangerous red tide
species based on wavelet utilized classification methods is conducted [26]. Meanwhile,
multi spectral image classification method with selection of independent spectral features
through correlation analysis is proposed [27].
Image retrieval and classification method based on Euclidian distance between normalized features including wavelet descriptor is proposed [28]. On the other hand, gender classification method based on gait energy motion derived from silhouettes through
wavelet analysis of human gait moving pictures is proposed [29] together with gender classification method based on gait energy motion derived from silhouettes through
wavelet analysis of human gait moving pictures [29].
Human gait skeleton model acquired with single side video camera and its application and implementation for gender classification is proposed [30]. Meanwhile, gender classification method based on gait energy motion derived from silhouette through
wavelet analysis of human gait moving pictures is proposed [31]. In the meantime, human
gait gender classification using 3D discrete wavelet transformation feature extraction is
proposed [32].
Image classification considering probability density function based on Simplified
beta distribution is proposed in particular for SAR image classifications [33]. On the other
hand, Maximum Likelihood: MLH classification based on classified result of boundary
mixed pixels for high spatial resolution of satellite images is proposed and overviewed
[34].
Context classification based on mixing ratio estimation by means of inversion theory is proposed [35]. Optimum spatial resolution of satellite-based optical sensors for
maximizing classification performance is investigated [36].
3 Proposed Method
3.1 Chaotic Neuron Model
Figure 1 shows the chaotic neuron model that constitutes CHAOS NN. In normal neurons,
Aj (input is input, x (t + 1) is output, but in chaos neurons, feedback (t) from the output
is added to the input.
This feedback causes the behavior of chaos neurons. It becomes chaotic. Also,
because of this chaotic behavior, there is a possibility of getting out of the local solution
in the space of the solution, and the possibility of converging to the global optimum
solution increases. The reason for introducing chaotic neurons into image classification
is that BPNN with normal neurons is likely to fall into a local solution, that is, misclassification, but CHAOS NN with chaotic neurons is more likely to break out of the local
solution and classify it into a global optimal solution, that is, a correct class.
Image Classification Method Based on Chaos Neural Network
59
Fig. 1. Chaos Neuron model
The chaotic neuron model is expressed by the following equations:
xi (t + 1) = f (μi (t) + ηi (t) + ζi (t))
μi (t + 1) = κe μi (t) +
M0
(1)
Vip Ap (t)
(2)
Wij h(xi (t))
(3)
ζi (t + 1) = κr ζi (t) − αxi (t) − θi (1 − κr )
(4)
ηi (t + 1) = κf ηi (t) +
p=1
N0
j=1
where the symbols used here are as follows:
x (t): Output of the first neuron at time t
k e : Time decay constant for external input term (0 < k e < 1)
k f : Time decay constant (0 < k f < 1) for the combined input term from another neuron
k r : i-th neuron’s own refractory time decay constant (0 < k r < 1)
Ap (t): The p-th external input at time t
V ie : Coupling coefficient from the external input formula A to the first neuron
W ij : Coupling coefficient from j-th neuron to i-th neuron
α: Non-negative parameter that scales the refractory term
θ i : Threshold of i-th neuron
M 0 : Number of external inputs
N 0 : Number of neurons in the network
μi (t): Term related to external input in the i-th neuron
ηi (t): Term related to connection input from other neurons in the network at the i-th
neuron
ξ i (t): Term related to refractoriness in the i-th neuron
60
K. Arai
Of these parameters, n, island, and n are coefficients that indicate the contribution of
the external input, feedback input, and the neuron itself, respectively, and were fixed at
1/2 in this study. Increasing these will increase the contribution of each input and speed
up the learning process, but it also increases the possibility of diverging and finding a
solution or falling into a local solution. On the other hand, if these are made smaller,
there is a higher possibility that learning will be collected in the global optimal solution,
but the speed of collection will be slower. It is necessary to determine the coefficients
based on these trade-offs, but since the purpose of this study is to show the superiority
of CHAOS NN in image classification in comparison with BPNN usually accompanied
by neurons, these coefficients were fixed. In addition, the function h, and the function f
are expressed by the following equations:
1(x > 0.5)
h(x) =
(5)
0(x < 0.5)
1
(6)
f (y) =
1 + exp(−εy)
A CHAOS NN is a combination of chaotic neurons that are interconnected or
hierarchically connected.
3.2 Chaotic Neuron Model
Network Configuration. Figure 2(a) shows the configuration diagram of the proposed
CHAOS NN. Figure 2(b) shows a configuration diagram of the error backpropagation
neural network BPNN used for ratio comparison.
3.3 Chaos NN Parameter Learning
The parameters used for CHAOS NN are more than those for BPNN with normal neurons.
These parameters are determined by learning using a training sample and a teacher set
(data set of correct input/output) as in the error back propagation neural network. In this
learning, the weighting factor for the external input, V, and the weighting factor for the
feedback input from the chaos neuron itself, W are updated.
The following two types of learning methods have already been proposed [4].
• V is a method using an error backpropagation algorithm [1], and w is a method using
the first order autoregressive model which is expressed as follows:
Vij (n + 1) = Vij (n) − η
∂E
∂Vij
Wn+1 = AWn
(7)
(8)
• V is constant and W is a method using an error back propagation algorithm.
Vij = 1.0
Wij (n + 1) = Wij (n) − η
(9)
∂E
∂Wij
(10)
Image Classification Method Based on Chaos Neural Network
61
Fig. 2. BPNN and CHAOS NN
The variable obtained by the error backpropagation algorithm is either V or W. In
the former, it is difficult to prove the validity of expressing w by an autoregressive
model, and in the latter, the degree of freedom of the solution is reduced by fixing V,
and the possibility of reaching the global optimum solution is small. There is a risk of
becoming. However, it is sufficient if there is enough chaotic behavior to get out of the
local solution when the chaotic behavior becomes weak, and if there is feedback input,
the chaotic behavior will vary to some extent.
Since it is clear to show, it is decided to adopt the latter. Furthermore, as will be
described later, it was judged that the sensitivity of CHAOS NN to the image classification
accuracy by V is small, and that there are few problems in image classification accuracy
even if V is fixed.
The method of determining α, which is another important parameter, has never
been clarified in previous studies. Here, it is proposed a method for determining α
62
K. Arai
as a result of sensitivity analysis for image classification accuracy. That is, for α and
V, the index obtained from the target image in advance, and the value obtained by
dividing the interclass variance by the intraclass variance from the training sample of
each class (indicating the ease of classification, classification. It is proposed a method of
determining from (called indicators) and using them in a fixed manner. The relationship
between α and V, which are optimal for these target images, and the classification index
shall be investigated in advance using simulation data of the seed maiden. It is also
proposed a method for obtaining W by the error back propagation method using the
same training sample.
3.4 CHAOS NN Condition Setting Based on Simulation Data
The simulation data was generated based on “an improved method of Neyman-Scott’s
method [5]” as follows,
1) Set the class center (intra-class average) using uniform random numbers in the range
of [0,1] for each feature dimension.
2) With that as the center, a normal distribution N {class mean m, class variance is used
to generate individuals in the class}.
3) The distance between the center of the generation class was calculated, and it was
assumed that the class individuals were generated until they reached the set value (β)
and replaced with the already generated individuals.
4) Convert class individuals to pixel values of [0–255] similar to LANDSAT TM satellite
images.
In the simulation experiment, the number of classes was set to 2 and the number of
bands was set to 2. In addition, 20 types of random numbers with different distances
were prepared by changing the initial value of random numbers.
The optimum values of the CHAOS NN parameter, α, and weight V obtained from
the experiment using the simulation data are different for each image. These optimum
values are considered to be related to the classification index. These relationships are
shown in Fig. 3 and 4.
Figure 3 shows the relationship between the optimum CHAOS NN, the parameter, α
and the classification index. The vertical axis is the optimum α value in the training sample. A place where multiple points are plotted indicates that there are multiple optimum
points. The solid line is an approximation of each plot point in the least square method.
Figure 4 shows the relationship between the classification index and V. From Fig. 4, the
optimum V is within the range of 0.1 per person, and it can be seen that fixing V does
not affect the classification accuracy, while the optimum α is in the range of 1 to 7. It
varies, and it seems better to change it depending on the image than to fix it. However,
looking at this variation, I could not find a systematic relationship with the classification
index, so I decided to use a fixed value.
Image Classification Method Based on Chaos Neural Network
63
Fig. 3. Relationship between the optimum CHAOS NN, the parameter, α, and the classification
index
Fig. 4. Relationship between the classification index and V
Experiment on the number of neurons in the hidden layer shows the followings.
In constructing a neural network, it is necessary to consider the effect of the number
of neurons in the intermediate layer (hidden layer) on learning. In his simulation experiment, it is investigated the relationship between the difficulty of separation between
classes and the number of neurons in the hidden layer.
In this experiment, it is found the number of neurons that showed the most efficient
classification results when the number of neurons in the hidden layer of the chaotic neural
64
K. Arai
network was changed. In Fig. 5, the horizontal axis shows the difficulty of classification,
and the vertical axis shows the number of neurons when the classification efficiency is
the best.
From this figure, the more difficult it is to classify, the more neurons in the hidden
layer need to be increased.
Fig. 5. Relation between classification index and the number of neurons
4 Experiment
4.1 Results Using Simulation Data
The following three simulation data were generated by the method described in the
previous section. Figure 6 shows three typical scatter plots of the data used. In each
scatter plot, the horizontal axis is band 1 and the vertical axis is band 2. Table 1 shows
the between-class distance and within-class variance of each of the data used. That is, it
is simulation data that can be easily classified into J-tops of data 1 to 3.
Table 1. Between-class distance and within-class variance of each of the data used
_
within-class-variance
between-class-distance
between-class-distance/within-class-variance
Data#1
0.0339,0.0366
0.0783
0.556
Data#2
0.0828,0.0955
0.3963
1.111
Data#3
0.0895,0.0807
0.6052
1.778
Image Classification Method Based on Chaos Neural Network
Fig. 6. Typical scatter plots of the data used
65
66
K. Arai
4.2 Comparison of Classification Efficiency
After the learning was completed, the classification accuracy (Percent of Correct Classification: PCC) was calculated. The results are shown in Tables 2 and 3. From Tables 2
and 3, it was found that the classification accuracy of the chaotic neural network exceeds
that of the error backpropagation neural network in all cases.
Table 2. Confusion matrix for BPNN
Data 1
Data 2
Data 3
_
class#1
class#2
_
class#1
class#2
_
class#1
class#2
class#1
63
37
class#1
93
7
class#1
100
0
class#2
9
91
class#2
0
100
class#2
0
100
PCC = 77.0
PCC = 64.5
PCC = 100
Table 3. Confusion matrix for CHAOS NN
Data 1
Data 2
Data 3
_
class#1
class#2
_
class#1
class#2
_
class#1
class#2
class#1
91
9
class#1
100
0
class#1
100
0
class#2
9
91
class#2
1
99
class#2
0
100
PCC = 91.0
PCC = 99.5
PCC = 100
4.3 Learning Speed Comparison
Figure 7 shows the learning speed for each simulation data. From Fig. 7, it was found
that the learning speed of CHAOS NN is slower than that of BPNN, but the learning
achievement is high for all simulation data.
There is no big difference between BPNN and CHAOS NN for images that are easy
to classify, such as data 2 and 3. However, in the case of data 1 which is difficult to
classify, BPNN collected the data after learning about 1800 times with an error of 0.325,
whereas in the case of CHAOS NN, learning progressed further. At about 3000 times,
the error reached about 0.26 and the data was collected. In other words, CHAOS NN is
more likely to search for a global optimal solution (correct class) than BPNN, and the
tendency is so great that it is difficult to classify, but it can be seen that learning takes
about twice as long as BPNN.
Image Classification Method Based on Chaos Neural Network
Fig. 7. Learning speed for each simulation data
67
68
K. Arai
4.4 Classification Index and Classification Efficiency
Figure 8 shows the relationship between the classification index and classification efficiency. From Fig. 8, it can be seen that the smaller the value on the horizontal axis, that
is, the more difficult it is to classify, the smaller the PCC value. It can also be seen that
the classification accuracy of the chaotic neural network is higher than that of the error
backpropagation neural network in all cases.
Fig. 8. Relationship between the classification index and classification efficiency
4.5 Evaluation Using Real Satellite Images
The Landsat-5 TM (Thematic Mapper) image observed around Saga City on 12 May
1986 was used as the actual image (Fig. 9). Landsat-5 TM covers the wavelength range
from visible to near infrared and consists of 6 bands. The average vector of each band
and variance-covariance matrix is as follows:
Band 1 Band 2 Band 3 Band 4 Band 5 Band 7
96.19 44.75 46.16 93.43 82.18 35.89
⎡
⎤
193.91 103.95 179.35 −143.94 168.90 171.33
⎢ 103.95 60.04 101.94
−60.38 105.74 98.21 ⎥
⎢
⎥
⎢
⎥
⎢ 179.35 101.94 179.30 −122.74 186.55 174.14 ⎥
⎢
⎥
⎢ −143.94 −60.38 −122,74 473.32 93.35 −52.39 ⎥
⎢
⎥
⎣ 168.90 105.74 186.55 92.35 530.00 330.38 ⎦
171.33 98.21 174.14 −52.39 330.38 252.62
The designated classes are forest, waterbody, paddy field, bare soil and urban.
Image Classification Method Based on Chaos Neural Network
69
Fig. 9. Landsat-5 TM image data used
4.6 Setting CHAOS NN Parameters
From Fig. 3 and 4, it is decided to use α = 3.0 and V = 0.98 as the optimum CHAOS
NN parameters for classification based on the ratio of the interclass distance and the
intraclass variance. From Fig. 5, the number of neurons (nodes) in the hidden layer was
set to 4.
4.7 Classification Result
Figure 10(a) and (b) show the classification result images of each neural network after
learning. Table 4 shows a comparison of classification accuracy. From these results, in the
error back propagation neural network, a large improvement was observed in the image
classification by CHAOS NN, as opposed to the class in which the misclassification was
large as seen in Forest, Paddy, etc.
Compared with PCC, BPNN is 73.4% while CHAOS NN is 83.2%, it is found
that the CHAOS NN achieved 13.8% improvement of the classification performance in
comparison to that of BPNN. This is thought to be due to the fact that the behavior of the
chaotic solution increased the probability that CHAOS NN would classify it as a global
optimal solution, that is, a correct class.
70
K. Arai
Fig. 10. Classification result for CHAOS NN and BPNN
Table 4. Classification accuracies for BPNN and CHAOS NN
_
Urban
Forest
Paddy
Soil
Water
(a) Back Propagation Neural Network
Urban
65
18
3
2
12
Forest
11
69
3
3
14
Paddy
12
26
55
4
3
(continued)
Image Classification Method Based on Chaos Neural Network
71
Table 4. (continued)
_
Urban
Forest
Paddy
Soil
Water
Soil
10
6
8
78
6
Water
0
0
0
0
100
(b) Chaos Neural Network
Urban
73
12
4
1
10
Forest
8
79
3
1
9
Paddy
7
11
76
4
2
Soil
6
2
3
88
1
Water
0
0
0
0
100
5 Conclusion
In this paper, it is proposed a method for determining internal parameters in a chaotic
neural network (CHAOS NN) and tried to apply it to image classification. Using various
simulation data with different ease of classification, the relationship between the optimal
parameter of CHAOS NN and the ease of classification (the average ratio of the interclass
distance calculated from the teacher set to the intraclass variance).
Also, it is investigated and made it possible to determine the parameters according
to the ease of classification of the target image. Image classification using CHAOS NN
with the proposed parameter determination was evaluated for classification accuracy for
both her and real images. The classification accuracy of the image classification method
using this CHAOS NN was compared with that of the error back propagation neural
network (BPNN).
The degree of improvement in this classification accuracy varies depending on the
ease of classification of the target image, but it is 18.2 to 54.3% for the simulation
data and 13.4% for the actual image (Landsat-5 TM image). In addition, when the time
required for evaluation learning was also evaluated, CHAOS NN is more likely to search
for a global optimal solution (correct class) than BPNN, and the tendency is so great
that it is difficult to classify, but for learning. It is found to take about twice as long as
BPNN.
6 Future Research Works
Further study is required for validation of the proposed chaotic neural network with the
other dataset of the images. Also, the other application of the proposed chaotic neural
network is used for not only classification but also for regressive analysis in the future.
Acknowledgment. The author would like to thank Prof. Dr. Hiroshi Okumura and Prof. Dr.
Osamu Fukuda of Saga University for their valuable comments and suggestions. The author
also would like to thank Mr. Soichiro Takamoto of former student of Saga University for his
experimental effort to this study.
72
K. Arai
References
1. Baba, N., Kojima, F., Ozawa, S.: Basics and Applications of Neural Nets. Kyoritsu Shuppan
Co., Ltd. (1994)
2. Aihara, K.: Chaos in Neural Systems. Tokyo Denki University Press (1993)
3. Aihara, K.: Neural computer Learning from the brain and nerves. Tokyo Denki University
Press (1988)
4. Arai, K.: How to use Earth Observation Satellite Data by Kohei Arai. Morikita Publishing
Co., Ltd. (2000)
5. Neyman, J., Scott, E.L.: Statistical approach to problems of cosmology. J. Roy. Stat. Soc. Ser.
B 20, 1–43 (1958)
6. Arai, K.: A sea surface temperature estimation method for ocean areas and seasons using a GIS
as neural network. In: Kusuda, T., Yamanishi, H., Spearman, J., Gailani, J.Z. (eds.) Sediment
and Ecohydraulics INTERCOH 2005, pp. 43–51. Elsevier (2008). ISBN 978-0-444-53184-1
7. Arai, K.: Sea Surface Temperature estimation method for ocean areas and seasons using Geographic Information System as a neural network. In: Sediment and Ecohydraulics, Proceedings
in Marine Science, vol. 9. Elsevier (2007). ISBN 978-0-444 53184-1
8. Arai, K.: Chapter 4: a sea surface temperature estimation method for ocean areas and seasons
u sing a GIS as neural network. Proc. Mar. Sci. 9, 43–51 (2008)
9. Arai, K.: Maximum entropy method based blind separation by means of neural network with
feature enhancements by wavelet analysis. Int. J. Res. Rev. Comput. Sci. 2(4), 1116–1122
(2011)
10. Arai, K.: Visualization of learning process for back propagation neural network clustering.
Int. J. Adv. Comput. Sci. Appl. 4(2), 234–238 (2013)
11. Arai, K.: Convolutional neural network considering physical processes and its application to
disaster detection. Int. J. Adv. Comput. Sci. Appl. IJACSA 10(12), 105–111 (2019)
12. Arai, K.: Classification by re-estimating statistical parameters based on auto-regressive model.
Can. J. Remote. Sens. 16(3), 42–47 (1990)
13. Arai, K.: Multi-temporal texture analysis in TM classification. Can. J. Remote. Sens. 17(3),
263–270 (1991)
14. Arai, K.: Maximum likelihood TM classification taking into account pixel-to-pixel correlation. J. Int. GEOCARTO 7, 33–39 (1992)
15. Arai, K.: A supervised TM classification with a purification of training samples. Int. J. Remote
Sens. 13(11), 2039–2049 (1992)
16. Arai, K.: TM classification using local spectral variability. J. Int. GEOCARTO 7(4), 1–9
(1992)
17. Arai, K.: A classification method with spatial spectral variability. Int. J. Remote Sens. 13(12),
699–709 (1992)
18. Arai, K.: TM classification using local spectral variability. Int. J. Remote Sens. 14(4), 699–709
(1993)
19. Arai, K.: Application of inversion theory for image analysis and classification. Adv. Space
Res. 21(3), 429–432 (1998)
20. Arai, K., Wang, J.: Polarimetric SAR image classification with maximum curvature of the
trajectory in eigen space domain on the polarization signature. Adv. Space Res. 39(1), 149–154
(2007)
21. Okumura, H., Yamaura, M., Arai, K.: A hybrid supervised classification method for multidimensional images using color and textural features. J. Image Electron. Eng. Japan 38(6),
872–882 (2009)
22. Arai, K.: Polarimetric SAR image classification with high frequency component derived from
wavelet multi resolution analysis: MRA. Int. J. Adv. Comput. Sci. Appl. 2(9), 37–42 (2011)
Image Classification Method Based on Chaos Neural Network
73
23. Arai, K.: Comparative study of polarimetric SAR classification methods including proposed
method with maximum curvature of trajectory of backscattering cross section in ellipticity
and orientation angle space. Int. J. Res. Rev. Comput. Sci. 2(4), 1005–1009 (2011)
24. Arai, K., Andrie, R.: Human gait gender classification using 2D discrete wavelet transforms
energy. Int. J. Comput. Sci. Netw. Secur. 11(12), 62–68 (2011)
25. Arai, K., Asunara, R.A.: Human gait gender classification in spatial and temporal reasoning.
Int. J. Adv. Res. Artif. Intell. 1(6), 1–6 (2012)
26. Arai, K.: Comparative study on discrimination methods for identifying dangerous red tide
species based on wavelet utilized classification methods. Int. J. Adv. Comput. Sci. Appl. 4(1),
95–102 (2013)
27. Arai, K.: Multi spectral image classification method with selection of independent spectral
features through correlation analysis. Int. J. Adv. Res. Artif. Intell. 2(8), 21–27 (2013)
28. Arai, K.: Image retrieval and classification method based on Euclidian distance between
normalized features including wavelet descriptor. Int. J. Adv. Res. Artif. Intell. 2(10), 19–25
(2013)
29. Arai, K., Asmara, R.A.: Gender classification method based on gait energy motion derived
from silhouettes through wavelet analysis of human gait moving pictures. Int. J. Inf. Technol.
Comput. Sci. 6(3), 1–11 (2014)
30. Arai, K., Asmara, R.A.: Human gait skeleton model acquired with single side video camera
and its application and implementation for gender classification. J. Image Electron. Eng. Soc.
Japan Trans. Image Electron. Vis. Comput. 1(1), 78–87 (2014)
31. Arai, K., Asmara, R.A.: Gender classification method based on gait energy motion derived
from silhouette through wavelet analysis of human gait moving pictures. Int. J. Inf. Technol.
Comput. Sci. 5(5), 12–17 (2013)
32. Arai, K., Asmara, R.A.: Human gait gender classification using 3D discrete wavelet
transformation feature extraction. Int. J. Adv. Res. Artif. Intell. 3(2), 12–17 (2014)
33. Arai, K.: Image classification considering probability density function based on simplified
beta distribution. Int. J. Adv. Comput. Sci. Appl. IJACSA 11(4), 481–486 (2020)
34. Arai, K.: Maximum likelihood classification based on classified result of boundary mixed
pixels for high spatial resolution of satellite images. Int. J. Adv. Comput. Sci. Appl. 11(9),
24–30 (2020)
35. Arai, K.: Context classification based on mixing ratio estimation by means of inversion theory.
Int. J. Adv. Comput. Sci. Appl. 11(12), 44–50 (2020)
36. Arai, K.: Optimum spatial resolution of satellite-based optical sensors for maximizing
classification performance. J. Adv. Comput. Sci. Appl. 12(2), 363–369 (2021)
Human-Created and AI-Generated Text: What’s
Left to Uncover?
Steven Salter, Phoey Lee Teh(B) , and Richard Hebblewhite
Faculty of Art, Computing and Engineering, Wrexham University, Wrexham, UK
{phoey.teh,r.hebblewhite}@glyndwr.ac.uk
Abstract. The advent of generative Artificial Intelligence (AI) has brought about
profound changes in society, education, and the professional realm. Machine
Learning models have created remarkably sophisticated language generators,
blurring the line between human-authored content and AI-generated text. This
poses a challenge for educators and professionals in distinguishing authentic work
from instances of plagiarism. This study investigates the fundamental distinctions
between human and AI texts by analyzing perspectives from human points of
view. It aims to present the outcome of those five questions that we examined
about human perceptions of text composition and contrasting them with computergenerated text. By exploring this, it aids in upholding academic integrity and
contributes to advancing our comprehension of Natural Language Processing.
In essence, this research strives to maintain academic credibility in a landscape
transformed by AI and nurtures the growth of more equitable AI technologies.
Keywords: Artificial intelligent · Chat GPT · Human generated-text · AI
generated text
1 Introduction
Machine Learning (a key part of AI) has led to the creation of advanced Large Language
Models (LLMs). These models can produce very realistic text that resembles human
writing. It’s getting harder to tell if a text was written by a person or generated by AI,
and this puzzle has attracted a lot of media focus [1]. A model like ChatGPT has gained
widespread attention and quickly drawn millions of users [2]. ChatGPT is like a smart
tool that understands and generates human-like text. It uses a bunch of clever techniques
to quickly create clear and logical sentences. It can match the way humans write, and
this is the important focus of this paper. The tool uses model that learned from a lot
of text. The new version, ChatGPT 4.0, released in April 2023, is even better and can
handle a lot of information at once.
Generative AI has a lot of good things it can do for society, but there’s a worry about
fake academic work made by this kind of technology. There is a need to ensure that
assignments completed by AI are not included in assessments. Additionally, it is crucial
to be able to distinguish between AI-generated work and human-produced content.
ChatGPT and its associated bots have smoothly become a part of many popular
websites. Chances are you’ve come across their capabilities without even realizing it.
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2024
K. Arai (Ed.): SAI 2024, LNNS 1017, pp. 74–80, 2024.
https://doi.org/10.1007/978-3-031-62277-9_5
Human-Created and AI-Generated Text: What’s Left to Uncover?
75
Of particular importance, ChatGPT has created a versatile tool that can be utilized by a
wide range of websites, software, and applications for tasks such as generating art and
composing music. This expansion has considerably increased ChatGPT’s popularity,
drawing in users from various backgrounds, extending beyond computer scientists and
engineers, who can enhance their work processes and more through its features.
Our objective is to unravel the distinctions between texts authored by humans and
those crafted by machines. To achieve this, we commence with a questionnaire, which
serves as our initial step. Concurrently, we gather human-generated texts for further elaboration in subsequent stages. Employing a unique approach to language comprehension,
we seek to shed light on this differentiation process. It’s a small study that looks into
this topic, and it’s important because it can help make sure that when we use AI like
ChatGPT to write, we also give credit to the right sources. This research also gives us
some good ideas about how tech companies, researchers, and schools can work together
to stop cheating and copying. This can help students use AI in a positive way to help the
world or augmenting their works.
The research question is the same as what we first asked in the title, but because
the article is meant to present a smaller outcome of the study, we will only present the
responses pertaining to the five questions discussed in Sect. 3.3.
2 Literature Review
Researcher has mentioned that the long-term consequences of generative AI are uncertain; it could either greatly improve our lives or lead us into a disastrous future [3]. Having
spent the past year experimenting with ChatGPT, I’ve come to realize that allowing students to use ChatGPT for their school assignments or other tasks is actually a beneficial
idea.
Generative AI can be a helpful tool for students to learn more about different theories
and enhance their overall understanding. It can answer many questions without students
feeling embarrassed in front of their classmates. This technology lets students explore
ideas quickly and satisfies their curiosity [4]. Moreover, it allows students to ask multiple
questions without burdening teachers with excessive demands. This extra help gives
teachers more time to focus on important tasks. This approach can improve students’
knowledge and assist with their schoolwork, broadening their understanding.
When used like this, the risk of cheating is quite low. In a paper titled “Should Using
an AI Generator be Plagiarism?”, Frye [5] brings up an interesting idea. He says that if
students can get good answers using AI, then the questions they’re asked might be too
easy. Interestingly, ChatGPT is a coauthor of his paper, adding a clever twist. Though
his comment might seem a bit critical, there are valid points to think about [5].
To delve deeper into how ChatGPT operates, here is a list of nine key points to consider. 1) Structured Responses: ChatGPT maintains an organized writing style, presenting ideas coherently. It typically starts by summarizing the main points of the questions
and then provides a comprehensive response. This is often followed by a concluding summary [6]. 2) Thorough Explanations: ChatGPT’s tendency to offer detailed responses is
shaped by Reinforcement Learning with Human Feedback (RLHF). It usually provides
thorough explanations unless specifically asked for brevity [7], in other words, ChatGPT tends to give long and detailed answers because of the way it learns from human
76
S. Salter et al.
feedback. It usually provides these detailed answers unless you ask for a shorter one.
Sometimes, the way it writes can affect how human perceive it [8]. 3) Neutral Handling
of Sensitive Topics: When discussing sensitive subjects, ChatGPT remains neutral. This
quality is especially useful for addressing politically charged issues, as it avoids bias
or harmful content [9]. 4) Recognizing Knowledge Limits: ChatGPT acknowledges its
limitations and refrains from answering when it lacks the necessary information [10].
5) Occasional Fabrication of Information: At times, ChatGPT may generate incorrect
details, particularly when responding to queries beyond its expertise [11]. 6) Use of
Repetitive Phrases: The model might use certain terms repeatedly, resulting in lack of
variety in its output [12]. 7) Following Set Patterns: ChatGPT’s sentences can adhere to
specific templates, leading to predictable language patterns [13]. 8) Contextual Accuracy: ChatGPT’s responses might lack accuracy in the context, leading to logically or
contextually irrelevant information [14]. 9) Filling Information Gaps: Even if ChatGPT
doesn’t fully grasp the context, it attempts to fill in gaps using patterns from its training
data [15].
Text analysis researcher has demonstrated the capability of human text to convey
nuanced sentiment [16], particularly in the context of text messaging and written conversations, effectively capturing the intricacies of human communication through written
text. As human beings, our communication goes far beyond the literal, influenced by
the nuances of language and culture. For instance, studies have explored the detection
of sarcasm [17] in human language and the concealed that emotions within text could
potentially reverse the intended sentiment the polarity of expression [18]. Others study
also stated that the repetition of words could affect the intended sentiment [19]. As
part of composing this paragraph, a straightforward query was presented to ChatGPT,
prompting the generation of a sentence that incorporates both sarcasm and emotion, as
demonstrated in the response: “Oh, wonderful! Another Monday morning at work, my
absolute favorite time of the week.” This raises the question: Can the emotions conveyed
through the way words are used in writing help us tell the difference between text written
by humans and text generated by ChatGPT?
3 Methodology
3.1 Data Collection
The study uses a combination of methods. It looks at both the text itself and uses numbers
to analyze and model the data. The focus is on understanding how AI-generated text
differs from human-written text. The study involves comparing 66 summaries of research
papers abstract made by AI to those made by human. These summaries are taken from
abstract from one specific author, keeping the consistency of topic/theme on check. The
aim is to compare how well AI and people can summarize the same research.
To engage participants, a focus group was organized at University, where an advertisement was placed in the main foyer. This group was informed about the purpose of
study, which would take around 15 min to complete. A computer lab was arranged,
equipped with computer for respondents to respond. This setup allowed for in-person
informed consent, immediate response to participant queries, and clear instructions.
Additionally, participants were invited to a ChatGPT workshop, designed by OpenAI,
Human-Created and AI-Generated Text: What’s Left to Uncover?
77
which showcased the benefits of generative AI and its practical applications. The workshop aimed to teach attendees, including teaching staff, students, and non-teaching personnel, how to utilize ChatGPT effectively. It demonstrated how to write simple prompts
and showcased the system’s capabilities. Participants were enthusiastic about joining the
research, especially since attendance at the workshop was offered in exchange for their
participation. Light refreshments were also provided as a token of appreciation for their
time.
3.2 Participant
A total of 66 participants took part in the study, with 59% being male and 41% female.
Among them, approximately 42% had a postgraduate background, 30% were undergraduates, and 28% had an educational background of A levels and GCSEs. Participants from
various educational backgrounds were engaged in summarizing the texts. Following the
summarization task, participants answered five personal perspective questions. They
were given a 15-min window for summarization to encourage natural responses without
overthinking. Throughout the study, participants’ identities were kept confidential. Ethical approval for the study was obtained before conducting the experiment. Participants
provided basic information such as gender, education, age group, and whether English
was their first language.
3.3 Survey/Questionnaire Design
To gauge participants perspective towards text disguising, participant was asked to rate
the score on all the following questions on a scale on 1 to 5 (with 1 being “strongly
disagree”, 2 being “disagree”, 3 being “don’t know or neutral”, 4 being “agree” and 5
being “strongly agree”).
1) Errors in grammar are key indicators that text has been generated by humans?
2) Repetition in text is a key indicator that text has been generated by humans?
3) Emotional meaning (underlying emotional nuances, tones, or sentiments conveyed
through the choice of words, phrasing, and overall expression of the written content)
is a key indicator that text has been generated by humans?
4) Mistakes around capitalisation, grammar and syntax are key indicators that text has
been generated by humans?
5) Connotations (the emotional or cultural associations that are evoked by certain words,
phrases, or expressions beyond their literal or dictionary definitions) are key indicators
that text has been generated by humans?
4 Results
Figure 1 displaying the outcomes of the perspectives on the five questions posed earlier.
Out of the total participants, 61 responded to the questionnaire, as five participants
did not provide answers, rendering their responses invalid. The mean ratings for each
question are as follows: Q1 - 3.39, Q2 - 2.87, Q3 - 3.80, Q4 - 3.49, Q5 - 3.70. Generally,
there is not a significant difference in the ratings, except for Q2 where there seems to
78
S. Salter et al.
be more disagreement regarding the notion that text repetition is a clear sign of AIgenerated content. The responses also indicate a higher level of agreement concerning
the influence of education on writing style and sentiment towards the choice of words
and phrasing.
Fig. 1. Viewpoint from human created text
It’s interesting that about half of the people strongly agree that when there are mistakes in things like capital letters, grammar, and how sentences are put together, it’s a
good clue that a human wrote the text. So, these little errors might be like a fingerprint
that shows if a person or a machine wrote something.
Also, a lot of the participants think that some words or phrases can make you feel
certain emotions or remind you of specific things from your culture. It’s like words can
carry extra meanings beyond what you find in the dictionary. This kind of special feeling
in the words seems to be something that humans are really good at, and it’s not easy
for machines to do the same. The fact that many people agree with these ideas helps us
understand that there are still parts of how humans use language that machines haven’t
quite figured out yet.
Human-Created and AI-Generated Text: What’s Left to Uncover?
79
5 Conclusion
In conclusion, our aim was to provide insights based on the responses to the questions,
shedding light on society’s viewpoint. Despite the limited responses from the UK, we
managed to gain valuable insights into the aspects agreed upon by human respondents
regarding their perspective on human-generated language. In the future, further studies
will delve into the topic, focusing on the determination to uphold academic integrity. Furthermore, this research aims to increase awareness and organize workshops for informing
different stakeholders, particularly those in academia, about the ways AI can enhance
various aspects of their work. It also encourages exploration of what distinctions remain
between human-generated and machine-generated text.
References
1. Zohny, H., McMillan, J., King, M.: Ethics of generative AI. J. Med. Ethics 49, 79–80 (2023)
2. Karaarslan, E., Aydın, O.: Is ChatGPT Leading Generative AI? What is Beyond Expectations?
Social Science Research Network (2023)
3. Andersen, E.: The Technological Singularity, Ubiquity Symposium (2014)
4. Crawford, J., Cowling, M., Allen, K.: Leadership is needed for ethical ChatGPT: character,
assessment, and learning using artificial intelligence (AI). J. Univ. Teach. Learn. Pract. 20(3),
Article 02 (2023)
5. Frye, B.L.: Should using an AI text generator to produce academic writing be plagiarism.
Fordham Intell. Prop. Media Ent. LJ 33(947), 949–969 (2023)
6. Holden Throp, H.: ChatGPT is fun, but not an author. Science 379(6630), 313 (2023)
7. Benzon, W.: A note about story grammars in ChatGPT, Independent, no. SSRN (2023) https://
ssrn.com/abstract=4324840
8. Teh, P.L., et al.: Textual variations affect human judgements of sentiment values. Electron.
Commerce Res. Appl. 53, 101149 (2022)
9. Thunstrom, A.O., Steingrimsson, S.: Can GPT-3 write an academic paper on itself, with
minimal human input. Hal Open Science, no. 03701250 (2022)
10. OpenAI: ChatGPT (2022). https://chat.openai.com/. Accessed 05 June 2023
11. Khalil, M., Er, E.: Will ChatGPT get you caught? Rethinking of Plagiarism (2023)
12. Uddin, M. S.: Stochastic Parrots: A Novel Look at Large Language Models and Their Limitations, towardsai.net (2023). https://towardsai.net/p/machine-learning/stochastic-parrots-anovel-look-at-large-language-models-and-their-limitation. Accessed 17 June 2023
13. Lund, B.D., Wang, T.: Chatting about ChatGPT: how may AI and GPT impact academia and
libraries? Library Hi Tech News (2023)
14. Rudolph, J., Tan, S., Tan, S.: ChatGPT: bullshit spewer or the end of traditional assessments
in higher education? J. Appl. Learn. Teach. 6(1), 342–363 (2023)
15. Perkins, M.: Academic integrity considerations of AI large language models in the postpandemic era: ChatGPT and beyond. J. Univ. Teach. Learn. Pract. 20(2) (2023). L. S. G. ltd,
Baldur’s Gate 3, Larian Studios Games ltd, 2023. https://baldursgate3.game/. Accessed 11
Aug 2023
16. Teh, P.L., Pak, I., Rayson, P., Piao, S.: Exploring fine-grained sentiment values in online
product reviews. In: 2015 IEEE Conference on Open Systems (ICOS), Melaka, Malaysia,
pp. 114–118 (2015)
17. Teh, P.L., Ooi, P.B., Chan, N.N., Chuah, Y.K.: A comparative study of the effectiveness of
sentiment tools and human coding in sarcasm detection. J. Syst. Inf. Technol. 20(3), 358–374
(2018)
80
S. Salter et al.
18. Teh, P.L., Rayson, P., Pak, I., Piao, S., Yeng, S.M.: Reversing the polarity with emoticons. In:
Métais, E., Meziane, F., Saraee, M., Sugumaran, V., Vadera, S. (eds.) NLDB 2016. LNCS,
vol. 9612, pp. 453–458. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-417547_48
19. Pak, I., Lee Teh, P., Cheah, Y.-N.: Hidden sentiment behind letter repetition in online reviews.
J. Telecommun. Electron. Comput. Eng. (JTEC) 10(3–2), 115–120 (2018)
AraXLM: New XLM-RoBERTa Based Method
for Plagiarism Detection in Arabic Text
Mona Alshehri(B) , Natalia Beloff, and Martin White
Department of Informatics, University of Sussex, Brighton, UK
{ma2250,n.beloff,m.white}@sussex.ac.uk
Abstract. Significant advancements have been made over the previous decade in
the development of plagiarism detection tools and software. This has expanded
application beyond identification of plagiarized articles and research papers in
academia, safeguarding authors’ literary rights across various domains, including digital marketing and libraries. However, the emergence of cross-language
plagiarism poses both a significant challenge and a crucial area for further development. This primarily stems from the complexity of different languages, as well
as the accuracy and interpretation of automatically translated words and sentences,
which can prove more complex than the identification of plagiarism in a monolingual corpus. This position paper addresses the obstacles encountered in crosslanguage detection of plagiarism relating to Arabic-English and English-Arabic
works. It outlines the impact of vowel marks on the semantic meaning of Arabic
sentences. The paper also introduces a novel approach utilizing XLM-RoBERTa
(XLMR) for enhancing the detection of plagiarism in Arabic text based on semantic similarity, referred to as AraXLM. This approach considers the influence of
Arabic diacritical marks during text processing, as these can play a crucial role
in improving translation accuracy and the interpretation of sentences. The proposed framework incorporates a cross-language approach utilizing features such
as Facebook AI Similarity Search (FAISS), as well as semantic similarity and
diacritization, to detect plagiarism in Arabic text at the sentence level of SemEval
dataset. This research is ongoing, with the following stage being to evaluate the
proposed framework by testing metrics such as Precision, Recall, and F-Measure.
We anticipate that the employment of AraXLM for the detection of plagiarism in
Arabic will address the challenges associated with, and enhance the effectiveness
of, Natural Language Processing (NLP).
Keywords: Transformer · XLM-RoBERTa · Cross-Language plagiarism
detection (CLPD) · Natural language processing (NLP) · Deep learning
1 Introduction
Online education, including the use of digital libraries and access to online written
work, is in a continuous process of expanding and evolving [1]. For example, search
engines such as Google can be employed to find documents and online topics related
to specific titles. This has resulted in a focus on effective plagiarism detection tools in
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2024
K. Arai (Ed.): SAI 2024, LNNS 1017, pp. 81–96, 2024.
https://doi.org/10.1007/978-3-031-62277-9_6
82
M. Alshehri et al.
both academic and non-academic fields, in order to protect the information integrity of
original sources [2]. Furthermore, the ease with which information can be accessed by
means of the internet, alongside the availability of Machine Translation (MT) software
has acted to increase plagiarism in professional environments, i.e. universities [3]. A
manual approach can be used to detect plagiarism in a document, but this is only practical
if the number of suspicious documents is manageable and monolingual.
Plagiarism is defined as the appropriation of the ideas or work of others, without
clearly referencing the original source [4]. However, a distinction is made between
similarities and plagiarism, with the former referring to non-exact matching between
submitted work and analogous other sources, while the latter is defined as work copied
without any acknowledgement, either intentionally or unintentionally [5]. Moreover,
plagiarism arises in different types and formats within diverse scenarios (see Fig. 1),
so requiring a number of methods to detect plagiarism fragments in documents lacking appropriate citation. Accordingly, the detection of text plagiarism can be classified
into two categories: firstly, literal (Direct) and secondly, intelligent (Indirect). Natural
Language Processing (NLP) techniques have been revealed as having the potential to
automatically detect plagiarism, extracting portions of text from large volumes of data
[6], including processing language elements and tasks, such as: Parts of Speech (POS);
Translation; Named Entity Recognition (NER); Diacritics; and tasks linked to Plagiarism
Detection (PD).
Fig. 1. Plagiarism classification drawn by the authors and adapted from [7–9]
Although multiple techniques and algorithms are currently employed for detecting
plagiarism, there has been only limited research into improving such systems, particularly in relation to the Arabic language [10]. This has led to PD applications and tools
only being developed to produce reliable results when used with the English language
[11]. For example, most detection methods (i.e. content-based) have a preprocessing
phase requiring the removal of diacritic marks as a normalization step. Diacritization
AraXLM
83
in Arabic indicates the meaning of a word. However, based on the XML-RoBERTa
technique, the proposed prototype (AraXLM) can improve the capacity to match multilingual sentences based on semantic similarity, along with enhanced processing time
and reliability, by processing the whole word without needing to remove diacritic marks.
This paper is organized as follows: Sect. 2 provides a brief description of the main
characteristics of the Arabic Language in digital text format. Section 3 examines related
work concerning plagiarism detection systems (including challenges relating to detection) in both the Arabic and English languages. Section 4 addresses the challenges and
limitations of cross-lingual models. Section 5 reviews the proposed framework, designed
to solve the problem presented in Sect. 3. Section 6 outlines the reasons for choosing
this solution and dataset, along with indicating how the model’s performance will be
measured to enhance the PD system. Finally, Sect. 7 includes both the conclusions of
this current study and recommendations for future work.
2 Arabic Language Characteristics
Digital script is used to write and print letters, as well as to record and document data.
Arabic script has two features that make it particularly distinctive with respect to encoding. Firstly, Arabic text is written from Right To Left (RTL). Secondly, the shape of
a single letter changes depending on its position; i.e. whether the letter is isolated, or
occurs at the beginning, middle, or end of the word, as well as no use of capitalization.
However, there are a number of limitations and failures in capturing all of the language’s
unique characteristics [12]. This can lead to challenges when the Arabic language is
printed digitally, i.e. the proper representation of cursive linear extension between the
letters and marks [13]. The following points identify some of the chief characteristics
of Arabic writing when attempting to follow the rules of Arabic calligraphy in the form
of digital text.
2.1 General Features of the Arabic Language
Arabic scripts are referred to as alternates. This feature affords Arabic script a series of
basic shapes [14]. Furthermore, the letters in Arabic script merge, overlap and change
shape as a consequence of their cursive characteristics. Consequently, two or more letters
can therefore be merged to generate and represent a single character; i.e. ‘lamalif’,
meaning no. In Latin script, an example similar to the ‘lamalif ’ is the ‘ampersand’
character ‘&’, which denotes ‘and’. This is used for the purpose of justification in lines,
or for emphasis. In addition, one of the most important features of Arabic script consists
of the natural flow of cursive writing.
Kashida (Tatweel). In Latin text, words are justified by means of spacing between
words and characters, including through hyphenation. Arabic script, however, is justified
using Kashida (U + 0640, small flat line), which is simply elongations made to connect
letters in different degrees. Kashidas are slightly curved in shape, and have specific rules
for their formation [15]. In addition, they can be used to fix crowded letters in a word, and
create balance within lines. In Unicode-encoded Arabic text, the Kashida is a character
used for alignment, but not considered a part of Arabic alphabet. Therefore, the Kashida
84
M. Alshehri et al.
is removed or ignored during the NLP processing stages, such as preprocessing and
tokenization.
Diacritics (Vowels). There are two kinds of vowels (diacritics) in Arabic script. The first
are the letter vowels (called long vowels) used to stress a given vowel, which are Alef “ ”,
Waw “ ”, and Ya “ ”. The second are vowel marks (also called short vowels or diacritic
marks), which are Fattha “ ”, Dhamma “ ”, Kasra “ ”, Sokoon “ ”, Shadda “ ”,
Tanween “ ”, and Maddah “ ” (see Table 1). These are signs or symbols combined
with the letters of the Arabic alphabet to indicate varying pronunciation [16, 17]. However, while vowels undertake the crucial role of eliminating ambiguity, contemporary
Arabic script can be written without recourse to vowel diacritics. In Unicode-encoded
Arabic text, many detection plagiarism approaches tend to remove diacritics during the
preprocessing step [18, 19]. However, diacritic marks play crucial role in the sematic
meaning of Arabic text, being the vowels of the words capturing linguistic patterns and
distinguishing between the homograph words.
Table 1. Example of diacritic marks
Arabic Punctuation Marks. Punctuation assists in rendering sentences more readable
and understandable, including eliminating any ambiguity between phrases [20]. This
contrasts with diacritic marks, which can be used to represent the sound and pronunciation of letters. There are a number of differences between sentence boundaries in
Arabic and English text. In English and Latin text, NLP applications identify complete
sentences in a paragraph through the capitalization of initial letters and the use of the full
stop to punctuate the end of sentences. However, the punctuation used in Arabic differs
from that used in English, whereby it is mirrored in Arabic since the writing direction
is reversed [21].
2.2 Summary
All of the characteristics discussed above have been employed to clarify the basic features
of Arabic, in order to consider how best to apply NLP methods to Arabic text. The first
step is to perform a normalization task with Latin text, and so this is applied to Arabic
text, i.e. the removal of Arabic punctuation marks and the Kashida have no impact on
processing in an NLP task. This is due to punctuation and Kashida being used only to
arrange the sentences in a document according to a specific style, without influencing the
meaning of a text. This leads them to being generally removed from an Arabic language
AraXLM
85
sentence to prepare it for NLP. However, the removal of Arabic vowels marks impacts
meaning and can result in ambiguity.
The pre-processing of text in NLP is undertaken to transform the written manuscript
into a format capable of being interpreted using a software model such as Embedding
Vector. Furthermore, as examined in the following section, Arabic languages possess
general features that differ from other natural languages, with the potential to create a
variety of issues during processing.
3 Related Work
Arabic NLP (ANLP) techniques that include diacritic marks in their approaches have
recently emerged to enhance the performance of language processing tools, i.e. Machine
Translation [22]. Alongside diacritization, there is the process of restoring the short
vowels to Arabic text [23]. This process was previously side-lined, as Arabic readers
can usually obtain the exact meaning from the context when Arabic text is written without
short vowels. However, Arabic text can be misunderstood in contexts where diacritical
marks are lacking, particularly when translated into languages such as English. MT
simultaneously translates a sentence to reproduce meaning, while respecting the target
language structure [24]. Thus, processing a word in the Arabic language with no diacritic
marks ignores the significance of morphological and orthographic features assisting
recognition of the written text, so placing the onus on the reader to interpret meaning
from context and language structure.
Arabic languages have a more complex morphological system than English. Therefore, many plagiarism detection systems use approaches reliant on a normalization step
to prepare standard format sentences for the calculation [18, 19], during which diacritic
marks are removed. This step prevents noise and supports normalization of text retrieval
[25, 26]. Altering or removing diacritics can alter the semantic meaning of a word in
Arabic affecting the process of calculating semantic similarity between sentences, and
can render sentences inaccurate and lead to ambiguity during the preprocessing phases
in ANLP [27, 28]. A number of studies have used diacritized sentences to produce a
good performance with large datasets [29], taking into account the fact that words with
differing structures in their sentences can lead to different diacritization.
Diacritization and MT in Arabic tend to encounter many additional challenges in
comparison to English. Cross-Language (Translation) plagiarism refers to the translation
of text from one language to another without citing the source of the original text [30].
Most plagiarism detection systems are designed to identify plagiarism and similarity in
the same language as that used in the target document [31, 32]. Thus, the performance
of cross-language plagiarism detection needs to be enhanced, as it is harder to detect
for several reasons [33]. The first is that translation between languages can disguise
syntactical similarities [34]. The second is that MT can only translate sentences, being
incapable of detecting an exact match between translated documents. The third is that the
limitation on the number of words in a given vocabulary in different language corpora,
because they typically only store a common vocabulary and ignore new and scientific
words [35]. Moreover, Arabic is considered a low-resource language [36, 37] and the
features of digital and modern Arabic data are remain under-represented relative to the
English language.
86
M. Alshehri et al.
It has recently emerged that sentence embedding models (e.g., XLM-RoBERTa)
produce more effective classification of text in Arabic Sentences [38]. However, these
systems still do not include unique Arabic language characteristics, i.e. diacritics (vowels) [39–41]. The detection of cross-language plagiarism can be undertaken through
various approaches, each employing its own specific similarity analysis model to obtain
matching words between languages. For example, a syntax-based model uses tokens
(or a portion of the text) to determine any overlap with the source text and calculates
similarities such as n-grams. This approach is used without MT software. The obtained
results create a greater number of matching words in closely related languages, compared to those sharing the same meaning. On the other hand, a related task has revealed
[42] that fine-tuning the multilanguage model XLM-R on English data can produce good
results when tested on other languages, including Arabic. The task addressed concerned
a XLM-R transformer model trained on an English dataset, following which the accuracy
of classification was found to be enhanced after testing the model on Arabic. However,
the result failed to reveal whether the model improved the ability to capture the specific
and main features of Arabic. Zhang et al. used semantic text similarity analysis (i.e.
Laten Semantic Analysis) at paragraph level, combined with the fine-tune BERT model
at sentence level. Their study was implemented on Chinese and demonstrated a good
result when it came to the detection of plagiarism at the sentence level with a transformer
such as the BERT model [43].
In addition, we identified several studies proposing the detection of cross-language
plagiarism between Arabic and English, i.e. an Arabic document translated to English
language by MT, followed by the application of the preprocessing phase to prepare the
sentences for the fingerprinting step. The detection of similarity was measured by means
of the fingerprint index [44].
In summary: the detection of plagiarized text remains complex and challenging, with
accurate identification of various types of plagiarism necessitating diverse approaches
and techniques. Additionally, few studies have focused on detecting plagiarism in Arabic
text as cross-language with English translated sentences. Moreover, there also remains a
lack of fine-tuning transformer to capture Arabic semantic text with diacritization. This
has resulted in the current study employing Arabic-English word embedding pairs to
detect plagiarism, by calculating the cosine and distance similarity between sentences,
in order to find semantic correlation of the text.
4 Challenges and Limitations
While cross-lingual models can transfer knowledge between languages, the complexities of language as well as the richness of semantics creates challenges and limitations
that affect NLP tasks. For example, cross-lingual plagiarism detection, involving the
identification of similarities between sentence pairs from different languages presents
challenges and limitations [45, 46]. In this section we explain the most common challenges that arise when processing sentence pairs involving Arabic (Ar) and English
(En).
AraXLM
87
4.1 Translation Quality
Cross-lingual pre-trained models, such as XLM-R, can improve the quality of sentences
translated between different languages [47]. However, the translation quality of Arabic
text differs from that in other languages, because Arabic has words with multiple meaning. In NLP this type of ambiguity is called Semantic Ambiguity [48]. In addition, the
absence of diacritic marks in Arabic text leads to ambiguities during the information
extraction process [49–51]. Therefore, the process of determining the correct token, or
identifying individual words can be a barrier to an accurate translation.
4.2 Fine-Tuning Model
Transformer-based pretrained models, such as XLM-R, are designed to handle multiple
languages, including Arabic and English [42]. However, they are still not optimized
for low resource languages such as Arabic. Also, the lack of resources for Arabic can
limit the development and accuracy of models to capture some language features and be
fine-tuned for specific tasks, such as sentence similarity between cross-lingual.
4.3 Cross-Lingual Semantic Variability
In cross-lingual word embedding, words that share similar meanings in the different
languages are mapped to nearby vectors. Therefore, identifying similarity between texts,
such as when paraphrased or translated, is a complex task [52, 53]. There is no word to
word or phrase to phrase correspondence. Cross-lingual semantic variability therefore
needs to understand meaning and context in both languages, and this can be challenging
when dealing with multiple linguistic structures and contexts.
5 The Proposed Solution
This section outlines proposed solutions to detecting plagiarism in Arabic language text,
using the Transformer known as a Cross-Language Model, based on RoBERTa (XLMRoBERTa) [12]. This is designed to detect semantic similarities from among the available
cross-language datasets [54], addressing plagiarism in paraphrasing or summarizing, as
well as translation from differing languages either manually or automatically, as well as
the writer passing off text as his/her own work without citation. Our research therefore
specifically addresses the issue of Cross-Language Plagiarism Detection (CLPD) from
Arabic to English, including the proposal of an external plagiarism solution. This section
is formed of two subsections: firstly, the proposed Framework AraXLM and secondly,
a demonstration of the system’s workflow.
5.1 The AraXLM Framework
Our proposed plagiarism detection framework AraXML includes a cross-language transformer. This processes whole sentences without a pre-processing step, i.e. the removal of
diacritization, assisting in reducing cases of ambiguity that share the same homograph.
88
M. Alshehri et al.
Therefore, our model contains three main components for overcoming the challenges
discussed above, as well as the issues associated with the current pre-processing of sentences. The main components are: (1) Dataset; (2) XLMRoBERTa (XLMR) Transformer;
and (3) Similarity Calculation.
Dataset. The dataset used in our proposed framework is from SemEval-2017 Task1
(Track2) and contains pairs of 1364 sentences of Arabic-English. The dataset was derived
from original sentences in English in the previous experimental task [55, 56], which
were subsequently translated into Arabic for cross-language tasks. For each pair of
English sentences from the examined dataset, two pairs of English-Arabic sentences
were extracted, with each pair containing a semantic similarity score ranging from 0 to
5. The reason for this similarity scale is to identify each pair of sentences within the
dataset with the obtained score of the output results in the proposed framework [57].
The annotation scale of 0 means the two sentences S1(En) and S2(Ar) are dissimilar. By
contrast, a score of 5 indicates that the two sentences share an identical meaning.
Transformer. XLMR is a pre-trained language model and a version of the RoBERTa
transformer, which is trained on differing multilingual sets of texts. The model was
introduced in 2019, demonstrating improved performance with low resource languages
after training the model with the English language, as well as exploring effectiveness
resulting in cross-lingual natural language inference (XNLI) [12, 58]. XNLI is a task
in NLP used to determine the semantic relationship between two sentences in separate
languages, i.e. MT and PD.
Similarity Calculation. The length of sentences tends to vary, while the process of
finding similarity between a large size of vectors can be time consuming. Therefore,
the Facebook team has devised the Facebook AI Similarity Search (FAISS) as an opensource library, which can be used to develop an efficient similarity search across large
number of indexes [59]. Moreover, some cases have a large distance, while the meaning
is related [60]. Hence, as outlined in the following section, cosine similarity is used
to calculate the angles between the two vectors and demonstrate whether the meaning
moves in the same direction.
The aim of the AraXLM framework is to enhance results obtained from plagiarism detection, including a consideration of Arabic language characteristics (i.e. vowel
marks) to reduce cases of obfuscation. Thus, the inclusion of diacritics (vowel marks)
in Arabic sentences can impact the meaning and interpretation of text during translation
and semantic similarity calculation.
5.2 AraXLM Workflow
Figure 2 shows the high-level framework representing our proposed CLPD system for
use with Arabic and English texts. The workflow of the AraXML framework commences
with the selection of a dataset of sentences in the Arabic and English languages, which
are then uploaded to the transformer XLMR to generate embedding vectors as matrices.
Following this, the FAISS library is used to calculate the similarity between the English
and Arabic vectors and report the findings.
AraXLM
89
Fig. 2. High-level view of the proposed AraXLM plagiarism detection framework
The dataset component contains files of Arabic text prepared as follows: firstly, the
original Arabic sentences from the SemEval-2017 [54] are automatically diacritized
using the application called Deep Arabic Diacritizer, available as open-source code on
GitHub [61], so enabling the production of two Arabic files for the transformer, with and
without diacritization sentences. Secondly, the files in Arabic are translated into English
by Google Translate. Thirdly, four files are prepared for XLMR, i.e. an Arabic diacritized
sentences file with an English translated sentences file, and an original Arabic sentences
file with an English translated file. The four files are then used with the English sentence
files from the SemEval-2017.
The XLM-RoBERTa consists of a sentence encoder capable of processing data from
the previous component. This encoder is trained on English and Arabic datasets from
SemEval2017, in order to understand the semantic representation of the Arabic and
English languages. This encoder is self-attention, which can be used in NLP tasks and
handle the interaction information between different languages [62]. The encoder is then
utilized to generate numerical vectors known as sentences embedding, ready for use by
the next component (Similarity Calculation) to determine the similarities between the
stored sentences.
90
M. Alshehri et al.
In the Similarity Calculation, FAISS (which is a search library in C++) can be used in
the Python Library. It is employed to search a set of vectors of any size for similarities and
calculate the distance of the vectors coordinated from the original using the Euclidean
distance formula (see Eq. 1). The p and q here represent different sentences, which
are stored in vectors according to their coordinates. The produced embedding vectors
of the XLM-R are dynamic and contain contextual information between words in a
sentence across multiple languages. Therefore, accuracy when calculating the similarity
of sentence pairs can be enhanced by leveraging XLM-R embeddings in a multilingual
context [63]. When calculating semantic similarity, the smallest number denotes the
shortest distance between their meaning. Additionally, cosine similarity is computed
with the Euclidean distance to determine the angle between the vector q and p (see
Eq. 2).
n
(qi − pi )2
(1)
d (q, p) = i=1
n
i=1 qi pi
n
2
2
i=1 qi
i=1 pi
Similarity = cosθ = n
(2)
In order to identify any relationship between the results obtained from Eqs. 1 and 2,
we need to find a measurement capable of unifying the relationship results between the
cosine similarity and Euclidean distance. This is due to the SemEval using the Pearson
Correlation Coefficient measurement of semantic text similarity to evaluate the system’s
performance with other participants approaches in SemEval.
5.3 Summary
This section has demonstrated that the proposed AraXLM framework includes three
components designed to ensure an external plagiarism detection solution between Arabic and English sentences, taking into account Arabic language features (e.g., diacritic
marks). In addition, the paper has outlined the workflow of the framework, in order to
clarify the processing of sentences between English and Arabic languages. This enables
the use of the framework’s performance to measure the plagiarism detection reliability
of AraXLM.
6 Discussion
Our proposed approach employs a small-scale corpus as a local dataset based on
SemEval2017 corpora. The aim of this test is to investigate the ability to detect plagiarism utilizing the proposed technique. Following this, the output results can be evaluated by comparing individual and combined techniques. In the dataset component, we
are restoring the diacritical marks for preparing our dataset to achieve semantic balancing between homograph words in Arabic and their translated words in English. This
demonstrates how to investigate the complexity of diacritic pattern words if we have the
same word with a different meaning. For example, in the used dataset SemEval-2017,
AraXLM
91
we found the words ‘explain’ “
”, and ‘dissect’ “
”. The former contains the
same letters with different meanings, but when we translated them without diacritics, the
Google translation and the dataset introduced only one meaning, i.e. to explain. In this
case, we used Word Error Rates (WER) and Diacritic Error-Rate (DER) to measure the
performance of the proposed framework, due to having MT and diacritization tasks to
capture the semantic meaning in Arabic texts and therefore needing to track the changes
during the text processing along with English texts.
Tables 2 and 3 show our experiment using XLM-R with the Sentence Transformer
Variant. This shows that the XLM-R encoder obtained good results when it came to
Arabic sentences without diacritic marks. However, the similarity calculated with diacritized Arabic-English pairs sentences proved inaccurate, because diacritic marks are
calculated as separated characters, i.e. letters.
), and is presented
In Table 2, the source sentence was in Arabic (
without diacritics. In addition, the Arabic source sentence was compared with different
sentences in the English Language. The model “paraphrase-xlm-r-multilingual” was
used to convert input texts into embedding vectors, which captured their semantic content
and computed the cosine similarity and distance between them. This model is trained on
multilingual Semantic Textual Similarity data and is an example of the XLM-R variant
that can utilize the sentence transformer library [64–66]. It is based on the XLM-R model
and fine-tuned on a multiple language dataset for paraphrasing tasks. Science XLMR uses SentencePieces, which is fine-tuned for generating sentence embeddings and
comparing sentence similarity. The sentence pair with the highest cosine score, ranging
from 0 to 1, and the lowest distance most closely aligns in meaning. The English sentence
“A little girl is playing a guitar” semantically corresponds to the source Arabic sentence.
) to monitor the calculation
We modified the source Arabic text to (
process. The model assigned a higher score to the English sentence “A kid is playing
” translates to “little girl”
a guitar”. As demonstrated in Table 2, the Arabic word “
” to “kid,” both of which are accurate translations in the respective languages.
and “
”, which is a diacritized
In Table 3, the source Arabic sentence is “
sentence with vowel marks. This is then compared with various sentences in the English
language, and the comparison uses the embedding vectors generated by the former pretrained variant of the XLM-R model. As illustrated in Table 3, the highest similarity
scores were observed between the Arabic source text and the English sentence “A cat
is playing a guitar”. The sentence “A cat is playing a guitar” doesn’t have an accurate
”, which is closer to “A
translation of the Arabic sentence provided “
little girl is playing the guitar”.
Moreover, we sought to investigate the performance of our framework when measuring semantic similarity on Arabic-English cross-language sentences using machine
translation and vector embedding generated by AraXLM. This was followed by comparing the results with best systems proposed in the SemEval-2017 Arabic-English
cross-lingual evaluation dataset [57]. We then used the performance comparison Pearson
Correlation Coefficient measurements of semantic text similarity in SemEval-2017.
92
M. Alshehri et al.
Table 2. Variant of XLM-R introduces the embedding vector for each sentences then calculates
the similarity between Arabic-English pair (without Diacritic Marks)
Table 3. Variant of XLM-R introduces the embedding vector for each sentence, then calculates
the similarity between Arabic-English pair (with Diacritic Marks)
7 Conclusion and Future Work
This position paper commenced with an overview of the characteristics of the Arabic
language in the form of digital text. We highlighted a number of basic Arabic features,
such as Kashida, diacritics, punctuations marks and Unicode. In addition, we identified
that some features have no impact on the meaning of a text during NLP task processing. For example, removal of the Kashida and punctuations marks does not impact the
semantic meaning of the Arabic text during the tokenization step of XLM-R. As the
former transformer produced word embedding vectors capturing the semantic meaning
of the text, this confirms that the diacritics feature impacts the meaning of the Arabic
text. In addition, we addressed the issue of calculating similarity scores for Arabic text
with diacritic marks.
AraXLM
93
Secondly, we proposed a new framework based on transformer XLM-R to detect
plagiarism in Arabic text under cross-language plagiarism (i.e. Arabic to English). In
addition, the paper presented the processing of Arabic sentences, while considering
diacritization marks during the process.
We propose that future work should involve implementing the proposed AraXLM
framework after fine-tuning the model on the SemEval-2017 benchmark containing
Arabic and English sentences with Arabic diacritization text, in order to understand
the semantic of the words in the text. This could potentially be expanded to include
languages other than English and Arabic. We also aim to test the accuracy and efficiency
of the proposed framework’s performance with Precision, Recall and F-Measures for
each of the plagiarized sentences. The primary aim of this framework is to enhance the
results obtained during plagiarism detection by accounting for vowel marks in Arabic
languages. In addition, we will calculate the Word Error rate and Diacritic Error Rate to
measure the enhanced PD framework, with or without the diacritization process.
References
1. Zalat, M.M., Hamed, M.S., Bolbol, S.A.: The experiences, challenges, and acceptance of
e-learning as a tool for teaching during the COVID-19 pandemic among university medical
staff (2021). https://doi.org/10.1371/journal.pone.0248758
2. Shafer, S.L.: You will be caught. Anesth. Analg. 112, 491–493 (2011). https://doi.org/10.
1213/ANE.0B013E3182095C73
3. Gasparyan, A.Y., Nurmashev, B., Seksenbayev, B., Trukhachev, V.I., Kostyukova, E.I., Kitas,
G.D.: Plagiarism in the context of education and evolving detection strategies. J. Korean Med.
Sci. 32, 1220–1227 (2017). https://doi.org/10.3346/jkms.2017.32.8.1220
4. Hopp, C., Speil, A.: How prevalent is plagiarism among college students Anonymity preserving evidence from Austrian undergraduates. Account. Res. Polic. Qual. Assur. 23, 133–148
(2021). https://doi.org/10.1080/08989621.2020.1804880
5. Memon, A.R., Mavrinac, M.: Knowledge, attitudes, and practices of plagiarism as reported
by participants completing the AuthorAID MOOC on research writing. Sci. Eng. Ethics 26,
1067–1088 (2020). https://doi.org/10.1007/s11948-020-00198-1
6. Khanbhai, M., Anyadi, P., Symons, J., Flott, K., Darzi, A., Mayer, E.: Applying natural
language processing and machine learning techniques to patient experience feedback: a systematic review. BMJ Health Care Inform. 28 (2021). https://doi.org/10.1136/bmjhci-2020100262
7. Wakil, K., Ghafoor, M., Abdulrahman, M., Tariq, S.: Plagiarism detection system for the
Kurdish language. Int. J. Inf. Technol. Comput. Sci. 9, 64–71 (2017). https://doi.org/10.5815/
ijitcs.2017.12.08
8. Alzahrani, S.M., Salim, N., Abraham, A.: Understanding plagiarism linguistic patterns, textual features, and detection methods. IEEE Trans. Syst. 42, 133–149 (2012). https://doi.org/
10.1109/TSMCC.2011.2134847
9. Osman, A.H., Salim, N., Abuobieda, A.: Survey of text plagiarism detection. Comput. Eng.
Appl. 1, 37–45 (2012)
10. Khan, I.H., Siddiqui, M.A., Jambi, K.M.: Towards building an Arabic plagiarism detection
system: plagiarism detection in Arabic. Int. J. Inf. Retrieval Res. 9, 12–22 (2019). https://doi.
org/10.4018/ijirr.2019070102
11. Menai, M.E.B.: Detection of plagiarism in Arabic documents. Int. J. Inf. Technol. Comput.
Sci. 4, 80–89 (2012). https://doi.org/10.5815/ijitcs.2012.10.10
94
M. Alshehri et al.
12. Kaoudja, Z., Kherfi, M.L., Khaldi, B.: A new computational method for Arabic calligraphy
style representation and classification. Appl. Sci. 11, 4852 (2021). https://doi.org/10.3390/
app11114852
13. Ibrahim, H., Mohamed, I., In Calligraphy-Associate, A.P.: Arabic typography development
and technological compatibility. Am. J. Linguist. 6, 19–26 (2018). https://doi.org/10.5923/j.
linguistics.20180602.01
14. Smitshuijzen, E.: Arabic Font Specimen Book. De Buitenkant, Amsterdam (2009)
15. Elyaakoubi, M., Lazrek, A.: Justify just or just justify. J. Electron. Publ. 13 (2010). https://
doi.org/10.3998/3336451.0013.105
16. Hssini, M.: Problem of multiple diacritics design for Arabic script. IOSR J. Eng. 02, 48–53
(2012). https://doi.org/10.9790/3021-021234853
17. Rjeily, R.: Cultural Connectives: Bridging the Latin and Arabic Alphabets. Mark Batty (2021)
18. Singh, M., Gupta, V.: Review of extrinsic plagiarism detection techniques and their efficiency
comparison. In: Woungang, I., Dhurandher, S.K., Pattanaik, K.K., Verma, A., Verma, P. (eds.)
ANTIC 2021. CCIS, vol. 1534, pp. 609–624. Springer, Cham (2022). https://doi.org/10.1007/
978-3-030-96040-7_46
19. Aliwy, A.H.: Tokenization as preprocessing for Arabic tagging system. Int. J. Inf. Educ.
Technol. 2(4), 348 (2012)
20. Lawler, J.: Punctuation From The International Encyclopedia of Language and Linguistics.
The International Encyclopedia of Language and Linguistics, 2nd edn. Elsevier (2006)
21. Awad, D.: The evolution of Arabic writing due to European influence: the case of punctuation.
J. Arabic Islamic Stud. 15, 117–136 (2015)
22. Fadel, A., Tuffaha, I., Al-Jawarneh, B., Al-Ayyoub, M.: Arabic Text Diacritization Using
Deep Neural Networks (2019). http://arxiv.org/abs/1905.01965
23. Bebah, M., Amine, C., Azzeddine, M., Abdelhak, L.: Hybrid approaches for automatic vowelization of arabic texts. Int. J. Nat. Lang. Comput. 3, 53–71 (2014). https://doi.org/10.5121/
ijnlc.2014.3404
24. Yang, S., Wang, Y., Chu, X.: A Survey of Deep Learning Techniques for Neural Machine
Translation. arXiv. abs/2002 (2020)
25. Alzahrani, S.: Arabic Plagiarism Detection Using Word Correlation in N-Grams with KOverlapping Approach. Taif (2015)
26. Ali, Y., Salih, A.K., Osman, I.M., Abdelrahman, Y.A., Khalid, A., Osman, I.M.: A method
for Arabic documents plagiarism detection. Int. J. Comput. Sci. Inf. Secur. (IJCSIS) 15, 79
(2017). https://www.researchgate.net/publication/315656767
27. Jarrar, M., Zaraket, F., Asia, R., Amayreh, H.: Diacritic-based matching of Arabic words.
ACM Trans. Asian Low-Resour. Lang. Inf. Process. 18 (2018). https://doi.org/10.1145/324
2177
28. Alqahtani, F., Dohler, M.: Survey of authorship identification tasks on Arabic texts. ACM
Trans. Asian Low-Resour. Lang. Inf. Process. (2022). https://doi.org/10.1145/3564156
29. Kübler, S., Mohamed, E.: Part of speech tagging for Arabic. Nat. Lang. Eng. 18, 521–548
(2012). https://doi.org/10.1017/S1351324911000325
30. Meuschke, N., Gipp, B.: State-of-the-art in detecting academic plagiarism (2013). http://
www.ojs.unisa.edu.au/journals/index.php/IJEI/
31. Köhler, K., Weber-Wulff, D.: Plagiarism DetectionTest 2010 (2011). https://plagiat.htw-ber
lin.de/software-en/2010-2
32. Al Sulaiman, M., Moussa, A.M., Abdou, S., Elgibreen, H., Faisal, M., Rashwan, M.: Semantic
textual similarity for modern standard and dialectal Arabic using transfer learning. PLoS One
17, e0272991 (2022). https://doi.org/10.1371/journal.pone.0272991
33. Corezola Pereira, R., Moreira, V.P., Galante, R.: A new approach for cross-language plagiarism analysis. In: Agosti, M., Ferro, N., Peters, C., de Rijke, M., Smeaton, A. (eds.) CLEF
AraXLM
95
2010. LNCS, vol. 6360, pp. 15–26. Springer, Heidelberg (2010). https://doi.org/10.1007/9783-642-15998-5_4
34. Potthast, M., Barrón-Cedeño, A., Stein, B., Rosso, P.: Cross-language plagiarism detection.
Lang. Resour. Eval. 45, 45–62 (2011). https://doi.org/10.1007/S10579-009-9114-Z
35. Zubarev, D., Sochenkov, I.: Cross-language text alignment for plagiarism detection based
on contextual and context-free models. In: Computational Linguistics and Intellectual Technologies: Proceedings of the International Conference. Skolkovo Institute of Science and
Technology, Moscow (2019)
36. Jarrar, M., Khalilia, M., Ghanem, S.: Wojood: nested Arabic named entity corpus and recognition using BERT. In: 2022 Language Resources and Evaluation Conference, LREC 2022,
pp. 3626–3636 (2022)
37. Darwish, K., et al.: A panoramic survey of natural language processing in the Arab world.
Commun. ACM (2021). https://doi.org/10.1145/3447735
38. Khallaf, N., Sharoff, S.: Automatic difficulty classification of Arabic sentences. In: Proceedings of the Sixth Arabic Natural Language Processing Workshop, pp. 105–114. Association
for Computational Linguistics, Kyiv, Ukraine (Virtual) (2021)
39. Setha, I., Aliane, H.: Enhancing automatic plagiarism detection: using Doc2vec model.
Presented at the November 4 (2022). https://doi.org/10.1109/icaase56196.2022.9931542
40. Alotaibi, N., Joy, M.: English-Arabic cross-language plagiarism detection. In: International
Conference Recent Advances in Natural Language Processing, RANLP, pp. 44–52. Incoma
Ltd. (2021). https://doi.org/10.26615/978-954-452-072-4_006
41. Alzahrani, S., Aljuaid, H.: Identifying cross-lingual plagiarism using rich semantic features
and deep neural networks: a study on Arabic-English plagiarism cases. J. King Saud Univ.
Comput. Inf. Sci. 34, 1110–1123 (2022). https://doi.org/10.1016/J.JKSUCI.2020.04.009
42. Conneau, A., et al.: Unsupervised cross-lingual representation learning at scale. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics,
pp. 8440–8451. Association for Computational Linguistics, Online (2020)
43. Zhang, R., et al.: Hierarchical and pairwise document embedding for plagiarism detection.
In: Advanced Data Mining and Applications, China, p. 161 (2020)
44. Aljohani, A., Mohd, M.: Arabic-english cross-language plagiarism detection using winnowing
algorithm. Inf. Technol. J. 13, 2349–2355 (2014)
45. Avetisyan, K., Malajyan, A., Ghukasyan, T., Avetisyan, A.: A Simple and Effective Method
of Cross-Lingual Plagiarism Detection (2023)
46. Litschko, R., Vulić, I., Ponzetto, S.P., Glavaš, G.: On cross-lingual retrieval with multilingual
text encoders. Inf. Retrieval J. 25, 149–183 (2022). https://doi.org/10.1007/s10791-022-094
06-x
47. Son, J., Kim, B.: Translation performance from the user’s perspective of large language models
and neural machine translation systems. Information 14, 574 (2023). https://doi.org/10.3390/
info14100574
48. Reena, M., Satpute, S., Agrawal, A.: A critical study of pragmatic ambiguity detection in
natural language requirements. Int. J. Intell. Syst. Appl. Eng. IJISAE 11, 249–259 (2023)
49. Boudchiche, M., Mazroui, A.: Evaluation of the ambiguity caused by the absence of diacritical
marks in Arabic texts: statistical study. In: 2015 5th International Conference on Information &
Communication Technology and Accessibility (ICTA), Marrakech, Morocco. IEEE (2015)
50. Alqahtani, S., Aldarmaki, H., Diab, M.: Homograph Disambiguation Through Selective
Diacritic Restoration. Presented at the (2019)
51. Azmi, A.M., Alnefaie, R.M., Aboalsamh, H.A., Aboalsamh, H.A.: Light diacritic restoration
to disambiguate homographs in modern Arabic texts. ACM Trans. Asian Low-Resour. Lang.
Inf. Process. 21, 60 (2021). https://doi.org/10.1145/3486675
96
M. Alshehri et al.
52. AL-Smadi, M., Jaradat, Z., AL-Ayyoub, M., Jararweh, Y.: Paraphrase identification and
semantic text similarity analysis in Arabic news tweets using lexical, syntactic, and semantic
features. Inf. Process. Manag. 53, 640–652 (2017). https://doi.org/10.1016/j.ipm.2017.01.002
53. Mahmoud, A., Zrigui, M.: Semantic Similarity Analysis for Paraphrase Identification in
Arabic Texts, pp. 274–281 (2017)
54. Data set contains English and Arabic sentences: Data and Tools < SemEval-2017 Task 1.
https://alt.qcri.org/semeval2017/task1/index.php?id=data-and-tools. Accessed 03 May 2023
55. Agirre, E., Cer, D., Diab, M., Gonzalez-Agirre, A., Guo, W.: *SEM 2013 shared task: semantic
textual similarity. In: International Workshop on Semantic Evaluation, pp. 32–43 (2013)
56. Cer, D., Diab, M., Agirre, E., Nigo Lopez-Gazpio, I., Specia, L.: SemEval-2017 task 1:
semantic textual similarity multilingual and cross-lingual focused evaluation. In: Proceedings
of the 11th International Workshop on Semantic Evaluation SemEval, Vancover, Canada,
pp. 1–14 (2017)
57. Dagan, I., Dolan, B., Magnini, B., Roth, D.: Erratum: recognizing textual entailment: rational,
evaluation and approaches. Nat. Lang. Eng. 16, 105 (2010). https://doi.org/10.1017/S13513
24909990234
58. Qi, K., Wan, H., Du, J., Chen, H.: Enhancing Cross-lingual natural language inference by
prompt-learning from cross-lingual templates. In: Proceedings of the 60th Annual Meeting
of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland,
pp. 1910–1923. Association for Computational Linguistics (2022)
59. facebookresearch/faiss: A library for efficient similarity search and clustering of dense vectors.
Accessed 11 July 2022
60. Marjai, P., Lehotay-Kéry, P., Kiss, A.: Document similarity for error prediction. J. Inf.
Telecommun. 5, 407–420 (2021). https://doi.org/10.1080/24751839.2021.1893496
61. GitHub: GitHub - BKHMSI/deep-diacritization: Official Repository of the Deep Diacritization Paper. Accessed 21 Feb 2023
62. Sun, K., Luo, X., Luo, M.Y.: A survey of sentiment analysis based on pretrained language
models. In: Proceedings - International Conference on Tools with Artificial Intelligence,
ICTAI, pp. 1239–1244. IEEE Computer Society (2022). https://doi.org/10.1109/ICTAI56018.
2022.00188
63. Srinivasarao, U., Karthikeyan, R., Bilal, M.J., Hariharan, S.: Comparison of different similarity methods for text categorization. Presented at the (2023). https://doi.org/10.1007/978981-99-0550-8_39
64. Gonzalez, S.M., Huertas, J., García, Á.H.: AIDA-UPM. Applied Intelligence & Data Analysis.
https://huggingface.co/AIDA-UPM. Accessed 07 Nov 2023
65. Reimers, N., Gurevych, I.: Making Monolingual Sentence Embeddings Multilingual using
Knowledge Distillation (2020)
66. Reimers, N., Gurevych, I.: Sentence-BERT: Sentence Embeddings using Siamese BERTNetworks (2019)
Boosting Customer Retention in Pharmaceutical
Retail: A Predictive Approach Based
on Machine Learning Models
Angel Espinoza-Vega and Henry N. Roa(B)
Facultad de Ingeniería, Pontificia Universidad Católica del Ecuador, Av. 12 de Octubre 1076 y
Vicente Ramón Roca, P.O. Box 17-01-2184, Quito, Ecuador
{aaespinozav,hnroa}@puce.edu.ec
Abstract. Prediction models have acquired significant importance in business
decision-making by allowing the use of data to generate strategic information. This
study focuses on creating and evaluating a predictive model to identify possible
customer abandonment in a pharmaceutical retail company to promote retention
strategies. The approach is based on applying the Cross-Industry Standard Process for Data Mining (CRISP-DM) method and using Python as the primary data
science tool. Three classification algorithms were considered to predict customer
abandonment: neural networks, decision trees, and logistic regression. These algorithms were trained and evaluated using historical data from a pharmaceutical
company. After an exhaustive analysis of the results, we concluded that the neural
network model proves the most appropriate, reaching an impressive accuracy of
0.99 in the classification. The implementation of this model will allow the pharmaceutical retail company to identify in advance the customers with the highest
risk of abandoning their services. This identification enables the marketing team
to implement personalized and timely retention strategies, thus reducing the churn
rate and improving the overall efficiency and effectiveness of the organization.
Keywords: Data science · Machine learning · Classification models
1 Introduction
In the retail business, companies are dedicated to mass marketing products or services
to many consumers. Due to the number and variety of products they sell and the large
number of transactions generated daily, it has become one of the most challenging sectors
to achieve customer loyalty [1]. Specifically, pharmaceutical retail companies are no
exception to this problem. Even though there are methods and strategies to maintain a
loyal customer, the repurchase abandonment rate is high in this industry.
Due to this problem, it is necessary to find new mechanisms that allow customers with
a high probability of abandonment to be identified promptly. In this way, the marketing
area can apply correct loyalty strategies to reduce the defection of these customers [2].
In this article, we present the development of a predictive model that allows us to
determine the clients with a high probability of abandonment in the repurchases. To do
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2024
K. Arai (Ed.): SAI 2024, LNNS 1017, pp. 97–117, 2024.
https://doi.org/10.1007/978-3-031-62277-9_7
98
A. Espinoza-Vega and H. N. Roa
so, we developed three predictive models using different machine-learning techniques
and selected the model with the best accuracy. All this was done by applying the CrossIndustry Standard Process for Data Mining (CRISP-DM) method and using Python as
a programming tool. Section 2 provides a theoretical description of some concepts and
terms, machine learning algorithms, and tools. Section 3 uses the CRISP-DM methodology to address the existing problem. In Sect. 4, we discuss the results obtained. Finally,
in Sect. 5, we present the conclusions and limitations of our study.
2 Background
2.1 Cross-Industry Standard Process for Data Mining (CRISP-DM)
CRISP-DM is a process model for data mining and analysis used to guide professionals
in developing data mining and data analysis projects [3]. This methodology provides a
complete framework from understanding the business to deploying prediction models.
The CRISP-DM model is made up of six fundamental phases:
1) Business understanding. The business problem is identified in this phase, and the
project objectives are established.
2) Understanding the data. The data is collected during this phase, and an initial
exploration is carried out to understand its quality and structure.
3) Data preparation. In this phase, the data is cleaned and prepared for analysis.
4) Modeling. Statistical models and algorithms are built to analyze the data during this
phase.
5) Evaluation. This phase involves the evaluation of the models and the selection of the
best model for use.
6) Deployment. In this phase, the implementation of the selected model in the production
environment is carried out.
2.2 Classification Models
Classification models are machine learning techniques that classify data into groups or
categories. Some examples of classification models are explained below.
Decision Trees. Decision trees are machine learning models used to make decisions or
predictions based on the analysis of characteristics or attributes of a data set [4]. Decision
trees are beneficial for classification and regression tasks [4]. They are a visually intuitive
way of representing a set of decision rules in a tree structure, where each internal node
represents a characteristic or attribute, each branch represents a possible decision or
outcome, and each leaf represents the conclusion or classification.
In a decision tree, the process starts at the root node, where a specific feature is
evaluated [4, 5]. Depending on the value of that feature, a corresponding branch is
followed to the next node, and so on, until a leaf is reached that provides the final
decision or prediction.
Building a decision tree involves selecting the most relevant features to make effective
decisions and dividing the data set into smaller, more homogeneous groups based on
Boosting Customer Retention in Pharmaceutical Retail
99
those features [4, 5]. Several algorithms for building decision trees are such as the ID3
algorithm, C4.5, CART, and Random Forest [4, 5].
While decision trees are easy to understand and can capture complex patterns in the
data, they can also be prone to overfitting (when the model overfits the training data and
does not generalize well to new data). To mitigate this, techniques such as tree pruning,
which involves the removal of specific branches to prevent overfitting, can be applied
[5].
Logistic Regression. Logistic regression is a statistical method used mainly for analyzing and predicting data in which the dependent variable is binary [6]. Although the name
includes the word “regression,” logistic regression is used for classification problems,
not for regression problems where you are trying to predict a continuous numerical value
[6].
In logistic regression, we seek to establish a relationship between one or more independent variables (also called predictor or characteristic variables) and the binary dependent variable [6]. The objective is to model the probability that the dependent variable
belongs to one of the two categories based on the independent variables [6]. This probability is transformed using the logistic function (the sigmoid function), ensuring that
the result is 0 to 1 [6].
The sigmoid function has the following form:
P(Y = 1|X ) =
1
1 + e−(β0 +β1 X1 +β2 X2 +···+βn Xn )
where:
P(Y = 1|X ) is the conditional probability that the dependent variable is one given
the independent variables X .
e is the base of the natural logarithm.
β0 , β1 , β2 , . . . , βn are coefficients that are estimated during the model fitting process.
X1 , X2 , . . . , Xn are the values of the independent variables.
These β coefficients are determined using optimization techniques to fit the model
to the training data. Once the model is fitted, it can predict the probability of category
membership for new data and apply a threshold (for example, 0.5) to make a final
classification decision.
Neural Networks. Neural networks are a set of algorithms and models inspired by
the structure and functioning of the human brain [7]. They are designed to process
information and recognize patterns in data, especially in tasks that can be difficult to
solve using conventional programming [7]. These networks consist of interconnected
units called “artificial neurons” or “nodes” organized in layers that allow data input,
internal processing, and output generation.
The basic structure of a neural network is made up of three types of layers [7]:
1) Input layer: Receives the initial data and passes it to the network for processing.
2) Hidden Layers: These intermediate layers process and transform data as it passes
through weighted connections.
100
A. Espinoza-Vega and H. N. Roa
3) Output layer: Provides the final response or prediction of the network after processing
the data.
Each connection between neurons has an associated weight, determining the information’s relative importance [7]. During network training, these weights are adjusted by
learning algorithms to minimize errors in predictions and improve the network’s ability
to generalize patterns to new data [7].
2.3 Data Science Tools
Python. Python is a high-level interpreted programming language widely recognized
for being easy to learn but capable of harnessing the power of systems-level programming
languages when needed [8]. Many libraries support the data science process developed
by different communities for this language. The libraries used in this study are described
below.
Pandas is a library used for data manipulation and analysis. Pandas is one of the most
popular libraries for data analysis in Python, and it is used for data loading, cleaning,
transformation, and exploration [9].
NumPy is a library that performs complex and efficient mathematical calculations
on large data sets. It is widely used in data science and scientific research due to its
efficiency and ability to work with multidimensional matrices and arrays [9, 10].
Matplotlib is a popular and widely used Python data visualization library in the
scientific community [11].
Seaborn library provides many tools for creating statistical graphs [12].
Scikit-learn is a machine-learning library that provides many tools and algorithms
for data modeling and analysis [13].
Keras is a Python-based deep learning API built on TensorFlow. It emphasizes quick
experimentation, prioritizing swift progression from ideas to outcomes, which is crucial
for effective research [14].
Jupyter Notebook. Jupyter Notebooks is an interactive programming environment that
allows users to combine code, text, and visualizations in a single interface [15].
Jupyter Notebooks are divided into cells, where each cell can contain programming
code, rich text, images, videos, and other media. Users can write code in code cells, run
the code, and see the results immediately in the same interface [15].
3 Research Approach
This study uses the CRISP-DM method due to its structured and systematic approach.
This method ensures complete coverage of the activities necessary for the survey’s
success, and a logical and orderly sequence of phases is guaranteed. In the following
sections, we explain each of the steps of this method. Additionally, it is essential to
mention that all the stages of the CRISP-DM model were implemented in a Jupyter
Notebook with the Python programming language, and we used several Python libraries
discussed in Sect. 2.3.1.
Boosting Customer Retention in Pharmaceutical Retail
101
3.1 Phase 1: Business Understanding
In this phase, we carried out the understanding of the business to establish the objectives,
evaluate the current situation, set the goals at the level of data mining, and obtain a project
plan. The activities are described below.
Business Objectives. In this study, we established three business objectives. First, to
identify the customers with a high probability of abandonment (churn) in the pharmaceutical company. Second, to optimize customer retention and loyalty actions. Finally, we
developed a prediction model to determine the profile of clients with a high probability
of abandonment.
Current Situation. We comprehensively assessed the pharmaceutical company’s existing customer retention and churn management situation. We analyzed historical customer
transaction data, and we identified areas for improvement. We found that the company
is facing challenges in customer retention and is seeking to implement more effective
strategies to identify those with a high likelihood of churn.
Objectives at the Data Mining Level. We established three data mining objectives.
First, to identify customer behavior patterns prone to churn based on their transaction
history and demographic characteristics. Second, to develop churn prediction models
based on neural network techniques, decision trees, and logistic regression. Finally, the
performance of the prediction models will be evaluated and compared to determine the
most effective in detecting customers prone to churn.
Project Plan. We prepared a detailed project plan that includes the necessary resources,
the schedule, and the specific activities to be carried out in the following phases. The
plan establishes the stages of data collection and preparation, modeling, evaluation, and
deployment of the selected model.
At the end of Phase 1, we clearly understood the business with established objectives
for the data mining project, an assessment of the pharmaceutical company’s current
situation, and a detailed project plan. These elements provide a solid foundation for
moving to the next phase of the CRISP-DM method.
3.2 Phase 2: Data Understanding
In this phase, we ran the data capture processes, described the data, performed data
exploration tasks, and managed data quality by identifying problems and providing
solutions. To follow, we tell each activity in this phase in detail.
Data Capture. In this study, we obtained transaction data from the customers of an
Ecuadorian pharmaceutical retailer. This data covers the period from March 2022 to
February 2023. The records of each purchase include data such as the customer identification number, the date of purchase, the subtotal value of the investment, whether the
customer is affiliated with the pharmaceutical retailer (“is_spoonity”), the customer’s
gender, and the customer’s age.
Data Base Description. The dataset consists of a total of 2,869,628 customer purchase
records. The variables available in the dataset are the following:
102
A. Espinoza-Vega and H. N. Roa
• numero_identificacion: Unique identifier of the client (mostly ID).
• fecha_compra: The date on which the purchase was made.
• subtotal: Value of the purchase made by the customer.
• is_spoonity: Indicates whether the customer is affiliated with the pharmaceutical
retailer (1 if affiliated, 0 if not).
• genero: Customer’s gender (Man, Woman).
• edad: Age of the client (customer data from 18 years old).
Data Exploration. In this activity, we carried out different tasks to explore and deeply
understand the data. First, we grouped the data to obtain the information centrally, that
is, for each client. This grouping allowed the generation of the following new variables:
• fecha_compra_primera: date of the first purchase of each customer.
• fecha_compra_ultima: date of the last purchase of each customer.
• fecha_compra_penultima: penultimate date of each customer’s purchase.
• cantidad_compras_count: number of purchases made by the customer.
• subtotal: sum of the total purchase values.
• fecha_max: maximum date of the dataset.
• dias_transcurridos: number of days elapsed between the date of the customer’s last
purchase and the maximum date of the dataset.
• generacion: the age variable is used to assign to a respective category:
“Silent_Generation” (age >= 75)
“Baby Boomers” (57 <= age <75)
“Generation_X” (41 <= age <57)
“Millennials” (25 <= age <41)
“Generation_Z” (9 <= age <25)
Second, we examined the distribution of variables such as gender (Fig. 1), generation
(Fig. 2), and subtotal (Fig. 3).
Fig. 1. Number of clients by gender
Boosting Customer Retention in Pharmaceutical Retail
103
Fig. 2. Number of clients per generation
Fig. 3. Subtotal distribution
Data Quality Verification. The collected data is not perfect. While exploring the data,
we identified inconsistent, duplicate, and irrelevant values. These identified data quality
failures are solved in the next phase of the CRISP-DM model.
3.3 Phase 3: Data Preparation
In phase 3 of the CRISP-DM method, data preparation is carried out for later use in
modeling and evaluation. This phase involves the selection, cleaning, transformation,
and consolidation of the data. The activities carried out in this phase are detailed below.
Selection of Relevant Data. In this activity, we selected the relevant attributes for the
churn-prone customer classification analysis. The variables that we considered are the
count of the number of purchases (cantidad_compras_count), the subtotal of purchases
(subtotal), the affiliation to the pharmacy (is_spoonity), the gender (gender), the days
elapsed (dias_transcurridos) and the generation (generation). The rest of the variables
were excluded from the dataset. Figure 4 shows the scatter plots of the variables above.
104
A. Espinoza-Vega and H. N. Roa
Fig. 4. Scatter plots of the variables
Data Cleaning. During this activity, we addressed the data quality problems detected in
the previous phase, such as missing values, inconsistencies, and errors. We fixed incorrect
or inconsistent values. In addition, we carried out actions to eliminate duplicates and
irrelevant records, thus guaranteeing the integrity of the data.
Outlier Analysis. Additionally, we searched for unusual or extreme values that may
affect subsequent analysis. In the data exploration process, we identified the presence of
outliers in the “subtotal” variable, which represents the value of customer purchases. To
determine the presence of outliers, we used the Shapiro-Wilk test to assess the normality
of the data distribution [16]. When we applied the Shapiro-Wilk test to the “subtotal”
variable, a significance value (p-value) less than 0.05 was obtained, indicating that the
data did not follow a normal distribution. This suggests the presence of outliers in the
variable. We used the interquartile range (IQR) method to address this issue for data
cleaning and outlier identification. The interquartile range is the difference between a
data set’s third quartile (Q3) and the first quartile (Q1). It is calculated using the following
formula:
IQR = Q3 − Q1
Boosting Customer Retention in Pharmaceutical Retail
105
We applied the interquartile range rule to identify outliers, considering any value
below Q1 − 1.5 * IQR or above Q3 + 1.5 * IQR as a potential outlier. These values
are considered outliers due to their significant distance from the data distribution. In this
case, the upper range equals 314.84, and the lower range equals −172.04.
Once the outliers were identified, we removed them from the dataset. This allows
for a more representative distribution that reduces the impact of outliers in subsequent
analyses.
Correlation Analysis. We explored the correlation between variables to identify significant relationships that could affect the predictive models. The results show that the
variables are not highly correlated, except for the number of purchases with the subtotal, which could be automatically regularized in the modeling phase with the proposed
models (Fig. 5).
Fig. 5. Variable correlation
Thresholds Definition (Tagged Churn). In this stage, we applied five methods to define
thresholds and identify clients with a high probability of abandonment. The objective
is that with these five techniques, in the end, a decision is made as to whether the
client is prone to abandon or not. The methods are manual threshold, median threshold,
distribution threshold, RFM analysis threshold, and behavior analysis threshold.
Manual Threshold. We do this threshold of elapsed days based on business experience.
For this work, a threshold value equal to 150 was chosen.
Median Threshold. We calculated the median of the days elapsed. In our dataset, the
median has a value of 67 days.
Distribution Threshold. To determine the abandonment threshold, we examine the distribution of days between the last purchase and today’s date. For example, if most
customers make purchases within 30 days, a threshold of 60 days is defined to consider
a customer abandoned. The 75th percentile, which is 140 days, is used for this (Fig. 6).
106
A. Espinoza-Vega and H. N. Roa
Fig. 6. Abandonment threshold
These thresholds follow basic principles such as the elbow rule to identify behaviors
and are motivated by business experience. With these first three thresholds, the distribution of customers identified as prone to abandonment and those who are not is obtained
(Fig. 7).
Fig. 7. Customers who abandoned vs. did not abandon
RFM Analysis Threshold. We conducted an RFM analysis (Recency, Frequency, Monetary) to understand customer behavior better. RFM analysis is a widely used technique
in marketing that classifies customers based on three key variables: the recency of their
purchases, the frequency with which they make purchases, and the monetary value of
their purchases [17].
In the context of the case study of classifying customers prone to Churn in pharmaceutical retail, RFM analysis was applied to segment customers according to their
purchasing behavior. The recency variable represents the time elapsed since the last
purchase made by a customer. In contrast, the frequency variable indicates the number
of purchases made in a given period. The monetary variable reflects the total value of
purchases made by each customer.
Through RFM analysis, we obtained customer segments based on their purchasing
behavior. These segments can provide valuable information for later modeling and classification of churn-prone customers. Clients with low recency, frequency, and monetary
value are considered clients with a higher probability of abandonment. On the other hand,
customers with high recency, high frequency, and high economic value are identified as
loyal and high-value customers for pharmaceutical retail.
Boosting Customer Retention in Pharmaceutical Retail
107
The R, F, and M values are given according to the data distribution and can be seen
in Fig. 8, Fig. 9, and Fig. 10, respectively.
Fig. 8. R Analysis
Finally, using the 80th percentile, we chose the threshold value of 9 (Fig. 11). Those
whose RFM = R + F + M is equal to or greater than nine are labeled customers with a
low probability of abandonment.
Behavior Analysis Threshold. The approach we propose uses linear regression to identify those customers who tend to decrease their level of purchases over time. The idea is
to analyze the relationship between the purchase date and the subtotal of each transaction
for each customer. Figure 12 shows, as an example, the evolution of purchases of a small
group of customers.
Using all the customer data, we performed a linear regression analysis for each customer, where the purchase date is considered the independent variable and the subtotal
is the dependent variable. From the regression coefficients, specifically the coefficient
associated with the date, it is determined if there is a negative trend in customer purchases. If the coefficient is negative, it indicates that purchases tend to decrease as time
progresses. On the other hand, if the coefficient is positive, it suggests no clear downward
trend in customer purchases.
The result is an additional variable that indicates a value of 1 if each customer tends
to decrease their purchases or 0 if not.
108
A. Espinoza-Vega and H. N. Roa
Fig. 9. F Analysis
Fig. 10. M Analysis
Of the records, 79,040 do not have a negative trend, and 41,312 do. This approach
helps to objectively identify those customers who are declining in purchases over time.
Boosting Customer Retention in Pharmaceutical Retail
109
Fig. 11. RFM distribution
Fig. 12. Evolution of purchases by customer
Final Target. In this process, we created a new variable that represents the final target
(target_tendencia_disminucion) to label with the value of 1 all the customers that were
marked in the thresholds as possible customers that abandon, and with the value of 0 in
the opposite case. Figure 13 shows the customers labeled as likely to leave or not.
Data Transformation. At this stage, we performed transformations on the data to
improve its quality and make it fit the model requirements, as we describe to follow.
First, we identified the categorical columns in the data set. These columns represent
variables such as gender and pharmacy affiliation. Next, we applied the One Hot Encoding technique to convert the categorical variables into a numerical representation suitable
110
A. Espinoza-Vega and H. N. Roa
Fig. 13. Churn result
for use in machine learning models. After the coding was complete, we removed the
original categorical columns from the dataset, as they are now adequately represented
by the coded columns. The result was a data frame called ‘df_encoded’ containing the
original numeric variables and the encoded columns.
Second, we split the data into training (80%) and test (20%) sets to continue the
modeling process. This allows you to assess the model’s predictive ability on previously
unseen data.
In addition, we performed the standardization of the numeric variables using the scaling technique through the ‘StandardScaler’ class of the ‘sklearn.preprocessing’ module.
This ensures that all numerical variables have a comparable scale, thus preventing one
variable from dominating the others due to its magnitudes. The training (‘X_train’) and
test (‘X_test’) sets were transformed using the same scaler to maintain consistency in
the scale of the data.
Finally, we obtained a data set prepared and ready for use in the modeling and
evaluation phases.
3.4 Phase 4: Modelling
In phase 4 of the CRISP-DM methodology, we selected and trained the machine learning
models using the data prepared in the previous stages. This study uses three models:
neural networks, decision trees, and logistic regression, as we explain below.
Neural Network. We set up a neural network with hidden layers and an output layer.
We use suitable activation functions and adjust the training parameters to obtain an
optimal model. The neural network used in the model is built using the Keras library,
which provides a high-level interface for building and training neural networks. The
architecture of the neural network we defined is shown in Table 1.
Boosting Customer Retention in Pharmaceutical Retail
111
Table 1. Neuronal network architecture
Layer
Type
Neuron Number
Activation Function
Input
Dense
64
ReLU
Hidden
Dense
32
ReLU
Output
Dense
1
Sigmoid
The build parameters used were:
• Loss function: Binary Crossentropy
• Optimizer: Adam
• Evaluation metrics: Accuracy
Decision Tree. We built a decision tree using the relevant attributes of the data set.
We apply pruning techniques and hyperparameter tuning to improve accuracy and avoid
overfitting. The decision tree model was implemented using the “DecisionTreeClassifier”
classifier from the “scikit-learn” library.
Additionally, we set the parameter random_state to 42. This parameter is used to
initialize the random number generator and ensure the reproducibility of the results.
Setting it to 42 provides the model is built the same way each time it is run.
We trained the decision tree model using the fit method with the training data set
(x_train and y_train).
Logistic Regression. We apply logistic regression to predict the probability of customer
churn. We fit the model coefficients using the training data, and regularization techniques
were applied to avoid overfitting and improve generalization.
3.5 Phase 5: Evaluation
We evaluated the three machine learning models trained in the previous steps in this
phase. For the evaluation, we used 20% of data separated from the initial dataset (x_test
and y_test), which was not used in training the models.
After a training phase of 25 epochs of the neural network, the results of the executions
are shown in Fig. 14 and Fig. 15.
The confusion matrices of the neural network, decision tree, and logistic regression
are shown in Fig. 16, Fig. 17, and Fig. 18, respectively.
The accuracy of the models was calculated with the following formula:
Accuracy =
TP + TN
TP + FP + TN + FN
The results of the accuracy of each of the models are:
• Neural Network Accuracy: 0.99
• Decision Tree Accuracy: 0.98
• Logistic Regression Accuracy: 0.97
112
A. Espinoza-Vega and H. N. Roa
Fig. 14. Model Loss
Fig. 15. Model Accuracy
The chosen model is the neural network since it presents better accuracy and
performance with the data.
3.6 Phase 6: Deployment
During the CRISP-DM deployment phase, we deployed a pre-trained neural network
model to classify a new database covering the months of March, April, and May 2023.
The primary objective is to pinpoint customers with a high likelihood of churning and
propose strategic alternatives for retaining them.
To accomplish this task, we applied the previously developed neural network model
to the test dataset, encompassing detailed transaction information for each sale made
Boosting Customer Retention in Pharmaceutical Retail
113
Fig. 16. Neural network confusion matrix
Fig. 17. Decision tree confusion matrix
Fig. 18. Logistic regression confusion matrix
over the three months. The model utilizes various customer attributes, including purchase history, gender, age, and pharmacy affiliation, for the classification task. Consistent preprocessing was employed to ensure the neural network processes the data in a
standardized manner.
114
A. Espinoza-Vega and H. N. Roa
Once the model categorizes customers as potential churn candidates or not, the outcomes can inform strategic decisions aimed at retaining those at a high risk of churning.
Possible strategies encompass tailored loyalty programs, customized discounts or special
promotions, and targeted marketing campaigns.
Figure 19 shows the result of this classification. Class 1 corresponds to clients with
a forecast of abandonment.
Fig. 19. Classification of new data
4 Discussion
Section 3.5 of this study deals with the evaluation stage, highlights the results of the
developed predictive models. These models exhibit a substantially high level of precision, exceeding a threshold of 0.9 in terms of accuracy. However, the model based on
neural networks stands out with prominence, with an exceptional accuracy of 0.99. This
superiority makes this specific model the choice for the Deployment phase.
Despite the selection of this trained neural network model for the classification of new
data, it is essential to recognize the need for monitoring and updating the model over time.
The dynamics of customer behavior can vary between different periods, underscoring
the importance of continuous monitoring and periodic renewal of the model with more
current data. This practice will be essential to ensure the accuracy and relevance of the
forecasts generated.
Another crucial aspect is the ability to interpret the model results, identifying the
variables or characteristics that significantly influence the abandonment prediction. This
detailed understanding will offer valuable insight for formulating more effective and
targeted retention strategies.
Likewise, it is essential to underline the importance of an ideal infrastructure for
implementing the model in a production environment or in real time. This includes configuring servers, seamlessly integrating the model into a legacy application or platform,
Boosting Customer Retention in Pharmaceutical Retail
115
and implementing robust security and privacy measures to safeguard the integrity of
customer data.
Finally, integrating the model with customer management systems deserves outstanding attention. This integration can maximize the impact of retention strategies by
enabling a comprehensive and coordinated approach to activities related to customer
retention. Collaboration between the model and management systems could foster a
cycle of continuous improvement, where model predictions inform and guide strategic
retention decisions. In turn, customer interactions enrich and optimize the accuracy of
the model.
5 Conclusions
In this study, we applied the CRISP-DM method to develop a classification model to
identify customers prone to abandon (churn) in a pharmaceutical company. Throughout
the various stages of the process, we achieved a deep understanding of the business
environment, which allowed the precise definition of the data mining objectives and,
consequently, a clear delimitation of the problem to be addressed.
We carry out an exhaustive analysis of the data, focusing on the quality and management of the data. Through transformation techniques, we managed to prepare the
data in an ideal way for processing. In a subsequent phase, we selected three classification models: a neural network, a decision tree, and a logistic regression. We train these
models using the data set of customer purchases collected over a year. To evaluate and
compare their performance, we use metrics like accuracy.
The results reveal that the neural network, configured with two hidden layers (64 and
32 neurons) and a ReLU activation function, excelled by exhibiting superior accuracy
and predictive ability performance. This outstanding efficiency confirms the usefulness
of the neural network as an effective tool for identifying customers prone to abandonment
in the context of retail pharmacies.
In the deployment phase, we used the trained neural network to classify a new
database covering the months of March, April, and May of 2023. This deployment
process enabled the early identification of customers with a high probability of churn,
which, in turn, facilitated the proposal of specific strategic alternatives to retain them.
These alternatives range from creating personalized loyalty programs to offering special
discounts adapted to each client’s needs.
Ultimately, successfully applying the CRISP-DM methodology and creating a churn
rating system has given pharmaceutical companies powerful tools for detecting and
retaining customers at higher risk of churn. Adopting a neural network model and implementing tailored retention strategies has the potential to have a substantial impact on
customer retention and the achievement of business objectives. However, the need to
focus on continuous improvement and adaptation as customer behavior and market
dynamics evolve is emphasized.
This study is not without limitations. Examples of these limitations include possible
bias in the data, the need to check the generalization of the model in different contexts, the
stability of the model to slight variations, the importance of selecting the right features,
the need to update the model regularly, challenges in the practical implementation,
116
A. Espinoza-Vega and H. N. Roa
complex interpretation of models such as neural networks, sensitivity to hyperparameter
settings, costs and resources required to implement and maintain the model. Despite
the limitations, the study is critical because it guides business decisions. The predictive
model of customer abandonment in pharmaceutical retail, developed with the CRISPDM methodology and machine learning, allows anticipating churn and taking preventive
measures. Although there are restrictions, this approach can optimize retention strategies
and improve efficiency.
Acknowledgment. We wish to express our sincere gratitude to the Pontificia Universidad Católica
del Ecuador for their support of our research, which was conducted as part of the activities of the
Applied Information Systems and Technologies research group (SITECIA), registered under GIQuito-071-2022. Furthermore, we extend our heartfelt appreciation to the Anonymous Reviewers
whose thoughtful and constructive recommendations greatly enhanced the quality of this paper.
Their expertise and feedback played a pivotal role in refining our research and ensuring its rigor
and credibility.
References
1. Beg, M., et al.: Using Jupyter for reproducible scientific workflows. Comput. Sci. Eng. 23,
36–46 (2021)
2. Bisong, E., Bisong, E.: Matplotlib and seaborn. Build. Mach. Learn. Deep Learn. Models
Google Cloud Platf. Compr. Guide Begin. 151–165 (2019)
3. Cordero Preciado, C.A.: Árbol de decisión para el consumidor en la ciudad de Bogotá en el
Hard Discount (2020)
4. Flores Alejos, M.J.: Marketing digital y fidelización de clientes en una empresa retail
mayorista del distrito de Santiago de Surco-2018 (2019)
5. Huber, S., Wiemer, H., Schneider, D., Ihlenfeldt, S.: DMME: data mining methodology for
engineering applications–a holistic extension to the CRISP-DM model. Procedia CIRP 79,
403–408 (2019)
6. Ketkar, N., Ketkar, N.: Introduction to keras. Deep Learn. Python Hands- Introd. 97–111 2017
7. Khajvand, M., Zolfaghar, K., Ashoori, S., Alizadeh, S.: Estimating customer lifetime value
based on RFM analysis of customer purchase behavior: case study. Procedia Comput. Sci. 3,
57–63 (2011)
8. Kotsiantis, S.B.: Decision trees: a recent overview. Artif. Intell. Rev. 39, 261–283 (2013)
9. Lemenkova, P.: Processing oceanographic data by Python libraries NumPy, SciPy, and Pandas.
Aquat. Res. 2, 73–91 (2019)
10. Oliphant, T.E.: Guide to numpy. Trelgol Publishing USA (2006)
11. Raschka, S., Patterson, J., Nolet, C.: Machine learning in python: main developments and
technology trends in data science, machine learning, and artificial intelligence. Information
11, 193 (2020)
12. Razali, N.M., Wah, Y.B.: Power comparisons of shapiro-wilk, kolmogorov-smirnov, lilliefors
and anderson-darling tests. J. Stat. Model. Anal. 2, 21–33 (2011)
13. Tran, M., et al.: Python-based scikit-learn machine learning models for thermal and electrical
performance prediction of high-capacity lithium-ion battery. Int. J. Energy Res. 46, 786–794
(2022)
14. Vicente Carbajal, C.G.: E-Commerce y Fidelización del cliente en productos de belleza de la
empresa Retail Natura cosméticos SA SMP, 2019 (2020)
Boosting Customer Retention in Pharmaceutical Retail
117
15. Wang, S.-C., Wang, S.-C.: Artificial neural network. Interdiscip. Comput. Java Program.
81–100 (2003)
16. Wright, R.E.: Logistic regression (1995)
17. Yim, A., Chung, C., Yu, A.: Matplotlib for Python Developers: Effective techniques for data
visualization with Python. Packt Publishing Ltd. (2018)
Introducing Prediction Concept into Data
Envelopment Analysis Using Classifier
in Economic Forecast
Guangzao Huang1 , Zijiang Yang2(B) , Grace Liu3 , and Guoli Ji4
1 College of Electrical and Electronic Engineering, Wenzhou University, Wenzhou 325035,
China
guangzh@wzu.edu.cn
2 School of Information Technology, York University, Toronto M3J 1P3, Canada
zyang@yorku.ca
3 Bayview Secondary School, Richmond Hill L4C 2L4, Canada
350840351@gapps.yrdsb.ca
4 Department of Automation, Xiamen University, Xiamen 361005, China
glji@xmu.edu.cn
Abstract. Using data envelopment analysis (DEA) to evaluate the relative efficiency of decision-making units (DMUs) needs pre-determining all the input and
output variables. However, this may not be the case in the practical applications.
This paper proposes a DEA-Classifier model to predict the efficiency level of
DMUs with partial input and output variables. This model first divides the historical DMUs into two classes (high efficiency and low efficiency) based on their
efficiency scores determined by DEA and then uses the machine learning method
to establish a mapping relationship between the class labels and the partial input
and output variables. For a new DMU, the trained classifier can predict whether
it belongs to a high or low efficiency class using its partial input and output
variables. The proposed DEA-classifier model is validated by theoretical analysis, simulation data, and real-life datasets. The experiment results indicate that
the proposed method has good adaptability and it can be applied in many areas
including economic forecast and prediction.
Keywords: Machine learning · Data envelopment analysis · Classifier ·
Prediction
1 Introduction
Data Envelopment Analysis (DEA) is an efficiency evaluation method based on the
concepts of relative efficiency. The main idea of DEA includes transforming the multiple
inputs and outputs of the decision making units (DMUs) into comprehensive efficiency
indicators, i.e. efficiency scores, ranking DMUs using efficiency scores, and indicating
the reasons for the inefficient ones. Since its first introduction by Charnes, Cooper, and
Rhodes in 1978 [1], DEA has developed rich literature. Many extensions to basic DEA
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2024
K. Arai (Ed.): SAI 2024, LNNS 1017, pp. 118–127, 2024.
https://doi.org/10.1007/978-3-031-62277-9_8
Introducing Prediction Concept into Data Envelopment Analysis
119
models were introduced and a considerable number of researchers have applied DEA
in different areas. The popular extensions include weight restriction, stochastic DEA,
network DEA, DEA with imprecise data, etc. In addition, managing the big data concept
in DEA is also investigated. Ding, Dong, Liang, and Zhu [2] introduced a quantitative
assessment method for organizational goal congruence and extended the functionality
of the DEA technique. Despotis, Koronakos, and Sotiros [3] presented a novel approach
to estimate unique and unbiased efficiency scores for the individual stages in a two-stage
network DEA. Wei and Wang [4] extended the robust efficiency analysis procedure to
the situation where precise information on some input and output data is unavailable.
Misiunas et al. [5] applied DEA to remove the outliers of the dataset used to train artificial
neural networks (ANN). Their proposed method is implemented in a large dataset for
predicting the functional status of patients in organ transplant operations and the results
validate the proposed method. Khezrimotlagh et al. [6] propose a new framework that
can significantly decrease the required DEA calculation time when a large set of DMUs
(e.g., 20,000 DMUs or more) is present. Zhu [7] proposes that network DEA is big
data enabled analytics (big DEA) when multiple metrics are linked through network
structures. He demonstrates that network DEA can derive insights and value from big
data. Tayal et al. [8] propose a novel 4-stage methodology using big data analytics,
machine learning, hybrid meta-heuristic, DEA, and K-mean clustering for designing
an energy-efficient sustainable sub-optimal layout under uncertain (stochastic) demand
over multiple periods. Zhu et al. [9] aim to establish a linkage between the DEA method
and machine learning (ML) algorithms and propose an alternative way that combines
DEA with ML (ML-DEA) algorithms to measure and predict the DEA efficiency of
DMUs.
In the production and economic forecast applications, it is very important to predict
the relative efficiency in advance in order to reduce the risk and increase returns. However,
it is a common phenomenon that not all the input and output variables of production and
investment can be determined in the early planning stages and therefore it is of great
significance to predict their relative efficiency levels with available input and output
variables. The production and investment processes can be seen as DMUs, therefore
studying how to use partial input and output variables to predict the relative efficiency
of DMU has great theoretical and practical significance.
As far as we know, there are no published papers on how to predict the relative efficiency of DMUs with partial input and output variables. We propose a model, named as
DEA-Classifier model, to predict the relative efficiency level of DMUs with partial input
and output variables. Classification is an important research area in machine learning,
which has a broad range of applications such as computer vision, natural-language processing, biological feature recognition, medical diagnostics, handwriting recognition,
etc. The classifier constructs the mapping relationship between the sample features and
the pre-defined class labels after training with samples. The trained classifier can predict
the class labels for the new samples. Usually, the class label of a sample is determined
by its inherent attribute and is not affected by other samples. However, the relative efficiency results of DMUs are the evaluation results calculated by DEA, which are affected
by the other DMUs involved in the DEA model. Hence, one of the foremost considerations is how to assign the DMUs with appropriate class labels, which is significantly
120
G. Huang et al.
different from traditional classification problems. In this paper, the DMUs are classified
into two efficiency levels (high efficient or low efficient) proportionally according to the
efficiency scores given by DEA. Then the historical data of DMUs are used to train the
classifier. For a new DMU that needs to be predicted, we only need to take its partial
input and output variables into the trained classifier to obtain the prediction.
This paper creatively integrates DEA and classifiers to predict the relative efficiency
level of DMUs. The BCC model is one of the most widely used DEA models and the
least squares support vector (LSSVM) [10] is one of the most representative machine
learning algorithms. Therefore, this paper takes the integration of BCC and LSSVM as an
example of the DEA-Classifier model to analyze the effect of the proposed method. The
result shows that the DEA-Classifier model is effective and it can provide the foundation
for economic forecast.
The remainder sections of this paper are as follows: Sect. 2 introduces the proposed
DEA-Classifier model and the testing method to validate the DEA-Classifier model;
Sect. 3 discusses the datasets and experimental method; Sect. 4 presents experimental
results and discussions; Sect. 5 concludes the paper.
2 Proposed Methodology
2.1 DEA Model
DEA is an optimization methodology to measure the relative efficiency of DMUs, which
is defined as the ratio of outputs and inputs. For the multi-input multi-output DMUs in
order to get the ratio of outputs and inputs, each input or output variable needs a weight
coefficient to transform the multiple inputs into a total input and the multiple outputs
into a total output. As the importance of each input and output varies, how to obtain
reasonable weight coefficients becomes the key problem [11]. DEA method obtains the
optimum weight coefficient by linear programming, which makes the relative efficiency
evaluation from optimization and avoids the effect of subjective factors. Therefore, the
DEA method has intrinsic objectivity. Meanwhile, the DEA method does not need to
determine the expression of the relationship between the inputs and outputs. Thus, it is
widely used to evaluate the relative efficiency of DMUs. The typical DEA application
is to select the input and output variables for a given set of DMUs and then obtain
the efficiency scores of these DMUs to evaluate their relative efficiency. DEA can not
only evaluate the relative efficiency of the DMUs but also further analyze the causes
of the inefficiency of DMUs and provide the basis for improvement directions. The
most representative DEA models are the CCR model (under the assumption of constant
returns to scale) and the BCC model (under the assumption of variable returns to scale). In
practice, the return to scale is often variable; hence, the BCC model is closer to practical
conditions than the CCR model. The DEA model can be divided into input-oriented
and output-oriented. The input-oriented model aims to minimize the inputs under the
condition of unalterable outputs while the output-oriented model means to maximize the
outputs under the condition of unalterable inputs. This paper chose the input-oriented
BCC model as an example to provide the formula for the DEA-Classifier model.
Introducing Prediction Concept into Data Envelopment Analysis
121
2.2 DEA-Classifier Model
The DEA model requires all the values of the input and output variables of DMUs
to evaluate the relative efficiency. Aiming at using partial input and output variables
to predict the relative efficiency, we divide all the input and output variables into two
groups. The available input and output variables used to predict the relative efficiency
are called known variables and the unavailable input and output variables are called
unknown variables.
Apparently, there is a relationship between the known and unknown variables.
Because the DMU’s relative efficiency is determined by both known and unknown
variables and the unknown variables are related to the known variables, we intuitively
think that there is a relationship between the known variables and DMU’s relative efficiency. The relationships of the DMU’s relative efficiency (y), known variables (xk ), and
unknown variables (xunk ) are shown below:
y = DEA(xk , xunk )
→ y = f2 (xk , f1 (xk )) = f3 (xk )
(1)
xunk = f1 (xk )
If the relationship y = f3 (xk ) can be solved, then the problem of using the partial
input and output variables to predict the relative efficiency of DMU can be perfectly
solved. Using the machine learning method to learn the relationship y = f3 (xk ) is a
natural idea. The learning machine is
y = f4 (xk , ω)
(2)
which treats the relative efficiency as the dependent variable and the known variables
as independent variables. The historical DMUs represent the DMUs that all the input
and out variables are known. The new DMUs represent the DMUs that only partial
input and output variables are known and their relative efficiency needs to be predicted.
The historical DMUs’ relative efficiencies are given by a specific DEA model, therefore
the learning machine in fact learns the evaluation method of this specific DEA model.
The evaluation method of a DEA model is related to the type of adopted DEA model
and the DMUs involved in the evaluation. For convenience, the DEA model involving
only the historical DMUs is called DEA1 and the DEA model involving the historical
and the new DMUs is called DEA2 . Both DEA1 and DEA2 are the same type of DEA
model. DEA2 involves one more DMU than DEA1 , which means the historical DMUs’
relative efficiencies given by the two models may not be the same. As the new DMU’s
relative efficiency is given by DEA2 rather than DEA1 , then the relative efficiencies of
historical DMUs given by DEA2 should be used as the dependent variable of training
data. However, the relative efficiencies of historical DMUs given by DEA2 are unknown
because only partial input and out variables of the new DMU are available. Therefore,
only the relative efficiencies of historical DMUs given by DEA1 can be used as the
dependent variable of the training data, which is equal to training the learning machine
with data whose dependent variable is polluted by noise. The loss function can be shown
as
L(yDEA2 , f4 (xk , ω)) = f5 ( yDEA2 − f4 (xk , ω)) = f5 ((yDEA2 − yDEA1 ) + (yDEA1
−f4 (xk , ω))) = f5 (y + (yDEA1 − f4 (xk , ω)))
(3)
122
G. Huang et al.
where yDEA1 and yDEA2 are the relative efficiencies given by DEA1 and DEA2 respectively, f (xk , ω) is the trained learning machine. y represents the noise. (yDEA1 –
f (xk , ω)) is related to the type of adopted learning machine. Therefore, the loss function
is related to both the noise and the type of the learning machine adopted.
Since the DEA model provides a single efficiency score, which is continuous, it
is a very natural idea to construct a fitting model that assumes the efficiency score as
the dependent variable and the known variables as independent variables. However,
the efficiency score is sensitive to the frontier change, which is affected by the DMUs
in the DEA analysis. Thus, the efficiency scores of historical DMUs depend heavily
on how much the new DMU has changed the frontier. The more the new DMU has
changed the frontier, the more serious the noise pollution is. Hence, the efficiency score
and noise are highly correlated, which is quite different from the traditional machine
learning problems. For the traditional machine learning problem, the dependent variable
and noise are assumed to be independent. Besides, with the available data, it is hard to
estimate the degree of noise pollution in advance. Thus, the efficiency score prediction
is very unstable and directly using the efficiency score as the dependent variable of the
learning machine is inappropriate.
The next thought is to change the efficiency score into another evaluation form,
which weakens the influence of noise pollution to make the prediction more stable. We
propose one of these new evaluation forms and define it as a binary efficiency level. Its
introduction is provided below.
Definition 1 (Binary efficiency level). A relative efficiency evaluation method
divides the DMUs involved in a DEA analysis into two efficiency levels proportionally according to the efficiency scores. The DMUs ranked based on the efficiency scores
in the top [lx%] are classified as high efficiency level and the rest l − [lx%] are classified
as low efficiency level. x% is the split ratio, [ ] is a rounding operator that rounds a value
up to the nearest integer, and l is the number of DMUs involved in the DEA model.
Predicting the efficiency level of a new DMU has direct economic meaning. Its
purpose is to help investors to decide whether they should invest or not. The DMU located
in the split point is deemed as a reference DMU. The DMUs with higher efficiency levels
mean that they can be invested and the DMUs with lower efficiency levels mean that the
investors should not invest in those DMUs.
The following analyzes the rationale for using efficiency level as the dependent
variable of the machine learning model. All the historical DMUs’ relative efficiency
rankings given by DEA models according to the efficiency scores should converge to a
consistent state. The historical DMUs’ relative efficiency rankings can also be affected
by the new DMU. However, it is possible that the historical DMUs’ efficiency scores
are changed by the new DMU while their efficiency rankings are not. Consequently,
the relative efficiency rankings are more stable than the efficiency scores. For DEA1 ,
the class boundary of efficiency
level is px% , and for DEA2 , the class boundary of
efficiency level is (p + 1)x% , where p is the number of historical DMUs. For the case
when the historical DMUs’ relative efficiency rankings are not affected by the new DMU,
there is at most one historical DMU not consistently evaluated by DEA1 and DEA2 using
the efficiency level, which means the degree of noise pollution is no more than 1/p. In
the other case, the historical DMUs’ relative efficiency rankings are affected by the new
Introducing Prediction Concept into Data Envelopment Analysis
123
DMU, which means there is a fluctuation in the historical DMUs’ relative efficiency
rankings. However, the influence of the fluctuation can be effectively weakened using
binary efficiency level. If the fluctuation leads to the situation where some historical
DMUs’ relative efficiency rankings are not consistent between DEA1 and DEA2 but
the efficiency levels remain the same, then the binary evaluation results of these DMUs
according to DEA1 and DEA2 are still consistent. Hence, for the binary efficiency level,
the average degree of noise pollution can be controlled within a small range, which is no
more than 1/p. Thus the binary efficiency level and noise are approximately independent.
In summary, it is more appropriate for the binary efficiency level to be the dependent
variable of the learning machine than the efficiency score. If we treat the binary efficiency
level as a dependent variable of the learning machine, then the learning machine is a
classifier. Therefore we call the integration of DEA and classifier as the DEA-Classifier
model.
The process of using the DEA-Classifier model to predict the binary evaluation of a
new DMU is shown below:
Step 1: Divide the input and output variables into the known and unknown variables
according to the needs to predict the new DMU’s efficiency level.
Step 2: Use the DEA model to obtain the efficiency scores for all the historical DMUs.
Step 3: Rank the historical DMUs based on the efficiency scores and choose a suitable
DMU as a reference DMU to decide the split portion according to the ranking result.
Step 4: Set the historical DMUs with class labels according to the binary efficiency level.
Step 5: Train the classifier using the historical data.
h
(4)
D = xi_k , yi i=1
where xi_k indicates the known variables of the ith DMU and y ∈ [1, −1] represents the
class labels. 1 indicates one class and –1 indicates the other class. h is the number of
historical DMUs.
Step 6: Test the performance of the trained classifier using the historical data (see next
section).
2.3 DEA-Classifier Model Validation
Before the DEA-Classifier model is applied in practice, its performance should be tested.
In this paper, we use the leave-one-out cross-validation [12] to test the performance of
the DEA-Classifier model. The number of total DMUs is t. The leave-one-out crossvalidation divides all the DMUs into two complementary subsets. One subset containing
t − 1 DMUs is used to train the classifier, which is equivalent to the historical DMUs.
Another subset containing one DMU is used to test the trained classifier, which is equivalent to the new DMU. This process is repeated for t times until all the DMUs have been
predicted once and only once. Then the performance of the DEA-Classifier is evaluated
by the average prediction accuracy of the t classifiers.
The process of using the leave-one-out cross-validation to test the performance of
the DEA-Classifier model is shown below:
t
t
Step 1: Set the total DMUs xi_k , xi_unk i=j with the class labels yi_DEA2 i=1 according
to the binary efficiency level where xi_k , xi_unk and yi_DEA2 indicate the known variables,
unknown variables, and class label of the ith DMU respectively.
124
G. Huang et al.
t
Step 2: Divide the total DMUs into two subsets xi_k , xi_unk i=j and xj_k , xj_unk , where
the former represents the historical DMUs and the latter
represents
the new DMUs.
t
Step 3: Set the historical DMUs with the class labels yi_DEA1 i=j according to the binary
efficiency level, where yi_DEA1 is the class label of ith DMU.
t
Step 4: Use the historical data xi_k , yi_DEA1 i=j to train the classifier.
Step 5: Take the known variable of the new DMU xj_k to the trained classifier to get the
predicted class label yj_DEA2 .
Step 6: Repeat Step 2 to Step 5 t times (j = 1, · · · , t) to make each DMU predicted
once and only once.
Step 7: Calculate the prediction accuracy
2
TPi
(5)
accuracy = i=1
n
where TPi is the number of the DMUs in ith class that are correctly classified. If yj_DEA2 =
yj_DEA2 , then the predicted result is correct, otherwise, the predicted result is wrong.
3 Experiments
3.1 Datasets
In order to verify our proposed algorithm, we chose five datasets from different fields:
a productivity dataset in the Canadian resource industry, a profitability dataset in the
Canadian resource industry, a dataset on the health-related research projects in New
York State from the National Institutes of Health (http://projectreporter.nih.gov/report
er.cfm), a dataset on one big Canadian bank branches in Quebec, and a dataset on one big
Canadian bank branches nationwide. Table 1 summarizes the details of the five datasets.
3.2 Implementation
In real-life scenarios, it is possible for each input and output variable to be an unknown
variable and therefore there are various forms of unknown variables. However, the way
to construct the DEA-Classifier is the same regardless of the form of unknown variables.
Therefore, this paper chooses the easiest case, only one input and output variable is
unknown. In order to study the influence of different split portions on the model’s performance, the split portions are set to three different values (60%, 50%, and 40%). A specific
form of the DEA-Classifier model, which combines the input-oriented BCC model and
the LSSVM classification algorithm is used as an example of the DEA-Classifier model
to analyze the effect of the proposed method. For the LSSVM classifier, we choose the
Gaussian RBF kernel function below:
K(xi , xj ) = exp −
xi − xj2
2σ 2
(6)
which is one of the most commonly used kernel functions and its parameter σ is optimized by the simulated annealing algorithm. The platform is MATLAB2012 and the
corresponding toolbox used is LSSVMLAB1.8.
Introducing Prediction Concept into Data Envelopment Analysis
125
Table 1. Data set summary
Dataset name
No of DMUs
Input variables
Output variables
Canadian resource
productivity dataset
63
Total assets/ revenue,
Total equity/revenue, No.
of employees/revenue
Return on Equity, Return
on Asset
Canadian Resources
Profitability dataset
63
Total assets, No. of
employees, Equity
Profit, Total revenue
Health-related research
project dataset
92
Research funds, Project
duration
Total SCI impact factors
of published papers,
Number of authorized
patents
One big Canadian bank
branches in Quebec
78
Personnel, Equipment,
Occupancy, Other
expenses
Term Deposits,
Non-Term deposit
One big Canadian bank
branches nationwide
134
Administrative_FTE,
Sales_FTE, Service_FTE
Mortgages, Personal
Loans, Secured Lines of
Credit, Small Business
Loans
4 Experimental Results and Discussions
The experimental results of the five real datasets are shown in Table 2. From Table 2,
it can be seen that the prediction accuracy of the DEA-Classifier for different datasets
is significantly greater than 50%. Therefore, the learning machine can indeed explain
the relationship between the partial input and output variables and the efficiency level
to a certain degree to meet the goal of predicting the efficiency level of the new DMU.
It can also be seen that the prediction effects of the DEA-Classifier are different for
different datasets, which is understandable from the perspective of machine learning.
The effect of a learning machine method is related to the involved data and the adopted
methods. The complexities of relationships between the input and output variables, the
efficiency levels determined by DEA, and the spilt portion are different for different
datasets. Although LSSVM is a robust learning algorithm, it cannot work well for all
scenarios. Besides, the importance of the unknown variables to explain the relationships
between the partial input and output variables and the efficiency levels is also different
for different datasets, which further increases the complexity of the problem. Obviously,
if the input or output variables that are more important are unknown, it leads to worse
prediction effects.
Above all, the DEA-Classifier is a data-driven modeling method. Thus, in real-life
applications, it should be optimized to improve its prediction effect according to real data.
Model optimization is a complex problem including the model selection and parameter
setting.
126
G. Huang et al.
Table 2. The experimental results
Data set name
Accury1
Accury2
Accury3
Canadian resource productivity
0.7302
0.8095
0.9048
Canadian resources profitability
0.7143
0.9206
0.8254
Health-related research project
0.9239
0.8587
0.8043
One big Canadian bank branches in Quebec
0.8333
0.8077
0.7564
One big Canadian bank branches nationwide
0.9851
0.9552
0.9403
Note: Accury1, Accury2, and Accury3 are the prediction accuracies corresponding to the split
portion of 60%, 50%, and 40% respectively. The last output variable of each dataset is chosen as
an unknown variable.
5 Conclusion
In order to evaluate the relative efficiency of DMUs in advance, this paper first introduced
the concept of predicting the relative efficiency of DMUs with partial input and output
variables using a machine learning method. In order to get a stable prediction result, we
proposed to use efficiency level instead of efficiency scores. Then the proposed DEAClassifier model is applied to construct a mapping relationship between the efficiency
levels and the partial input and output variables. Both theoretical analysis and real-life
datasets are used to verify the effectiveness of the proposed DEA-Classifier model.
The result shows that the DEA-Classifier model is effective and it can provide the
foundation for economic forecast. In addition, the DEA-Classifier model discretizes the
continuous efficiency scores into discrete binary efficiency levels, which opens a new
way to improve prediction accuracy at the cost of reducing the distinguishing power
of relative efficiency. Obviously, discretization of the continuous efficiency scores into
more efficiency levels can improve the distinguishing degree of relative efficiency but
also increase the difficulty of prediction. How to make a good compromise between the
difficulty of prediction and the distinguishing power of relative efficiency is a big issue
and future work will concentrate on it.
Acknowledgment. This work was supported by Natural Sciences and Engineering Research
Council of Canada.
References
1. Charnes, A., Cooper, W.W., Rhodes, E.: Measuring the efficiency of decision making units.
Eur. J. Oper. Res. 2(6), 429–444 (1978)
2. Ding, J., Dong, W., Liang, L., Zhu, J.: Goal congruence analysis in multi-division organizations with shared resources based on data envelopment analysis. Eur. J. Oper. Res. 263,
961–973 (2017)
3. Despotis, D.K., Koronakos, G., Sotiros, D.: Composition versus decomposition in two-stage
network DEA: a reverse approach. J. Prod. Anal. 45, 71–87 (2016)
Introducing Prediction Concept into Data Envelopment Analysis
127
4. Wei, G., Wang, J.: A comparative study of robust efficiency analysis and Data Envelopment
Analysis with imprecise data. Expert Syst. Appl. 81, 28–38 (2017)
5. Misiunas, N., Oztekin, A., Chen, Y., Chandra, K.: DEANN: a healthcare analytic methodology
of data envelopment analysis and artificial neural networks for the prediction of organ recipient
functional status. Omega 58, 46–54 (2016)
6. Khezrimotlagh, D., Zhu, J., Cook, W.D., Toloo, M.: Data envelopment analysis and big data.
Eur. J. Oper. Res. 274(3), 1047–1054 (2019)
7. Zhu, J.: DEA under big data: data enabled analytics and network data envelopment analysis.
Ann. Oper. Res. 309, 761–783 (2022)
8. Tayal, A., Solanki, A., Singh, S.P.: Integrated frame work for identifying sustainable manufacturing layouts based on big data, machine learning, meta-heuristic and data envelopment
analysis. Sustain. Cities Soc. 62, 102383 (2020)
9. Zhu, N., Zhu, C., Emrouznejad, A.: A combined machine learning algorithms and DEA
method for measuring and predicting the efficiency of Chinese manufacturing listed
companies. J. Manag. Sci. Eng. 6(4), 435–448 (2021)
10. Suykens, J.A., Vandewalle, J.: Least squares support vector machine classifiers. Neural.
Process. Lett. 9, 293–300 (1999)
11. Podinovski, V.V., Bouzdine-Chameeva, T.: Consistent weight restrictions in data envelopment
analysis. Eur. J. Oper. Res. 244, 201–209 (2015)
12. Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, New York (2006)
Investigating Machine Learning Techniques
Used for the Detection of Class Noise in Data:
A Systematic Literature Review
Cindy van den Berg1 and Sunet Eybers2(B)
1 University of Pretoria, Private Bag X20, Hatfield, Pretoria 0028, South Africa
2 University of South Africa (UNISA), Science Campus, Florida, Johannesburg, South Africa
eeyberss@unisa.ac.za
Abstract. Data provides valuable information and insights and assists in making
strategic decisions. The quality of the data is distorted by noise, which negatively
affects information, insights, and decisions made. It also influences a machine
learning model’s complexity, accuracy, results, and ability to learn. Therefore,
the correct model should be married with the appropriate dataset whilst taking
cognisance of data noise. The study’s objective is to identify machine learning
techniques for class noise detection by considering various strengths and weaknesses to aid in selecting the correct model. A systematic literature review was used
as a research approach and resulted in a taxonomy for noise treatment to classify
machine learning class noise detection techniques whilst considering strengths and
weaknesses of the techniques. Further research should focus on machine learning
techniques to make selecting the appropriate data noise detection or managing
approach easier.
Keywords: Data noise · Noise detection · Machine learning techniques
1 Introduction
Data are at the heart of every organisation, and an organisation needs to understand
the data they have and communicate the importance thereof. Due to the continuous
digitisation of organisations and the volume of data collected daily, organisations are
compelled to become data-driven [1]. The digital economy converts large volumes of
data into business assets [1]. The success of an organisation depends on information
and insights that can be gathered from data. Therefore, quality data are imperative to
create useful information and insights that can assist the organisation in making quality
decisions and gaining a competitive advantage in the market [1–3].
Organisations need to be data-driven and ensure that quality data are collected.
Quality data can assist in strategic decision-making. Any real-world dataset will not be
perfect as data quality is influenced by incorrect data, poor definition, and inconsistency
[4]. This phenomenon is referred to as ‘dirty data’, which is also referred to as noise [5].
Noisy data influences decision-making, information, and insight gathered from data.
Data noise is the number one reason for bad quality data and the presence of class noise
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2024
K. Arai (Ed.): SAI 2024, LNNS 1017, pp. 128–147, 2024.
https://doi.org/10.1007/978-3-031-62277-9_9
Investigating Machine Learning Techniques
129
harms the quality of models built, insights derived from the model, problem formulation, and decision-making [6–8]. Therefore, it is important to identify and handle noise
effectively to improve the output delivered by machine learning [9].
When dealing with noise, two approaches are highlighted in the literature, namely
noise detection and noise handling. It is important to understand the machine learning
techniques available for each approach. The challenge is to choose the most appropriate
noise detection or noise handling technique for class noise [10]. When the connection
between data quality and the accuracy of the outcome is understood, an appropriate
technique can be selected [10]. To choose a technique, one should be aware of the
different machine learning techniques available from current literature. Literature as an
existing body of knowledge serves as an explanation of available techniques for both
class noise detection and handling. In this study, only machine learning techniques for
the detection of class noise will be investigated since noise handling techniques replicate
the work done by the noise detection techniques [11].
Therefore, the main research question for this paper is: What noise detection machine
learning techniques exist to detect class noise? To answer this research question, a
systematic literature review process was used to provide an overview of machine learning
techniques, their classification, and their strengths and weaknesses.
The remainder of the paper is organised as follows: an overview of class noise,
attribute noise, class noise detection, and class noise handling techniques are presented
in Sect. 2, the research approach in Sect. 3, followed by the results and findings of the
systematic literature review in Sect. 4. The paper is concluded in Sect. 5.
2 Background: Noise Types
Data noise harms the value of information extracted from data, the outcomes of machine
learning models created, and the choices made on how to engage with the data [7, 8,
12]. The presence of data noise minimises classification precision and increases the
complexity and time of the machine learning model [12–14]. Therefore, data needs to
be cleansed by removing data noise to improve the quality of the data [4, 15]. Two
categories of noise exist within data, namely class noise and attribute noise, both of
which are discussed in the following sections [11, 16].
2.1 Class Noise
Class noise, also called label noise, refers to instances where records are labelled incorrectly or where the same instance is labelled differently [11, 17]. Records labelled incorrectly may cause harmful outcomes like subjective learning and decreased prediction
precision [18]. Common causes of class noise are partiality at the time of label assignment, data capturing errors, and insufficient information to assign labels correctly [13].
Classes are important in machine learning as they are the categories derived from the
data for which insights and predictions are required [7]. The process of assigning classes
is referred to as classification and it aims to allocate a class from a predetermined list of
classes to unclassified records [19]. These classes are also used to identify relationships
130
C. van den Berg and S. Eybers
and patterns. Classification in the fields of data mining and machine learning is continually being researched [20]. Classification is applied in various industries in terms of
fault identification, image identification, network inconsistency discovery, and disease
judgement [20]. Class noise is assumed to impact machine learning techniques more
than attribute noise since it influences classification [21, 22].
2.2 Attribute Noise
Attribute noise, also called feature noise, refers to instances where the value assigned to
an attribute is incorrect [16, 23]. Barreto and Barros [24] explained that attribute noise
influences the perceived values of relationships in models. Gupta and Gupta [11] defines
three kinds of attribute noise, namely (1) erroneous attribute values, (2) missing or don’t
know values, and (3) incomplete or don’t care values. Missing values are either removed
or derived. Using a derived value implies that an alternative value resembling the missing
value, is used. Attribute noise is seen as complex and very few techniques for detecting
attribute noise have been studied [11]. Class noise is perceived to have a bigger impact
than attribute noise [15, 22].
2.3 Class Noise Detection and Class Noise Handling Techniques
Class noise detection techniques are utilised in scenarios where the data analytics model
is not noise tolerant or robust to noise. This implies that once class noise is detected,
it will be filtered out or edited to correct erroneous labels. This is referred to as a
data level approach [21]. A data level approach forms part of the pre-processing of
a dataset to remove (filter out) or correct class noise and has proven to be effective in
improving results. Removing noisy records using filtering before the model is developed
is advantageous as it removes the risk of noisy records influencing the structure of the
model [25]. Editing is also referred to as polishing [25]. Editing aims to increase the
quality of data rather than removing the data [19]. Editing entails replacing the noisy
label with an approximate label [21, 26]. The edited records are then reinstated into the
dataset [25]. The aim of noise detection is also to understand the data and the problems
that exist in the data collection process [10, 25, 27]. Xia et al. [22] described two main
types of machine learning techniques for the detection of class noise as classifier-based
and distance measure-based. In comparison, Gupta and Gupta [11] proposed three types
of class noise detection techniques, namely distance-based, single learning-based, and
ensemble-based. The focus of these techniques is to improve the quality of data and
increase the usefulness of the process of extracting insights [28].
Class noise handling techniques refer to machine learning techniques that can deal
with label noise. These machine learning techniques do not remove or edit class noise. It
concentrates on the effect of class noise and on how to explain it [14]. Machine learning
techniques have been shown to overcome the deficiencies of conventional statistical
categorisation [29].
Prati et al. [21] developed a taxonomy for class noise based on the taxonomy created by Frenay and Verleysen [14]. The taxonomy as depicted in Fig. 1 specifies two
approaches to class noise treatment, namely (1) data level and (2) algorithm level [21].
Investigating Machine Learning Techniques
131
Data level approaches relate to the class noise detection techniques discussed in the previous section. Algorithm level approaches speak to class noise handling techniques that can
be categorised into the three categories discussed above, i.e. (1) machine learning techniques that are robust to class noise, (2) probabilistic noise tolerant, and (3) model-based
noise tolerant. The taxonomy classifies the algorithm level as three different categories
of machine learning techniques.
Fig. 1. Taxonomy of noise treatment methods [21]
3 Research Approach
A systematic literature review approach was used to gather academically relevant articles focusing on the machine learning techniques used for the detection of class noise,
including its strengths and weaknesses.
The Preferred Reporting Items for Systematic Reviews and Meta-Analyses
(PRISMA) are suggested to be used as a means of recording the review process for
systematic literature reviews [30]. The four steps followed for review are:
Step 1: Identification of articles using the following keywords: class noise, attribute
noise, noise detection, as well as the following machine learning related keywords:
k-nearest neighbours, Naïve Bayes, support vector machines, random forest, distancebased techniques, density-based spatial clustering of application with noise, k-means
clustering, learning vector quantization, ensemble-based techniques, bagging, boosting,
and stacking. The inclusion criteria were based on language, type of article, and time
frame. The search was focused on English peer-reviewed articles with full text and
citation and where the keywords were present in the title, content, abstract, and article
keywords. Books and dissertations were excluded. Articles from 2010 onwards were
132
C. van den Berg and S. Eybers
taken into consideration for class noise, attribute noise, and noise detection. Articles from
2019 onwards were considered for the specific machine learning techniques. Backward
topic searching and author searching articles were considered. In total 195 peer reviewed
articles were found as part of the database search process and 10 other relevant articles
were found from other sources.
Step 2: Screening of identified articles. Hundred and thirteen articles remained after
duplicate articles were removed. Hundred and four articles were screened. Sixteen of the
104 articles were excluded. Articles that did not mention one of the following keywords
in the content of the article were excluded.
Step 3: The eligibility of the remaining 88 articles was reviewed based on the following conditions: (1) does the article provide a definition for data noise or class noise, (2)
does the article discuss a machine learning technique that can be used for the detection
of class noise, and (3) does the article list strengths and weaknesses for the machine
learning techniques discussed.
Step 4: The 88 eligible articles were assessed for their quality. Rousseau [31] classifies six universal sets of criteria for quality assessment, namely, (1) construct validity, (2)
internal validity, (3) effect size, (3) generalisability, (4) intervention compliance, and (6)
contextualisation [31]. Table 1 below lists the quality assessment criteria and questions
for this study.
Table 1. Quality assessment criteria (based on [32])
Number Quality assessment criteria Quality question
QA1
Reporting
Does the article clearly state the purpose and findings
of the study?
QA2
Rigor
Does the article clearly state the research method
applied?
QA3
Credibility
Does the article state limitations to the validity of the
findings?
QA4
Relevance
Does the article relate to class noise detection
techniques?
QA5
Relevance
Does the article highlight the strengths and weaknesses
of the machine learning technique in the study?
Most articles (90.91%) scored four or more. Sixty-five articles attained a score of
four, constituting 73.86% of the articles reviewed. Fifteen articles attained a score of
five, constituting 17.05% of the articles used. The eight residual articles scored three and
only made up 9.09% of the articles reviewed. The total number of articles in the final
research article pool was 88.
3.1 Data Analysis
Thematic analysis was used to identify key themes across the abstracts of the final
research article pool. From the final research article pool, fourteen articles provided a
Investigating Machine Learning Techniques
133
definition or explanation of data noise and subsequently identified two types of data
noise, namely class and attribute noise. These articles also discussed algorithms for the
treatment of data noise. One of the articles discussed a taxonomy for the treatment of
class noise. Furthermore, topics of noise detection and noise handling were identified
as topics for further reading. As a result, twenty-three additional articles were read to
gain a better understanding of techniques proposed for the detection and handling of
class noise. As part of the analysis process, the researcher used the taxonomy for the
treatment of class noise as proposed by Prati et al. [21] (see Fig. 1) as lens to identify
machine learning techniques. This resulted in three main categories of machine learning
techniques used for the detection of class noise, including classifier-based, distancebased, and ensemble-based [21, 22, 33, 34]. Using these categories as starting point, an
additional 56 articles were identified. Each of the articles was applicable to either the
identification of machine learning techniques or explained strengths and weaknesses of
machine learning techniques relevant to the detection of class noise in data.
4 Discussion of Results
Noise in data is different from having outliers in a dataset. Noisy data refers to records
that hold no value in data analysis or modelling. Noise in data distorts data quality and
the value that can be derived from the data [35]. The main causes of noise are data of
incorrect format entered, incorrect value entered, and unspecified values. Noise needs
to be eliminated from a dataset whereas an outlier is a deviation and could hold valuable
information [36]. Class noise causes decreased accuracy of the results delivered by a
machine learning model [37].
It is imperative to select a suitable machine learning technique for the type of data
being studied [38–40]. The aim is to select the best technique to achieve more exact and
efficient models [35, 41]. Three main machine learning techniques were identified in
this study that can assist in detecting class noise [42], namely, classifier-based, distancebased, and ensemble. Each of these categories with their techniques is discussed in the
section below.
4.1 Classifier-Based Techniques
Classifier-based techniques aim to sort data into categories and appoint a class label
to each category [34]. Common classifier-based techniques are logistics regression,
Naïve Bayes (NB), k-nearest neighbours (K-NN), decision tree (DT), and support vector
machines (SVM) [20, 22]. Classifier-based techniques are effective for the identification
of class noise [22]. These techniques are seen as supervised learning techniques and are
utilised in scenarios where the data is structured [41]. The goal of supervised learning
techniques is to anticipate the output of new data provided as input [33, 34]. Supervised
learning techniques are used in scenarios where regression or classification is required
[33]. Classifier-based techniques are assessed based on the correctness of predictions,
speed, sturdiness, scalability, and understandability [41, 43]. The main challenge is to
build the best model to handle the amount of data in the dataset [41]. These techniques
134
C. van den Berg and S. Eybers
provide a ruleset that utilises different levels of computing power and the higher the
amount of data, the more complex the rules become [36, 41].
Decision Tree (DT)
Strengths. DT is a supervised machine learning technique that is well known and widely
used [44–47]. The technique is primarily used for classification purposes and regression
[39–41, 43, 45, 46, 48]. DT is not dependent on parameters and does not make presumptions about associations, it uses labelled records to accurately categorise new data [45,
47–49]. The technique is easy to understand and provides a set of rules as output [39, 40,
43–45, 47, 50]. DT does not require data preparation and can be applied to unnormalised
data [47]. It is reported to be stable and robust with a high accuracy and low error rate
[49, 51]. DT is also beneficial in the process of data discovery as it is less dependent on
clean data and assists in identifying possible features [39–41, 45, 48, 49]. The technique
can be used for scenarios where multiple predictions are required for a single feature
[49]. The technique can accommodate different data whether values are fixed or variable
[40, 41, 45, 48, 49]. In addition, the technique has a reasonably low computational cost
and a high classification speed [40, 43, 48].
Weaknesses. DT’s predictive precision is lower than other machine learning techniques
[44, 46]. A small change in the dataset can cause an alteration in the decision tree structure which makes the technique volatile [39–41, 48]. The technique is also prone to
misclassification and the application can become complicated when trying to accommodate a large amount of feature labels [40, 44]. It is highly influenced by class noise which
increases complexity [47]. DT is prone to overfitting and the technique misses important
facts while grouping variables [41, 45, 47–49, 51]. The technique is rated low on accuracy when it comes to complex datasets [52]. Cervantes et al. [46] report the technique
to be inflexible to the modelling of features. Meaningless features negatively impact
performance [53]. In addition, the technique lacks in predicting continuous values and
applying regression [39].
K-nearest Neighbours (K-NN)
Strengths. K-NN is a commonly used method for classification and is a supervised
learning technique [35, 41, 44, 45, 54, 55]. The technique is reported to be a reliable
classifier with good accuracy [44, 55]. K-NN is an easy technique that can be implemented in different applications or fields of analysis [43–46, 49, 56]. The technique
is easy to understand and implement [40, 45, 55, 57]. K-NN can be used in scenarios
where regression or classification needs to be applied and the results are determined by
the application [35, 44, 45, 55, 57]. The technique can also be used to allocate absent
values based on labelled training data and can be applied to various samples of a training
dataset [45, 49, 55]. It can handle large datasets [53]. In addition, K-NN can ignore
the skewness of the dataset size and has a high accuracy rate [43]. K-NN can also be
utilised with data where relationships exist and where no features can be identified [57].
It can predict relationships between new and existing data based on the prediction of the
model [45]. Bo and Haiyan [56] report K-NN as part of the top 10 methods utilised in the
field of data mining and machine learning. The technique provides performance similar
to other advanced classification techniques like decision trees (DT) and support vector
Investigating Machine Learning Techniques
135
machines (SVM) [56]. K-NN has been found to perform better than random forest (RF)
[49]. The technique can be used together with other machine learning techniques like
support vector machines (SVM) and k-means [55]. The technique’s weakness in class
noise is now used as an advantage in that the technique is effective in identifying class
noise to remove noise and address labelling challenges [35].
Weaknesses. Abdulrahman et al. [44] found K-NN to be less precise and efficient than
support vector machines (SVM). K-NN is found to perform less efficiently with complicated classification datasets [35, 49, 57]. The technique is also reported to be a weaker
model than random forest (RF) in terms of validity [58]. The performance and effectiveness decline with larger datasets [35, 40, 43, 44, 46, 55, 57]. Predictability decreases
as the number of features in the dataset grows [40, 44, 49, 57]. K-NN does not perform
well on datasets with skewed proportions and is highly influenced by exceptions [44,
56]. The results of the model are also highly influenced by the existence of class noise
or unnecessary features [35, 41, 46, 57, 58]. The technique requires data without noise
and that is labelled to provide a model [35, 45]. It can therefore be utilised to identify
the class noise in the dataset. The classification of unlabelled records is also expensive
[35, 40, 41]. K-NN accuracy is dependent on parameter adjustment to determine the best
value for K [45, 55]. Another disadvantage of K-NN is that the technique implements
various functions to calculate distance which is time consuming [54, 55]. K-NN is more
costly than other machine learning techniques.
Naïve Bayes (NB)
Strengths. NB is a supervised learning technique and can be used to predict probabilities
of classes [37, 40, 41, 43, 44, 54, 59, 60]. The technique can be used for multiple
applications, is robust and can handle large datasets efficiently while delivering accurate
results [44, 59–61]. Performance is much higher on smaller datasets [61]. It is an effective
technique to be used for datasets where regression or classification is required [44]. NB
has a well-grounded arithmetic base, is uncomplicated and highly efficient [40, 43, 50,
53, 54, 60]. In addition to the technique being easy to build and understand it does not
require intricate programs for repeated parameter inference and is easily adaptable to
the introduction of new features [37, 39, 44, 59]. NB only requires a small sample set
to determine required parameters and uses memory and processing power moderately
[37, 40, 41, 44]. The technique consists of methods that facilitate rapid learning and
testing and can be trained on small datasets [44, 59]. NB also performs well on datasets
where dependencies exist between features in the dataset [59]. The technique can be
applied to continuous or discreet datasets and is insensitive to unnecessary features [40].
It can also handle class noise and missing data [41, 61]. NB is more suitable for datasets
where several categories are present [40, 43, 61]. Al-Aidaroos et al. [59] report that
scholars found NB to be precise and that it performs better than other machine learning
techniques.
Weaknesses. The technique’s classification is built on the hypothesis that variables are
independent, which causes noise and unnecessary features to receive the same weighting
as important features [37, 39, 43, 44, 50, 53, 59, 60]. When multiple features show a
strong relationship, it gets a higher decision rating in terms of class classification of
136
C. van den Berg and S. Eybers
the results which causes reduced precision [37, 59–61]. NB is a probability classifier,
which is dependent on incidence [53]. The technique is not suitable for scenarios where
a feature needs to be applied as a quantitative variable [40]. Tuning and training play a
very important role in the technique [41, 50]. The technique can present distorted results
in large datasets [61]. NB is also reported to have a lower level of prediction accuracy
and requires more computational power [41, 61].
Support Vector Machines (SVM)
Strengths. SVM is a supervised machine learning technique which is a simple method
and widely used [39, 44, 46, 47]. The method finds structures in the data that are utilised
for classification, it can also be used in scenarios where regression is required [39, 44–
46, 49, 61–64]. SVM is a powerful and precise technique that can also be utilised in
data mining and pattern recognition [44, 46, 49, 61]. The technique provides accurate
classifications and the technique is stable and robust [50, 65]. SVM can be used on
small to medium sized datasets [65, 66]. Mohd et al. [49] report SVM to be a technique
that minimises bias in predictions. The technique outperforms other machine learning
techniques when few variables or features exist [67]. The results of SVM are easy to
interpret [61]. The technique’s classification strategy reduces classification errors, and
the error acceptance can be adjusted [46, 61, 63, 68]. SVM can handle complex problems
with the selection of a suitable kernel function and it is less likely to be overfitting [40, 41,
63]. Ahmad Sobri et al. [68] report that SVM can be implemented on very unstructured
data while Cervantes et al. [46] report SVM to only be effective on well-structured
datasets. Nalini and Usha [41] and Ray [40] report SVM to be effective on both semi
and structured data. Boyko and Boksho [54] report SVM works effectively on datasets
with few instances and where data issues are evenly distributed. SVM works more
efficiently on datasets where there are many features and classes that are not similar
[34, 39, 49]. The technique still performs effectively in scenarios where the number of
characteristics is more than the samples [34]. SVM is rated well on how accurately it
classifies new data based on the classifications identified in the training data [33, 64].
Weaknesses. SVM has a high computational cost in large datasets which causes the
technique to be slow [34, 39–41, 43, 46, 49, 50, 61, 63–65, 68]. Abdulrahman et al. [44]
and Mohd et al. [49] report SVMs to be slower than KNN. The technique is developed
for binary classification efforts and is difficult to use in multiclass scenarios [46, 47,
50, 63, 66, 67]. In addition, selecting the most suitable weighting function is difficult
[33, 40, 41, 44, 46, 63, 66]. SVM is vulnerable to parameter tuning and changes to the
weighting function [43, 63, 66]. The technique is disposed to misclassification and does
not perform well on datasets with a large volume of class noise or incorrectly labelled
data [39–41, 45, 46, 62, 67, 68]. SVM can only be applied to labelled data [47]. It is also
sensitive to outliers [62]. Overlapping classes causes decreased performance [34]. The
technique does not offer a likelihood score and the understanding of the resulting model
is complex [40]. It does not perform optimally on skewed datasets [46]. In addition,
SVM cannot handle numerous data types and is sensitive to parameter changes [61].
Random Forest (RF)
Investigating Machine Learning Techniques
137
Strengths. RF is a supervised machine learning technique and makes use of decision
trees [49, 52, 58, 66, 68]. It can be applied in classification and regression scenarios [34,
49, 61, 69]. RF is an easy technique and provides consistent predictions [34, 65, 69].
The accuracy rate remains high when new data is introduced to be classified based on
training data [65]. The technique can be used for large datasets or small datasets and, in
addition, it can handle datasets with multiple variables and many predicted values [49,
50, 52, 58, 61, 66, 68]. The technique is not sensitive to class noise or anomalies [69].
RF’s dependence on multiple decision trees provides a defence against misclassification
and overfitting which leads to higher accuracy [34, 49, 50, 52, 58, 61, 68]. The technique
is a strong and good practical model and can be applied to unnormalised data [58, 61,
65]. It can also be applied to an unbalanced dataset [65]. Another benefit of RF is the
ability to handle numerous input features [34, 58]. The technique can detect irregular
associations between features whether independent or dependent, which increases the
accuracy [58]. Feature normalisation is not required since results are calculated based on
distance [34]. RF has also been proven as an excellent technique to determine missing
values in data while preserving the soundness of the dataset [49, 58].
Weaknesses. RF makes use of many computations and requires plenty of computational
power [34, 52, 61, 68]. Training is time consuming due to the numerous decision trees
used in the process [34]. The classification precision is shaped by the dimension, size,
file type, and correctness of the dataset [58]. The technique struggles to interpret datasets
which influences its ability to decide on the importance of a feature [34, 61]. RF chooses
the data as well as the features for each tree at random [49, 50, 65].
4.2 Distance-Based Techniques
Distance-based techniques are also known as clustering techniques or models [10, 22,
41]. Clustering techniques aim to separate data into groups where the similarity within
a group is high and the similarity to other groups is lower [70–72]. These techniques are
known as unsupervised learning techniques and are used in scenarios where the data is not
filtered and unstructured [41, 70, 72, 73]. Unsupervised learning can handle complicated
jobs where data is not labelled [34]. Unsupervised learning aims to identify patterns
within a dataset by grouping data with similarities together [33]. These techniques help
detect class noise [33]. Overall, these techniques are complex and not appropriate for
use on large datasets [36, 70]. Distance measures used in machine learning techniques
include density-based spatial clustering of application with noise (DBSCAN), k-means,
and learning vector quantization (LVQ) [10, 20, 22].
Density Based Spatial Clustering of Application with Noise (DBSCAN)
Strengths. DBSCAN works effectively on datasets with noise and exceptions and determines results based on the density of observations in a cluster [70, 72, 74]. The technique,
as the name implies, is a spatial clustering technique and can handle different forms of
clusters, which makes it more beneficial than the K-Means Clustering Algorithm [34,
38, 70, 72, 75]. DBSCAN can form any number of clusters and is not dependent on the
user defining the number of clusters as is the case with k-means [34, 38, 70, 73, 75]. The
technique intends to determine the groups and noise in the dataset [36, 70, 73, 74]. The
138
C. van den Berg and S. Eybers
result of the technique identifies the border, core, and noise, which makes the technique
effective for the identification of class noise [38, 70, 73, 75]. DBSCAN can be combined
with other techniques to achieve improved performance and results [70]. The technique
can be implemented on large datasets and is not influenced by anomalies in the data [34,
74].
Weaknesses. DBSCAN does not perform effectively on datasets where clusters have
different densities of observations [34, 36, 70, 72, 74]. The technique has a high computational cost in large datasets [70, 74]. DBSCAN is reliant on two input variables (Eps
and MinPoints) which requires a thorough understanding of the data to ensure that effective clusters are formed [36, 70, 72, 75]. The variables greatly influence the outcome and
amending the variables is intricate [74]. It is also challenging to determine the correct k
value for a dataset [36, 75].
K-Means Clustering
Strengths. The K-means algorithm is widely used due to its simplicity and usefulness
[34, 39, 44, 74]. The technique is a cluster-based technique and is classified as an
unsupervised machine learning technique [33, 34, 40, 72, 74]. K-means can handle
large variables [40, 44]. The technique works best with unique datasets [39, 72]. It is
easy to apply, algorithmically efficient, and the results are easy to interpret [40, 73, 74].
The technique can handle qualitative and quantitative data and can be applied to large
datasets [34, 72, 73]. In addition, the technique is adjustable to missing values [73].
Weaknesses. The technique requires the number of groups to be specified [34, 72, 73]. It
is difficult to calculate the number of groups or k-value and the outcome is highly affected
by the first iteration of the technique [33, 34, 39, 40, 72, 74]. The technique is sensitive
to noise and anomalies [34, 40, 72–74]. The performance of the technique declines when
the number of characteristics is more than the records [40, 73]. The performance also
declines when the dimensions and concentration of the clusters are different [40, 72,
73].
Learning Vector Quantization (LVQ)
Strengths. LVQ is an easy and effective technique that is widely used [76]. The technique is also simple to interpret [76]. LVQ can be used in binary and multiclass classification scenarios [34]. Multiclass classification is ideal for unstructured data and provides
increased clustering efficiency [34, 76, 77]. The technique has the functionality to cluster
new data without having to redo training [77]. Common properties of the classification
item are trained automatically [76, 78]. LVQ classification results are achieved by comparing data to numerous prototype data structures [76, 78]. New data is classified by
deriving the class from the nearest prototype data structure [76]. LVQ is especially
valuable where pattern recognition is required.
Weaknesses. LVQ is resource intensive and has a decreased success rate on large datasets
[77]. The technique requires labelled data [34].
Investigating Machine Learning Techniques
139
4.3 Ensemble-Based Techniques
Ensemble-based techniques also referred to as ensemble learning are one of two
approaches that work better than conventional machine learning techniques [79–81].
These techniques combine several techniques into one structure to achieve a more effective model [42, 79–84]. Ensemble-based techniques can be homogenous or heterogenous
[85]. Homogenous refers to techniques from the same group and heterogenous refers
to techniques from different groups [85]. Each of the techniques employed works independently [79, 80, 86]. The benefit of ensemble-based techniques is determined by how
the techniques employed are trained and how they are joined [79]. Predictions are made
based on the output of each of the techniques employed in the ensemble [42, 80, 86].
Ensemble-based techniques can overcome overfitting by combining several techniques
[79, 82]. The output of the ensemble becomes one model [42, 80]. It can be used in
various applications and works better than employing a single machine learning technique [42, 79, 80, 82]. Popular ensemble-based techniques are bagging, boosting, and
stacking [42, 69, 79–82, 86]. These techniques aim to enhance predictive ability [42,
79, 80, 86]. The success of an ensemble method is measured by looking at prediction
accuracy, resource usage, and understandability [79]. The primary cause for variations
in predictions when using a single machine learning technique is class noise, imbalance,
and bias [80]. Ensemble techniques can overcome this and provide higher accuracy by
employing multiple machine learning techniques [69, 80, 82, 86]. These techniques are
more robust to class noise [82]. Ensemble techniques are resource-intensive and require
good data for learning [82, 86].
Bagging
Strengths. Bagging is also referred to as bootstrap aggregating [79, 80, 82, 85–88].
The technique is data-centric and creates small subdivisions of the same size within the
dataset which creates various predictive models [42, 69, 79–82, 85, 86]. Bagging uses
several decision trees in parallel and trains each decision tree with a different subsection
of the training dataset [69, 80, 81, 86, 87, 89]. Each technique works independently [80,
85, 88]. The subsections for the decision trees are randomly selected [42, 80, 81, 83,
85, 87–89]. The different decision trees strengthen recognition capacities in that a set
incorrectly classified in one tree can be correctly classified in another tree [79, 89]. The
use of several decision trees inhibits overfitting, which is a weakness of making use of
a single decision tree [79, 80, 85, 89]. The ultimate prediction is then determined using
the outcome of most of the decision trees [69, 79, 81, 82, 86–89]. It can be used for
classification or regression [69, 79, 81, 83]. Bagging is a strong and scalable technique
with improved efficiency by decreasing the differences between predictions [69, 81–83].
It is an easy and effective technique [81, 83]. Bagging can handle large datasets [79].
The technique is reported to be more fitting for small datasets [81, 83].
Weaknesses. Bagging can only employ multiple models from the same machine learning
technique, it is called a homogeneous technique [42, 69, 80]. The technique is resource
intensive and prone to bias [79, 85]. The technique also causes the model to lose its
understandability [79, 80, 82]. When the model is intricate or when subdivisions are
highly correlated it can cause overfitting [82]. The difficulty of bagging is to determine
the ideal amount of base machine learning techniques to employ, the subdivisions of the
140
C. van den Berg and S. Eybers
dataset, and the number of samples required within a subdivision [79]. With bagging it
is also not known which attributes are used during the random selection process [80].
Bagging is resource-intensive with complex models and large datasets [82, 83]. The
technique is reported to be an ineffective learner [83].
Boosting
Strengths. Boosting can be used for classification or regression [79, 80, 82, 85]. It makes
use of weighted averages to produce the results from all the repetitions [69, 80–83, 85,
87]. The technique processes the base machine learning techniques in chronological
order on the same random datasets [42, 79–81, 83, 86, 88]. The aim of the subsequent
model is to pick up and fix any errors produced by the previous [42, 69, 79–81, 83, 85,
88]. The wrongly classified samples of each iteration get the highest weighting [82, 87].
Boosting is easy to understand, and it decreases prejudice [79–82]. The technique aims
to improve precision and efficiency [81]. It is strong and decreases overfitting [80–82].
Boosting is reported to have the ability to handle data restrictions [83].
Weaknesses. Boosting can only employ multiple models from the same machine learning technique, it is called a homogeneous technique [42, 69, 83]. Due to chronological
processing, a data dependency exists [79]. It also causes issues with normalising chronological learning [79, 80]. Boosting is resource intensive and prone to overfitting when
the number of repetitions increases [79, 82]. The technique learns slower than bagging
[79, 83]. A large number of variables influence the performance of the model [79]. The
technique is sensitive to anomalies and class noise [80, 82].
Stacking
Strengths. Stacking is also referred to as Stacked Generalisation [69, 79]. It is a flexible
technique as it can use any machine learning technique as a base [82, 86]. Stacking is
called a heterogeneous technique since it can use different machine learning techniques
together in the ensemble [69, 80, 88]. The technique joins the results from various
machine learning models to create a new model [42, 69, 79–81, 83, 85–88]. This new
model is referred to as a meta-model [42, 79, 80, 83, 85]. Different machine learning
techniques are applied to the same datasets [80, 83, 86]. The aim is to join the best
predictions into the meta-model [69, 79–82]. Stacking utilises a different model to decide
how output should be joined effectively [69, 80, 83, 85, 87]. The technique delivers better
results than any other learning model [69, 79, 80, 83, 86, 88]. The technique forms a
better understanding of the datasets which makes stacking more accurate and efficient
[79–81, 83]. Stacking can be used for classification or regression [80, 82]. The technique
is robust to class noise and anomalies [82]. Other advantages are improved efficiency
and resilience [82, 83]. Stacking can make use of the advantages of different techniques
to minimise drawbacks [82].
Weaknesses. Stacking is prone to overfitting because predictions are made on the same
dataset and the predictions of the different techniques are then joined [79, 82]. Stacking
is expensive because of the amount of data needed for learning [79, 82]. In addition,
the technique is also time intensive [69, 79, 86]. The understanding of the final model
is difficult since it is the combination of predictions from multiple techniques [79, 82].
Investigating Machine Learning Techniques
141
The ideal amount of base machine learning techniques needed to achieve more accurate
predictions is difficult to determine [79, 83]. The number of machine learning techniques
employed influences efficiency [69]. Stacking has a high computational cost for large
data sets [80].
Table 2 presents a taxonomy of machine learning techniques that were identified and
described, namely classifier-based, distance-based, and ensemble-based. Furthermore,
the machine learning techniques for each category were identified.
Table 2. Proposed taxonomy of machine learning categories and techniques
Machine learning category
Machine learning technique
Classifier-based
Decision tree, K-nearest neighbours, Naïve Bayes, Random
Forest, and Support Vector Machines
Distance-based
Density Based Clustering of Application with Noise, K-Means
Clustering, and Learning Vector Quantization
Ensemble-based
Bagging, Boosting, and Stacking
5 Conclusion
Class noise damages the accuracy of models and insights. The presence of class noise
also complicates the resulting model of machine learning techniques. Machine learning
techniques can be employed to identify and detect class noise. The aim is to remove the
class noise identified to improve the quality of the data used as input to derive insights
or make predictions.
It is important to choose the appropriate machine learning technique to achieve optimal efficiency and results. In this study, three categories of machine learning techniques
were identified, namely classifier-based, distance-based, and ensemble-based, with a
further breakdown of the techniques for each category. Understanding the strengths and
weaknesses of machine learning techniques assists in choosing the optimal technique
for the specific dataset and therefore included in the study.
This study is limited to the noise detection techniques available for class noise. The
detection of attribute noise is not discussed. Noise handling techniques for data noise
are also not be discussed in this study.
Further research can further add value by considering, identify and discussing the
strengths and weaknesses of the machine learning technique that can be classified on
algorithm level of the taxonomy for class noise treatment.
References
1. Bratasanu, V.E.: Leadership decision-making processes in the context of data driven tools.
Calitatea. 19(S3), 77–87 (2018)
142
C. van den Berg and S. Eybers
2. Marcinkowski, B., Bartlomiej, G.: Data-driven business model development insights from
the facility management industry. J. Facilit. Manag. 19(2), 129–149 (2021)
3. McAfee, A., Brynjolfsson, E.: Big data: the management revolution. Harv. Bus. Rev.. Bus.
Rev. 90(10), 60–66 (2012)
4. Redman, T.C.: Data Driven: Profiting From Your Most Important Business Asset. Harvard
Business Press, Boston (2008)
5. Scarisbrick-Hauser, A., Rouse, C.: The whole truth and nothing but the truth? the role of data
quality today. Direct Mark. Int. J. 1(3), 161–171 (2007). https://doi.org/10.1108/175059307
10779333
6. Sluban, B., Gamberger, D., Lavrač, N.: Ensemble-based noise detection: noise ranking and
visual performance evaluation. Data Min. Knowl. Disc.Knowl. Disc. 28(2), 265–303 (2014).
https://doi.org/10.1007/s10618-012-0299-1
7. Garcia, L.P.F., Lorena, A.C., Matwin, S., de Carvalho, A.C.P.L.F.: Ensembles of label noise
filters: a ranking approach. Data Min. Knowl. Disc.Knowl. Disc. 30(5), 1192–1216 (2016).
https://doi.org/10.1007/s10618-016-0475-9
8. Li, M., Huang, C., Wang, D., Hu, Q., Zhu, J., Tang, Y.: Improved randomized learning
algorithms for imbalanced and noisy educational data classification. Computing 101(6), 571–
585 (2019). https://doi.org/10.1007/s00607-018-00698-w
9. Nettleton, D.F., Orriols-Puig, A., Fornells, A.: A study of the effect of different types of noise
on the precision of supervised learning techniques. Artif. Intell. Rev. Int. Sci. Eng. J. 33(4),
275–306 (2010). https://doi.org/10.1007/s10462-010-9156-z
10. Qi, Z., Wang, H., Wang, A.: Impacts of dirty data on classification and clustering models:
an experimental evaluation. J. Comput. Sci. Technol.Comput. Sci. Technol. 36(4), 806–821
(2021). https://doi.org/10.1007/s11390-021-1344-6
11. Gupta, S., Gupta, A.: Dealing with noise problem in machine learning data-sets: a systematic
review. Procedia Comput. Sci. 161, 466–474 (2019). https://doi.org/10.1016/j.procs.2019.
11.146
12. Sluban, B., Gamberger, D., Lavrač, N.: Advances in class noise detection. Proceedings (2010)
13. Sáez, J.A., Galar, M., Luengo, J., Herrera, F.: Analyzing the presence of noise in multi-class
problems: alleviating its influence with the One-vs-One decomposition. Knowl. Inf. Syst..
Inf. Syst. 38(1), 179–206 (2014). https://doi.org/10.1007/s10115-012-0570-1
14. Frenay, B., Verleysen, M.: Classification in the presence of label noise: a survey. IEEE Trans.
Neural Netw. Learn. Syst. 25(5), 845–869 (2014). https://doi.org/10.1109/TNNLS.2013.229
2894
15. Khoshgoftaar, T.M., Hulse, J.V., Napolitano, A.: Supervised neural network modeling: an
empirical investigation into learning from imbalanced data with labeling errors. IEEE Trans.
Neural Netw.Netw. 21(5), 813–830 (2010). https://doi.org/10.1109/TNN.2010.2042730
16. Segata, N., Blanzieri, E., Delany, S.J., Cunningham, P.: Noise reduction for instance-based
learning with a local maximal margin approach. J. Intell. Inf. Syst.Intell. Inf. Syst. 35(2),
301–331 (2010). https://doi.org/10.1007/s10844-009-0101-z
17. Guan, D., Yuan, W., Lee, Y., Lee, S.: Identifying mislabeled training data with the aid of
unlabeled data. Appl. Intell.Intell. 35(3), 345–358 (2011). https://doi.org/10.1007/s10489010-0225-4
18. Liu, J., Ren, Z., Lu, R., Luo, X.: GMM discriminant analysis with noisy label for each class.
Neural Comput. Appl.Comput. Appl. 33(4), 1171–1191 (2021). https://doi.org/10.1007/s00
521-020-05038-8
19. Ougiaroglou, S., Evangelidis, G.: Efficient editing and data abstraction by finding homogeneous clusters. Ann. Math. Artif. Intell.Artif. Intell. 76(3), 327–349 (2016). https://doi.org/
10.1007/s10472-015-9472-8
Investigating Machine Learning Techniques
143
20. Xiao, J., Tian, Y., Xie, L., Jiang, X., Huang, J.: A hybrid classification framework based on
clustering. IEEE Trans. Ind. Inf. 16(4), 2177–2188 (2020). https://doi.org/10.1109/TII.2019.
2933675
21. Prati, R.C., Luengo, J., Herrera, F.: Emerging topics and challenges of learning from noisy
data in nonstandard classification: a survey beyond binary class noise. Knowl. Inf. Syst.. Inf.
Syst. 60(1), 63–97 (2019). https://doi.org/10.1007/s10115-018-1244-4
22. Xia, S., Wang, G., Chen, Z., Duan, Y., Liu, Q.: Complete random forest based class noise
filtering learning for improving the generalizability of classifiers. IEEE Trans. Knowl. Data
Eng.Knowl. Data Eng. 31(11), 2063–2078 (2019). https://doi.org/10.1109/TKDE.2018.287
3791
23. Salekshahrezaee, Z., Leevy, J.L., Khoshgoftaar, T.M.: A reconstruction error-based framework for label noise detection. J. Big Data 8(1), 57 (2021). https://doi.org/10.1186/s40537021-00447-5
24. Barreto, G.A., Barros, A.L.B.P.: On the design of robust linear pattern classifiers based on
M-estimators. Neural. Process. Lett. 42(1), 119–137 (2015). https://doi.org/10.1007/s11063014-9393-2
25. Sluban, B., Gamberger, D., Lavrač, N.: Performance analysis of class noise detection
algorithms. In: Stairs 2010 (2011)
26. Smith, M.R., Martinez, T.: The robustness of majority voting compared to filtering misclassified instances in supervised classification tasks. Artif. Intell. Rev.. Intell. Rev. 49(1), 105–130
(2018). https://doi.org/10.1007/s10462-016-9518-2
27. Shehab, N., Badawy, M., Ali, H.A.: Toward feature selection in big data preprocessing
based on hybrid cloud-based model. J. Supercomput.Supercomput. 78(3), 3226–3265 (2022).
https://doi.org/10.1007/s11227-021-03970-7
28. Huy, T.N., Tong, B., Shao, H., Suzuki, E.: Transfer learning by centroid pivoted mapping in
noisy environment. J. Intell. Inf. Syst.Intell. Inf. Syst. 41(1), 39–60 (2013). https://doi.org/
10.1007/s10844-012-0226-3
29. Xia, S., Meng, F., Liu, B., Zhou, Y.: A kernel clustering-based possibilistic fuzzy extreme
learning machine for class imbalance learning. Cogn. Comput.. Comput. 7(1), 74–85 (2015).
https://doi.org/10.1007/s12559-014-9256-1
30. Mohamed Shaffril, H.A., Samsuddin, S.F., Abu Samah, A.: The ABC of systematic literature
review: the basic methodological guidance for beginners. Qual. Quant. 55(4), 1319–1346
(2020). https://doi.org/10.1007/s11135-020-01059-6
31. Okoli, C.: A guide to conducting a standalone systematic literature review. Commun. Assoc.
Inf. Syst. 37 (2015). https://doi.org/10.17705/1CAIS.03743
32. Zhou, Y., Zhang, H., Huang, X., Yang, S., Tang, H., Babar, M.A.: Quality assessment of systematic reviews in software engineering: a tertiary study. In: ACM International Conference
Proceeding Series, 27–29 April 2015 (2015). https://doi.org/10.1145/2745802.2745815
33. Aliramezani, M., Koch, C.R., Shahbakhti, M.: Modeling, diagnostics, optimization, and control of internal combustion engines via modern machine learning techniques: a review and
future directions. Prog. Energy Combust. Sci.. Energy Combust. Sci. 88, 100967 (2022).
https://doi.org/10.1016/j.pecs.2021.100967
34. Papakyriakou, D., Barbounakis, I.S.: Data mining methods: a review. Int. J. Comput.
Appl.Comput. Appl. 183(48), 5–19 (2022). https://doi.org/10.5120/ijca2022921884
35. Triguero, I., García-Gil, D., Maillo, J., Luengo, J., García, S., Herrera, F.: Transforming big
data into smart data: an insight on the use of the k-nearest neighbors algorithm to obtain
quality data. Wiley Interdisc. Rev. Data Min. Knowl. Disc. 9(2), 1289 (2019). https://doi.org/
10.1002/widm.1289
36. Smiti, A.: A critical overview of outlier detection methods. Comput. Sci. Rev. 38, 100306
(2020). https://doi.org/10.1016/j.cosrev.2020.100306
144
C. van den Berg and S. Eybers
37. Chen, H., Hu, S., Hua, R., Zhao, X.: Improved naive Bayes classification algorithm for traffic
risk management. EURASIP J. Adv. Signal Process. 2021(1), 30 (2021). https://doi.org/10.
1186/s13634-021-00742-6
38. Ravi, J., Kulkarni, S.: A critical review on density-based clustering algorithms and their
performance in data mining. Int. J. Res. Anal. Rev. (IJRAR). 9(1), 73–82 (2022)
39. Chippalakatti, S., Renumadhavi, C.H., Pallavi, A.: Comparative review on the machine learning algorithms for medical data. In: 2022 6th International Conference on Computation System
and Information Technology for Sustainable Solutions (CSITSS), pp. 1–6. IEEE (2022)
40. Ray, S.: A quick review of machine learning algorithms. In: 2019 International Conference
on Machine Learning, Big Data, Cloud and Parallel Computing (COMITCon), pp. 35–39.
IEEE (2019)
41. Nalini Durga, S., Usha Rani, K.: A perspective overview on machine learning algorithms. In:
Jyothi, S., Mamatha, D.M., Satapathy, S.C., Srujan Raju, K., Favorskaya, M.N. (eds.) CBE
2019. LAIS, vol. 15, pp. 353–364. Springer, Cham (2020). https://doi.org/10.1007/978-3030-46939-9_30
42. Tanveer, M., Rastogi, A., Paliwal, V., Ganaie, M.A., Malik, A.K., Del Ser, J., et al.: Ensemble
deep learning in speech signal tasks: a review. Neurocomputing 550, 126436 (2023). https://
doi.org/10.1016/j.neucom.2023.126436
43. Mindong, L., Qingwei, C., Panling, H., Jun, Z., Weike, G.: A review of classification algorithms for data mining. In: 2019 2nd International Conference on Mechanical, Electronic and
Engineering Technology (MEET 2019) (2019)
44. Abdulrahman, S.A., Khalifa, W., Roushdy, M., Salem, A.B.M.: Comparative study for 8
computational intelligence algorithms for human identification. Comput. Sci. Rev. 36, 100237
(2020). https://doi.org/10.1016/j.cosrev.2020.100237
45. Bansal, M., Goyal, A., Choudhary, A.: A comparative analysis of K-nearest neighbour,
genetic, support vector machine, decision tree, and long short term memory algorithms in
machine learning. Decis. Anal. J. 3, 100071 (2022)
46. Cervantes, J., Garcia-Lamont, F., Rodríguez-Mazahua, L., Lopez, A.: A comprehensive survey on support vector machine classification: applications, challenges and trends.
Neurocomputing 408, 189–215 (2020). https://doi.org/10.1016/j.neucom.2019.10.118
47. Pineda-Jaramillo, J.D.: A review of Machine Learning (ML) algorithms used for modeling
travel mode choice. DYNA 86(211), 32–41 (2019). https://doi.org/10.15446/dyna.v86n211.
79743
48. Fletcher, S., Islam, M.Z.: Decision tree classification with differential privacy: a survey. ACM
Comput. Surv. (CSUR) 52(4), 1–33 (2019). https://doi.org/10.1145/3337064
49. Mohd, T., Jamil, N.S., Johari, N., Abdullah, L., Masrom, S.: An overview of real estate
modelling techniques for house price prediction. In: Kaur, N., Ahmad, M. (eds.) Charting
a Sustainable Future of ASEAN in Business and Social Sciences: Proceedings of the 3rd
International Conference on the Future of ASEAN (ICoFA) 2019—Volume 1, pp. 321–338.
Springer, Singapore (2020). https://doi.org/10.1007/978-981-15-3859-9_28
50. Zhang, C., Jia, D., Wang, L., Wang, W., Liu, F., Yang, A.: Comparative research on network
intrusion detection methods based on machine learning. Comput. Secur.. Secur. 121, 102861
(2022). https://doi.org/10.1016/j.cose.2022.102861
51. Zhou, H., Zhang, J., Zhou, Y., Guo, X., Ma, Y.: A feature selection algorithm of decision tree
based on feature weight. Expert Syst. Appl. 164, 113842 (2021). https://doi.org/10.1016/j.
eswa.2020.113842
52. Speiser, J.L., Miller, M.E., Tooze, J., Ip, E.: A comparison of random forest variable selection
methods for classification prediction modeling. Expert Syst. Appl. 134, 93–101 (2019). https://
doi.org/10.1016/j.eswa.2019.05.028
Investigating Machine Learning Techniques
145
53. Aboalnaser, S.A.: Machine learning algorithms in arabic text classification: a review. In: 2019
12th International Conference on Developments in eSystems Engineering (DeSE), pp. 290–
295. IEEE (2019)
54. Boyko, N., Boksho, K.: Application of the naive bayesian classifier in work on sentimental
analysis of medical data. In: CEUR Workshop Proceedings, p. 230–239 (2020)
55. Taunk, K., De, S., Verma, S., Swetapadma, A.: A brief review of nearest neighbor algorithm
for learning and classification. In: 2019 International Conference on Intelligent Computing
and Control Systems (ICCS), pp. 1255–1260. IEEE (2019)
56. Bo, S., Haiyan, C.: A survey of k nearest neighbor algorithms for solving the class imbalanced
problem. Wirel. Commun. Mob. Comput.. Commun. Mob. Comput. (2021). https://doi.org/
10.1155/2021/5520990
57. Cunningham, P., Delany, S.J.: K-nearest neighbour classifiers - a tutorial. ACM Comput. Surv.
(CSUR). 54(6), 1–25 (2021)
58. Avand, M., Janizadeh, S., Naghibi, S.A., Pourghasemi, H.R., Bozchaloei, S.K., Blaschke, T.:
A comparative assessment of Random Forest and k-Nearest Neighbor classifiers for gully
erosion susceptibility mapping. Water (Switzerland). 11(10), 2076 (2019). https://doi.org/10.
3390/w11102076
59. Al-Aidaroos, K.M., Bakar, A.A., Othman, Z.: Naïve bayes variants in classification learning. In: 2010 International Conference on Information Retrieval & Knowledge Management
(CAMP), pp. 276–281 (2010)
60. Gao, H., Zeng, X., Yao, C.: Application of improved distributed naive Bayesian algorithms in
text classification. J. Supercomput. Int. J. High-Perf. Comput. Des. Anal. Use 75(9), 5831–
5847 (2019). https://doi.org/10.1007/s11227-019-02862-1
61. Gupta, A., Lohani, M.C., Manchanda, M.: Financial fraud detection using naive bayes algorithm in highly imbalance data set. J. Disc. Math. Sci. Cryptogr. 24(5), 1559–1572 (2021).
https://doi.org/10.1080/09720529.2021.1969733
62. Singla, M., Shukla, K.K.: Robust statistics-based support vector machine and its variants: a
survey. Neural Comput. Appl.Comput. Appl. 32(15), 11173–11194 (2019). https://doi.org/
10.1007/s00521-019-04627-6
63. Tanveer, M., Rajani, T., Rastogi, R., Shao, Y.H., Ganaie, M.A.: Comprehensive review on
twin support vector machines. Ann. Oper. Res. 1–46 (2022)
64. Tavara, S.: Parallel computing of support vector machines a survey. ACM Comput. Surv.
(CSUR) 51(6), 1–38 (2019). https://doi.org/10.1145/3280989
65. Zhu, L., Zhou, X., Liu, W., Kong, Z.: Total organic carbon content logging prediction based
on machine learning: a brief review. Energy Geosci. 4, 1–8 (2022)
66. Zheng, L., Li, J., Qu, X., Li, F.: A review of the application of machine learning technologies
in vehicle navigation and positioning. In: Proceedings of SPIE, the International Society for
Optical Engineering, p. 10. SPIE (2022)
67. Kok, Z.H., Mohamed Shariff, A.R., Alfatni, M.S.M., Khairunniza-Bejo, S.: Support vector
machine in precision agriculture: a review. Comput. Electron. Agric.e. 191, 106546 (2021).
https://doi.org/10.1016/j.compag.2021.106546
68. Ahmad Sobri, M.Z., Redhwan, A., Ameen, F., Lim, J.W., Liew, C.S., Mong, G.R., et al.: A
review unveiling various machine learning algorithms adopted for biohydrogen productions
from microalgae. Fermentation. 9(3), 243 (2023). https://doi.org/10.3390/fermentation903
0243
69. Zhang, Y., Liu, J., Shen, W.: A review of ensemble learning algorithms used in remote sensing
applications. Appl. Sci. 12(17), 8654 (2022). https://doi.org/10.3390/app12178654
70. Singh, H.V., Girdhar, A., Dahiya, S.: A Literature survey based on DBSCAN algorithms. In:
2022 6th International Conference on Intelligent Computing and Control Systems (ICICCS),
pp. 751–758. IEEE (2022)
146
C. van den Berg and S. Eybers
71. Jollyta D, Efendi S, Zarlis M, Mawengkang H. Analysis of an optimal cluster approach: a
review paper. Journal of Physics: Conference Series. 2023;2421(1). https://doi.org/10.1088/
1742-6596/2421/1/012015
72. Kangane, P., Joshi, V., Kacker, A., Jain, M.: A comprehensive survey of various clustering
paradigms. Int. J. Eng. Appl. Sci. Technol. 5(11), 1–7 (2021)
73. Daghigh, H., Tannant, D.D., Daghigh, V., Lichti, D.D., Lindenbergh, R.: A critical review
of discontinuity plane extraction from 3D point cloud data of rock mass surfaces. Comput.
Geosci.. Geosci. 169, 105241 (2022). https://doi.org/10.1016/j.cageo.2022.105241
74. Zhang, M.: Unsupervised learning algorithms in big data: an overview. In: Proceedings of the
2022 5th International Conference on Humanities Education and Social Sciences (ICHESS
2022), pp. 910–931. Atlantis Press SARL, Paris (2022)
75. Chandrasekar, R.S., Britto, G.: Comprehensive review on density-based clustering algorithm
in data mining. Int. J. Res. Anal. Rev. 6(2), 5–9 (2019)
76. Tang, F., Fan, M., Tino, P.: Generalized learning riemannian space quantization: a case study
on riemannian manifold of SPD matrices. IEEE Trans. Neural Netw. Learn. Syst. 32(1),
281–292 (2021). https://doi.org/10.1109/TNNLS.2020.2978514
77. Bilen, T., Canberk, B., Sharma, V., Fahim, M., Duong, T.Q.: AI-driven aeronautical ad hoc
networks for 6G wireless: challenges, opportunities, and the road ahead. Sensors 22(10), 3731
(2022). https://doi.org/10.3390/s22103731
78. Liu, C., Liu, Y.: Energy management strategy for plug-in hybrid electric vehicles based on
driving condition recognition: a review. Electronics 11(3), 342 (2022)
79. Mohammed, A., Kora, R.: A comprehensive review on ensemble deep learning: opportunities
and challenges. J. King Saud Univ. Comput. Inf. Sci. 35(2), 757–774 (2023). https://doi.org/
10.1016/j.jksuci.2023.01.014
80. Mahajan, P., Uddin, S., Hajati, F., Moni, M.A.: Ensemble learning for disease prediction: a
review. Healthcare (Basel, Switzerland). 11(12), 1808 (2023). https://doi.org/10.3390/health
care11121808
81. Zounemat-Kermani, M., Batelaan, O., Fadaee, M., Hinkelmann, R.: Ensemble machine learning paradigms in hydrology: a review. J. Hydrol.Hydrol. 598, 126266 (2021). https://doi.org/
10.1016/j.jhydrol.2021.126266
82. Aman, R.S.C., Chhillar, U.: Disease prediction in healthcare: an ensemble learning perspective. In: National Conference on Computational Intelligence and Data Science (2023)
83. Fadli, V.F., Soesanti, I., Nugroho, H.A.: Performance review of ensemble learning method use
in COVID-19 case detection. In: 2022 IEEE International Conference of Computer Science
and Information Technology (ICOSNIKOM), pp. 1–7. IEEE (2022)
84. Sarajcev, P., Kunac, A., Petrovic, G., Despalatovic, M.: Artificial intelligence techniques for
power system transient stability assessment. Energies 15(2), 507 (2022)
85. Ganaie, M.A., Hu, M., Malik, A.K., Tanveer, M., Suganthan, P.N.: Ensemble deep learning:
a review. Eng. Appl. Artif. Intell.Artif. Intell. 115, 105151 (2022). https://doi.org/10.1016/j.
engappai.2022.105151
86. Kumar, S., Kaur, P., Gosain, A.: A comprehensive survey on ensemble methods. In: 2022
IEEE 7th International conference for Convergence in Technology (I2CT), pp. 1–7. IEEE
(2022)
87. Lasantha, D., Vidanagamachchi, S., Nallaperuma, S.: Deep learning and ensemble deep learning for circRNA-RBP interaction prediction in the last decade: a review. Eng. Appl. Artif.
Intell.Artif. Intell. 123, 106352 (2023). https://doi.org/10.1016/j.engappai.2023.106352
Investigating Machine Learning Techniques
147
88. Sharma, T., Jatain, A., Bhaskar, S., Pabreja, K.: Ensemble machine learning paradigms in
software defect prediction. Procedia Comput. Sci. 218, 199–209 (2023). https://doi.org/10.
1016/j.procs.2023.01.002
89. Gupta, N., Jindal, V., Bedi, P.: LIO-IDS: handling class imbalance using LSTM and improved
one-vs-one technique in intrusion detection system. Comput. Netw.. Netw. 192, 108076
(2021). https://doi.org/10.1016/j.comnet.2021.108076
The Role of Chatbots in Data Analytics:
An Evaluation of Functional Abilities
Preeti Patel and Sowgol Shooshtarian(B)
London Metropolitan University, London N7 8DB, UK
p.patel@londonmet.ac.uk, sos0651@my.londonmet.ac.uk
Abstract. In the fast-paced evolution of artificial intelligence, in which chatbots
are becoming vital tools in many areas, this study is focused on the various functional roles of ChatGPT-4 in data analytics. The primary objective is to assess the
technological advancements and capabilities of ChatGPT-4 in analyzing data. The
research methodology employs three distinct datasets and the analysis compares
various prompt engineering techniques with ChatGPT-4 against human-driven
analysis. This comparison is grounded in profile optimization techniques, the
CRISP-DM framework, and a structured Data Science Project Plan. The findings
reveal the diverse capabilities of ChatGPT-4. On one hand, basic prompts yield
quick, initial insights, positioning ChatGPT-4 as a valuable asset for those with
limited technical expertise. Conversely, refined and engineered prompts enable
ChatGPT-4 to deliver a deeper, more detailed analysis. Yet, when contrasted with
traditional human-driven analysis, typically utilizing tried and tested techniques,
tools and experience, all combined with a deep knowledge of machine learning
and data analysis; the human approach remains unsurpassed in depth, granularity, and accuracy. Conclusively, while ChatGPT-4 stands out for preliminary data
explorations and general insights, complicated, nuanced analytical tasks are best
tackled using time-tested, traditional data science methodologies and tools. While
ChatGPT-4 is versatile, it has not yet matched the expertise of seasoned data
analysts.
Keywords: Chatbot · ChatGPT-4 · Data analytics · Generative AI
1 Introduction
OpenAI’s text-generating chatbot, ChatGPT-4 is a conversational AI interface that uses
natural language processing and machine learning algorithms to generate high-quality
responses to a wide range of queries. In the rapidly evolving landscape of data science,
chatbots have emerged as a significant player, revolutionizing the way we interact with
data and derive insights. The evolution and utilization of chatbots, particularly ChatGPT, have permeated various sectors, marking a significant shift in how interactions and
data analyses are carried out. ChatGPT is an advanced Large Language Model (LLM)
that uses deep learning techniques to produce human-like responses to natural language
inputs. It is part of the family of generative pre-training transformer (GPT) models
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2024
K. Arai (Ed.): SAI 2024, LNNS 1017, pp. 148–168, 2024.
https://doi.org/10.1007/978-3-031-62277-9_10
The Role of Chatbots in Data Analytics: An Evaluation of Functional Abilities
149
developed by OpenAI and is currently one of the largest publicly available LLM. ChatGPT can capture the nuances and intricacies of human language, allowing it to generate
appropriate and contextually relevant responses across a broad spectrum of prompts [5].
The emergence and integration of chatbots in data analytics present both opportunities and challenges. They offer potential for enhanced user interaction, democratization
of data analysis, and increased efficiency. However, concerns about their accuracy, depth
of analysis, and suitability for complex tasks remain. This research aims to address these
issues and evaluate the functional abilities of chatbots, with a specific focus on their role
in data analytics.
This study is motivated by the need to understand the extent of the lack of comprehensive studies on the accuracy and performance of advanced chatbots such as ChatGPT-4
in processing and analyzing data in comparison to traditional data analysis methods. The
study involves the design and implementation of multiple experiments with ChatGPT-4
on three unique datasets, selected for their distinct nature and significance in data science issues. The research problem focuses on evaluating the accuracy and performance
of ChatGPT-4 in handling data analysis processes, such as its ability in automating data
preparation, analysis, and interpretation tasks, and how these capabilities align with the
current and future needs of data analytics.
This paper is structured to first present an overview of the current state of chatbots in
data analytics, followed by a detailed examination of ChatGPT-4’s abilities. The methodology section describes the experimental approach taken and the multiple experiments
conducted with ChatGPT-4 on diverse datasets. The results and discussion will analyze
and compare the findings, and the conclusion will summarize the study’s implications
and potential future directions for chatbot technology in data analytics. In doing so, this
paper not only fills a critical gap in existing research but also charts a path forward for
the effective utilization of ChatGPT-4 in data analytics, potentially revolutionizing how
we approach data-driven decision-making.
2 Related Work
An extensive overview [1] of the technology and applications of chatbots emphasizes
the historical development of chatbots, tracing their evolution from rule-based systems
to sophisticated AI-driven entities. AI-powered chatbots have adopted technologies such
as Natural Language Processing (NLP), Machine Learning (ML), and Artificial Intelligence (AI), facilitating their application in various domains, including data analytics.
In the realm of data analytics, chatbots now function as tools for data querying, visualization, and exploration, facilitating conversational interaction with data. They provide
an accessible platform for users with limited technical expertise, enabling them to analyze data through intuitive natural language queries. This democratizes data analysis,
making it more accessible and user-friendly. Nonetheless, there remain challenges to the
widespread adoption of chatbots in data analytics. These include issues of data privacy
and security, as well as the need for more sophisticated contextual understanding and
more human-like interaction capabilities [1, 10, 11].
Past research has addressed the significant challenge of evaluating chatbot performance in a quantitative manner [9] in which the application of data analytics to generate
150
P. Patel and S. Shooshtarian
measurable and comparable metrics, thereby fostering an objective evaluation of a chatbot’s functional abilities in projects and assessments. In this analysis, key performance
indicators (KPIs) are identified such as response time, accuracy of response, and user
satisfaction, demonstrating the extensive potential of data analytics to enhance the understanding of chatbot performance. The work further highlights the transformative role of
chatbots in data analytics, echoing other works in emphasizing the growing importance
of chatbots in facilitating user-friendly data interactions. Jongerius in [9] emphasizes
the potential of chatbots to democratize data analysis, in line with findings from [7].
However, Jongerius also underscores potential challenges associated with quantifying
chatbot performance, such as data privacy and security concerns, further extending the
ongoing discourse on the limitations of chatbot applications in data analytics. Khlaif
et al. also address concerns regarding ChatGPT’s accuracy and reliability. They note
instances where the chatbot may provide responses that lack depth or are not entirely
accurate, underscoring the importance of human oversight in data analytics processes
[11].
Others have delved into the transformative role of ChatGPT in the data science field
[7], emphasizing the significant revolution brought about by AI-assisted conversational
interfaces. The authors highlight how ChatGPT can assist data scientists in automating
various aspects of their workflow, including data cleaning and preprocessing, model
training, and result interpretation. Alshater’s paper notes that the ChatGPT’s ability to
process and simplify complex datasets can be helpful in teaching and learning data analytics, thereby improving overall academic performance [2]. It also can provide new
insights and improve decision-making processes by analyzing unstructured data. However, there are also limitations and issues associated with using ChatGPT. For example,
while it can save time and resources compared to training a model from scratch, it may
not perform well on certain tasks if it has not been specifically trained for them and lead
to biased or not concise results. Additionally, the output of ChatGPT may be difficult to
interpret, which could pose challenges for decision-making in data science applications.
The benefits of using ChatGPT outweigh the costs and can enhance the productivity
and accuracy of data science workflows. There is also a suggestion that ChatGPT has
become an increasingly crucial tool for intelligence augmentation in data science [7].
There appears to be considerable value in using ChatGPT in healthcare, particularly
in relation to improving accessibility to medical information and services [5]. The use
of chatbots enhances overall patient satisfaction through instant responses and service
availability, thus offering a promising outlook for chatbot integration in healthcare. It is
also argued that ChatGPT has made significant strides in facilitating efficient interaction
with complex medical datasets, thereby improving the ease and accessibility of data analysis. By transforming data interpretation into a more conversational format, ChatGPT
offers the potential to democratize data analytics across various professional spheres, not
just medicine. The study by Kalla et al. underscores the versatility of ChatGPT in various
fields, including but not limited to healthcare, cyber security, and business. They note
that the chatbot’s ability to provide quick, informed responses can help to automate some
aspects of different fields [10]. However, the authors also identify notable limitations,
specifically, the potential for inaccuracies and biases in complex data interpretation.
They point out that while ChatGPT has made substantial advancements in the handling
The Role of Chatbots in Data Analytics: An Evaluation of Functional Abilities
151
of natural language queries, there is room for improvement in the context of intricate
data analysis [5, 10]. Moreover, Nah et al. notes that ChatGPT’s output could potentially
exhibit reflections of any inaccuracies, disproportionate representation of information
sources, or inherent biases present within the training dataset [14].
Chatbots have been leveraged in educational contexts, enabling students and educators to engage with complex data in a more accessible, conversational manner [15]. They
illustrate the functional abilities of chatbots in managing and interpreting educational
data in interactive manner, making them vital tools in assessments and project management. This underlines the wider applicability of chatbots in diverse fields requiring data
analysis and interpretation, beyond the educational context. [6] In the context of data
analytics, the authors draw attention to the use of chatbots in personalized learning, datadriven student assessments, and automated feedback, all of which rely heavily on the
analysis and interpretation of large data sets. This illustrates how chatbots can enhance
the efficiency and precision of data analysis, reinforcing their crucial role in this field.
However, potential limitations and highlights include issues around data privacy and
the potential for inaccuracies or misunderstandings in chatbot responses, particularly in
complex or ambiguous queries [6, 14, 15]. These concerns align with discussions in the
wider literature around the ethical and practical challenges associated with chatbot use
in data analytics.
Chatbots have become essential tools in project management, providing intuitive,
interactive platforms for complex data analysis. This aligns with the role of chatbots
in data analytics by enhancing the understanding of intricate project data, facilitating
decision-making processes, and promoting project efficiency [3]. Significantly, the chatbots’ ability to provide real-time data insights, ensuring more efficient tracking and monitoring of project progress. This functionality could lead to more accurate and timely
decision-making, reinforcing the broader role of chatbots in data analytics. Moreover,
potential challenges are associated with the implementation of chatbots, including the
need for ongoing training and development to ensure chatbots can adequately understand and respond to user queries. These considerations offer further insights into the
evaluation of chatbots’ functional abilities in data analytics and the need for continuous
improvement [3].
The role of chatbots, particularly ChatGPT, in the data science community is widely
praised. It is suggested that the most significant advantage lies in the ability of these AI
models to automate and streamline data processing, leading to improved efficiency and
effectiveness in data analysis. Liu et al. in [13] provide a comprehensive analysis of the
prospects of large language models such as ChatGPT highlighting that with the increasing complexity and sophistication of these models, their ability to generate insightful
data and analytics is continually growing.
As chatbots become more accurate, reliable, and able to understand complex queries,
they will become more widely adopted in data analytics projects. Automating tasks,
providing personalized assistance, answering direct questions and providing assessment
of skills are valuable to data analysts.
152
P. Patel and S. Shooshtarian
3 Methodology
Data science, an interdisciplinary domain, has transformed the way data is analyzed and
interpreted. This research employs the Cross-Industry Standard Process for Data Mining
(CRISP-DM) as its foundational methodology. The research also leverages modern tools,
with a particular emphasis on ChatGPT-4, to explore its capabilities in data analytics.
We adopt an inductive and empirical approach, rooted in experimentation. The aim is to
conduct experiments using ChatGPT-4 on three distinct datasets to assess its adaptability
and efficiency in diverse data scenarios. For this research, the focus is exclusively on
ChatGPT-4, a state-of-the-art chatbot based on the GPT-4 architecture. Renowned for its
ability to generate human-like text and its prowess in understanding context, ChatGPT-4
will be the focus of our experiments. Beta features of ChatGPT-4 include plug-ins and
Advanced Data Analysis which allows users to retrieve real-time information. This is a
critical issue since ChatGPR-3.5 has access to the data before 2021 [8]. Furthermore,
the recently introduced feature allowing for instruction customization enhances prompt
accuracy. ChatGPT-4 plug-ins are additional capabilities or tools that can be integrated
with the ChatGPT-4 model to enhance its functionality. Plug-ins have been selected
for their utility in data analysis and include Data Interpreter, Notable, WebPilot and
Wolfram.
3.1 Datasets
Three datasets are selected from the different domains of finance, health and energy;
enabling scope for comparison. Each dataset provides a unique perspective on the
capabilities of ChatGPT-4. Figure 1 shows the framework of this methodology.
• Dataset A: This dataset is focused on structured data of five stocks in the US market downloaded from the website www.finance.yahoo.com. The dataset provides the
stock prices for five tickers: “AVGO”, “OXY”, “FSLR”, “CAH”, and “LW” from
01/06/2020 to 01/05/2023.
• Dataset B: This dataset contains medical measurements related to heart health
downloaded from https://www.kaggle.com/datasets/johnsmith88/heart-disease-dat
aset?resource=download.
• Dataset C: A synthetic dataset is generated by “Noteable” plug-ins of ChatGPT-4
based on a prompt and requirements and afterwards, the CSV file of this dataset is
downloaded for further investigation in Python.
3.2 Prompt Engineering and Evaluation
Prompt engineering is crucial to maximize ChatGPT-4’s potential. It involves crafting
input prompts to obtain precise and relevant responses. An adeptly engineered prompt
can steer ChatGPT-4 to generate outputs that resonate with the specific prerequisites
of a data science task. Conversely, ambiguous, or poorly structured prompts may yield
generic or misaligned responses, potentially compromising the quality of data analysis.
ChatGPT-4 offers two primary avenues for prompt engineering: its user interface
(UI) and its API. The UI provides an interactive platform for manual prompt input and
The Role of Chatbots in Data Analytics: An Evaluation of Functional Abilities
153
Fig. 1. The framework of methodology
refinement, facilitating real-time response observation. This approach is particularly
beneficial during the nascent stages of experimentation. Conversely, the API provides
a more automated, scalable methodology, allowing for programmatic prompt input and
response retrieval. For this research, prompt engineering was conducted via the UI.
For the two types of prompts, baseline prompt and engineered prompts, a systematic
approach was taken for each dataset in order to aid comparison. A baseline prompt is a
generic prompt that asks ChatGPT-4 to analyze the dataset without any specific guidance.
This type of prompt is more open-ended and general. It gives the model freedom to
provide a broad answer without specific guidelines. The response might be more generic
and might not delve deep into specific details or insights. Engineered prompts allow
the design of a series of prompts that guide ChatGPT-4’s analysis, focusing on specific
aspects of the dataset or asking for types of insights. This prompt is crafted with precision,
providing clear instructions to the model about what is expected in the response. It guides
the model towards generating a more detailed and specific output, tailored to the user’s
requirements. The most effective prompts are relevant, clear, complex, scalable and
include role-play. After designing a set of prompts for each dataset, the chatbot is fed
with the dataset for setup. The chatbot is then queried using these prompts, and the results
are evaluated based on metrics like precision, recall, execution speed, and relevance.
4 Results
This section evaluates the analytical capabilities of ChatGPT-4 across three distinct
datasets: A, B, and C. Each dataset offers unique challenges, from portfolio optimization to heart health metrics and energy performance certifications. The approach is to
154
P. Patel and S. Shooshtarian
assess ChatGPT-4’s responses using basic and engineered prompts, and then benchmark its findings against human-based analyses. The goal is to understand ChatGPT-4’s
capabilities in the data analysis field and its potential for future applications.
4.1 Dataset A: Stock Market Analysis
Using the structured data of five stocks in the US market, ChatGPT-4 was tasked with
guiding potential investors in choosing the most efficient portfolio. The analysis incorporates various metrics, performance indicators, and visual representations. The basic
prompt, shown in Fig. 2, gives back a general exploration of the dataset, identifying
potential errors, and then proceeding to extract actionable insights.
Prompt: “Check this dataset for errors in data and provide insights.”
Fig. 2. ChatGPT-4 answer for the prompt
One of the indications of this analysis was the visualization of stock price trends
over time for each company. This was accompanied by the presentation of the Efficient
Frontier curve and its insights—a key tool in financial analysis that portrays the best
possible return for a given level of risk (see Fig. 3 and 4).
Fig. 3. Stock prices over time for each company
Average Daily Returns and Daily Standard Deviations, the Correlation Matrix and
the Covariance Matrix of Each Asset are also calculated. Furthermore, the iterative
nature of the analysis, especially when factoring in varying risk-free rates, underscores
the versatility and adaptability of ChatGPT-4 in handling complicated financial datasets.
The Role of Chatbots in Data Analytics: An Evaluation of Functional Abilities
155
Fig. 4. Plotted efficient frontier curve by ChatGPT-4 on the left and plotted in excel by a data
analyst on the right
To recalculate the “Sharpe Ratio” of the portfolio, another detailed prompt is given to
calculate this value for equally weighted assets portfolio and optimized portfolio.
Prompt: “We have a portfolio with all the above assets. Use an appropriate Solver
function to determine the portfolio risk and the percentage of investment in each asset in
your portfolio for a target return of your choice. First try it with equally weighted assets
in the portfolio and then optimize the weights to have the highest “Sharpe Ratio”. (see
Fig. 5).
Capital Market Line (CML) is plotted through a prompt by ChatGPT-4 (see Fig. 6).
To verify the result, prompts are optimized and revised iteratively. Each prompt yielded
varied outcomes, and after several iterations, the free versions of the “Data Interpreter”
and “Noteable” analysis plugins were finished.
To calculate the Beta Coefficient for each asset, the “SPY” ticker, representing the
S&P 500, is used as a benchmark. UI prompts for ticker details and prices are ineffective
for ChatGPT-4’s calculation. Hence, an updated “SPY” price dataset was uploaded again,
prompting the calculation of the Beta Coefficient for each asset (see Fig. 7).
156
P. Patel and S. Shooshtarian
Fig. 5. Optimized portfolio’s weights of assets, return and standard deviation
Fig. 6. Capital market line (CML) plotted by ChatGPT-4
5% value at risk (VaR) is also crucial as it measures the potential loss in value of a
risky asset that is calculated by ChatGPT-4. However, ChatGPT-4 attempts for estimating
the volatility of assets using ARCH/GARCH fail.
4.2 Dataset B: Heart Health Analysis
ChatGPT-4 was engaged to automate the six steps of the CRISP-DM process using a
structured heart health dataset. A supervised model is assessed in this experiment. The
analysis aims to understand the data, identify patterns, and model the results. A prompt
is used for data extraction and analysis, with “Business Understanding” results in Fig. 8.
The Role of Chatbots in Data Analytics: An Evaluation of Functional Abilities
157
Fig. 7. Beta coefficient for asset
Prompt: “This dataset needs to be analyzed. Consider CRISP-DM as your main methodology to do that. Conduct each step separately on this dataset and give back your
result.”
Fig. 8. ChatGPT-4 answer for “Business Understanding”
Beginning with an in-depth exploration, the data is examined for attributes, anomalies, and trends. Box plots detect outliers in continuous variables, like cholesterol (see
Fig. 9). Yet, ChatGPT-4 avoided manipulating data at this stage due to the following reason: “We’ll keep the outliers intact for now. If they prove problematic during modeling,
we’ll address them accordingly.”
The dataset’s dependent variable is “Target”, indicating if a patient has developed
heart disease or not. ChatGPT-4 considers feature scaling for Logistic Regression analysis, essential for magnitude-sensitive algorithms. Model assessment includes a confusion
matrix, Receiver Operating Characteristic (ROC) curve, Area Under the Curve (AUC),
and classification report (see Fig. 10 and 11).
4.3 Dataset C: Synthetic Energy Performance Analysis
Highlighting the innovative capabilities of ChatGPT-4, this section focused on the creation and analysis of a synthetic dataset. Therefore, for a broader understanding of its
features, this dataset is created by ChatGPT-4 plug-ins called “Noteable”. Moreover,
158
P. Patel and S. Shooshtarian
Fig. 9. Exploring the data (left) exploring the data with box plots to detect outliers (right) by
ChatGPT-4
this experiment will use a deeper methodology than CRISP-DM called Data Science
Project Plan and an unsupervised model is assessed in this experiment. ChatGPT-4
checks the attributes of the dataset and then, it checks the data for any missing values
and summarizes the statistics (see Fig. 12).
The Role of Chatbots in Data Analytics: An Evaluation of Functional Abilities
159
Fig. 10. Confusion matrix for logistic regression model: left: pre-outlier basic prompt, middle:
engineered prompt and right: human-based analysis
ChatGPT-4 determines that feature selection or engineering isn’t needed, but categorical variables are required to be converted to numerical. It visualizes distributions
and box plots (see Fig. 13).
For the predictive model and data mining phase, ChatGPT-4 selects K-Means and
scales the data. Both basic and engineered prompts identified 4 as the optimal cluster count. While basic prompts did not perform any performance metrics evaluations,
Silhouette Score or Elbow Method, engineered prompts did. Based on these methods,
ChatGPT-4 chooses four clusters. In human-based analysis, the optimal number of clusters is evaluated by plotting the Elbow Method for both SSE (Sum of Squared Errors)
and WCSS (Within-Cluster Sum of Squares) (see Fig. 14).
Since ChatGPT-4 did not provide all the required evaluation of the model’s performance, a separate prompt was crafted to achieve these values. To find the best metrics
for evaluation, different numbers of clusters are tested and assessed (k = 2, 4, 6), by key
160
P. Patel and S. Shooshtarian
Fig. 11. ROC curve for logistic regression model: left: pre-outlier and post-outlier basic prompts,
middle: engineered prompt and right: human-based analysis. The AUC for the engineered prompt
is 4% lower in ChatGPT-4’s report compared to the image.
Fig. 12. Checking null values and providing basic statistics by ChatGPT-4
The Role of Chatbots in Data Analytics: An Evaluation of Functional Abilities
161
Fig. 13. ChatGPT-4 visualization such as distribution and identifying outliers
metrics in the human-based analysis. Table 1 summarizes all models’ performance metrics. Although the six-cluster model performs the best, the four-cluster model is selected
for consistency with other models.
5 Discussion
This section serves as a bridge between the raw results obtained and their interpretation in
the context of broader research. This chapter delves deep into the themes of ‘comparative
approach’, ‘efficiency and depth’, ‘integration’, ‘accuracy and trust’, and more. Each
theme is explored in relation to the results obtained from different datasets, providing
a comprehensive understanding of the capabilities and limitations of the ChatGPT-4 in
data analysis.
5.1 Dataset A
The results present an exploration of ChatGPT-4’s capabilities in automating the stages
of the portfolio optimization.
Comparative Approach. The three analysis methods offer a gradient in terms of complexity and depth. The basic prompt approach provides a surface-level analysis, ensuring data integrity and preliminary insights. The engineered prompt method dives deeper,
leveraging specific plug-ins and detailed questions to extract more granular data insights.
Moreover, several principles, such as clarity, precision, and role-play, were considered
to enhance the effectiveness of these prompts. The human-based approach, being the
most detailed, serves as the accuracy benchmark for this article.
162
P. Patel and S. Shooshtarian
Fig. 14. Elbow method (Left) and Silhouette Score (Middle) by ChatGPT-4 (Top), SSE Elbow
method (Right) using python
Efficiency and Depth. The chatbot-driven approaches, especially the basic prompt, are
efficient for quick insights and data validation. They enable users without a deep technical
background to access and understand data. In contrast, the human-based method in Excel
and R offers depth and is more suitable for in-depth analysis and generating metrics.
Integration. The engineered prompt approach highlights ChatGPT-4’s ability to integrate with platforms like Google Docs. Such integrations can streamline data uploading
and preprocessing stages, making the analysis more seamless. However, there are some
The Role of Chatbots in Data Analytics: An Evaluation of Functional Abilities
163
Table 1. Comparison of metrics evaluation for each approach
Approach
k
Calinski-Harabasz
Index
Davies-Bouldin
Index
Inertia
Silhouette score
Basic prompt
4
--
--
--
--
Detailed prompt
4
183.30
1.92
2740.78
0.135
Human-based
2
701.49
1.16
4296.92
0.34
Human-based
4
498.59
1.42
3293.97
0.27
Human-based
6
558.21
1.09
2652.67
0.38
limitations for using ChatGPT-4. ChatGPT-4 has a limited service and cap for the number of queries and messages during a specific period of time. For example, every 3 h,
it is possible to send 50 messages. This restriction might slow analysis. Additionally,
timeout errors can disrupt analysis if there is a delay between prompts, requiring dataset
reloading and metric recomputation.
Accuracy and Trust. The human-based method, as the benchmark, is compared against
ChatGPT-4’s accuracy and reliability. In the basic prompt approach, while less detailed,
the data had fewer errors and values such as correlation and variance-covariance matrix
aligned with the benchmark. However, the engineered prompts had more errors, occasionally failed to answer, and certain technical analyses, like the Beta Coefficient, did
not match the benchmark.
Dataset A Recommendations. Chatbots like ChatGPT-4, with their expanding capabilities and integrations, signal a bright future. They can swiftly handle initial data analysis
stages, especially preliminary stages, allowing analysts to tackle more complex analyses. Recent advancements such as Excel-Python integration, can handle more advanced
data analysis which expedite and simplify the process [12]. Moreover, using the OpenAI’s API playground for engineered prompting is suggested for further and deeper
research. This feature allows users to have more options to customize their prompts
such as adjusting the model’s temperature to influence its randomness.
5.2 Dataset B
The results explore ChatGPT-4’s capabilities to automate the six CRISP-DM stages
using a heart health dataset.
Comparative Approach. The three analysis methods vary in complexity and depth.
The basic prompt provides preliminary observations. The engineered prompt goes further with specialized tools and precise queries for enhanced data interpretations. This is
attributed to tailored prompts that are clear, precise, and include a role-play (Data Analyst) for each CRISP-DM step. Meanwhile, the detailed human-based method establishes
the benchmark in this study.
164
P. Patel and S. Shooshtarian
Efficiency and Depth. ChatGPT-4 offers a time-saving solution for initial data analyses. The chatbot, especially using the basic prompt, provides rapid insights and data
validation, making it user-friendly for those without technical background. While it may
not match the depth of human analysts, its speed in data processing is invaluable for
early project stages. Compared to the “human-based”, ChatGPT-4 excels in many data
analysis areas but can struggle with many such as data visualization, possibly due to
text-platform limitations on visualizing complex data.
Integration. The integration of chatbots in data science workflows, as demonstrated by
this experiment, could significantly streamline, and automate various stages of the data
analysis process. By handling tasks such as data preprocessing, modeling, and evaluation,
chatbots can free up experts to focus on higher-level analytical tasks, interpretation, and
strategy formulation.
Accuracy and Trust. Using the human-based method as the benchmark, the basic
prompt approach was not deep, particularly in “Business Understanding” and “Evaluation” phases. Moreover, it was the least accurate among all three approaches. The
visualizations show incremental accuracy from basic to detailed prompts, with human
analysis as the benchmark. While ROC curves for both chat-based analysis are notably
close, their documented values were lower, indicating caution when relying on ChatGPT4 visuals that do not match the report. This discrepancy further suggests higher value of
human analysis (see Fig. 15).
Fig. 15. Comparing accuracy and ROC curves of three different approaches
Conversely, the engineered prompts encountered errors or failed to respond altogether. Therefore, it needed several iterations to show results that can be used. The
accuracy of the model increased slightly, but there was still a huge gap between this
value and human-based model. Additionally, there was a lack of technical analysis,
such as how results might differ with an 80–20% training-testing data split, or a report
detailing feature importance after its application.
Limitations. While the findings are promising, it is essential to consider potential limitations. Some easy tasks such as visualizing confusion matrices take a long time for
The Role of Chatbots in Data Analytics: An Evaluation of Functional Abilities
165
ChatGPT-4 and may lead to errors. Furthermore, some visualizations, such as a confusion matrix are plotted empty of any results. This suggests that while ChatGPT-4 might
be proficient at data analysis, it might face challenges with more complex visualization
tasks. Additionally, any errors might necessitate restarting the entire process or even
initiating a new chat, potentially inconveniencing the user.
Logistic Regression does not require data standardization due to its insensitivity
to variable magnitudes. Yet, during the Chat-GPT4 analysis, data standardization was
sometimes applied, suggesting the chosen model required it. This is important as preprocessing steps should be chosen based on the specific machine learning model in
use. Over processing or unnecessary preprocessing can sometimes skew data or waste
computational resources.
Dataset B Recommendations. In this experiment the dataset was small, ChatGPT-4
has some limitations regarding large scale datasets. There are new services like Google’s
new service which allows users to edit various programming languages, including SQL,
Python, and Spark, to conduct analytics and machine learning tasks on a massive scale.
Such services simplify user experience, facilitate data discovery, exploration, analysis,
and prediction [17]. Moreover, as machine learning models and platforms evolve, there
is a potential for future versions of platforms like ChatGPT to overcome the highlighted
limitations. Integration with visualization libraries or enhanced capabilities can make
them more versatile for end-to-end data science tasks.
5.3 Dataset C
The results thoroughly examine ChatGPT-4’s ability to generate datasets and automate
the seven stages of the Data Science Project Plan using a residential houses’ energy
efficiency dataset.
Comparative Approach. Analytically, the three approaches offer distinct depths and
nuances. The basic prompt serves as an introductory exploration, providing a broad
overview of the dataset. The engineered prompt technique, based on its precise principles
and data analyst role-play, provides more complex insights and a better understanding
of the data. The human-based analysis, deeply established traditional methods, stands
as the most comprehensive, ensuring a thorough and detailed examination of the dataset
and serves as the benchmark.
Efficiency and Depth. The basic prompt, while simple, gives quick insights, making
it great for those new to data analysis. ChatGPT-4 is fast and covers many areas of data
study, but it does not match the detailed understanding a human analyst brings. While
the human-driven approach is seen as the most thorough, ChatGPT-4 struggles in some
parts, especially showing data visually. This might be because text-based and complex
visuals are harder to capture.
Integration. Datasets sometimes lack adequate sample sizes. Chatbots like ChatGPT4 can generate synthetic data. Following guidelines, synthetic data, as evidenced in
contemporary research, provides a robust mechanism for data scientists to engage with
centrally archived data, all while preserving confidentiality of the data [16]. Moreover,
166
P. Patel and S. Shooshtarian
incorporating chatbots into data science routines can effectively optimize and automate
several aspects of the data evaluation process such as data preprocessing, exploring
data, and general data profiling. This automation enables data experts to focus on deeper
analysis and interpretations, and strategy creation.
Accuracy and Trust. Table 1 shows that the basic prompt approach yielded no results.
Therefore, the result is unreliable. Comparing the detailed prompt to human-based methods, while the first set has tighter clusters as indicated by the lower inertia, the latter
outperforms in terms of the Silhouette Score, Calinski-Harabasz Index, and DaviesBouldin Index. This suggests that the human-based approach has better-defined and
more distinct clusters than the first. Moreover, by comparing different metrics of optimal clusters, the model that has six clusters has the best evaluation among all models.
Therefore, none of ChatGPT-4’s analyses met the desired accuracy.
Limitations. Several errors occurred while using ChatGPT-4, instances with no
responses, the need for reruns, and occasional loss of computed results. These resulted in
iterations to achieve the desirable result. Another issue was when ChatGPT-4 wanted to
choose the optimal number of clusters, the basic-prompt approach did not provide performance evaluations or reasons why it chose the certain number of clusters. Moreover,
the more detailed-prompt approach did not compare different methods such as SSE and
WCSS and did not show why it chose a certain number of clusters.
Dataset C Recommendations. Using the synthetic dataset for this experiment was on
purpose to challenge the ability of ChatGPT-4. However, a real-world dataset, especially
when the dataset is extremely large, is more challenging. It is suggested to use real-world
data that is freely accessible for further challenges. For example, some institutes such as
Allen Institute for AI (AI2) has provided a vast repository of unstructured (text-based)
data [4].
6 Conclusion
The exploration of chatbots’ role, especially ChatGPT-4, in data science has revealed
a diverse landscape. ChatGPT-4, with its varying analytical approaches, offers a spectrum of insights ranging from preliminary to in-depth. While the basic prompt approach
provides swift insights suitable for users without extensive technical expertise, the engineered prompts delve deeper, but have limitations for sophisticated metrics, correlations
and complex calculations. Given the occasional errors and discrepancies observed in
ChatGPT-4’s analysis, it is crucial for users to continuously monitor and validate the
results, especially when relying on the platform for detailed prompts and complex calculations. The ideal approach for data analysis might be a blend of chatbot-driven methods for preliminary tasks and human expertise for in-depth analysis. This collaborative
approach ensures accuracy, depth, and nuance in data interpretation. Organizations and
researchers should leverage the integration capabilities of ChatGPT-4 with repository
platforms to streamline the data uploading and preprocessing stages. The potential of
chatbots in creating synthetic data, a feature that can enhance sample sizes without compromising data confidentiality can also be leveraged. This capability, combined with
The Role of Chatbots in Data Analytics: An Evaluation of Functional Abilities
167
ChatGPT-4’s automation capability, can free human experts to focus on more complex
analytical tasks.
While ChatGPT-4 demonstrated efficiency in providing insights, there were instances
where its results and accuracy did not align with the benchmark set by human-based
methods, especially in basic prompts. Users might also encounter operational challenges
such as time out errors, limitations on the number of queries, word limitations, and the
need to reload datasets, which can disrupt the analytical process. ChatGPT-4 faces challenges in data visualization tasks, indicating that it might not yet be a complete substitute
for platforms that specialize in advanced visual representations. The platform’s decisionmaking process, such as in tasks like optimal cluster selection, lacked comprehensive
evaluation and clarity, necessitating a cautious approach when relying on it for complex
analytical tasks.
Overall, the integration of chatbots in data science workflows promises a streamlined
and efficient approach. However, recognizing its limitations is crucial. As chatbot technology matures, the next stage lies in a wholly collaborative approach, where chatbots
handle preliminary tasks, setting the stage for human experts to ensure accuracy and
nuance in data interpretation.
References
1. Adamopoulou, E., Moussiades, L.: Chatbots: history, technology, and applications. Mach.
Learn. Appl. (2020). https://doi.org/10.1016/j.mlwa.2020.100006. Accessed 14 Dec 2023
2. Alshater, M.: Exploring the role of artificial intelligence in enhancing academic performance:
a case study of ChatGPT. In: SSRN (2022)
3. Cı̄rule, D., Bērziša, S.: Use of chatbots in project management. In: Damaševičius, R., Vasiljevienė, G. (eds.) Information and Software Technologies. CCIS, vol. 1078, pp. 33–43. Springer,
Cham (2019). https://doi.org/10.1007/978-3-030-30275-7_4
4. Coldewey, D.: AI2 drops biggest open dataset yet for training language models. TechCrunch
(2023). https://techcrunch.com/2023/08/18/ai2-drops-biggest-open-dataset-yet-for-traininglanguage-models/. Accessed 14 Dec 2023
5. Dave, T., Athaluri, S.A., Singh, S.: ChatGPT in medicine: an overview of its applications,
advantages, limitations, future prospects, and ethical considerations. Front. Artif. Intell.
(2023). https://doi.org/10.3389/frai.2023.1169595
6. Gill, S.S., et al.: Transformative effects of CHATGPT on modern education: emerging era of
AI chatbots. Internet Things Cyber-Phys. Syst. 4, 19–23 (2024). https://doi.org/10.1016/j.iot
cps.2023.06.002
7. Hassani, H., Silva, E.S.: The role of ChatGPT in data science: how AI-assisted conversational
interfaces are revolutionizing the field. Big Data Cogn. Comput. 7(2), 62–71 (2023). https://
doi.org/10.3390/bdcc7020062
8. Hamilton, K.: ChatGPT will now have access to real-time info from Bing search. Forbes
(2023). https://www.forbes.com/sites/katherinehamilton/2023/05/23/chatgpt-will-now-haveaccess-to-real-time-info-from-bing-search-report-says/. Accessed 14 Dec 2023
9. Jongerius, C.: Quantifying chatbot performance by using data analytics. Master’s thesis (2023)
10. Kalla, D., Smith, N., Samaah, F., Kuraku, S.: Study and analysis of chat GPT and its impact
on different fields of study. Int. J. Innov. Sci. Res. Technol. 8(3) (2023)
11. Khlaif, Z., et al.: The potential and concerns of using AI in scientific research: ChatGPT
performance evaluation. JMIR Med. Educ. (2023). https://mededu.jmir.org/2023/1/e47049.
https://doi.org/10.2196/47049. Accessed 14 Dec 2023
168
P. Patel and S. Shooshtarian
12. Lardinois, F.: Microsoft is bringing Python to Excel. TechCrunch (2023). https://techcrunch.
com/2023/08/22/microsoft-is-bringing-python-to-excel/. Accessed 14 Dec 2023
13. Liu, Y., et al.: Summary of ChatGPT/GPT-4 research and perspective towards the future of
large language models (2023)
14. Nah, F.F.-H., Zheng, R., Cai, J., Siau, K., Chen, L.: Generative AI and ChatGPT: applications,
challenges, and AI-human collaboration. J. Inf. Technol. Case Appl. Res. 25(3), 277–304
(2023). https://doi.org/10.1080/15228053.2023.2233814
15. Okonkwo, C.W., Ade-Ibijola, A.: Chatbots applications in education: a systematic review.
Comput. Educ. Artif. Intell. (2021). https://doi.org/10.1016/j.caeai.2021.100033
16. Singh, K.: Synthetic data—key benefits, types, generation methods, and challenges. Towards
Data Science. https://towardsdatascience.com/synthetic-data-key-benefits-types-generationmethods-and-challenges-11b0ad304b55. Accessed 14 Dec 2023
17. Wiggers, K.: Google launches BigQuery Studio, a new way to work with data. TechCrunch
(2023). https://techcrunch.com/2023/08/29/google-launches-bigquery-studio-a-new-way-towork-with-data/. Accessed 14 Dec 2023
An Analytical Investigation into the Impact
of Product Color on the Price of Retail Products
and Purchasing Decisions of Consumers
Zaid M. Altukhi and Nasser F. Aljohani(B)
Faculty of Computer and Information Systems, Islamic University of Madinah, Madinah,
Saudi Arabia
naljohani@iu.edu.sa
Abstract. Product owners have a role in determining the effective colors when
launching new products or repackaging existing ones. Both consumers and product
owners often wonder about the influence of colors on pricing and purchasing
decisions. Therefore, analytics play a role in understanding how different colors
impact consumer behavior and evaluations. The purpose of this study was to
investigate whether the color of a product influences both its price and consumer
behavior. By utilizing data exploration, statistical models and visualizations, the
proposed hypothesis has been confirmed. This article provides an analysis of the
data used in project specifications, data sources and acquisition methods, data
analytics techniques and data visualizations to assist with decision making related
to themes. There exists a strong correlation and statistically significant association
between the colors and pricing of the products. Those that are orange in hue have
greater confidence intervals for receiving more likes compared to those of other
colors. Brown products have higher average pricing compared to other products.
Purple products have a greater average discount compared to other products. Black
is the predominant color utilized in these products. Yet, it’s important to note that
these results can be further refined by considering additional sources and a wider
range of products. Therefore, gathering data from e-commerce websites becomes
necessary as there is a scarcity of accessible information and obtaining permission
from site owners is required to do so.
Keywords: Colors · Decision-making · Product · Price · Analysis · Consumers
1 Introduction
One of the essential questions that appear when a product owner wants to launch a
product or who has markets and wants to buy products to sell is the suitable color
that could be used or bought. Can product colors determine the product cost and the
customer’s decisions? Anyone who works in the marketing or sales department can tell
how product color is an essential factor affecting customer behavior. However, some
details need analytics to be answered; such as if the product has more than one color,
will this affect the customer’s actions? What are the most men’s, women’s, and kid’s
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2024
K. Arai (Ed.): SAI 2024, LNNS 1017, pp. 169–186, 2024.
https://doi.org/10.1007/978-3-031-62277-9_11
170
Z. M. Altukhi and N. F. Aljohani
product colors? If there is a product with several colors, will this influence customer
reviews? Analyzing how colors affect consumers’ behavior will provide a good idea for
those seeking to sell many products and gain profits. Although the designers working
on creating the product details know how important it is to pick the right color for each
piece of the product, knowing how the color can influence the product price and people’s
desires will help the decision-maker make the right decisions.
This study is more than frameworks since it sheds light through empirical probes.
The authors detail their data, not just presenting the project’s prerequisite conditions but
also explaining the direction that descriptions take the users toward the eyes and ears of
some publics and their tools to tap those sources of information. How they went on with
this investigation into details has added to the validity and reliability of the explained
findings in this study. The research also demonstrates the competence in data analysis.
The authors employ methods and tools to illustrate the interrelation and tendencies
between the color of a product, its price, and the behavior of consumers. Regarding the
use of these methods, the authors make the research more reliable. Make more precise
and veracious statements.
Moreover, the incorporation of data visualizations in this research paper enhances its
importance since the purpose of data visualization is to present information in an easily
understandable way. Visually, this study had appealing charts, graphs, and diagrams
that aided in enhancing the comprehending capacity of the findings. The essay used
visual aids in a way that readers should quickly understand and recall the information.
What makes this research overbid others in the area is methodology in analyzing the
effects of the product color on pricing and consumer behavior. The authors contribute to
marketing and consumer research via their methods of data collecting, rigorous analysis,
and persuasive visual aids.
Therefore, this article aims to address the inquiry by investigating the concept that the
color of a product can influence both its pricing and the behaviors shown by customers.
In addition, answering these questions needs to consider many factors, such as the target
audiences and the product nature. The following sections will show the data used, project
requirements, data acquisition and sources, data analytics, and data visualizations.
2 Analytical Aspects
2.1 Requirements
After the hypothesis was determined, the search for datasets started by looking for the
pertinent data to use in multiple data sources that provide the datasets we can analyze.
Since the hypothesis focuses on the product’s characteristics, the dataset we are looking
for is a must-have list of products with specific kinds of features that can be examined
to test the hypothesis. To select the dataset, it must contain at least 10,000 products with
different categories, colors, prices, and rates done by customers. Moreover, the data type
must be categorical, numerical, and nominal.
On the other hand, finding the proper analytical tools is essential for the analysis
operation. The tools used to do this analysis are Python and Tableau. Python can be used
to explore, clean, prepare, and visualize. In addition, it can be used to apply statistical
models to find if there is a relationship between the features. At the same time, Tableau
An Analytical Investigation into the Impact of Product Color
171
can visualize and create dashboards that show the trends in the data, answer the primary
questions, and find patterns and insights. Using Python and Tableau can help determine
if the hypothesis we test is right or wrong using different data analytics methods.
2.2 Data Acquisition and Data Sources
Many data sources provide this kind of data—for example, Kaggle.com and
data.world.com. After looking at many datasets provided by those websites. Most of
the data were available in CSV files, common file types used in the data analytics field.
However, most of these datasets have broken at least one of our data requirements until
datasets found on data.world.com fulfil the requirements.
The datasets were found in data.world.com have more than four datasets that show
the list of products from an e-commerce website called NewChic.com [1]. Each dataset
contains around 11,000 records. Also, I found that the data provided are appropriate
for examining the hypothesis. For example, product categories, product names, price,
discount percentage, number of likes the product gets, and product colors.
2.3 Datasets Overview
Five datasets will be used in this project. Each dataset contains specific kinds of products,
clothes, bags, and shoes. The datasets were used in this project as follows:
• Women products
• Men products
• Children products
• Shoes
• Bags
After exploring the data, some variables were found to work. For example:
• Raw price: Original product price
• Current price: The price after the discount is applied.
• Colors: Some products contain only primary colors, and others have two main colors.
• Likes count: Number of likes that product got from customers.
• Discount: the discount percentages per product.
In addition, the datasets contain different data types as follows:
• Categorical features: main product category, product sub-category, primary color,
secondary color.
• Numerical features: product price, product discount percentage, number of likes.
• Nominal features: product name, product IDs.
Although most of the data are in good shape and quality (defined, measurable, unitized, relatable, normalized, and quality), there is a country column with multiple values
that need to be handled, and the color columns have many different forms for the same
color.
172
Z. M. Altukhi and N. F. Aljohani
2.4 Prepare Data
Many tools are used to prepare and clean data: Microsoft Visual Studio Code [2], python
programming language [3]. The Python language supports powerful libraries that help
the developers perform the analytics processes. This project uses multiple software and
packages to complete the data cleaning, preparation, and analysis of the data:
• Anaconda [4].
• Pandas [5].
• Numpy [6].
• Sklearn [7].
• SciPy [8].
Each dataset contains 22 columns. All datasets were concatenated together to prepare
the datasets to perform analyzing models and visualizations. The final concatenated data
set consists of 47,193 records and 22 columns.
2.5 Feature Engineering
The feature engineering process was done on the data by dropping columns, processing
null and unified values, and adding a new column. There are ten features in the dataset
that have no relation to our project: currency, brand_url, is_new variation_0_thumbnail,
variation_0_image, variation_1_thumbnail, variation_1_image, image_url, URL, brand,
and country; those columns were removed.
The color columns need to be cleaned. The main color column ‘variation_0_color’
has 559 null values. In addition, after exploring the datasets, some color values could not
be used, as shown in Fig. 1, and these values were also removed. Moreover, the columns
Variation_0_color and Variation_1_color contain 433, 460 unique values, respectively.
Some of these values have typos or were written in different formats, such as the red
color written in various formats (RED, Red, red, red1). This issue needs to be unified
because it may affect the final analyzing results. Similar colors are aggregated in one
color to fix this issue the color columns. For example, there are around 60 values that
indicate the brown color, and 21 colors belong to the black color. Moreover, a two_color
column was added to indicate whether the items have more than one color, so we need
a Boolean column to identify this information.
Fig. 1. Sample of the values located in the color columns [Python]
An Analytical Investigation into the Impact of Product Color
173
The shape of the cleaned and final dataset is 45,425 records and 12 columns as
follows in Table 1:
Table 1. The final dataset features a description
No.
Column Name
Data Type
Description
1
category
String
Shows the item’s category. There are five different
categories: women, men, kids, bags, and shoes)
2
subcategory
String
Under each category there are different
sub-categories. The dataset contains 178
subcategories
3
name
String
Items name
4
Current_price
Integer
Items price after discount
5
Raw_price
Integer
Items price before discount
6
Discount
Integer
Discount percentage
7
Likes_count
Integer
Number of likes the item got from customers
8
Variation_0_color
String
Item main color
9
Variation_1_color
String
Item secondary color
10
Id
String
Item id, as primary key
11
model
string
Item model
12
Two_colors
Boolean
Indicates if the item has only one color or more
Fig. 2. Final dataset shape [Python]
174
Z. M. Altukhi and N. F. Aljohani
Five operations are performed on the datasets. First, aggregate the datasets into one
data frame. Second, drop the unrelated columns and not understandable values. The
third process is to unify the columns with the item’s color. The number of unique color
values after unified is 12 unique values presents in Table 2. Lastly, add new column to
indicate if the item has more than one color. 0 shows the final dataset information after
the cleaning and preparation processes were performed.
Labeling the data is a useful technique to analyze the categorical data. The variables need to convert from strings to numerical values. The statistical models cannot
understand the string values. Five columns were converted: category, subcategory, variation_0_color, variation_1_color, and model. The table below shows the categorical
values with the numerical ones (Fig. 2).
Table 2. Color code in the labeled data
No
Color Name
Color number
1
Black
0
2
Blue
1
3
Brown
2
4
Gray
3
5
Green
4
6
Orange
5
7
Pink
6
8
Purple
7
9
Red
8
10
White
9
11
Yellow
10
12
other
11
3 Analysis
3.1 Conduct Exploratory Analysis
It is important to ascertain the interrelationships among various aspects to comprehend
the data. Correlations are useful statistical models used to find the relationships between
the positive or negative features and how these relationships are strong [9]. 0 shows the
correlations between the different variables in the dataset. It can be noticed that there are
many strong correlations between most variables. The strong correlations are between
−1.0 and −0.5 or 1.0 and 0.5, and the correlations from −0.5 to −0.3 or 0.3 and 0.5 are
considered moderate [9]. The variation_0_color groups the correlation results shown in
0 from the final dataset as an independent variable (Fig. 3).
0 shows the values in a statistical format. This kind of data can provide useful
information about how the data are distributed and give an idea (Fig. 4).
An Analytical Investigation into the Impact of Product Color
175
Fig. 3. Data correlation [Python]
Fig. 4. Final dataset quantitative data description [Python]
3.2 Exploratory Visualizations
To facilitate data exploration, it is imperative to generate visual representations that
aid in comprehending the underlying information. Figure 5 shows the different colors
with the number of items. 0 represents the number of items for each category. All these
visualizations give us an idea of how the information is distributed in the dataset, allowing
us to understand the data content profoundly. By the way, all the visualizations shown
in this section were produced using the Altair library [10] (Fig. 6).
176
Z. M. Altukhi and N. F. Aljohani
Fig. 5. Number of items pre-color
Fig. 6. Number of items pre-category [Python by Altair]
3.3 Exploratory Statistical Model
Confidence Intervals. A confidence interval (CI) is a set of numbers expected to encompass a population figure with a high degree of certainty. When a population means falls
between two intervals, it is commonly stated as a percentage [11]. Table 3 below represents the different confidence interval values for the different colors with likes count,
current price, raw price, and discount percentages. Also, we highlighted the highest
values for each variable (see Fig. 7 and 8).
One-Way-ANOVA. After looking at the strongest relations, either positive or negative,
it can perform the statistical models to look deeply into the data. In this project, we will
An Analytical Investigation into the Impact of Product Color
177
Table 3. Confidence intervals summary
Color
Likes_count
mean
ci95_hi
Current_price
ci95_lo
mean
Raw_price
ci95_hi
ci95_lo
mean
ci95_hi
discount
ci95_lo
mean
ci95_hi
ci95_lo
Black
193.04
202.86
183.22
31.83
32.14
31.52
66.74
67.77
65.71
51.82
52.00
51.63
Blue
253.53
273.21
233.86
28.03
28.42
27.64
59.79
60.63
58.95
52.42
52.69
52.15
Brown
222.44
242.65
202.24
33.44
34.09
32.79
70.78
72.06
69.51
52.79
53.14
52.43
Gray
221.10
241.25
200.95
29.81
30.49
29.12
61.96
63.28
60.64
51.59
52.01
51.17
Green
259.10
287.87
230.33
26.99
27.50
26.47
57.47
58.55
56.38
52.26
52.62
51.91
Orange
284.63
359.77
209.50
26.64
27.84
25.44
57.56
60.28
54.83
52.61
53.46
51.75
Pink
272.48
294.59
250.37
25.54
25.95
25.12
55.95
56.78
55.12
54.29
54.59
53.99
Purple
188.42
218.01
158.83
25.20
26.10
24.31
56.19
58.25
54.12
54.54
55.16
53.92
Red
231.14
253.65
208.63
31.12
31.71
30.53
66.56
67.81
65.32
52.52
52.87
52.17
White
201.80
214.51
189.09
24.09
24.38
23.79
49.61
50.17
49.05
51.22
51.48
50.95
Yellow
255.93
282.78
229.07
28.27
28.83
27.72
59.94
61.10
58.79
52.35
52.73
51.97
Fig. 7. Total of the current price per color [Python by Altair]
examine the data using one-way ANOVA because we test 13 groups of data (Table 4).
The library used to perform the model is statsmodels [12].
Ordinary Least Squares Regression (OLS) Regression. In [13], the authors introduce
the ordinary least squares regression (OLS) as a statistical method that estimates the
relationship between independent and dependent variables by minimizing the sum of
squares in the difference between observed and predicted values, represented as a straight
line. The least-squares mistake is called the least-squares error (SSE). This analysis was
178
Z. M. Altukhi and N. F. Aljohani
Fig. 8. Total of the like counts per category [Python by Altair]
Table 4. One-Way ANOVA results
Independent column
Dependent
column
sum_sq
F
PR(>F)
variation_0_color
Current_price
12966.597
773.670567
7.499581e−169
Raw_price
2.310318e−122
variation_1_color
9375.489
556.77557
Discount
5.757
0.337798
0.561106
Likes_count
85.970
5.044197
0.024713
Current_price
2055.451
153.503942
3.485611e−35
Raw_price
1023.543
76.282336
2.558763e−18
Discount
404.791
30.131003
4.064432e−08
Likes_count
14.179
1.054614
0.304453
performed using statsmodels [12] in Python. The OLS analysis results are shown in
Table 5 below:
Chi-Square. One technique to illustrate a link between two category variables is to use
a chi-square statistic. Numerical (countable) variables and non-numerical (categorical)
variables are the two categories of variables in statistics [14]. The following table shows
the chi-square test results for the main colors with the current price, likes count, raw
price, and discount percentage. From 0 Table 6, we can find there are strong relationships
between the independent variables and the main colors because the p-values for all
variables are under 0.05, which is the value that we can reject the null hypothesis in it.
An Analytical Investigation into the Impact of Product Color
179
Table 5. OLS results
Independent column
Dependent column
R-squared
Coefficient
Std error
variation_0_color
Current_price
0.346
25.2131
8.624
Likes_count
0.042
10.0836
6.984
Raw_price
0.397
25.4963
7.838
variation_1_color
Discount
0.097
41.1466
34.483
Current_price
0.095
10.8319
5.366
Raw_price
0.256
13.8619
4.607
Discount
0.686
55.6206
10.756
Likes_count
0.167
10.2022
3.445
Table 6. Chi-Square summary
Independent variable
Dependent variable
Statistic
p-value
variation_0_color
Current_price
222.953
1.2515490198e-
Likes_count
2619.180
0.0
Raw_price
587.810
5.66777868258e-119
Discount
504.247
3.982598225824e-
4 Data Visualizations
Visualizations are considered one of the most important ways to explore and understand
the data. In addition, visualizations could be used to see the data distribution and get a
general idea about it. Visualizing a variable’s distribution is critical to quickly comprehend properties like frequencies, peaks, skewness, center, modality, and how variables
and outliers behave in the data range [15]. There are six variables were used to create
the charts below as in Tables 7 and 8:
All charts were generated by Tableau software [16], making creating charts more
flexible and manageable. In addition, all charts were sorted descending from high to low.
For more data understanding, here are the numerical variables described:
4.1 The Number of Products Based on Color
This bar chart in 0 demonstrates the color distribution for the entire dataset. The bar graph
is maybe the most popular type of data visualization in both scientific and popular media
[17]. It can be noticed that the black color has the highest frequency with 11,818 products.
Conversely, the lowest number of product colors is the orange ones, with 493 products.
However, the average number of products grouped by color is 3,785 products per color.
The average helps us to find what is the products’ colors above and under the mean.
There are four colors above the average: black, white, blue, and pink. Understanding
180
Z. M. Altukhi and N. F. Aljohani
Table 7. Variables are used in the visualizations
Variable Name
Data Type
Data Range
Describe
Category
String
Women, men, kids, shoes,
and bags
The product category
Main color
String
Black, blue, brown, gray,
green, orange, pink, purple,
red, white, yellow, and other
The product main color. There
is 0 Null value in this column
Secondary color
String
Black, blue, brown, gray,
green, orange, pink, purple,
red, white, yellow, and other
The product additional color.
There are some Null values in
this column
Current price
Number
0.14–314.59
The product price
Discount
Number
0%–100%
The discount percentage
Likes count
Number
0–21547
Number of likes
Two colors
Boolean
True – False
Indicates if the product has just
main color or more than one
color
Table 8. Numerical variables describe
Variable
name
count
mean
Std
min
25%
50%
75%
max
Current
price
45425
28.73852
16.02538
0.14
18.04
24.99
35.69
314.59
Discount
45425
52.22261
10.40863
0
47
52
59
100
Likes
count
45425
224.2966
631.9663
0
29
75
189
21547
the distribution of the product’s color can help the decision-maker to make the right
decisions about the color of their products and whether they want the product to be
above the average or not. For example, if they want to show that the product is different
from most products, they can pick the less used colors. Nonetheless, if they want to use
standard colors, they can pick colors above the average.
However, this distribution could not be generalized to the entire market because this
sample was collected from one e-commerce website and for a few products. Nevertheless,
the sample could give an idea about how the population would be (Fig. 9).
4.2 The Average Number of Likes Based on the Product’s Color
The bar graph in 0 demonstrates the average number of likes per color. This visualization
surprised us because the color that gets the highest average number of likes was the lowest
color used, which is orange. The orange color showed in the dataset 493 times and got
An Analytical Investigation into the Impact of Product Color
181
Fig. 9. Product counts based on color [generated by Tableau]
284.6 average number of likes. While the most color shown in the dataset, which is black,
got 193 average number of likes, which is less than the overall average likes count. Also,
the average number of likes for all colors is 229.1 likes. In addition, it can be noticed
that the lowest average number of likes was for the product that does not have a specific
color. The average number of likes can help marketers and decision-makers understand
how the color can impact the product likes rate and what customers like the colors most
(Fig. 10).
Fig. 10. Mean of likes count based on product’s color [generated by Tableau]
This chart supports the project hypothesis by showing how the product color can
influence customer behaviors. Although the orange color shows a small amount of product, it got the highest average number of likes. Also, the pink color is used most in the
kid’s products but got the average number of likes more than the mean.
4.3 Average Likes Count for the Products that Have More Than One Color
0 depicts the average number of likes for products containing multiple colors compared
to those with only one primary color. Based on the graph, the products that contain more
than one color got a higher average number of likes. However, the difference between
the number of likes between the products containing one color and the more than one
182
Z. M. Altukhi and N. F. Aljohani
color is not too significant. This plot can help the marketing department present multiple
colors to get customer attention. In addition, the result helps the decision-maker buy
or produce products with more than one color than those with one color because the
customers like products containing multiple colors (Fig. 11).
Fig. 11. Average number of likes based on the product’s color counts [generated by Tableau]
4.4 The Average Number of Likes for Products that Have More Than One Color
Based on the Product’s Category
This bar chart in 0 shows the number of products that contain more than one color and
the average number of likes divided by product category. The most significant number of
likes went to the women’s and shoe products, followed by the men’s and bags products.
The lowest average number of likes was for the kid’s products. This chart indicates that the
people like the kid’s products to be in one color; however, women like the products with
multiple colors. The chart can help the decision-maker make the right decisions about
the product design and understand the customer’s desires. In addition, how much the
number of colors can impact customer behavior, which supports the project hypothesis
(Fig. 12).
4.5 Average Product’s Price Based on Color
0 depicts the average price per color. The products with brown color got the maximum
average price of 33.44 USD, followed by the black products at 31.83 USD. At the
same time, the lowest average price is for the product that has white colors at 24 USD.
Moreover, it can be observed that the blue color has almost the average price. This
chart supports the hypothesis used in this research by showing how color can affect
the product price. It is obvious that there are differences in the price distribution based
An Analytical Investigation into the Impact of Product Color
183
Fig. 12. Product count based on category and the average likes count [generated by Tableau]
Fig. 13. Average product’s price based on product’s color [generated by Tableau]
on color. The chart emphasizes the hypothesis’s correctness by representing the price
differences based on the product’s color (Fig. 13).
184
Z. M. Altukhi and N. F. Aljohani
5 Challenges
Certain challenges were encountered while examining the hypothesis. Nevertheless,
they were successfully overcome. The initial obstacle encountered was the disparate
formats associated with each hue, necessitating a labor-intensive process to standardize
the colors. The second challenge we faced was that the data was not normally distributed.
Normality is essential to perform the statistical models. To make the data in normal shape,
we take the mean values per category to perform the statistical models.
Also, finding the proper statistical model to examine the data and understand the
results was a beneficial challenge. This made me read more about the statistical models
and understand the purpose of each model. In addition, finding ways to apply these models to the data was another challenge that required searching the libraries and packages
of each model to perform these models.
Furthermore, we attempted to gather data from several sources but could not locate
accessible datasets from many sources.
6 Future Work
To augment the findings presented in this study, it is necessary to obtain data from many
sources. Numerous e-commerce platforms, like Shein.com, Gap.com, and Zara.com,
provide a comparable range of items. However, in order to obtain the necessary product
data from these websites, it is imperative to employ web scraping techniques, as no
readily available data can be found from these sources. In order to gather this data, it
may be necessary to get consent from the data owners for its utilization.
Conversely, it is more advantageous to utilize a five-point rating scale rather than
aggregating the total number of likes, as is currently implemented in this project. The
utilization of this rating structure is prevalent among several e-commerce websites.
7 Conclusion
In conclusion, the hypothesis investigated in this study posits that there exists a relationship between product colors and both product pricing and consumer behavior. In
order to evaluate the validity of this hypothesis, it is imperative to establish certain
data requirements, define the methodology for data exploration, execute statistical and
analytical models, and generate a variety of visualizations. The project outcomes will
provide valuable opportunities for acquiring knowledge and skills in utilizing advanced
analytical tools and methodologies. Additionally, individuals will be able to leverage the
obtained results to enhance their work and make informed judgments on the selection
of appropriate colors for the product.
Upon thorough examination of the statistical findings and visual representations, it
can be concluded that the hypothesis under investigation is substantiated. This assertion
is supported by the identification of significant associations between the colors employed
and the corresponding pricing and number of likes garnered by the items. The summary
of the results from the visualizations and models are as follows:
An Analytical Investigation into the Impact of Product Color
185
• There is a high correlation and significant relationship between the colors and the
product’s prices.
• Products with orange color have higher confidence intervals to get more likes than
other colors.
• Brown products have higher mean prices than other products.
• Purple products have a higher mean discount percentage than other products.
• The black color is the most used in the products.
• Most products have more than two colors.
• The highest average number of likes went for the products with orange color.
• The products with multiple colors get a high average number of likes.
• Women and shoe products with more than one color got more likes than others.
• The highest average price was for the brown products, followed by the black ones.
• There is a slightly different discount rate between different colors.
• There is no significant difference in the average price and discount between products
with only one color or multiple ones.
However, it is important to consider that this distribution pertains just to a single ecommerce platform and is limited to certain product categories, namely clothes, shoes,
and bags. It is essential to refrain from generalizing based on these findings until further
sources are obtained, preferably with a larger sample size of items.
References
1. Mabilama, J.: Products Catalog, data.world, 7 September 2020. https://data.world/jfreex/pro
ducts-catalog-from-newchiccom
2. Visual Studio Code (1.61.0): Visual Studio Code, Microsoft (2019). https://code.visualstudio.
com
3. van Rossum, G., Drake, Jr. F.L.: Python tutorial, Centrum voor Wiskunde en Informatica,
Amsterdam, The Netherlands (1995)
4. Anaconda Distribution: Anaconda Documentation, Anaconda Inc. (2020). https://docs.ana
conda.com/
5. T. p. d. team: pandas-dev/pandas: Pandas (3.8.8) [Python library] (2020). https://doi.org/10.
5281/zenodo.3509134
6. Millman, K., et al.: Array programming with NumPy. Nature 585(7825), 357–362 (2020)
7. Varoquaux, G., Buitinck, L., Louppe, G., Grisel, O., Pedregosa, F., Mueller, A.: Scikit-learn.
GetMobile Mob. Comput. Commun. 19(1), 29–33 (2015)
8. Virtanen, P., et al.: SciPy 1.0: fundamental algorithms for scientific computing in Python.
Nat. Methods 17(3), 352 (2020)
9. Wilson, L.T.: Statistical Correlation. Explorable 2, 5 (2009). https://explorable.com/statis
tical-correlation
10. VanderPlas, J., et al.: Altair: interactive statistical visualizations for Python. J. Open Source
Softw. 3(32), 1057 (2018)
11. Mcleod, S.: Z-score: what are confidence intervals in statistics? Simply Psychol. (2019).
https://www.simplypsychology.org/confidence-interval.html. Accessed 27 Nov 2021
12. Seabold, S., Josef, P.: Statsmodels: econometric and statistical modeling with Python. In: 92
Proceedings of the 9th Python in Science Conference (2010)
13. Dudley, L., Poston, J.: Ordinary least squares regression. J. Mod. Appl. Stat. Methods 1(2),
387–396 (2002)
186
Z. M. Altukhi and N. F. Aljohani
14. Glen, S.: Chi-square statistic: how to calculate it/distribution. StatisticsHowTo.com,
20 November 2021. https://www.statisticshowto.com/probability-and-statistics/chi-square/.
Accessed 27 Nov 2021
15. Iyim, C.: Recipes for the visualizations of data distributions. Medium, 19 April 2020. https://
towardsdatascience.com/recipes-for-the-visualizations-of-data-distributions-a1527a0faf77.
Accessed 6 Nov 2021
16. Chabot, C., Beers, A., Hanrahan, P.: Tableau (2021.3.3), Tableau [Computer Software] (2021).
https://www.tableau.com
17. Newman, G., Scholl, B.: Bar graphs depicting averages are perceptually misinterpreted: the
within-the-bar bias. Psychon. Bull. Rev. 19, 601–607 (2012)
A Numerical Approach for the Fractional
Laplacian via Deep Neural Networks
Nicolás Valenzuela(B)
Department of Mathematical Engineering, Faculty of Physical
and Mathematical Sciences, University of Chile, Santiago, Chile
nvalenzuela@dim.uchile.cl
Abstract. In this paper we consider the linear fractional elliptic problem with Dirichlet boundary conditions on a bounded and convex domain
D of IRd , with d ≥ 2. This work is devoted to the approximations of such
a PDE solutions via deep neural networks, even if the solution does not
have an analytical expression. We have used that the PDE solution has a
stochastic representation, and therefore we can use Monte Carlo approximations to generate train data related to the solution. With such a train
data we perform a stochastic gradient descent (SGD) like algorithm to
find a deep neural network that approximates the PDE solution in the
whole domain D. The DNN found will be considered to have a fixed
number of hidden layers and neurons per hidden layer. Additionally, we
provide four numerical examples with different settings on the boundary condition and source term to test the efficiency of the approximation
given by our algorithm, and each example will be studied for many values
of α ∈ (1, 2) and d ≥ 2.
Keywords: Fractional laplacian · Deep neural networks ·
Approximation
1
Introduction
1.1
Motivation
Deep Learning (DL) techniques [15] have become principal actors on the numerical approximation of partial differential equation (PDE) solutions. Among them,
linear PDEs [17,32], semi-linear PDEs [14,23,24], fully nonlinear PDEs [29,30]
and partial integrodifferential equations [16] have been the main objectives
of study. In particular, there exist numerical [3,20] and theoretical [17,23,32]
results that suggest a good performance of Deep Neural Networks (DNNs) in
the approximation of some PDE solutions. Previous references imply that DNNs
N. Valenzuela—Partially funded by Chilean research grants ANID 2022 Exploration
13220060, FONDECYT 1191412, 1231250, Basal CMM FB210005, ANID -Subdirección
de Capital Humano/Doctorado Nacional/2023-21231021 and Latin America Google
Ph.D. Fellowship Program.
c The Author(s), under exclusive license to Springer Nature Switzerland AG 2024
K. Arai (Ed.): SAI 2024, LNNS 1017, pp. 187–219, 2024.
https://doi.org/10.1007/978-3-031-62277-9_12
188
N. Valenzuela
can approximate (in suitable conditions) PDE solutions with arbitrary accuracy.
Moreover, the requested DNN do not suffer of the so-called curse of dimensionality, that is to say, the total number of parameters used to describe the DNN
is at most polynomial on the dimension of the problem and on the reciprocal of
the accuracy.
On the other hand, the fractional Laplacian has attracted considerable interest in past decades due to its vital role in some scientific disciplines such as
physics, engineering and mathematics. This operator describes phenomena that
exhibit anomalous behavior: is used to describe some diffusion and propagation
processes. Starting from the foundational work by Caffarelli and Silvestre [6],
the study of fractional problems has always required a great amount of detail
and very technical mathematics. The reader can consult the monographs by
[1,5,18,26,27].
In addition, Guilian, Raissi, Perdikaris and Karniadakis proposed in 2019 a
machine learning approach for fractional differential equations [19]. This approach comes from the fact that for linear differential equation there exists a
framework where the linear operator can be learned from Gaussian processes
[31]. The framework uses given data on the solution u and the source term f ,
assuming that both functions are Gaussian processes with some parameters that
depend on the linear operator. Once the operator is learned, it is possible to find
the Gaussian processes that describe u and f . Finally, the framework proposed
in [19] is a generalization of the linear case.
In this paper we develop an alternative algorithm for approximating solutions of the high-dimensional fractional elliptic PDE with Dirichlet boundary
condition over a bounded, convex domain D ⊂ IRd , using deep learning techniques. To do so, we treat with the unique continuous solution of the fractional
elliptic PDE. In 2018, Kyprianou, Osojnik and Shardlow proved in [26] that the
previously mentioned solution can be written in terms of the expected value of
some stochastic processes. To obtain this result the authors assume hypotheses
on the source and the boundary term, in addition with the setting on the domain
D. With this and a relation between the fractional Laplacian and stochastic processes called isotropic α-stable processes, the stochastic representation can be
found.
That stochastic representation allows to give a Monte Carlo approximation
of the solution for every point in the domain D. Each term of the Monte Carlo
operator can in turn be approximated by a deep neural network. At the end, the
average of the obtained DNNs is indeed an approximation of the solution of the
PDE, and moreover, it is also a DNN.
The advantage of this methodology is that the deep neural network found
can be evaluated in each point on IRd , and there is not necessary to generate a
Monte Carlo iteration for every point on IRd , but in some of them for the training
of the deep neural network. Furthermore, no data on the solution is needed in
this approach since all the training data is generated in the algorithm, not so in
the case of [19].
Numerical Fractional Laplacian via DNNs
1.2
189
Recent Works on DL Techniques
In the recent years different DL techniques involving DNNs and PDEs have
been studied. Some of them are: Monte Carlo methods [17,32], Multilevel Picard
(MLP) iterations [2,23,24], Forward Backward Stochastic Differential Equations
(FBSDE) [7,20,22], DeepONets [8,9,28] and Physic Informed Neural Networks
(PINNs) [10,11]. Monte Carlo methods are used to approximate linear PDE
solutions. The principal key is the Feynman-Kac representation for some PDE
solutions, where the solution can be written as the expectation of a random
variable (or a stochastic process), and it is approximated by means of such
random variable. Finally, each term of the mean can be approximated by a
suitable DNN.
On other hand, MLP iterations are used in semilinear PDE solutions. Here,
the Feynman-Kac representation can be seen as a functional with the PDE
solution as the unique fixed point of such functional. Then, via Picard iterations,
the obtained terms are approximated by DNNs. Also, the PDEs are usually
connected with stochastic differential equations by the Itô Formula. The FBSDEs
use this relation to obtain stochastic processes which are related with the PDE
solution. Then, with a deep learning based algorithm is possible to have an
approximation of the PDE solution.
Furthermore, there is also the DeepONets, which are approximations of infinite dimensional operators, for instance, the operator that given a Dirichlet
boundary condition, return the solution of a fixed PDE. The idea is to reduce the
infinite dimensional problem to a finite dimensional space. Then, make a DNN
between to finite dimensional spaces, and finally extend the DNN to the image
set of the operator, which is infinite dimensional. From this technique, one of
the most recent works is the approximation of the direct and inverse Calderón’s
mapping via DeepONets, given in [8]. Finally, but not less important, there are
the PINNs, approximations via DNNfs whose have in the loss function some
constraints that comes from the nature of the PDE (such as the differential
equation, boundary conditions, among others).
Organization of This Work. This work is organized as follows. In Sect. 2 we
introduce the necessary preliminary definitions and results for the understanding
of the numerical problem. Section 3 is devoted to the modeling of the numerical
problem, such as the simulation of the involved random variables, the generation of the training set, the use of the SGD algorithm to find an optimal deep
neural network and the errors used to evaluate the accuracy of the algorithm.
Section 4 deals with four numerical examples, each one with a rigorous analysis,
and some discussion considering the four examples. Finally, Sect. 5 is devoted to
the conclusions obtained in this paper and future related work.
190
N. Valenzuela
2
Preliminaries
2.1
Setting
PDE Setting. Along this work we will deal with the fractional Laplacian operα
ator. Let α ∈ (0, 2) and d ≥ 2. The fractional Laplacian operator (−Δ) 2 is
formally defined in IRd as:
u(y) − u(x)
− (−Δ)α/2 u(x) = cd,α lim
dy,
x ∈ IRd ,
(1)
ε↓0 IRd \B(0,ε) |y − x|d+α
α
Γ ((d+α)/2)
where B(0, ε) = {x ∈ IRd : |x| < ε}, cd,α = − 2πd/2
and Γ (·) is the
Γ (−α/2)
classical Gamma function. For d ≥ 2, let D ⊂ IRd be a convex and bounded
domain, and consider the following Dirichlet boundary value problem
(−Δ)α/2 u(x) = f (x),
x in D,
u|Dc = g,
(2)
where we consider the following assumptions on the functions f and g:
– g : Dc → IR is a continuous function in L1α (Dc ), Lg > 0, that is to say
|g(x)|
dx < ∞.
1
+
|x|d+α
Dc
(3)
– f : D → IR is a continuous function, such that
f ∈ C α+ε0 (D)
for some fixed
ε0 > 0.
(4)
Remark 1. For the approximation Theorems in Subsect. 2.2, we assume additionally that g is a Lg -Lipschitz continuous function in Assumption (3), and that f
is a Lf -Lipschitz continuous function in Assumption (4).
Under Assumptions (3), (4), Kyprianou, Osojnik and Shardlow proved in [26]
that there exists a representation formula for the solution of (2) via stochastic
processes (Xt )t≥0 called isotropic α-stable processes. The above is summarized
in the next Theorem
Theorem 1 ([26], Theorem 6.1). Let d ≥ 2 and assume that D is a bounded
domain in IRd . Suppose that g is a continuous function which belongs to L1α (Dc ).
Moreover, suppose that f is a function in C α+ε (D) for some ε > 0. Then there
exists a unique continuous solution to (2) in L1α (IRd ) given by
σD
u(x) = IEx [g(XσD )] + IEx
f (Xs )ds ,
x ∈ D,
(5)
0
where σD = inf{t > 0 : Xt ∈
/ D}.
An equivalent representation of (5) can be stated in terms of the so-called Walkon-Spheres (WoS) processes. We need to provide additional objects that will be
used in the new representation. First of all, the WoS process (ρn )n∈IN0 is defined
as follows:
Numerical Fractional Laplacian via DNNs
191
1. ρ0 = X0 ,
2. for all n ∈ IN, denote rn = dist(ρn−1 , ∂D). Let Yn be an independent copy of
XσB(0,1) . Then, ρn is defined recursively as follows
ρn = ρn−1 + rn Yn .
(6)
/ D, let N = n and stop the algorithm. Otherwise go to (2).
3. If ρn ∈
Remark 2. The following remarks about the WoS processes can be stated:
/ D}.
1. The random variable N is formally defined as N = min{n ∈ IN0 : ρn ∈
If α ∈ (0, 2) and D ⊂ IRd is a convex, bounded domain then N is a random
variable finite a.s. [26].
2. For further explanation of the WoS processes the reader can consult the work
of Kyprianou et. al. [26] and the paper of the author [32].
3. It is easier to work with the WoS process instead of the α-stable process for
the reason that we do not need to simulate the entire trajectory of (Xt )t≥0 ,
but a finite set of points of this trajectory.
The next theorem gives the distribution of XσB(0,1) , which allow us to simulate
efficiently the copies Yn in the definition of the WoS processes. This theorem
was proved in first instance in 1961 by Blumenthal, Getoor and Ray [4]
Theorem 2 ([4,26]). Let B(0, 1) the unit ball centered at the origin and write
σB(0,1) = inf{t ≥ 0 : Xt ∈
/ B(0, 1)}. Then
IP0 (XσB(0,1) ∈ dy) = π −(d/2+1) Γ (d/2) sin(πα/2)|1 − |y|2 |−α/2 |y|−d dy,
|y| > 1.
On other hand, we define the expected occupation measure of the stable process
prior to exiting a ball of radius r > 0 centered in x ∈ IRd as follows:
∞
Vr (x, dy) :=
IPx Xt ∈ dy, t < σB(x,r) dt, x ∈ IRd , |y| < 1, r > 0.
0
The next result is valid for V1 (0, dy).
Theorem 3 ([26]). The measure V1 (0, dy) is given for |y| < 1 by
−2
|y| −1
−α −d/2 Γ (d/2)
α−d
−d/2 α/2−1
V1 (0, dy) = 2 π
|y|
(u + 1)
u
du dy.
Γ (α/2)2
0
Remark 3. In [32] we obtain the following equivalent form of V1 (0, dy):
V1 (0, dy) = 2−α π −d/2
Γ (d/2)
B
Γ (α/2)2
d
α α
− ,
2
2 2
|y|α−d
d
α α
dy,
1 − I |y|2 ; − ,
2
2 2
where B(z, w) is the beta function
1
uz−1 (1 − u)w−1 du,
B(z, w) :=
0
192
N. Valenzuela
which satisfies
Γ (z)Γ (w)
,
Γ (z + w)
B(z, w) =
and I(x; z, w), x < 1 is the incomplete beta function
x
1
I(x; z, w) :=
uz−1 (1 − u)w−1 du.
B(z, w) 0
With an abuse of notation, for any bounded measurable function f denote
Vr (x, f (·)) =
f (y)Vr (x, dy).
B(x,r)
Now we are able to write the representation of u in terms the WoS process,
which proof is available in [26]:
Lemma 1 ([26]). For x ∈ D, g ∈ L1α (Dc ) and f ∈ C α+ε0 (D) we have the
representation
N
rnα V1 (0, f (ρn−1 + rn ·)) .
u(x) = IEx [g(ρN )] + IEx
(7)
n=1
Notice that we can determine a probability measure over B(0, 1) from the measure V1 (0, dy). Indeed, denote
κd,α :=
V1 (0, dy),
(8)
B(0,1)
then, μ defined as
μ(dy) := κ−1
d,α V1 (0, dy),
is a probability measure over B(0, 1). With this normalization, the representation
(7) can be written as:
N
rnα κd,α IE(μ) [f (ρn−1 + rn v)] ,
u(x) = IEx [g(ρN )] + IEx
(9)
n=1
where v is a random variable with density μ and IE(μ) represent the expectation
given by the probability measure μ.
DNN Setting. For the identification of deep neural networks, we will use the
same notation as in [23]. In particular, we consider the following setting for the
architecture of a fully connected ReLu DNN. For d ∈ IN define
Ad : IRd −→ IRd
z −→ Ad (z) = (max{z1 , 0}, ..., max{zd , 0}).
the ReLu activation function. Let also
Numerical Fractional Laplacian via DNNs
193
(NN1) H ∈ IN be the number of hidden layers;
(NN2) (ki )H+1
i=0 be a positive integer sequence;
(NN3) Wi ∈ IRki ×ki−1 , Bi ∈ IRki , for any i = 1, ..., H + 1 be the weights and
biases, respectively;
(NN4) x0 ∈ IRk0 , and for i = 1, ..., H let xi = Aki (Wi xi−1 + Bi ).
We call
H+1
Φ := (Wi , Bi )H+1
i=1 ∈
IRki ×ki−1 × IRki ,
(10)
i=1
the DNN associated to the parameters in (NN1)–(NN4). The space of all DNNs
in the sense of (10) is going to be denoted by N, namely
N=
H∈IN (k0 ,...,kH+1 )∈INH+2
H+1
IRki ×ki−1 × IRki
.
i=1
Define the realization of the DNN Φ ∈ N as
R(Φ)(x0 ) = WH+1 xH + BH+1 .
(11)
Notice that R(Φ) ∈ C(IRk0 , IRkH+1 ). For any Φ ∈ N define
H+1
P(Φ) =
kn (kn−1 + 1),
D(Φ) = (k0 , k1 , ..., kH+1 ),
(12)
n=1
and
|||D(Φ)||| = max{k0 , k1 , ..., kH+1 }.
(13)
The entries of (Wi , Bi )H+1
i=1 will be the weights and biases of the DNN, P(Φ)
represents the total number of parameters used to describe the DNN, working
always with fully connected DNNs, and D(Φ) represents the dimension of each
layer of the DNN. Notice that Φ ∈ N has H + 2 layers: H of them hidden, one
input and one output layer.
2.2
Approximation Theorem
In [32] it has been proved that the solution u of the problem (2), written as
(9), can be approximated via ReLu DNNs with arbitrary accuracy. In order
to obtain that result, additional assumptions on the functions f, g and on the
structure of the domain D need to be stated. In particular, we need the following
assumptions:
Assumption 1. Let d ≥ 2, D ⊂ IRd be a convex and bounded domain. Let
a, b ≥ 1, p ∈ (1, α), B > 0 some constants which do not depend on d. Let
g : Dc → IR satisfying (3).Assume that for any δg ∈ (0, 1], the function g can be
approximated by a ReLu DNN Φg which satisfies
1. R(Φg ) : Dc → IR is continuous, and
194
N. Valenzuela
2. The following are satisfied:
|g(y) − (R(Φg )) (y)| ≤ δg Bdp (1 + |y|)p ,
| (R(Φg )) (y)| ≤ Bdp (1 + |y|)p ,
|||D(Φg )|||
∀y ∈ Dc .
∀y ∈ Dc .
≤ Bdb δg−a .
(14)
(15)
(16)
Assumption 2. Let d ≥ 2. Let α ∈ (1, 2), a, b ≥ 1 and B > 0 as in Assumption 1. Suppose that D⊂ IRd is a convex, bounded domain that enjoys the following structure:
1. For any δdist ∈ (0, 1], the function x → dist(x, ∂D) can be approximated by
a ReLu DNN Φdist ∈ N such that
(17)
sup dist(x, ∂D) − R(Φdist ) (x) ≤ δdist ,
x∈D
and
−1
b
a
D(Φ
dist ) ≤ Bd log(δdist ) .
(18)
2. For all δα ∈ (0, 1) there exists a ReLu DNN Φα ∈ N, R(Φα ) ∈ C(IR, IR) such
that
|(R(Φα )) (x) − xα | ≤ δα ,
(19)
sup
|x|≤diam(D)
and
|||D(Φα )||| ≤ Bdb δα−a .
(20)
Moreover, R(Φα ) is a Lα -Lipschitz function, Lα > 0, for |x| ≤ diam(D).
Assumption 3. Let d ≥ 2, D ⊂ IRd be a convex and bounded domain. Let
a, b ≥ 1 and B > 0 as in Assumption 1. Let f : D → IR a function satisfying
(4). Assume that for any δf ∈ (0, 1), the function f can be approximated by a
ReLu DNN Φf ∈ N that satisfies
f -Lipschitz continuous, L
f > 0, and
1. R(Φf ) : D → IR is L
2. The following are satisfied:
|f (x) − (R(Φf )) (x)| ≤ δf ,
x ∈ D.
b −a
|||D(Φf )|||
≤ Bd δf .
(21)
(22)
With these three assumptions, one can state the following Theorem, as in [32]:
Theorem 4 ([32]). Let d ≥ 2, D ⊂ IRd be a convex and bounded
domain,
α ∈ (1, 2), a, b ≥ 1, p ∈ (1, α) as in Assumption 1 and let q ∈ 1, αp . Assume
that (3) and (4) are satisfied. Suppose that for every δα , δdist , δf , δg ∈ (0, 1)
there exist ReLu DNNs Φg , Φα , Φdist and Φf satisfying Assumptions 1, 2 and
3, respectively. Then there exist B̂, η > 0 such that for every ∈ (0, 1], the
solution of (2) can be approximated by a ReLu DNN Ψ such that the realization
R(Ψ ) : D → IR is a continuous function which satisfies:
Numerical Fractional Laplacian via DNNs
195
1. Proximity in Lq (D): If u is the solution of (2)
q
|u(x) − (R(Ψ )) (x)| dx
q1
≤ .
(23)
D
2. Bounds:
η dη −η .
P(Ψ ) ≤ B|D|
(24)
depends on f ∞ , the Lipschitz constants of g, R(Φf )
The constant B
L (D)
and R(Φα ), and on diam(D). The constant η depends on a, b, p and q.
Remark 4. This paper tries to face this problem in a numerical way. We will
propose an algorithm to find the ReLu DNN that exists from Theorem 4 without
the curse of dimensionality.
Remark 5. Convexity is necessary for the law of large numbers, in order to do a
well approximation of the solution via Monte Carlo iterations.
3
Numerical Problem Modeling
In this section we propose an algorithm that finds a ReLu Deep Neural Network
that fits the solution of (2). For this, we need preliminary work that we explain in
each subsection. In particular, we need to compute efficiently each term involved
in the representation (9).
3.1
Simulation of Random Variables
First of all, we will give an analytical value for the expression κd,α defined in
(8). The next Lemma talk about this quantity.
Lemma 2. For any d ≥ 2 and α ∈ (0, 2),
κd,α =
Γ (d/2)
21−α
.
α Γ (α/2)Γ (d/2 + α/2)
Proof. From Remark 3 and the definition of κd,α in (8) is easy to see that
d α α α
−α −d/2 Γ 2− 2
α−d
2 d
κd,α = 2 π
−
,
|y|
;
1
−
I
|y|
dy.
2
2 2
Γ α2
B(0,1)
Using spherical coordinates we have
d α 1
α α
−α −d/2 Γ 2− 2
α−d d−1
2 d
r
r
1−I r ; − ,
drdS,
κd,α = 2 π
2
2 2
Γ α2
Sd−1 0
196
N. Valenzuela
where Sd−1 is the surface area
of the
unit (d − 1)-sphere embedded in dimension
d−1 2πd/2
d. This measure is given by S = Γ d . Then
(2)
d
α
Γ( − ) 1
κd,α = 2−α π −d/2 Γ2 α 2 Sd−1 0 rd−1 1 − I r2 ; d2 − α2 , α2 dr
(2)
1 α−1 2 d α α Γ(d−α)
1
= 21−α Γ α2 Γ 2 d
−
r
I r ; 2 − 2 , 2 dr .
0
( 2 ) (2) α
1
To work with the above integral, denote J = 0 rα−1 I r2 ; d2 − α2 , α2 dr. In J we
can use integration by parts to obtain
1
α
1
1
d α α
rd−2 (1 − r2 ) 2 −1 2rdr.
J= −
α αB 2 − 2 , 2 0
Then, by change of variables r̃ = r2 ; dr̃ = 2rdr and the definition of Beta
function it follows that
1 d −1
α
1
r̃ 2 (1 − r̃) 2 −1 dr̃
J = α1 − αB d −
α α
(2 2 , 2 ) 0
B( d , α )
= α1 − αB d 2− α2 , α
(2 2 2 )
d
Γ
Γ ( d )Γ ( α )
( )
= α1 − αΓ d − α2 Γ α Γ 2d + α2 .
(2 2 ) ( 2 ) (2 2 )
From this we obtain
2
Γ d2
1
.
J= −
α αΓ d2 − α2 Γ d2 + α2
Replacing this value in κd,α we conclude that
Γ d2
21−α
.
κd,α =
α Γ α2 Γ d2 + α2
Now we will work with each random variable of the representation (9). First we
will give a scheme to simulate XσB(0,1) and find an algorithm to simulate a WoS
process (ρn )N
n=0 starting at a point x ∈ D. Then we make a scheme in order to
do the simulations of the random variables v with probability measure μ. In both
random variables, we will notice that the densities only depend on the radius of
the initial point, and then one can simulate those random variables using the
marginal radial density of each random variable.
Simulation of X σ B(0,1) . Denote by fRX the marginal radial density of XσB(0,1) .
From Theorem 2 we can obtain fRX (r) explicitly for r > 1. Indeed, for all r > 1
by spherical coordinates one has
d
α
fRX (r) = π −( 2 +1) Γ d2 sin πα
(r2 − 1)− 2 r−d rd−1 dSdr
2
Sd−1
d−1 2
d
α
= π −( 2 +1) Γ d2 sin πα
S (r − 1)− 2 r−1 dr.
2
Numerical Fractional Laplacian via DNNs
197
Then, for any r > 1
fRX (r) =
α
πα 2
2
sin
(r − 1)− 2 r−1 dr.
π
2
From the density fRX , one can calculate the distribution FRX (r) for r > 1.
Indeed
r
α
πα r 2
2
FRX (r) =
fRX (s)ds = sin
(s − 1)− 2 s−1 ds.
π
2
1
1
By a change of variables t = 1s , one obtains
1 1
− α
− 1 2 t−1 dt
FRX (r) = π2 sin πα
2
1/r t2
1
= π2 sin πα
(1 − t2 )−α/2 tα−1 dt.
2
1/r
One can do a new change of variables, of the form s = t2 to obtain
1
α/2−1
= π2 sin πα
(1 − s)1−α/2−1 s 2 ds
FRX (r)
2
1/r 2
α
B 2 , 1 − α2 1 − I r12 ; α2 , 1 − α2 ,
= π1 sin πα
2
where we have used the definitions of Beta and Incomplete Beta functions. Therefore FRX (r) can be written, for r > 1, as
πα α
α
1 α
α
1
Γ
Γ 1−
FRX (r) = sin
; ,1 −
1−I
.
π
2
2
2
r2 2
2
Notice that FRX (r) has an explicit inverse by using the well known inverse of
the Incomplete Beta function I −1 . First of all notice that for all α ∈ (0, 2), the
Euler’s reflection formula [21] is valid:
α
π
α
.
Γ 1−
=
Γ
2
2
sin πα
2
Then, FRX (r) can be simply written as
1 α
α
FRX (r) = 1 − I
; ,1 −
.
r2 2
2
Therefore,
u=1−I
α
α
1
; ,1 −
−1
2
2
2
FXR (u)
1
; α , 1 − α2
−1
FX
(u)2 2
=⇒ I
=1−u
R
=⇒ F −11(u)2 = I −1 1 − u; α2 , 1 − α2 .
XR
This implies that
−1
FX
(u) =
R
I −1
1
1 .
1 − u; α2 , 1 − α2 2
(25)
198
N. Valenzuela
The fact that FR−1
can be expressed explicitly implies that a copy of a real
X
valued random variable with density fRX can be simulated as FR−1
(U ), where
X
U ∼ Uniform(0, 1).
By the radial symmetry of the process XσB(0,1) , one can state the following
algorithm for the simulation of copies of XσB(0,1) :
Algorithm 1. Simulation of copies of XσB(0,1)
Input: α ∈ (0, 2), d ≥ 2
Output: Y a copy of XσB(0,1) ∈ IRd
Simulate U ∼ Uniform(0, 1)
Simulate Θ1 , ..., Θd−2 ∼ Uniform(0, π) and Θd−1 ∼ Uniform(0, 2π)
(U ) as in (25)
R ← FR−1
X
w1 ← cos(Θ1 )
for i = 2, ..., d − 1do
wi ← cos(Θi ) i−1
k=1 sin(Θk )
end for
d−1
wd ← k=1 sin(Θk )
w ← (w1 , ..., wd )
Y ← Rw
Remark 6. For Algorithm 1, the point w generated is a random point on
B(0, 1) ⊂ IRd , chosen uniformly.
Algorithm 1 and recursion (6) allow us to write an algorithm in order to obtain
a copy of the WoS process (ρn )N
n=0 , starting at a point x ∈ D.
Algorithm 2. Simulation of copy of WoS process starting at x ∈ D
Input: α ∈ (0, 2), d ≥ 2, x ∈ D
N
Output: (ρn )N
n=0 a copy of the WoS process, (rn )n=1 the sequence of radius
ρ0 ← x
n←0
while ρn ∈ D do
n←n+1
rn ← dist(ρn−1 , ∂D)
Simulate Y with Algorithm 1.
ρn ← ρn−1 + rn Y
end while
N ←n
Simulation of v . Denote by fRv the marginal radial density of the random
variable v with probability measure μ. From Theorem 3 and Remark 3 we obtain
Numerical Fractional Laplacian via DNNs
199
fRv (r) explicitly for r < 1. Indeed, for all r < 1, by spherical coordinates one
has
−α
−d
π 2
fRv (r) = 2 κd,α
−α
Γ(d
2)
Γ(
d
π− 2
= 2 κd,α
α
2
)
2
d
α α
2 − 2, 2
B
Γ(d
2)
2
Γ(α
2)
B
Sd−1
d
α α
2 − 2, 2
rα−d 1 − I r2 ; d2 − α2 , α2 rd−1 dSdr
|Sd−1 |rα−1 1 − I r2 ; d2 − α2 , α2 dr.
d
Recall the value of κd,α from Lemma 2 and |Sd−1 | = Γ2πd2 . Therefore
(2)
Γ(d+α)
fRv (r) = α Γ d2 Γ 2α B d2 − α2 , α2 rα−1 1 − I r2 ; d2 − α2 , α2 dr
(2) ( 2 )
B( d − α , α )
= α B2 d ,2α 2 rα−1 1 − I r2 ; d2 − α2 , α2 dr.
(2 2 )
Now we can calculate the distribution FRv (r) for r < 1. Indeed
FRv (r) =
r
0
fRV (s)ds = α
B
d
−α
,α
2
2 2
B
d α
,
2 2
α
r
r
α α
d
−
ds .
sα−1 I s2 ; − ,
α
2
2 2
0
r
Denote J1 = 0 sα−1 I s2 ; d2 − α2 , α2 ds. We integrate by parts J. Following the
α
notation u = I s2 ; d2 − α2 , α2 , with du = B d −2 α , α sd−α−1 (1 − s2 ) 2 −1 ds and
(2 2 2 )
α
dv = sα−1 ds with v = sα , one obtains
r
α
rα
d α α
2
d α α
I r2 ; − ,
sd−1 (1 − s2 ) 2 −1 ds.
J1 =
−
α
2
2 2
αB 2 − 2 , 2 0
r
α
Denote J2 = 0 sd−1 (1 − s2 ) 2 −1 ds. With a change of variables u = s2 , J2 can
be written as
2
B d2 , α2
α
1 r d −1
d α
−1
I r2 ; ,
u 2 (1 − u) 2 du =
J2 =
.
2 0
2
2 2
Therefore
J1 =
rα
d α α
d α
1 B d2 , α2
d α α I r2 ; ,
I r2 , − ,
−
.
α
2
2 2
α B 2 − 2, 2
2 2
Finally, for r < 1, the distribution FRv (r) is
d α α d α α
2 d α
αB 2 − 2, 2
d α
FRv (r) = I r ; ,
1 − I r2 ; − ,
+r
.
2 2
2
2 2
B 2, 2
. Instead, by
Due the form of FRv (r), it is difficult to calculate its inverse FR−1
v
the knowledge of fRv and FRv we will simulate random variables with density
fRv using the Newton-Raphson method:
200
N. Valenzuela
Algorithm 3. Newton-Raphson algorithm
Input: r, δ ∈ (0, 1)
Output: R a copy of a random variable with density fRv
Simulate U ∼ Uniform(0, 1)
R←r
while |FRv (R) − U | ≥ δ do
F
(R)−U
R ← R − RfvR (R)
v
end while
It is important to recall that the tolerance δ must be small to have an
improved copy. For the next numerical examples, we set δ = 10−3 . The simulation of copies of a random variable V with density μ is a quite similar to
the case of XσB(0,1) . We will use the same way to generate the random vector
w ∈ IRd and the fact that the copy of v is the multiplication of R and w, with
the difference that R will be a copy obtained from Newton-Raphson algorithm.
3.2
Monte Carlo Training Set Generation
To obtain train data, we will use Monte Carlo simulations. Let M, P ∈ IN.
Choose randomly P points in a set containing D, namely (xk )P
k=1 . Each value
u(xk ) will be approximated by a Monte Carlo simulation with M iterations. The
advantage of use Monte Carlo in the train data is that we can create the data
even when the solution u is unknown. Denote by D the training set. D can be
written as
D = {(xk , ûk ) : k ∈ {1, ..., P }} ⊂ IRd × IR,
(26)
where for each k = 1, ..., P , ûk has the form
⎧
Ni
M
⎪
i i
⎨ 1
α
i
g ρNi +
κd,a rn f ρn + rn vi
ûk = M
n=1
i=1
⎪
⎩
g(xk )
Here,
i
(ρin , rni )N
n=0
if xk ∈ D,
(27)
if xk ∈
/ D.
M
i=1
are M copies of the WoS process starting at xk , with
their respective radius, and (vi )M
i=1 are M copies of the random variable v with
probability measure μ.
3.3
DNN Approximation
Recall from Theorem 4 the existence of a ReLu DNN that approximates the
solution of PDE (2). This DNN may have different size of the hidden layers H
or vector D(Φ) for different settings of the problem (2).
In the next simulations, in order to simplify the numerical computation, we
set a fixed number of hidden layers H and their dimensions. We will find the
optimal DNN that approximates the solution u of problem (2) for the previous
Numerical Fractional Laplacian via DNNs
201
conditions. In order to do this, we set an initial ReLu DNN Φ, and we train that
DNN using the stochastic gradient descent (SGD) method. The training of the
DNN is made with a loss function that depends on the training set defined in
(26). In the Sections below we talk about each one in detail.
DNN Generation. Under the notation in Sect. 2.1, we fix H = 7 with ki = 110
for all i = 1, ..., H. This will be the number of hidden layers and the dimension
of such layers. We set also k0 = d to be the dimension of the input layer and
kH+1 = 1 to be the dimension of the output layer. With this setting, we generate
an initial ReLu DNN Φ = ((Wi , bi ))8i=1 , where
– W1 ∈ IR110×d , b1 ∈ IR110 ;
– For all i = 1, .., 7, Wi ∈ IR110×110 , bi ∈ IR110 ;
– W8 ∈ IR1×110 , b8 ∈ IR.
The values of (Wi , bi )8i=1 are chosen in an arbitrary way, and they will become
parameters of an optimization problem. The generation of the ReLu DNN in
Python is made with the Tensorflow library.
SGD Algorithm Implementation. In order to find the optimal values of the
parameters (Wi , bi )8i=1 we perform a SGD algorithm. Consider a training set
with the form of (26) of size |D| = P ∈ IN and let NIter ∈ IN, γ ∈ (0, 1) be the
number of iterations of the SGD algorithm and its learning rate, respectively.
For each j = 1, ..., NIter take Sj an arbitrary subset of D with fixed size |Sj | =
L
L < P , and consider the mean square error between (ûl )L
l=1 and ((R(Φ)) (xl ))l=1 ,
where ((xl , ûl ))L
l=1 are the elements of Sj , that is
Loss
Φ, (xl , ûl )L
l=1
1
=
L
L
2
((R(Φ)) (xl ) − ûl ) .
(28)
l=1
In each step j = 1, ..., NIter , the values of weights and biases (Wi , bi )8i=1 are
updated via the minimization of the loss function (28). We will do the minimization step with the optimizer ADAM implemented in the Tensorflow library.
We can state therefore the following SGD meta-algorithm in order to obtain
an optimal neural network Φ∗ .
202
N. Valenzuela
Algorithm 4. SGD algorithm with batches
Input: Niter , M, P ∈ IN, γ ∈ (0, 1), L < P , L ∈ IN
Output: Φ∗ := (Wi∗ , b∗i )8i=1
Create D of size P as equal as in (26)
Initialize Φ := (Wi , bi )8i=1
for j = 1, .., NIter do
– Denote Sj ⊂ D as a set of L random points of the training set,
– Calculate the loss function as in (28) with the data set Sj ,
– Update (Wi , bi )8i=1 via the minimization of the loss function using the ADAM
optimizer.
end for
Φ∗ ← (Wi , bi )8i=1
Loss Function for Radial Solutions. For the cases where the solution is
radial, i.e., it only depends on |x|, we perform a reinforcement of the loss function,
by adding a term that controls the radiality on the realization of the obtained
DNN. The loss function we will consider for the cases where the solution is radial
is
Loss
Φ, (xl , ûl )L
l=1
1
=
2L
L
2
2
((R(Φ)) (xl ) − ûl ) +((R(Φ)) (xl ) − (R(Φ)) (−xl )) .
l=1
(29)
3.4
Method’s Error Estimates
In the next numerical examples, we will work with PDEs which have explicit
solutions v ∈ C(IRd , IR). For the quantification of the following errors, we sample 5000 points xk ∈ D or near the boundary ∂D. Each point will have the
corresponding solution v(xk ).
We will compare the optimal neural network Φ∗ with the solution of the
sampled points in the following two ways:
1. Mean Square Error:
M SE(Φ∗ , (xk , v(xk ))5000
k=1 ) =
1
5000
5000
((R(Φ∗ ))(xk ) − v(xk ))2 .
(30)
k=1
2. Mean Relative Error:
M RE(Φ∗ , (xk , v(xk ))5000
k=1 ) =
1
5000
5000
k=1
|(R(Φ∗ ))(xk ) − v(xk )|
.
v(xk )
(31)
This comparison is made in order to study the accuracy of the trained neural
network obtained with the SGD algorithm in each example.
Numerical Fractional Laplacian via DNNs
4
203
Numerical Examples
All the numerical examples will be performed in Python on a 64-bit MacBook
Pro M2 (2022) with 8 GB of RAM. The codes are available in the Github [33].
In this paper we provide four different examples of the fractional Dirichlet
problem, and they differs in the settings of f and g. Each example will be considered over the domain D = B(0, 1) ⊂ IRd , for different values of the dimension
d. Every figure and table of those examples are shown in the Appendix.
In addition, in each example the points xk of the training set D are chosen
such that |xk | ≤ 1.5, so we are sampling points on D or outside D but near the
boundary ∂D.
4.1
Example 1: Constant Source Term
Let d ≥ 2. Consider the following problem
Γ α2 + d2 Γ α2 + 1
α
α
(−Δ) 2 u(x) = 2
,
Γ d2
x ∈ B(0, 1),
u|B(0,1)c = 0.
This problem has an explicit continuous solution u given by [12].
2
α
u(x) = (1 − x )+2 ,
(32)
where (·)+ := max{0, ·}. The solution (32) is radial, then we use the loss function
defined in (29) in the SGD algorithm. For this example we may consider three
different values of d: d = 2, 5, 15. Also, we will set NIter = 1000, P = 2000,
L = 400, M = 100 and a learning rate γ = 5 × 10−3 .
First we set d = 2. In Fig. 1 we compare the realization of the optimal DNN
with the solution (32) for α = 0.1, 1, 1.9 in a grid of 5000 points between 0 and
√1 + 0.1. The x-axis is the value of both components x1 = x2 . In Fig. 2 we do
2
the comparative in 3D in a grid of 1000 × 1000 points in [−1, 1] × [−1, 1]. The
x-axis is the value of x1 and the y-axis is the value of x2 .
In both Figs. 1 and 2 we see that the optimal neural network fits pretty well
the solution (32). For the case of α = 0.1 the algorithm takes 22.02 s to obtain
the optimal DNN. When α = 1 it takes 21.38 s and for the case α = 1.9 it takes
1 min and 23.11 s.
In Fig. 1 the most visible difference between the DNN and the solution (32 is
near the boundary ∂B(0, 1), principally in the cases α = 0.1 and α = 1.0. This
can also be seen in Fig. 2, in the points where one component is near zero and
the other component is near 1 or −1.
Now we set d = 5. Figure 3 shows the comparative between the optimal DNN
and the solution (32) for α = 0.1, 1, 1.9 in a grid of 5000 points between 0 and
√1 + 0.1. The x-axis is the value of the components x1 = ... = x5 . Figure 4 shows
5
the comparative in 3D in a grid of 1000×1000 points in [−1, 1] × [−1, 1]. The
x-axis is the value of x1 = ... = x4 and the y-axis is the value of x5 .
204
N. Valenzuela
As we can see in Fig. 3 the approximation via DNN is good. The results have
the most visible difference between the prediction and the solution is near the
boundary ∂B(0, 1) for the three cases. In Fig. 4 we see the same behavior that
in the case d = 2, that is, near 0 in the x-axis and near {−1, 1} in the y-axis
there are a notable difference between the optimal DNN and the solution (32).
In order to improve the approximation, we perform a study of the algorithm
when we increase the number of iterations M of the Monte Carlo simulation and
the size of the training set P . In particular, for the same values of α as before,
we calculate the elapsed time of the algorithm when M = 10, 100, 1000, 2000 and
P = 10, 100, 1000, 2000. The results are shown in seconds in Tables 1a, b and c.
As we can see in the three tables, the setting that takes the longest elapsed
time is when M = 2000 and P = 2000. This seems reasonable because we need
to do a more expensive Monte Carlo simulation for a larger number of points in
IR5 .
Another relationship between elapsed time and PDE parameters can be seen
in Tables 1a, b and c. In particular, for small values of α the algorithm takes less
time than when α is larger.
Also, from Tables 1a, b and c, we can deduce that for fixed M , the value of
the elapsed time when P increases is similar to the value of the elapsed time
when M increases and P is fixed.
Now, for the same values of M and P , and for α = 0.1, 0.5, 1, 1.5, 1.9 the
Fig. 5 shows the mean square error calculated as in (30 for 5000 random points
in D. On the other hand, Fig. 6 shows the mean relative error calculated as in
(31) for the same 5000 points in D.
Notice from Fig. 5 that for small values of α, one needs big values of P . For
example, when α = 0.1, P = 1000 is not enough to have a mean square error
of the order of 10−4 , even with M = 2000. On the other hand, when α = 1.9,
P = 1000 gives the order of 10−4 on the mean square error for M ≥ 100. Also
notice that, despite that the high elapsed time when P = 2000, in almost all
the choices of M and α, this setting ensures a mean square error of the order of
10−3 , and the order of 10−4 if M is greater than 100.
In Fig. 6 we see that for the mean relative error is more difficult to obtain a
value small order. In particular, only for P ≥ 1000 we can ensure an error lower
than the order 10−1 for almost all the choices of M and α. The excluding case is
when α = 1.9, where the errors are greater than 10−1 , but they are still in that
order.
Finally, we set d = 15 and we change the value of γ to 5 × 10−4 . In Fig. 7
we compare the realization of the optimal DNN with the solution (32) for α =
0.1, 1, 1.9 in a grid of 5000 points between 0 and √115 + 0.1. The x-axis is the
value of the components x1 = ... = x15 . In Fig. 8 we do the comparative in 3D
in a grid of 1000×1000 points in [−1, 1] × [−1, 1]. The x-axis is the value of
x1 = ... = x14 and the y-axis is the value of x15 .
Notice from Fig. 7 that the behavior is a quite similar that when d = 5. In
the sense that for all the values of α, the optimal DNN is not a good prediction
of the solution near the boundary ∂B(0, 1), specially when α = 0.1 and α = 1.9.
Numerical Fractional Laplacian via DNNs
205
Despite the not-so-good approximation mentioned above, in Fig. 8 we see that
the optimal DNN still preserves the solution’s form.
Comparison of Loss Functions. Recall from (32) the radiality of the solution in this example. As said before, in previous figures we perform the SGD
algorithm with the loss function defined in (29). In this section we compare the
optimal DNNs of the SGD algorithm for the two different losses described in
Sect. 4. The comparison will be made for α = 0.1, 1, 1.9, d = 15, γ = 5 × 10−4 ,
P = 2000, L = 400 and M = 100.
As we can see in Fig. 9, the performance of the SGD algorithm is better when
we are using the loss function (29) instead of function (28). Despite there are
not so much difference between the errors obtained with both loss functions, in
the visualization on Fig. 9 is remarkable the use of the loss function with radial
term.
4.2
Example 2: Non Constant Source Term
Let d ≥ 2. Consider the following problem
α
α 2
|x| , x ∈ B(0, 1),
(−Δ) 2 u(x) = bα,d 1 − 1 +
d
where
αΓ
bα,d = 2
α
d
2 + 2
Γ
Γ
α
d
u|B(0,1)c = 0, (33)
2 +2
.
2
This problem has an explicit continuous solution u given by [13].
2 1+
α
u(x) = (1 − x )+ 2 .
For this example, is also radial, then the loss function (29) will be used. We
consider d = 2 and d = 5. For both dimensions, we set M = 500, P = 1000,
L = 200, γ = 5 × 10−3 and NIter = 1000.
First we set d = 2. In Fig. 10 we compare the realization of the optimal DNN
with the solution (33) for α = 0.5, 1.5, 1.9 in a grid of 5000 points between 0 and
√1 + 0.1. Figure 11 we do the comparative in 3D in a grid of 1000×1000 points
2
in [−1, 1] × [−1, 1].
As in Example 1, the optimal DNN fits pretty well the solution (33) when
d = 2, for all studied cases of α, but unlike in previous example, in this case there
are no problems near the boundary and there are not major visible differences
between the optimal DNN and the solution (33). On the other hand, Fig. 11
shows that the optimal DNN has the same order and the same form of the
analytical solution.
For d = 2 we compare the elapsed time for different choices of M and P . This
was made for α = 0.5 (Table 2a), α = 1.5 (Table 2b) and for α = 1.9 (Table 2c).
In those tables we can see that the elapsed time increases when we increase the
206
N. Valenzuela
value of M and P . For fixed values of M and P the elapsed time also increases
when we increases the value of α, but the major gap is for larger values of M
and P .
Finally we compare the mean square error and the mean relative error in
a grid of the Monte Carlo iterations M , for each α ∈ {0.1, 0.5, 1, 1.5, 1.9} and
P = 10, 100, 1000, 2000. The results are summarized in Figs. 12 and 13.
Figure 12 shows that the mean square error decreases when we increases P
and M , achieving the order 10−4 for the cases P = 1000 and P = 2000 when
M ≥ 100 and for any value of α. In Fig. 13 we see a pretty similar behavior of
the mean relative error than the MSE, but with different order: for P = 1000
and P = 2000, we have an error of order 10−1 only for small values of α, α ≤ 1,
and for α > 1, only in the case M = 1000. For the other cases, the MRE is
greater than 1. In other words, the mean square error is small for all the studied
values for α, but only for small values of α we reach a small value on the mean
relative error.
Now we set d = 5. We will compare the optimal DNN and the solution (33)
in 2D (Fig. 14) and in 3D (Fig. 15) for α = 0.5, 1.5, 1.9. This will be done in the
same way as in Example 1. In both the figures, we can see a good approximation
of the solution (33) with the optimal DNN obtained with the SGD algorithm. In
the three settings of α the most visible difference of the approximation is near
the center of the domain, i.e., near 0, and near the boundary ∂B(0, 1), specially
in the cases α = 1.5 and α = 1.9. Moreover, Fig. 15 shows that the form of the
optimal DNN is the same of the solution (33).
4.3
Example 3: Non Zero Boundary Term
Let d ≥ 2 and recall that D = B(0, 1). For y ∈ Dc fixed let the boundary condition g be a translation of the fundamental solution of the fractional Laplacian,
that is,
Γ d2 − α2
g(x) =
|x − y|α−d .
d
2α π 2 Γ α2
Now consider the problem
α
(−Δ) 2 u(x) = 0,
x ∈ B(0, 1),
u|B(0,1)c = g.
(34)
From the choice of y outside B(0, 1), one obtains that one exact solution for
previous problem is u(x) = g(x) for every x ∈ IRd . In this particular case, we
consider the vector y = (2, 0, ..., 0) ∈ IRd .
For this example, we will set the values M = 300, P = 1000, L = 200,
γ = 5 × 10−3 and NIter = 1000. First we study the algorithm in dimension 2.
Figure 16 shows the comparison of the optimal DNN and the solution of problem
(34) for α = 0.5, 1.5, 1.9 in a grid made in the same way as in previous examples.
Figure 17 shows the comparison in 3D.
From Fig. 16 we can see a poorly approximation of the optimal DNN when
α = 0.5. The approximation is improved by increasing α. By de order of the
Numerical Fractional Laplacian via DNNs
207
solution of (34), we see that the optimal DNN still preserves the form of the
analytic solution in the square [−1, 1] × [−1, 1].
Now we study the elapsed time by changing the values of M and P for
α = 0.5, 1.5, 1.9. The elapsed times are shown in Tables 3a, b and c.
From those values of M and P , and for α = 0.1, 0.5, 1, 1.5, 1.9 we calculate
the mean square error and the mean relative error of the optimal DNN. These
errors are in Figs. 18 and 19.
Figure 18 shows a small mean square error in all settings of M , P and α, but
this small error is a bit tricky, because for small values of α, the order of the
solution is also small.
The errors shown in Fig. 19 are very different of errors in Fig. 18. Figure 19
shows an error near of (or lower than) the order 100 for the cases α = 0.1, 0.5.
When α = 1, for larger values of M and P we have an error lower than the order
10−2 , but in the other cases, the error is also near of the order 10−1 . The lowest
errors are in the cases of α = 1.5 and α = 1.9. The error is of the order 10−3 for
large values of M and P , and near of the order 10−2 for small values of M and
P.
4.4
Example 4: A Counter Example for the Algorithm
Let d ≥ 2 and D = B(0, 1). Consider the following problem
d
α
(−Δ) 2 u(x) = 0,
x ∈ B(0, 1),
u|B(0,1)c =
xi
(35)
i=1
Property 2 in [25] ensures that every harmonic function must be α-harmonic for
all α ∈ (0, 2). In particular, the function
d
g(x) =
xi ,
x∈
/ B(0, 1),
i=1
is harmonic, and then it is α-harmonic. Therefore, for all α ∈ (0, 2), the solution
of problem (35) is
d
xi ,
u(x) =
x ∈ IRd .
i=1
For this Example we only consider dimension 2. The performances of the algorithm are similar for different choices of d. Figure 20 shows a comparison between
the optimal DNN and the solution of problem (35) in two dimensions. Figure 21
shows the same comparison, but in three dimensions.
We can notice from Fig. 20 and 21 that for almost all the settings of α, the
algorithm does not perform a good approximation of the solution of problem
(35). We only have a well approximation when α is near 2.
208
4.5
N. Valenzuela
Discussion
This section is devoted to do some comments about our algorithm and the examples proposed. First of all we highlight the behavior of the WoS process and we
compare this behavior with our results. For small values of α, the isotropic αstable process exits far from the boundary ∂B(0, 1). This implies that the WoS
process exits the set D in a few steps, that means, the value of N is also small.
On the other hand, when α is near 2 (i.e. α is large), the isotropic α-stable process exits very close to the boundary ∂B(0, 1). This means that the WoS process
needs more steps to exit the set D. In other words, the value of N is large. This
makes sense, because in the case α = 2, one can see in [17,26] that the WoS
process in this case never exits the set D, and then the value of N is infinite a.s.
In terms of time, we note that the previously mentioned behavior of the WoS
process has a strong influence in the elapsed time of the SGD algorithm. Indeed,
we can see in the different tables on each example that for the same values of P
and M , the algorithm takes more time for large values of α than for small ones.
Notice also that in all the examples, there is a good approximation for large
values of α. In the case of small values of α, the approximation has some issues,
specially in the examples where the boundary condition is non-zero. This problem is caused principally due the same behavior of the WoS process: If α is small,
the last element in the WoS process, namely ρN , is a point far away the set D,
and if the boundary condition is non-zero, the Monte Carlo iterations have the
terms involving g(ρN ). This leads to many errors if a small number of Monte
Carlo iterations are considered, that is our case. This has not happened in [26],
because the order of the Monte Carlo iterations in that paper is much higher
(210 iterations) compared to the order considered in this paper (maximum 2000
iterations).
5
Conclusions
In this section we talk about conclusions about our proposed algorithm. With
this method we can estimate the value of the solution of the fractional PDE in
every point of IRd once the deep neural network is trained, contrary to the case
of Monte Carlo, where we need to do a different Monte Carlo realization for each
point of IRd , and that is an expensive computation. The same happens in the
case of finite differences, where we know the value of the solution only on a grid,
and the computational cost is larger the finer grid is. For this method, and in the
examples presented, we only needed a small number of Monte Carlo iterations,
for a given set of points on IRd , with maximum size of 2000.
Although in all examples we perform a good approximation of the PDE
solution, we only focussed in the cases where an analytical solution can be found
(principally to ensure that the approximation via DNNs is indeed near the PDE
solution). Those examples restrict our PDE in the case where the domain is
the unitary ball. A potentially next step of this work is to give a setting where
the domain is any bounded and convex set, and compare the efficiency of the
Numerical Fractional Laplacian via DNNs
209
approximation with another approximation method (e.g. finite differences or
finite elements in low dimensions). Finally, we recall that we only considered
DNNs having 7 hidden layers and 110 neurons per layer. An interesting question
is if one can let the hidden layers and neurons to be variables in our model, in
order to obtain an optimal DNN with an optimal number of hidden layers and
neurons, that may be distinct for different settings of the PDE.
Appendix
A
Figures
Fig. 1. Example 1: Two dimensional comparison of DNN and exact solution when
d = 2.
Fig. 2. Example 1: Three dimensional comparison of DNN and exact solution for d = 2.
In each Fig., left: exact solution, right: DNN solution.
210
N. Valenzuela
Fig. 3. Example 1: Two dimensional comparison of DNN and exact solution when
d = 5.
Fig. 4. Example 1: Three dimensional comparison of DNN and exact solution for d = 5.
In each Fig., left: exact solution, right: DNN solution.
Fig. 5. Example 1: Mean square error against number of Monte Carlo iterations M for
5 values of α.
Numerical Fractional Laplacian via DNNs
211
Fig. 6. Example 1: Mean relative error against number of Monte Carlo iterations M
for 5 values of α.
Fig. 7. Example 1: Two dimensional comparison of DNN and exact solution when
d = 15.
Fig. 8. Example 1: Three dimensional comparison of DNN and exact solution for d =
15. In each Fig., left: exact solution, right: DNN solution.
212
N. Valenzuela
Fig. 9. Example 1: Two dimensional comparison of two optimal DNN solutions with
different loss functions, for d = 15.
Fig. 10. Example 2: Two dimensional comparison of DNN and exact solution when
d = 2.
Fig. 11. Example 2: Three dimensional comparison of DNN and exact solution for
d = 2. In each Fig., left: exact solution, right: DNN solution.
Numerical Fractional Laplacian via DNNs
213
Fig. 12. Example 2: Mean square error against number of Monte Carlo iterations M
for 5 values of α.
Fig. 13. Example 2: Mean relative error against number of Monte Carlo iterations M
for 5 values of α.
214
N. Valenzuela
Fig. 14. Example 2: Two dimensional comparison of DNN and exact solution when
d = 5.
Fig. 15. Example 2: Three dimensional comparison of DNN and exact solution for
d = 5. In each Fig., left: exact solution, right: DNN solution.
Fig. 16. Example 3: Two dimensional comparison of DNN and exact solution when
d = 2.
Fig. 17. Example 3: Three dimensional comparison of DNN and exact solution for
d = 2. In each Fig., left: exact solution, right: DNN solution.
Numerical Fractional Laplacian via DNNs
215
Fig. 18. Example 3: Mean square error against number of Monte Carlo iterations M
for 5 values of α.
Fig. 19. Example 3: Mean relative error against number of Monte Carlo iterations M
for 5 values of α.
216
N. Valenzuela
Fig. 20. Example 4: Two dimensional comparison of DNN and exact solution when
d = 2.
Fig. 21. Example 4: Three dimensional comparison of DNN and exact solution for
d = 2. In each Fig., left: exact solution, right: DNN solution.
B
Tables
Table 1. Example 1: Elapsed time in seconds of the SGD algorithm with d = 5
M\P 10
100
1000
2000
M\P 10
100
1000
2000
10
5.44 5.62
8.35
10.49
10
5.50 5.76
8.67
10.90
100
5.45 6.25
14.13
22.26
100
5.54 6.56
15.13
23.54
1000 5.94 12.46 70.90
139.47
1000 6.47 13.32 79.53
151.38
2000 6.61 17.29 140.75 274.69
2000 7.00 20.19 151.30 294.83
(a) α = 0.1
(b) α = 1.0
M\P 10
100
1000
2000
10
5.60
6.72
17.86
29.27
100
6.09
14.40
101.61
203.21
1000 18.27 101.77 981.54
2012.45
2000 22.54 197.30 2012.85 5556.19
(c) α = 1.9
Numerical Fractional Laplacian via DNNs
217
Table 2. Example 2: Elapsed time in seconds of the SGD algorithm with d = 2.
M\P 10
100
1000
2000
M\P 10
10
5.44 5.62
100
5.45 6.25
100
1000
8.35
10.49
14.13
22.26
1000 5.94 12.46 70.90
139.47
2000
10
5.50 5.76
8.67
10.90
100
5.54 6.56
15.13
23.54
1000 6.47 13.32 79.53
151.38
2000 6.61 17.29 140.75 274.69
2000 7.00 20.19 151.30 294.83
(a) α = 0.1
(b) α = 1.0
M\P 10
100
1000
2000
10
5.60
6.72
17.86
29.27
100
6.09
14.40
101.61
203.21
1000 18.27 101.77 981.54
2012.45
2000 22.54 197.30 2012.85 5556.19
(c) α = 1.9
Table 3. Example 3: Elapsed time in seconds of the SGD algorithm with d = 2.
M\P 10
100
1000
2000
M\P 10
10
5.44 5.62
100
5.45 6.25
100
1000
8.35
10.49
14.13
22.26
1000 5.94 12.46 70.90
139.47
2000
10
5.50 5.76
8.67
10.90
100
5.54 6.56
15.13
23.54
1000 6.47 13.32 79.53
151.38
2000 6.61 17.29 140.75 274.69
2000 7.00 20.19 151.30 294.83
(a) α = 0.1
(b) α = 1.0
M\P 10
100
1000
2000
10
5.60
6.72
17.86
29.27
100
6.09
14.40
101.61
203.21
1000 18.27 101.77 981.54
2012.45
2000 22.54 197.30 2012.85 5556.19
(c) α = 1.9
References
1. Acosta, G., Borthagaray, J.P.: A fractional Laplace equation: regularity of solutions
and finite element approximations. SIAM J. Numer. Anal. 55(2), 472–495 (2017)
2. Beck, C., Hornung, F., Hutzenthaler, M., Jentzen, A., Kruse, T.: Overcoming the
curse of dimensionality in the numerical approximation of Allen-Cahn partial dif-
218
N. Valenzuela
ferential equations via truncated full-history recursive multilevel Picard approximations. J. Numer. Math. 28(4), 197–222 (2020)
3. Berner, J., Grohs, P., Jentzen, A.: Analysis of the generalization error: empirical risk minimization over deep artificial neural networks overcomes the curse of
dimensionality in the numerical approximation of Black-Scholes partial differential
equations. SIAM J. Math. Data Sci. 2(3), 631–657 (2020)
4. Blumenthal, R., Getoor, R., Ray, D.: On the distribution of first hits for the symmetric stable processes. Trans. Am. Math. Soc. 99(3), 540–554 (1961). https://
doi.org/10.2307/1993561
5. Bonito, A., Borthagaray, J.P., Nochetto, R.H., Otárola, E., Salgado, A.J.: Numerical methods for fractional diffusion. Comput. Vis. Sci. 19(5), 19–46 (2018)
6. Caffarelli, L., Silvestre, L.: An extension problem related to the fractional Laplacian. Comm. Partial Differ. Eq. 32(7–9), 1245–1260 (2007)
7. Castro, J.: Deep Learning schemes for parabolic nonlocal integro-differential equations. Partial Differ. Equ. Appl. 3(6), 77 (2022)
8. Castro, J., Muñoz, C., Valenzuela, N.: The Calderón’s problem via DeepONets.
arXiv preprint arXiv:2212.08941 (2022)
9. Chen, T., Chen, H.: Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems.
IEEE Trans. Neural Netw. 6(4), 911–917 (1995)
10. De Ryck, T., Mishra, S.: Error analysis for physics-informed neural networks
(PINNs) approximating Kolmogorov PDEs. Adv. Comput. Math. 48(6), 79 (2022).
https://doi.org/10.1007/s10444-022-09985-9
11. De Ryck, T., Jagtap, A.D., Mishra, S.: Error estimates for physics informed
neural networks approximating the Navier-Stokes equations. arXiv preprint
arXiv:2203.09346 (2022)
12. Dyda, B., Kuznetsov, A., Kwaśnicki, M.: Fractional Laplace operator and Meijer
G-function. Constr. Approx. 45(3), 427–48 (2016)
13. Dyda, B.: Fractional calculus for power functions and eigenvalues of the fractional
Laplacian. Fractional Calc. Appl. Anal. 15(4), 536–555 (2012). https://doi.org/10.
2478/s13540-012-0038-8
14. Weinan, E., Han, J., Jentzen, A.: Deep learning-based numerical methods for highdimensional parabolic partial differential equations and backward stochastic differential equations. Commun. Math. Stat. 5(4), 349–380 (2017)
15. Elbrächter, D., Perekrestenko, D., Grohs, P., Bölcskei, H.: Deep neural network
approximation theory. IEEE Trans. Inf. Theor. 67(5), 2581–623 (2021). https://
doi.org/10.1109/tit.2021.3062161
16. Gonon, L., Schwab, C.: Deep ReLU neural networks overcome the curse of dimensionality for partial integrodifferential equations. Anal. Appl. 21(01), 1–47 (2022)
17. Grohs, P., Herrmann, L.: Deep neural network approximation for high-dimensional
elliptic PDEs with boundary conditions. IMA J. Numer. Anal. (2021). https://doi.
org/10.1093/imanum/drab031
18. Gulian, M., Pang, G.: Stochastic solution of elliptic and parabolic boundary value
problems for the spectral fractional Laplacian. arXiv preprint arXiv:1812.01206
(2018)
19. Gulian, M., Raissi, M., Perdikaris, P., Karniadakis, G.: Machine learning of
space-fractional differential equations. SIAM J. Sci. Comput. 41(4), A2485–A2509
(2019). https://doi.org/10.1137/18m1204991
20. Han, J., Jentzen, A., Weinan, E.: Solving high-dimensional partial differential
equations using deep learning. Proc. Nat. Acad. Sci. 115(34), 8505–8510 (2018).
https://doi.org/10.1073/pnas.1718942115
Numerical Fractional Laplacian via DNNs
219
21. Havil, J.: Gamma: Exploring Euler’s Constant. Princeton University Press (2003)
22. Huré, C., Pham, H., Warin, X.: Deep backward schemes for high-dimensional nonlinear PDEs. Math. Comp. 89(324), 1547–1579 (2020)
23. Hutzenthaler, M., Jentzen, A., Kruse, T., Nguyen, T.A.: A proof that rectified deep
neural networks overcome the curse of dimensionality in the numerical approximation of semilinear heat equations. Partial Differ. Equ. Appl. 1, 10 (2020)
24. Hutzenthaler, M., Jentzen, A., Kruse, T., Nguyen, T.A., Von Wurstemberger, P.:
Overcoming the curse of dimensionality in the numerical approximation of semilinear parabolic partial differential equations. Proc. R. Soc. A Math. Phys. Eng.
Sci. 476(2244), 20190630 (2020)
25. Itô, M.: On α-harmonic functions. Nagoya Math. J. 26, 205–221 (1966)
26. Kyprianou, A.E., Osojnik, A., Shardlow, T.: Unbiased ‘walk-on-spheres’ Monte
Carlo methods for the fractional Laplacian. IMA J. Numer. Anal. 38(3), 1550–
1578 (2018). https://doi.org/10.1093/imanum/drx042
27. Lischke, A., et al.: What is the fractional Laplacian? arXiv preprint
arXiv:1801.09767 (2018)
28. Lu, L., Jin, P., Karniadakis, G.E.: Learning nonlinear operators via DeepONet
based on the universal approximation theorem of operators. Nat. Mach. Intell.
3(3), 218–229 (2021). https://doi.org/10.1038/s42256-021-00302-5
29. Lye, K.O., Mishra, S., Ray, D.: Deep learning observables in computational fluid
dynamics. J. Comput. Phys. 410, 109339 (2020)
30. Lye, K.O., Mishra, S., Ray, D., Chandrashekar, P.: Iterative surrogate model optimization (ISMO): an active learning algorithm for PDE constrained optimization
with deep neural networks. Comput. Meth. Appl. Mech. Eng. 374, 113575 (2021)
31. Raissi, M., Perdikaris, P., Karniadakis, G.E.: Machine learning of linear differential equations using Gaussian processes. J. Comput. Phys. 348, 683–693 (2017).
https://doi.org/10.1016/j.jcp.2017.07.050
32. Valenzuela, N.: A new approach for the fractional Laplacian via deep neural networks. arXiv preprint arXiv:2205.05229 (2022)
33. Valenzuela, N.: Fractional Laplacian codes. https://github.com/nvalenzuelaf/
DNN-Fractional-Laplacian
RBF-SC: A Fast Community Detection
Technique Using Radial Basis Functions
Fang Hu1 , Jia Liu2(B) , Lina Wu3 , Xingang Fang2 , Mingfang Huang2 ,
and Haotian Liu4
1
College of Information Engineering, Hubei University of Chinese Medicine,
Wuhan 430065, People’s Republic of China
naomifang@hbucm.edu.cn
2
University of West Florida, Pensacola, FL, USA
jliu@uwf.edu
3
Borough of Manhattan Community College, The City University of New York,
New York, USA
4
School of Mathematics and Statistics, University of Glasgow,
Glasgow G12 8QQ, UK
Abstract. We propose an optimized algorithm for detecting communities in complex networks. We apply the radial basis functions to the
corresponding similarity matrices according to the different networks.
The structure of the network is exploited to develop an efficient matrix
algorithm for the resulting systems. The proposed algorithm is utilized
with the Spectral Clustering method. We adopt a two-stage training
paradigm and propose a simple approach to finding the communities of
the complex networks: (1) finding the proper Radial Basis Function to
construct the weighted matrix; (2) finding the optimal shaping parameter
with respect to the conditioning of the corresponding weighted matrices.
Our approach achieves state-of-the-art accuracy on different types of
networks. The numerical experiments demonstrate the benefits of incorporating the shaping parameter into radial basis functions. The results
of simulations conducted on multiple data sets indicate that the optimized algorithm achieves exceptional performance, particularly for large
data sets. A software package to generate the RBF networks and perform
SC-RBF algorithms can be downloaded from GitHub.
Keywords: Complex networks · Community detection · Spectral
cluster · Radial basis function
1
Introduction
Community detection in complex networks has attracted numerous attention. It
is one of the most important features to find the communities in the different
complex networks in nature, society and technology. For example, [12,13] focus
on social networks and [15] has the research in the collaboration network. The
authors in [3,6,9] discussed the relations in the technological networks such as
c The Author(s), under exclusive license to Springer Nature Switzerland AG 2024
K. Arai (Ed.): SAI 2024, LNNS 1017, pp. 220–236, 2024.
https://doi.org/10.1007/978-3-031-62277-9_13
RBF-SC
221
the Internet, the World Wide Web and power grids. The complex networks also
include the biological networks [23]and metabolic networks [18]. Communities
of networks reveal the groups of related individuals in the specific communities.
Thus, community detection aims to analyze the structure of a given network
at the current time. It enables us to study the group-level features of various
networks. [19] made a study to identify fraud events on telecommunication networks by using community detection. The author in [22] provided the examples
of community detection in the area of social networks such as Facebook, Twitter,
LinkedIn, etc. [4] proposed an algorithm of community-based lung cancer detection and [2] applied the community detection methods in the political networks
to observe the influences of political ideologies or track evolution of the influence
over the time. Due to the wide range of applications of community detection in
complex networks, a large number of algorithms have been explored to solve
the community structures in all types of networks. For the Static communities,
most traditional algorithms are developed by four major categories: Statistical
Inference, such as Stochastic Block Model [1]; optimization method such as [5],
dynamics such as random walk [20], spin dynamics [10]; or spectral methods
[11,17,21].
Nevertheless, detecting communities in networks presents numerous challenges. The ultimate goal for designing an ideal detecting tool must have two
criteria: the algorithm must achieve high accuracy in terms of meaningful modules; the complexity of the algorithm could be low or linear with respect to the
size of the networks. The complexity of the algorithm is possible to compute and
the recent development of the hardware enables the computations with higher
cost. The first goal is more challenging. Most of the time, researchers compare
the proposed algorithm with the existing benchmark using the real networks
whose modules are known precisely. The majority of publications in the community detection use real-world data such as Karate network [24], Dolphin network
[14]. Those are small size real-world data with known structures. The question
remains whether the accuracy of the proposed algorithm will be consistent as
the dataset changes, especially as the size of the data set increases dramatically.
In this paper, we study the new detecting algorithm based on spectral methods.
We show that it is possible to provide a stable algorithm with optimized matrix
representations. We apply the Radial Basis Function (RBF) to derive a weighted
spectral clustering method and introduce the shaping parameters to the Radial
Basis Functions (RBF). With the optimized RBF, the network system will be
well-conditioned. By reducing the condition number of the network system, we
could increase the accuracy of the results, which leads to more accurate and
meaningful communities in the networks.
In the following sessions, we propose a new method for detecting community
structure and apply it to the analysis of a variety of real-world and artificial networks. As we will demonstrate, when applied to networks for which the community structure is already known, our method appears to achieve the expected performance levels. It yields promising results when applied to computer-generated
large networks, allowing us to comprehend the network structure of large data
sets with unknown structures.
222
F. Hu et al.
2
Related Work
Community detection in complex networks has attracted numerous attention. It
is one of the most important features to find the communities in the different
complex networks in nature, society and technology. For example, [12,13] focus
on social networks and [15] has the research in the collaboration network. [3,6,9]
discussed the relations in the technological networks such as Internet, the World
Wide Web and power grids. The complex networks also include the biological
networks [23] and metabolic networks [18]. Communities of networks reveal the
groups of related individual in the specific communities. Thus, community detection aims to analyze the structure of a given network at current time. It enable
us to study the group-level features of various of networks. [19] made a study to
identify fraud events on telecommunication network by using community detection. [22] provided the examples of community detect ions in the area of social
networks such as Facebook, Twitter, LinkedIn, etc. [4] proposed an algorithm of
community-based lung cancer detection and [2] applied the community detection
methods in the political networks to observe the influences of political ideologies
or track evolution of the influence over the time. Due to the wide range of applications of the community detection in the complex networks, a large number of
algorithms have been explored to solve the community structures in all types of
networks. For the Static communities, most traditional algorithms are developed
by four major categories: Statistical Inference, such as Stochastic Block Model
[1]; optimization method such as [5], dynamics such as random walk [20], spin
dynamics [10]; or spectral methods [11,17,21].
However, detecting communities in networks still has many challenges. The
ultimate goal for designing an ideal detecting tools must have two criteria: the
algorithm must achieve high accuracy in terms of meaningful modules; the complexity of the algorithm could be low or linear with respect to the size of the
networks. The complexity of the algorithm is possible to compute and the recent
development of the hardware enable the computations with higher cost. The fist
goal is more challenging. Most of the time, researchers compare the proposed
algorithm with the existing benchmark using the real networks whose modules
are known precisely. Majority publications in the community detection use the
real world data such as Karate network [24], Dolphin network [14]. Those are
small size real world data with known structures. The question remains that if
the accuracy of the proposed algorithm will be consistent as the dataset changes
especially as the size of the data set increases dramatically. In this paper, we
study the new detecting algorithm based on the spectral methods. We show
that it is possible to provide a stable algorithm with optimized matrix representations. We apply the Radial Basis Function (RBF) to derive a weighted
spectral clustering method and introduce the shaping parameters to the Radial
Basis Functions (RBF). With the optimized RBF, the network system will be
well-conditioned. By reducing the condition number of the network system, we
could increase the accuracy of the results which leads to more accurate and
meaningful communities in the networks.
RBF-SC
223
Here we summarize the main contributions of our proposed new algorithm:
– We provide a streamlined workflow for the community detection improving
quality and accuracy by reducing the condition number of resulting coefficient
matrix of the linear system.
– We address the application of RBFs that is common in scientific computing,
but not well-supported in the complex network area.
– We release our python codes as an open-source package that can be deployed
off-the-shelf with multiples models, and can be modified with multiple
machine learning algorithm applications across many scientific disciplines.
3
Radial Basis Function: Spectral Clustering
Consider a data set x1 , x2 , · · · , xn obtained from a certain complex network. We
represent the data in form of the similarity graph G = (V, E) [21]. Each vertex
vi in the graph represents a data point xi . If any two points in the data set
are related, we set the weight wij > 0 between the corresponding two points
in the graph and there is an edge between two points. If two data points are
not related, there is no edge between the corresponding two vertices in the
graph with wij = 0. The weighted adjacency matrix of the graph is the matrix
W = (wij )i,j=1,··· ,n . In the paper, we mainly consider the undirected graph,
which results in a symmetric weighted adjacency matrix W = W T . The structures of the networks could be studied by exploring the structures of the matrix
W [16]. The pattern of the vertices and edges of the adjacency graph or the corresponding adjacency matrix may reveal the information of the division, clusters,
and communities of the network. The first step is to transform the given data set
into a graph, which is called similarity graph. The goal to contact the similarity
graph is to model the local neighborhood relationships between the data points.
In this paper, we consider the construction based on the Radial Basis Function
(RBF) interpolations.
3.1
RBF Interpolation
n
For a given data set, {xi }i=1 , containing n distinct scattered data points in the
computational domain, Ω ⊆ Rd , the approximation utilizing RBF is assuming
that unknown function, f , can be expressed as linear combinations,
f (x) ≈ f˜ (x) =
n
αi Φ (x − xi ) ,
x ∈ Ω,
(1)
i=1
where · represents the Euclidean norm on Rd , αi is the coefficients, and
Φ : Rd → R is called a Radial Basis Function (RBF) [7] if
Φ (x) = Φ (y) ,
whenever
x = y,
x, y ∈ Rd .
(2)
224
F. Hu et al.
In Table 1, we list the most common RBFs that have been used in neural
networks and the numerical approximations. The symbol r in the table denotes
the Euclidean distance,
r = x2 = x21 + x22 + · · · + x2d , x ∈ Rd ,
(3)
and c is a free variable which is also called shape parameter.
Table 1. Some common choices of RBFs
Choice of RBF
Definition
√
Multiquadric (MQ)
Φ(r, c) = c2 + r2
√
Inverse Multiquadric (IMQ) Φ(r, c) = 1/ c2 + r2
Inverse quadric (IQ)
Φ(r, c) = 1/ c2 + r2
2 2
Gaussian (GA)
Φ(r, c) = exp r /c
Φ(r, c) = rk , k = 1, 3, 5; Φ(r, c) = rk ln k,k = 2, 4, 6,
Polyharmonic spline
Follows Eq. (1) with the collocation scheme,
f˜ (xi ) = f (xi ) ,
i = 1, 2, · · · , n,
(4)
leads to a system of linear equations,
Wα = f,
T
(5)
T
where α = [α1 , · · · , αn ] , f = [f (x1 ) , · · · , f (xn )] , and a matrix Wij =
Φ (rij ) ∈ Rn×n . The distance matrix, rij , contained within W can be expressed
as follows,
⎤
⎡
x1 −x12 · · · x1 −xn2
⎥
⎢
..
..
..
rij = ⎣
(6)
⎦ , i, j = 1,· · ·, n.
.
.
.
xn −x12 · · · xn −xn2
After applying RBF to the data sets, we obtain the similarity graph and the
weighted matrix W in Eq. 7. The matrix W with entries wij = Φ(xi − xj 2 ),
i, = 1, · · · , N is called the interpolation matrix. W consists of the functions
serving as the basis of the approximation space. For distinct data points in
the data sets and a constant shape parameter c, W is a non-singular matrix.
Both the choice of the RBF and its corresponding shaping parameter plays
an important role in calculating the final interpolations and partition in the
resulting graph. In the neural networks, the most common RBF or similarity
− 2
function is the Gaussian similarity function e2cr2 which is equivalent to the GA
RBF in Table 1. In the geographical data sets, the MQ RBF is used, and the
local data set’s density determines the shaping parameter. Which RBF is used
for the interpolation matrix is strongly problem-dependent. On the other hand,
RBF-SC
225
the interpolation matrix which is the same weight matrix we use in the complex
network, is high ill-conditioned.
⎤
Φ (x1 −x12 ), Φ (x1 −x22 ) · · · Φ (x1 −xN2 )
⎥
⎢
.
.
.
.
⎥
⎢
.
.
.
.
⎥
⎢
.
.
.
.
⎥
⎢
⎢
Φ
x
−x
),
Φ
x
−x
)
.
.
.
Φ
x
−x
)
Ni
Ni
2 2
Ni
N 2 ⎥
⎥
⎢ 1 2 W =⎢
⎥
⎢Φ xNi+1 −x1 ), Φ xNi+1 −x2 ) · · · Φ xNi+1 −xN )⎥
⎢
2
2
2 ⎥
⎥
⎢
.
.
.
.
⎥
⎢
.
.
.
.
⎦
⎣
.
.
.
.
Φ (xN −x12 ), Φ (xN −x22 ) · · · Φ (xN −xN2 )
⎡
3.2
(7)
Trade-Off Principle
The Trade-Off Principle (or the Uncertainty), named by R. Schaback, describes
an interpolation utilizing a RBF which contains a shape parameter, c, (RBFs
in Table 1), where a compromise must be made between achieving accuracy and
achieving stability while on choosing an optimal shape parameter.
To achieve highly accurate numerical results, an interpolate utilizing MQ can
be improved by adjusting the value of the shape parameter, c. Here we introduce
the condition number of the interpolation matrix, which is denoted by κ (W ):
κ (W ) = W W −1 =
σmax
σmin
(8)
where σmax and σmin are the largest and smallest singular values of W .
The condition number can be used to estimate the degree of the stability to
which a numerical solution will be affected by rounding errors. In other words,
the optimal numerical solution of RBF interpolations utilizing MQ relies on
choosing an appropriate value of the shape parameter, c. Similarity, the optimal
parameters are chosen when we use IMQ, IQ, GA RBFs so that we can achieve
the lowest condition number.
In order to improve the accuracy of the algorithm, we would expect a wellconditioned system. It would be crucial to reduce the condition number of the
resulting weighted matrix W . The first strategy is to choose a reasonable RBF.
In general, we expect the RBF to have similar features as the local data set. For
example, the most neural network would use the Gaussian kernel to interpolate
the data since the GA models local neighborhoods most of the time. Due to the
different structures and applications of the complex networks, the choices of RBF
should not be the same. The next strategy is to adjust the shaping parameters
to reduce the condition number of the resulting interpolation matrix.
Figure 1, 2, and 3 show the condition number changes versus the different
choices of the shaping parameters. As seen, the condition number decreases
from 10125 to 1027 when the parameter sharing is changed from 0.02 to 0.1. In
contrast, if we consider the other two RBFs, IMQ or MQ RBF, we can observe the
high oscillations of the condition numbers in response to changes in the shaping
parameters. In numerical experiments, the following results are consistent: The
Gaussian RBF performs better with the Karate network when we employ an
226
F. Hu et al.
Fig. 1. Condition number vs shaping parameter values in the Karate network with
Gaussian RBF
unsupervised learning algorithm such as Spectral Clustering, and the optimal
value of the shaping parameter could improve the model’s accuracy, as measured
by evaluation metrics such as Modularity or NMI.
3.3
Spectral Clustering with Optimized RBF
In the following we will introduce spectral clustering with optimized RBF. Spectral clustering algorithms has become one of the most popular community detection algorithms after Shi & M, Ng et al., and M & SH. The method, first published by [8] in 1973, is based on spectral properties of the graph and focuses on
the eigenvectors of the matrices related with the dataset. Again Let G = (V, E)
be an undirected graph with vertex set V = {v1 , v2 , · · · , vn }. Each vi corresponds
the data point xi . The main tool for spectral clustering is the graph Laplacian
matrix. The unnormalized Laplacian matrix is defined by L = D − W , where D
is the degree matrix with the degrees d1 , d2 , · · · , dn on the diagonal. Here di is
the degree of a vertex vi ∈ V .
n
di = Σj=1
wi,j .
(9)
We apply the RBF interpolation and optimal shaping parameters to spectral
clustering. The specific process is illustrated in Fig. 4.
The pseudo-code of the algorithm is presented below in Algorithm 1. The
implementation of the algorithms in this paper is publicly accessible on GitHub.
In Algorithm 1, Line 2 involves the two stage paradigm: (1) finding the best RBF;
RBF-SC
227
Fig. 2. Condition number vs shaping parameter values in the Karate network with
IMQ RBF
(2) finding the optimal shaping parameter. Then, collect eigenpairs (eigenvalue,
eigenvector) of K and arrange them in decreasing order by eigenvalues (Lines
7). The algorithm returns a list of clustered node labels.
4
Experimental Results
Our experiments were performed on the real-world data sets and large synthetic
data sets. In the present work, three widely used RBFs in Table 1: MQ, IMQ and
GA have been implemented in the network. The data sets are from the Stanford
data center (). For the real-world data sets, we considered six different benchmark networks: Karate, Dolphins, Football, Les Miserables, and Power networks.
For the synthetic data sets, we employed the Lanchichinetti-Fortunato-Radicchi
(LFR) benchmark to test all algorithms. By testing the real-world and LFR
models, we unveil the relations between the RBFs, shaping parameters, and the
condition numbers of the corresponding adjacency matrices. In this chapter, we
present the experiments for the real-world data sets. Then, we applied the algorithms to the LFR benchmark data sets show the relationship between the RBFs
and data sets. Last, we discuss the comparisons of community detection algorithms in terms of accuracy using the Molecularity and other evaluation metrics.
The experiments were performed using Python on a 16 GB RAM 3.6 GHz Intel
(R) Core (TM) CPU.
228
F. Hu et al.
Fig. 3. Condition number vs shaping parameter values in the Karate network with MQ
RBF
Table 2. The number of communities in various real-world data sets
No. Networks
4.1
Nodes Edges Communities
1
Karate
34
78
2
2
Dolphins
62
159
2
3
Football
115
613
12
4
Les Miserables 77
254
11
5
Polbooks
441
3
6
Jazz
198
2742
5
7
Email
1133
10903 23
8
Protein
2375
11693 13
9
Power
4941
6594
105
836
Real Data Sets
In the following experiments, a total of nine real world data sets were tested
with SC with three different RBFs. We used the optimal shaping parameters in
each choice of RBFs. We compared the results with three state-of-art algorithms:
Spectral Clustering, K-means and Infomap. Table 2 lists the nodes, edges, and
number of the communities of all real data sets we have tested. Among those
network sets, Zachary’s karate club (Karate), Dolphins, and American college
football teams are extensively used in many community detection algorithms.
Both Karate and Dolphins networks have small nodes, edges and community
numbers. Email, Protein, and Power networks are relatively larger networks
with more than 1000 nodes and more than 10 communities.
RBF-SC
229
Fig. 4. Flowchart of spectral clustering with optimized RBF interpolation
Algorithm 1. Spectral Clustering with optimized RBF interpolation
Require:
G (V, E) // Undirected graph
k // Number of clusters
Ensure:
Clusters: C = {C1 , C2 , . . . , Ck } // Community list
1: G (V, E) −→ S // Similarity matrix
2: Find the optimal shaping parameter and RBF to finalize S
3: S −→ W, D // Weighted adjacency matrix and degree matrix
4: L = D − W // Laplacian matrix
5: D−1/2 LD1/2 // Normalized Laplacian matrix
6: D−1/2 LD1/2 −→ First k minimum eigenvalues
7: Eigenvector −→ F // Eigenmatrix
8: return C // Community list
The detection and the characterizing the community structure is tested by
the quality function “Modularity”. In 2004, Newman and Girvan proposed modularity Q metrics. It represents the difference between the actual number of connection and the excepted number of connection in random conditions. It is used
to assess the quality of the network cluster structure. The equation is as follows:
k
2
ds
ls
−
(10)
Q=
l
2l
s=1
where k is the number of communities in the network, l is the sum of all
edges in the network, ls is the sum of all edges in community s, and ds is the
sum of the degree of all nodes in s. The modularity Q metric is used to verify
the accuracy of community detection in complex networks. Modularity contains
an intrinsic scale that depends on the total number of links in the network.
Modules that are smaller than certain scales may not be resolved. Table 3 lists
the Modularity numbers using optimized RBF with SC, K-means, Infomaps. In
the column of RBF-SC, the optimized RBF and optimal shaping parameter are
230
F. Hu et al.
Table 3. The Modularity values in various real-world data sets
Modularity
SC
K-means Infomap
RBF1-SC RBF2-SC RBF3-SC
Karate
0.359961 0.37147
0.17891
0.11389
0.45581
0.49957
Dolphins
0.37870
−0.02781 0.39605
0.37298
0.40001
Football
0.600517 0.60052
Networks
0.30495
0.01600
0.62897
0.51171
0.75694
0.39340
0.08327
0.17345
0.57855
0.64896
Polbooks
0.50175 0.33652
0.34627
0.46743
0.47838
0.10457
Jazz
0.35742 0.34974
0.01545
0.00337
0.00212
0.01042
Les Miserables 0.51409
Email
0.30293
0.22834
0.53669
0.28311
0.00378
0.60111
Protein
0.29926
0.34441
0.00584
0.09163
0.07361
0.62979
Power
0.68283
0.10081
0.51647
0.74255
0.84309
N/A
chosen based on the optimization of the Modularity. Seven out of nine network
data sets obtained largest Modularity. The two exceptions are Jazz network and
Polbooks network. In those two cases, SC without RBF implementation obtained
the largest Modularity. Among the seven real-world data sets, Karate, Dolphins,
Football, Les Miserables, Email, and Protein network data have the largest Modularity values when we implemented the Gaussian RBF; Power network works
best with the IMQ RBF. MQ RBF, which is also widely used, does not work well
in all real-world data sets. Data shows the Modularity of the seven real world
networks with three different RBFs. RBF1-SC represents SC with the MQ RBF,
RBF2-SC represents SC with the IMQ RBF and RBF3-SC represents SC with
the Gaussian RBF. The line associated with the Gaussian RBF maximize the
modularity in most datasets. For the Power dataset, RBF with IMQ RBF works
the best. The Polbooks and Jazz datasets are not working well with the RBF
interpolations. For all cases in the experiments, the optimal shaping parameter c
is used in each case. We will discuss the details about how to choose the optimal
shaping parameters in the next section (see Fig. 5, 6 and Tables 4, 5 and 6).
Table 4. The number of communities in various LFR (N, < k >, m, γ, β, μ)
No. Networks
Nodes Edges
Communities
1
LFR(1024, 20, 100, 2, 1, 0.5)
1, 024
19, 568
24
2
LFR(1024, 20, 100, 2, 1, 0.1)
1, 024
20, 022
27
3
LFR(1024, 20, 100, 2, 1, 0.4)
1, 024
19, 800
28
4
LFR(1024, 20, 100, 2, 1, 0.3)
1, 024
18, 746
29
5
LFR(2048, 30, 200, 2, 1, 0.2)
2, 048
62, 078
23
6
LFR(2048, 30, 200, 2, 1, 0.1)
2, 048
60, 728
28
7
LFR(2048, 30, 200, 2, 1, 0.6)
2, 048
62, 854
30
8
LFR(10000, 50, 500, 2, 1, 0.5) 10, 000 506, 722 48
RBF-SC
Fig. 5. The modularity in various synthetic network data sets
Fig. 6. The NMI in various synthetic network data sets
Table 5. The Modularity in various synthetic network data sets
No. Modularity
SC
K-means Infomap
RBF1-SC RBF2-SC RBF3-SC
1
−0.00454 0.17039
0.00001
0.39542
2
−0.00149 0.61007
0.00346
0.83426
0.20264
−0.00608
3
−0.00149 −0.00940 0.00233
0.00384
0.00193
−0.00431
4
0.00259
0.52767
0.00099
0.67594
0.01386
−0.00558
5
0.00153
0.49263
0.00138
0.89058
0.00265
−0.00589
6
−0.00203 0.55416
0.00294
0.59215
0.06012
−0.00510
7
−0.00186 0.04736
0.00000
0.25994
−0.00033 −0.00227
8
0.01122
−0.00004 0.66426
0.07341
−0.00179 −0.00221
−0.00128 0.00073
231
232
F. Hu et al.
Table 6. The NMI in various synthetic network data sets
No. NMI
SC
K-means Infomap RBF1-SC RBF2-SC RBF3-SC
1
0.10691 0.62547
0.09317
0.74727
0.09451
0.09649
2
0.10950 0.85988
0.11305
0.94062
0.67143
0.23479
3
0.12170 0.09431
0.12835 0.11230
0.12376
0.09200
4
0.13436 0.87388
0.13088
0.90758
0.24761
0.16648
5
0.04988 0.84671
0.04481
0.96053
0.13098
0.21470
6
0.06922 0.86648
0.06226
0.89185
0.54512
0.24874
7
0.07544 0.47766
0.75000 0.70103
0.08005
0.08045
8
0.27901 0.61156
0.03220
0.06379
0.04469
0.91117
Fig. 7. The comparison plots of the shaping parameters vs condition number of associated weighted matrices in the real-world network: Jazz network
4.2
The Role of the Shaping Parameters
In this section, we discuss the role of the shaping parameters for each RBF
interpolation. We will present the tests on the condition number of the network
matrices and the corresponding shaping parameters. For a thorough analysis, we
have considered nice real-world data sets and eighth synthetic LFR benchmark
data sets. We have investigated that we could improve the accuracy by choosing
the optimized shaping parameter for each RBF. Figure 7, 8 and 9 present the
plots of the shaping parameter values vs the condition numbers of the associated
linear systems. In all cases, the weighted matrices are extremely ill-conditioned.
The condition number could be 10119 as in the Protein network. With the shaping
parameter, the value of the condition number could be reduced rapidly if we
choose the good value interval. Most optimal shaping parameters are between
RBF-SC
233
Fig. 8. The comparison plots of the shaping parameters vs condition number of associated weighted matrices in the real-world network: Power network
Fig. 9. The comparison plots of the shaping parameters vs condition number of associated weighted matrices in the LFR benchmark: size 1024, 28 communities
[0.08, 0.1]. Therefore we could always pick the shaping parameter c = 0.1 when
we have the real world RBF networks. We noticed that even in general, Gaussian
RBF is the common choice is the radial basis neural network, the exceptions do
exist. For example, the Power data set works best with the inverse Multiquadritic
with the optimal shaping parameter = 0.08. Therefore we may give an estimate
that for the real world data sets, we could use the optimal shaping parameter
between 0.08 and 0.1. Most of cases, we may choose a simple value 0.1 which leads
to a reasonable small conditional number of the associated weighted matrix.
234
F. Hu et al.
5
Conclusions
The community detection is a crucial challenge for many real-world data sets due
to the large network size and complexity. Our work is motivated by the empirical
observation that we can improve the accuracy for most mathematical models by
reducing the conditioning numbers. We propose to learn the balanced condition
number via the shaping parameter. We have introduced this new algorithm using
the RBF and SC to identify communities in networks. The new algorithm is based
on the traditional spectral clustering algorithm but the idea could be expanded
to other similar community detecting methods. We found that by introducing
the shaping parameter to adjust the conditional number of the weighted matrix
of its associated network, the algorithm could get higher accuracy, such as a
higher modularity value or NMI value. We found that most real-world data sets
are more feasible to the GA RBF while MQ RBF works much better on the
LFR benchmark. The choice of the optimal shaping parameter could cost highly
if the network is huge. However, if the network results in a sparse matrix, we may
apply the iterative algorithms to estimate the largest and smallest eigenvalues
or singular values. The RBF interpolation and shaping parameter optimization
will work very well for a reasonable size of real-world network. Especially for the
real-world data sets, we could test different possibilities of the RBFs to find the
best function and approximately the best shaping parameter so that we expect
a relatively high accuracy in our community detection results. Furthermore,
we tested the proposed technique on the LFR benchmark. We found that the
RBF-SC with the multiquadratic function works best for the LFR benchmark.
Due to the construction of the LFR, we may apply the MQ RBF to similar
structured large complex networks. We found the optimal shaping parameters
on the small-size datasets and applied similar values to the large datasets with
similar properties. Our results indicate that larger size data set also achieve high
modularity and NMI values with the same shaping parameters.
Besides, we have not tested the other possible RBFs in Table 1. Based on the
networks, we may propose another adaptive shaping parameters instead of fixed
values. These approaches may reduce the condition number of the associated
weighted matrix so that we can improve the stability of the results.
A software package to generate the RBF networks and perform SC-RBF
algorithms can be downloaded from GitHub site: https://github.com/jialiu-uwf/
rbfscore.
Acknowledgment. Author Professor Lina Wu would like to thank for support provided by a PSC-CUNY Award, jointly funded by The Professional Staff Congress and
The City University of New York.
RBF-SC
235
References
1. Abbe, E., Sandon, C.: Community detection in general stochastic block models:
fundamental limits and efficient algorithms for recovery. In: 2015 IEEE 56th Annual
Symposium on Foundations of Computer Science, pp. 670–688. IEEE (2015)
2. Alothali, E., Zaki, N., Mohamed, E.A., Alashwal, H.: Detecting social bots on
twitter: a literature review. In: 2018 International Conference on Innovations in
Information Technology (IIT), pp. 175–180. IEEE (2018)
3. Amaral, L.A.N., Scala, A., Barthélémy, M., Stanley, H.E.: Classes of small-world
networks. Proc. Natl. Acad. Sci. 97(21), 11149–11152 (2000)
4. Bechtel, J.J., Kelley, W.A., Coons, T.A., Klein, M.G., Slagel, D.D., Petty, T.L.:
Lung cancer detection in patients with airflow obstruction identified in a primary
care outpatient practice. Chest 127(4), 1140–1145 (2005)
5. Chen, M., Kuzmin, K., Szymanski, B.K.: Community detection via maximization
of modularity and its variants. IEEE Trans. Comput. Soc. Syst. 1(1), 46–65 (2014)
6. Faloutsos, M., Faloutsos, P., Faloutsos, C.: On power-law relationships of the internet topology. ACM SIGCOMM Comput. Commun. Rev. 29, 251–262 (2003)
7. Fasshauer, G.E.: Meshfree Approximation Methods with MATLAB, vol. 6. World
Scientific, New Jersey (2007)
8. Fiedler, M.: Algebraic connectivity of graphs. Czechoslov. Math. J. 23(2), 298–305
(1973)
9. Flake, G.W., Lawrence, S., Giles, C.L., Coetzee, F.M.: Self-organization and identification of web communities. Computer 35(3), 66–70 (2002)
10. Fortunato, S., Hric, D.: Community detection in networks: a user guide. Phys. Rep.
659, 1–44 (2016). Community detection in networks: A user guide
11. Fang, H., Wang, M., Wang, Y., Hong, Z., Zhu, Y.: An algorithm J-SC of detecting
communities in complex networks. Phys. Lett. A 381(42), 3604–3612 (2017)
12. John, S.: Social Network Analysis: A Handbook. Sage Publications, London (2000)
13. Lusseau, D., Newman, M.E.J.: Identifying the role that animals play in their social
networks. Proc. Royal Soc. Lond. Ser. B Biol. Sci. 271(suppl 6), S477–S481 (2005)
14. Lusseau, D., Schneider, K., Boisseau, O.J., Haase, P., Slooten, E., Dawson, S.M.:
The bottlenose dolphin community of doubtful sound features a large proportion
of long-lasting associations. Behav. Ecol. Sociobiol. 54(4), 396–405 (2003)
15. Newman, M.E.J.: The structure of scientific collaboration networks. Proc. Natl.
Acad. Sci. 98(2), 404–409 (2001)
16. Newman, M.E.J.: Community detection and graph partitioning. EPL (Europhys.
Lett.) 103(2), 28003 (2013)
17. Newman, M.E.J.: Spectral methods for community detection and graph partitioning. Phys. Rev. E 88(4), 042822 (2013)
18. Palla, G., Derényi, I., Farkas, I., Vicsek, T.: Uncovering the overlapping community
structure of complex networks in nature and society. Nature 435(7043), 814–818
(2005)
19. Pinheiro, C.A.R.: Community detection to identify fraud events in telecommunications networks. In: SAS SUGI Proceedings: Customer Intelligence (2012)
20. Pons, P., Latapy, M.: Computing communities in large networks using random
walks (long version) (2005)
21. Von Luxburg, U.: A tutorial on spectral clustering. Stat. Comput. 17(4), 395–416
(2007)
22. Wang, C., Tang, W., Sun, B., Fang, J., Wang, Y.: Review on community detection
algorithms in social networks. In: 2015 IEEE International Conference on Progress
in Informatics and Computing (PIC), pp. 551–555. IEEE (2015)
236
F. Hu et al.
23. Williams, R.J., Martinez, N.D.: Simple rules yield complex food webs. Nature
404(6774), 180–183 (2000)
24. Zachary, W.W.: An information flow model for conflict and fission in small groups.
J. Anthropol. Res. 33(4), 452–473 (1977)
Improving Medication Prescription
Strategies for Discordant Chronic
Comorbidities Through Medical Data
Bench-Marking and Recommender
Systems
Tom Ongwere1(B) , Nimbalkar Rutuja2 , and Tam V. Nguyen1
1
University of Dayton, Dayton, OH 45469, USA
{tongwere1,tnguyen1}@udayton.edu
2
Amazon Web Services, Seattle, WA, USA
Abstract. Patients with Discordant Chronic Comorbidities (DCCs)
face the challenge of managing evolving treatment needs and conflicting
regimens, necessitating multiple medical appointments. The complexity arises from conflicting treatment plans and potential drug interactions, making it difficult for both patients and healthcare providers to set
an optimal treatment plan. To address this, Machine Learning-powered
tools offer significant solution by providing treatment recommendations
and detecting potential interactions. Despite the potential, there is a
lack of comprehensive documentation on how these algorithms effectively handle conflicting recommendations and evolving patient needs
in the context of DCCs. This research seeks explore and improve the
how Support Vector Machines (SVM), Neural Networks, and Random
Forests algorithms set and prioritize treatment regimens for DCCs. We
conducted survey to understand healthcare providers’ strategies for prioritizing medication regimens, we constructed a dataset from survey data
and deployed machine learning algorithms to predict optimal medication
regimens for DCC patients with specific treatment concerns. Following
benchmarking, SVM emerged as the top performer, achieving an impressive accuracy rate of 0.95.
Keywords: Complex chronic diseases · Discordant chronic
conditions · Bench-marching · Recommender systems · Machine
learning · Decision aids
1
Introduction
Discordant Chronic Comorbidities (DCCs) are health conditions in which
patients have multiple, often unrelated, chronic illnesses that need to be
addressed concurrently but are also associated with conflicting treatment
instructions [23]. Patients with DCCs often experience multiple obstacles when
c The Author(s), under exclusive license to Springer Nature Switzerland AG 2024
K. Arai (Ed.): SAI 2024, LNNS 1017, pp. 237–250, 2024.
https://doi.org/10.1007/978-3-031-62277-9_14
238
T. Ongwere et al.
prioritizing treatment plans and prescriptions [20]. Machine learning (ML)
and/or deep learning (DL) algorithms can provide these treatment recommendations and also identify drug interactions and/or conflicts. Examples of these
machine algorithms include; support vector machines (SVM), logistic regression
(LR), and k-means clustering. These algorithms are increasingly commonly used
in the classification, and diagnosis of chronic diseases. In addition, these algorithms could also be integrated into decision aids to capture patients’ concerns
and preferences upfront, before suggesting a prescription recommendation. However, to the best of our knowledge, we have yet to see good documentation of how
conflicting recommendations and/or evolving patients’ needs could be addressed
using machine learning techniques. Further, there are no recommendations for
algorithms suitable for addressing these complex needs that patients with DCCs
experience.
In this research, we first collected data regarding the concerns of patients with
DCCs (e.g., the cost of medication, side effects/symptoms, addiction, sleep, and
sexual drive) and prescription preference for each of their medical conditions
(e.g., type 2 diabetes). We then reviewed models and algorithms, tested and
created benchmarks of some of these algorithms using a set of criteria from our
data set. Benchmark outcomes were set to reveal the suitable algorithms, parameters, and testing criteria. Specifically, our research applies four (N = 4) popular
ML algorithms for identifying effective and efficient predictions of DCCs prescriptions, while taking into account patients’ concerns and drug interactions.
We assessed the accuracy of Nearest Neighbor (NN), Random Forest (RF), and
Support Vector Machine (SVM) using our dataset. SVM and RF demonstrated
better performance, showing a promising and greater capability to predict accurate prescriptions for patients with Discordant Chronic Comorbidities (DCCs)
when compared to Nearest Neighbor (NN).
This study not only presents the results of our predictive models but also
establishes benchmarks to evaluate the performance of each algorithm utilized
and makes the following contributions:
This finding not only highlights the efficacy of SVM and RF but also provides
valuable guidance for healthcare providers and researchers seeking to implement machine learning solutions in the management of patients with discordant
chronic comorbidities. By doing so, it makes the following contributions:
1. Algorithmic Prioritization in DCC Treatment Plans: By exploring and
improving the application of Support Vector Machines (SVM), Neural Networks, and Random Forests algorithms in setting and prioritizing treatment
regimens for patients with Discordant Chronic Comorbidities (DCCs).
2. Survey-Based Understanding of Healthcare Provider Strategies: Adds value
by conducting a survey to gain insights into healthcare providers’ strategies
for prioritizing medication regimens in the context of DCCs.
3. Benchmarking and Identification of Top-Performing Algorithm (SVM): By
benchmarking the performance of different machine learning algorithms in
predicting optimal medication regimens for DCC patients.
Hamiltonian Mechanics
239
The remainder of this paper is organized as follows. Section 2 provides a brief
overview of the related work. Section 3 presents the dataset and methods used in
our benchmarking study. In Sect. 4, we present the extensive evaluation results
obtained from the benchmark. Finally, in Sect. 5, we conclude our paper and
outline future research directions.
2
Related Works
Due to the limited work done on DCCs, this section highlights studies done on
prediction for a single chronic diseases and compares the performance of the
algorithms used in those predictions. Recently researchers in computer science
have began engaging the use Machine Learning techniques to predict disease and
drug interaction. Chengyin et at., used XGBoost machine learning algorithms
to predict the incidence of hypertension using Electronic Health Records (EHR)
data [34]. Nahit et al., classified respiratory sounds using AdaBoost combined
with Random Forest and artificial neural networks to analyze and accurately
diagnose an asthma patient based on the sounds acquired from the chest of
patient [10]. Gómez-Vallejo et al., used case-based reasoning algorithms to perform automatic diagnostics of Healthcare-Associated Infections (HAIs) using
collected samples [11]. Meanwhile, Li et al., developed a hybrid automated algorithm for discrepancy detection between patients’ discharge prescriptions and
medications documented in free-text clinical notes [14]. Majority of this algorithms are being deployed and used in medication reconciliation and recommender systems. Despite its wide acceptance, medication reconciliation is inadequately performed in current clinical practice [33]. Further, sustaining effective
and accurate reconciliation remains challenging. In addition to these challenges,
reports show that medication reconciliation is a resource-intensive, error-prone
task, and the resources to accomplish it may not be routinely available [30]. This
is because algorithms deployed in these systems make use of the data, yet a vast
amount of clinical data (especially for patients with DCCs) is scattered across
different sites on the Internet. Patients with DCCs, may not be able to find help
because of data fragmentation. Another challenge is the overload of medical
information (e.g., on drugs, medical tests, and treatment suggestions) resulting
in difficulties in making personalized and patient-oriented recommendations [21].
There is a need for work that applies the use of recommender systems in the
healthcare domain to help both end-users and medical professionals make more
efficient and accurate health-related decisions [31]. Such studies could strive to
address the accuracy, efficiency, and trustworthiness aspects that are currently
missing. We need to identify or create new algorithms capable of addressing
these complexities.
There are already studies taking that direction and are creating benchmarks
and comparisons of accuracy, precision, and performance of multiple algorithms:
Maseer et al., used 10 popular supervised and unsupervised ML algorithms for
identifying efficient intrusion detection systems of their networks and computers [16]. These algorithms include the artificial neural network (ANN), decision tree (DT), k-nearest neighbor (k-NN), naive Bayes (NB), random forest
240
T. Ongwere et al.
(RF), support vector machine (SVM), and convolutional neural network (CNN)
algorithms, expectation-maximization (EM), k-means, and self-organizing maps
(SOM) algorithms. The results of their classification evaluation showed that
Nearest Neighbor, DT, and NB algorithms performed exceptionally well compared to SOM and EM when implementing a normally-based intrusion detection system. Chang et al. used “Flaredown” data-sets generated from an app
designed to help patients with chronic diseases improve their symptoms and
conditions. The study tested the performance of seven machine learning algorithms when predicting a patient’s illness severity level [5]. The results revealed
that the XGBoost model (with an F1 score of 0.85) and LightGBM model (with
an F1 score of 0.84) had the best performance compared with other algorithms.
Mostafa et al. conducted exploratory research on the popular ML algorithms and
evaluated their capacity to accurately diagnose PD based on vocal symptoms.
They applied classifiers including Decision Tree (DT), Naı̈ve Bayes (NB), and
Neural Network (NN), in the diagnosis of Parkinson’s disease and found that
the Decision Tree produced the highest accuracy rate of 91.63%, followed by the
Neural Network, 91.01%, and the Naı̈ve Bayes produces the lowest accuracy rate
of 89.46%. [18]. Das [7] compared four classification algorithms, to accurately
diagnose Parkinson’s diseases using SAS-based software. The four classifiers
selected and implemented included; DMNeural, Neural Network, Regression, and
Decision tree respectively. The neural networks classifier yielded the best classification results (92.9% correct classification rate) among the four classifiers.
Charleonnan et al. explored multiple classifiers targeting increasing the accuracy of predicting a kidney disease [6]. They applied a support vector machine
(SVM), logistic regression (LR), decision tree (DT), and KNN on chronic kidney disease (CKD) data set consisting of 400 occurrences. Their results showed
that the SVM classifier attains higher accuracy when compared to the others for
CKD detection.
This review shows that ML and DL algorithms can predict the occurrences
and/or recurrence of chronic disease (i.e., kidney failures, or Parkinson’s). They
provide researchers with strategies for understanding when a patient is most
likely to develop a disease and suggestions for medical recommendations. Further, these algorithms and interventions that implement these algorithms (e.g.,
decision aids, recommender systems, and medications reconciliation systems)
show that specific strategies or constructs may work for one group of people
and data they provide while another algorithms work effectively for other populations (and data). However, these studies do not account for patients with
DCCs who have multiple and interacting experiences. There is no research that
tailors current algorithms for individuals who have multiple diseases, treatment
plans, and interacting medications. Creating Benchmarks of algorithms for these
multiple and interacting experiences is what this research will focus upon.
Hamiltonian Mechanics
3
Background on Machine Learning Algorithms
3.1
Nearest Neighbors
241
Nearest Neighbors is a non-parametric algorithm that is based on the assumption
that similar points lie close together in the data space. It works by taking a
test instance, finding the k-nearest neighbors in the training dataset, and then
using the labels of those neighbors to predict the label of the test instance. For
example, it works by calculating the distance between a new data point and all
the other data points in the training dataset, and then selecting the k closest
points [27]. The label of the new data point, is then determined by looking at
the labels of the k closest points. Nearest Neighbors (NN) is a popular machinelearning algorithm that has been used for various purposes, including disease
and drug-drug interaction prediction.
KNN algorithm has been successfully used to diagnose heart diseases using
clinical features such as age, blood pressure, and cholesterol levels [2]; diabetic
retinopathy with the accuracy of 80% [13]; breast cancer based on features such
as tumor size, lymph node status, and age with an accuracy of over 95% [12];
various skin diseases based on symptoms and signs with an accuracy of over 85%
[27].
Calculate Distance: The distance between the new point and each training point
consisting of d dimensions must first be calculated. There are several ways to
calculate this distance, the most well-known of which are the Euclidean and
Manhattan distances. Euclidean distance is The square root of
difference in
the
d
2
squared values between a new point (x) and an old point (y):
i=1 (xi − yi ) .
Meanwhile, Manhattan distance is the sum of their absolute differences is used
to
dcalculate the distance between real vectors:
i=1 |xi − yi |
3.2
Random Forest
Random Forest (RF) is supervised learning algorithm that creates a forest of
decision trees, each of which is trained on a different subset of the data. For
each of these trees, RF randomly selects a subset of the features to be used
in the tree. Making each tree slightly different from each other, thus helps to
reduce the chances of overfitting. When it comes to making a prediction, the
Random Forest algorithm looks at all the trees and selects the one that has
the most accurate prediction. Random Forest algorithms are used in a variety
of applications, including regression, classification, and outlier detection. For
example, i) the study used a Random Forest model to predict heart disease risk
and obtained is 93.3% in terms of accuracy [22]. Another study used the random
forest to predict diseases using Electronic Health Records (EHR) and found that
random forest was more accurate than other machine learning methods, such as
Support Vector Machines (SVMs) and Logistic Regression (LR), in terms of
predicting diseases [25].
242
T. Ongwere et al.
Additional studies have compared the performance of the random forests
worth other classifiers. For example, used random forest has been used for predicting breast cancer from mammogram images. The study found that random
forest was the most accurate machine-learning algorithm for predicting breast
cancer [17]; for predicting cardiovascular diseases from clinical data, their finding showed that random forest was more accurate than other machine learning
methods, such as decision trees and logistic regression, in terms of predicting
cardiovascular diseases [24]. Finally, random forest, naive Bayes, and logistic
regression algorithms were used to predict drug-drug interactions. The dataset
used in the study was obtained from DrugBank and consisted of 6,717 drugs and
5,942 drug pairs. The authors found that the random forest model was able to
accurately predict drug-drug interactions with an AUC of 0.988 [9].
3.3
Support Vector Machine (SVM)
Support Vector Machine (SVM) is a supervised machine learning algorithm used
for classification, regression, and outlier detection [1]. SVM finds a hyperplane
that maximizes the margin between two classes, such that the data points of each
class are as far away from the hyperplane as possible. To find this hyperplane,
SVMs first identify a set of training data points that are closest to the decision
boundary (support vectors). The hyperplane is then identified by maximizing
the distance between the support vectors and the decision boundary. Once the
hyperplane is found, it can be used to classify new data points [32]. The SVM
algorithm can also be used to identify outliers in a dataset [1]. Thus, perform well
on datasets with small sample sizes and can learn complex non-linear boundaries. This makes them particularly well-suited for problems in which data is not
linearly separable or when there are multiple classes.
A Support Vector Machine (SVM) has been used in several studies to predict
potential drug-drug interactions (DDI) [15]; predict the risk of Type 2 Diabetes
based on clinical laboratory data using collected data on height, age, weight,
and other clinical laboratory indicators [26].
3.4
Artificial Neural Networks (ANNs)
An artificial neural network (ANN) is a computing system composed of interconnected nodes or neurons, which use inputs to produce outputs. ANNs mimic
the behavior of biological neural networks in the brain and are used for a variety
of tasks such as classification, pattern recognition, prediction, and optimization
[3]. An artificial neural network is made up of three or more linked layers. Input
neurons make up the first layer. These neurons send input to deeper layers,
which then deliver the final output data to the final output layer. All of the
inner layers are concealed and are made up of units that use a series of transformations to modify the information received from layer to layer. Each layer
serves as both an input and an output layer, allowing the ANN to comprehend
increasingly complicated things. The neural layer is the collective name for these
inner layers. The neural layer’s units attempt to learn about the data collected
Hamiltonian Mechanics
243
by weighting it according to the ANN’s internal framework. ANNs have become
increasingly popular due to their ability to learn from data and identify complex
relationships between variables.
ANNs has been used to detect and classify diseases, such as cancer, cardiovascular diseases, and diabetes; For example, ANNs was used to classify breast
cancer tissues by a three-layer ANN model to classify samples into benign and
malignant classes [8]. Their results showed that the ANN model was able to
accurately classify the samples with an accuracy of 97.6%. ANNs has also been
used to predict the onset of diseases and provide early diagnosis. For example,
one study used ANNs to predic the progression of coronary artery disease (CAD)
with 94.6% accuracy using the data collected using coronary angiography, and
trained using a backpropagation algorithm [3]. Other studies have used ANNs to
predict the risk of stroke [29], risk of heart failure [19] and drug-drug interactions
[28].
4
Experiments and Results
In this section, we discuss how we collected data that represents patients with
DCCs concerns and we used that data to test selected classifiers in predicting
optional treatment plans
Fig. 1. The proposed system. The user inputs the information via a tablet/smart
phone. The collected data is used to train an AI model. Then, the trained AI model is
used to provide the treatment recommendation given the new input data.
4.1
Data Collection
To get representative data, we distributed an online survey Fig. 2 and Fig. 1
to understand how to understand patients’ and healthcare provider’s strategies
for setting treatment plans and collect data on the strategies they made. Thirty
four (N=34) participants responded to the survey. Six were healthcare provider
244
T. Ongwere et al.
and 28 were patients. All the patient participants reported having one or more
chronic diseases (including type-2 diabetes, arthritis, and/or depression). The
majority (N = 20) patients participants were female. The participants recorded
the conditions (i.e., type-2 diabetes, or arthritis, or depression, or all of them)
that applied to their situation, and any three concerns (e.g., sleep, cost, weight,
symptoms, addiction, or effects on sexual performance) that were most important
to them and most optimal prescription. These responses were then collected to
form a data-set which was then used for creating our benchmarks.
Fig. 2. Survey Form for patients with DCC’s
This study (survey) was approved by our University’s Institutional Review
Board (IRB).
4.2
Experimental Settings
In this work, all experiments carried out on Machine learning algorithms focused
on testing the accuracy as a metric for evaluation and bench-marking every
algorithm. Initially we focused on the Random forest algorithm. We collected a
dataset for Diabetes Type 2 condition with three medications using the data generated from the Google survey responses. This data-set has 74 samples,including
independent variables- sleep, cost, weight, symptoms, addictiveness, effects on
sexual performance and a dependent variable-class. We have three classes in this
data-set. Our data cleaning process is discussed in the prior section.
Hamiltonian Mechanics
245
By performing data cleaning we filled up the missing data by marking blank
entries as ‘no’. The cleaned data-set now had ‘yes’ or ‘no’ values for the independent variables and class name for the dependent variable. Using Label encoder
we converted these strings into numerical values like 0’s and 1’s representing ‘no’
and ‘yes,’ respectively. The class independent variables were converted to ‘0’, ‘1’
and ‘2’.
Then, we split the dataset into 70% training and 30% testing. Initially, Random Forest Classifier model yields an accuracy of 30% with number of decision
trees as four (N = 4). Later the accuracy rate reduces to 25% as the number of
trees were increased to seven (N = 7). Again we revamped the dataset by keeping
only the unique entries (removing the duplicates) and obtained an accuracy of
26%. Table 1 shows the best preferred medications according to the individual
feature for Type 2 Diabetes:
Table 1. Priority of medications/treatments for each concern
Sleep
Cost
Empagliflozin
Canagliflozin
Metformin Metformin
Empagliflozin
Weight
Pain/Symptoms
Addictiveness
Metformin Empagliflozin Empagliflozin
Metformin
Canagliflozin
Sexual performance
Metformin
A dataset with 0’s and 1’s combinations was created with the class based on
the best preferred medication for each respective combination. We kept 10% as
test data and trained the model by using this preferred medication table above.
We obtained 75% accuracy. Later, we again updated the data-set by keeping
minimum preferences selected to two (2) and maximum selected to four (4).
After updating, we had sixty two (N = 62) rows in our data-set. We kept 10% as
test data and trained the model by using this updated data-set. We obtained 71%
accuracy. We then updated our data-set by removing Canagliflozin medication
and added Victoza medication and selected the class in the updated data-set
based on the below best preferred medication for each respective combination
(Table 2). We obtained 82% accuracy for 30% Test data and 83% accuracy for
10% Test data.
Table 2. Revised priority of medications/treatments for each concern
Sleep
Cost
Empagliflozin
Metformin
Metformin Victoza Metformin/Victoza Empagliflozin
Empagliflozin
Metformin
Weight
Pain/Symptoms
Addictiveness
Sexual Performance
Empagliflozin
Metformin
Canagliflozin
Metformin/Victoza
246
5
T. Ongwere et al.
Discussion
A total of 74 samples, 54 for training and 20 for testing, were passed through the
Random Forest Classifier Model and an increased accuracy of 85% was achieved.
Further we trained the same data for SVM model and obtained an accuracy of
95% for Linear kernel. Meanwhile, training the data in Neural Networks and
Nearest Neighbor model gave an accuracy of 59% and 60%, respectively. Table 3
shows the accuracy for each algorithm on which the data was experimented.
A total of 74 samples were formed in the data set with 54 as training and 20
as testing data. As per the above table, SVM showed the highest accuracy of
95 percent (Linear kernel) followed by Random forest. The SVM has a very
important approach called kernel. We solved any difficult issue by using the
related kernel function. Kernel allows the selection of nonlinear functions that
take on multiple shapes depending on the data, making it a non-parametric
function. There is a strong assumption in classification issues that data has
linearly separable samples, however with the introduction of kernel, input data
may be turned into high-dimensional data, eliminating the necessity for this
assumption. On out-of-sample data, SVM performs well and generalizes well.
SVM proves to be quick since it operates well on out of generalization sample
data. This is because the kernel function is assessed and done for each and
every support vector in SVM for the classification of one sample. When there
is a clear evidence of distinction between classes, SVM does not suffer from
over-fitting and performs effectively. When the total number of samples is fewer
than the number of dimensions, SVM may be employed, and it has a good
memory performance. Support Vector Machine is useful in finding the separating
Hyperplane. Finding a hyperplane can be useful to classify the data correctly
between different groups. SVMs can be robust even when the training sample is
biased, and one of the reasons for this is that they are better at delivering unique
answers than Neural Networks, which provide several solutions for each local
minima for various samples. Meanwhile, the random forest built a decision tree
based on the list of input features. Note that only 6 dimensions are used for the
input features. SVM is less computationally demanding than Nearest Neighbor
and is easier to interpret but can identify only a limited set of patterns. On the
other hand, Nearest Neighbor can find very complex patterns but its output is
more challenging to interpret.
Table 3. Algorithms implemented and its corresponding accuracy. The best performance is marked in boldface.
Algorithm
Accuracy
SVM
95%
Neural Networks 57%
Random Forest
85%
Hamiltonian Mechanics
247
We can avoid the explicit assumption of a linear class boundary by using the
nearest Neighbors algorithm. This algorithm determines the class of an unclassified point by counting the majority class vote from its nearest Neighbor training
points. For example, a patient whose symptoms closely match those of patients
with a specific diagnosis would be classified with the same disease status. Because
nearest neighbor does not assume a particular boundary between the classes, its
boundary can be closer to the “true” relationship. However, for a given training
set, predictions may be less stable than for SVMs, especially when k is small,
and the algorithm will often overfit the training data. The value of k acts to
regularize kNN, and is generally selected by cross-validation. To avoid ties in
the vote, k can be chosen to be odd. Small k gives a finely-textured boundary
which is sensitive to outliers and yields a high model variance (k = 3). Larger k
gives more rigid boundaries and high model bias (k = 7), pooling the effect of
more distant neighbors. The largest possible value of k is the number of training
points-at this extreme, any new observation is classified based on the majority
in the entire training sample incurring maximal model bias.
Neither Support Vector Machine nor Nearest Neighbor make explicit model
specifications about the data-generating process such as normality of the data.
However, linear SVM is considered a parametric method because it can only produce linear boundaries. If the true class boundary is nonlinear, SVM will struggle
to find a satisfying fit even with increased size of the training set. To help the
algorithm capture nonlinear boundaries, functions of the input variables, such
as polynomials, could be added to the set of predictor variables. This extension
of the algorithm is called kernel SVM.
In contrast, Nearest Neighbor is a nonparametric algorithm because it avoids
a priori assumptions about the shape of the class boundary and can thus adapt
more closely to nonlinear boundaries as the amount of training data increases.
Nearest Neighbor has higher variance than linear SVM but it has the advantage
of producing classification fits that adapt to any boundary. Even though the true
class boundary is unknown in most real-world applications, Nearest Neighbor
has been shown to approach the theoretically optimal classification boundary as
the training set increases to massive data. However, because Nearest Neighbor
does not impose any structure on the boundary, it can create class boundaries
that may be less interpretable than those of linear SVM. The simplicity of the
linear SVM boundary also lends itself more directly to formal tests of statistical
significance that give P values for the relevance of individual variables. There
are also trade-offs in the number of samples and the number of variables that
can be handled by these approaches. SVM can achieve good prediction accuracy
for new observations despite large numbers of input variables.
By contrast, the classification performance of Nearest Neighbor rapidly deteriorates when searching for patterns using high numbers of input variables when
many of the variables may be unrelated to the classification or contribute only
small amounts of information. Because equal attention is given to all variables,
the nearest neighbors may be defined by irrelevant variables. This so-called curse
of dimensionality occurs for many algorithms that become more flexible as the
248
T. Ongwere et al.
number of predictors increases. Finally, computation and memory resources are
important practical considerations. SVM only needs a small subset of training
points (the support vectors) to define the classification rule, making it often more
memory efficient and less computationally demanding when inferring the class
of a new observation. In contrast, Nearest Neighbor typically requires higher
computation and memory resources because it needs to use all input variables
and training samples for each new observation to be classified [4].
6
Conclusion and Future Work
In this paper, we address the problem of how patients with DCC’s often experience multiple obstacles when prioritizing treatment plans and prescriptions.
Treatment suggestions and medication interactions can be provided by a variety of machine learning (ML) or deep learning (DL) techniques by resolving
the problem of conflicting drugs. We collected the data for the data-set from
the survey responses of the patients. The data-set consisted of a single disease
(Diabetes Type 2) with three treatments, depending on the replies of the individuals. There are 74 samples in this data-set, with six independent factors (sleep,
cost, weight, symptoms, addictiveness, and impacts on sexual performance) and
one dependent variable (class) (that is the medication). Regarding the evaluation, we investigated different machine learning algorithms like Random Forest,
SVM, Nearest Neighbor and Neural networks on the above dataset. Following
the benchmarking, SVM achieved the highest performance with an accuracy of
95%.
In the future, we aim to collect more data for different conditions like depression and arthritis and evaluate different machine learning algorithms to overcome
the problem of drugs interaction for multiple chronic diseases.
References
1. Amarappa, S., Sathyanarayana, S.V.: Data classification using support vector
machine (SVM), a simplified approach. Int. J. Electron. Comput. Sci. Eng 3, 435–
445 (2014)
2. Anusha, M., Suresh, K., Chandana, M.: Earlier prediction on the heart disease
based on supervised machine learning techniques. In: 2021 5th International Conference on Intelligent Computing and Control Systems (ICICCS), pp. 1696–1703.
IEEE (2021)
3. Ayat, S., Khosravanian, A.: Identification and classification of coronary artery
disease patients using neuro-fuzzy inference systems. J. Math. Comput. Sci. 13,
136–141 (2014)
4. Bzdok, D., Krzywinski, M., Altman, N.: Machine learning: supervised methods.
Nat. Methods 15(1), 5 (2018)
5. Chang, Y., Chen, X.: Estimation of chronic illness severity based on machine learning methods. Wirel. Commun. Mob. Comput. 2021 (2021)
Hamiltonian Mechanics
249
6. Charleonnan, A., Fufaung, T., Niyomwong, T., Chokchueypattanakit, W., Suwannawach, S., Ninchawee, N.: Predictive analytics for chronic kidney disease using
machine learning techniques. In: 2016 Management and Innovation Technology
International Conference (MITicon), pages MIT–80. IEEE (2016)
7. Das, R.: A comparison of multiple classification methods for diagnosis of parkinson
disease. Expert Syst. Appl. 37(2), 1568–1572 (2010)
8. Desai, M., Shah, M.: An anatomization on breast cancer detection and diagnosis
employing multi-layer perceptron neural network (MLP) and convolutional neural
network (CNN). Clin. eHealth 4, 1–11 (2021)
9. Ding, Y., Tang, J., Guo, F.: Identification of drug-target interactions via multiple
information integration. Inf. Sci. 418, 546–560 (2017)
10. Emanet, N., Öz, H.R., Bayram, N., Delen, D.: A comparative analysis of machine
learning methods for classification type decision problems in healthcare. Decis.
Anal. 1(1), 1–20 (2014)
11. Gómez-Vallejo, H.J., et al.: A case-based reasoning system for aiding detection and
classification of nosocomial infections. Decis. Support Syst. 84, 104–116 (2016)
12. Jain, P., Chaudhary, S.: High performance based disease prediction by RFNN
algorithm. In: 2019 3rd International Conference on Electronics, Communication
and Aerospace Technology (ICECA), pp. 1293–1300. IEEE (2019)
13. Lachure, J., Deorankar, A.V., Lachure, S., Gupta, S., Jadhav, R.: Diabetic
retinopathy using morphological operations and machine learning. In: 2015 IEEE
International Advance Computing Conference (IACC), pp. 617–622. IEEE (2015)
14. Li, Q., et al.: An end-to-end hybrid algorithm for automated medication discrepancy detection. BMC Med. Inform. Decis. Mak. 15(1), 1–12 (2015)
15. Maharao, N., Antontsev, V., Wright, M., Varshney, J.: Entering the era of computationally driven drug development. Drug Metab. Rev. 52(2), 283–298 (2020)
16. Maseer, Z.K., Yusof, R., Bahaman, N., Mostafa, S.A., Foozy, C.F.M.: Benchmarking of machine learning for anomaly based intrusion detection systems in the
CICIDS2017 dataset. IEEE Access 9, 22351–22370 (2021)
17. Massafra, R., et al.: Radiomic feature reduction approach to predict breast cancer by contrast-enhanced spectral mammography images. Diagnostics 11(4), 684
(2021)
18. Mostafa, S.A., Mustapha, A., Khaleefah, S.H., Ahmad, M.S., Mohammed, M.A.:
Evaluating the performance of three classification methods in diagnosis of Parkinson’s disease. In: Ghazali, R., Deris, M.M., Nawi, N.M., Abawajy, J.H. (eds.) SCDM
2018. AISC, vol. 700, pp. 43–52. Springer, Cham (2018). https://doi.org/10.1007/
978-3-319-72550-5 5
19. Myers, J., et al.: A neural network approach to predicting outcomes in heart failure
using cardiopulmonary exercise testing. Int. J. Cardiol. 171(2), 265–269 (2014)
20. Ongwere, T., Cantor, G., Martin, S.R., Shih, P.C., Clawson, J., Connelly, K.:
Design hotspots for care of discordant chronic comorbidities: patients’ perspectives.
In: Proceedings of the 10th Nordic Conference on Human-Computer Interaction,
pp. 571–583 (2018)
21. Ongwere, T., Stolterman, E., Shih, P.C., James, C., Connelly, K.: Design for discordant chronic comorbidities (DCCs): a DC3 model. In: Lewy, H., Barkan, R.
(eds.) PH 2021. LNICST, vol. 431, pp. 346–362. Springer, Cham (2022). https://
doi.org/10.1007/978-3-030-99194-4 22
22. Pal, M., Parija, S.: Prediction of heart diseases using random forest. J. Phys. Conf.
Ser. 1817, 012009 (2021)
23. Piette, J.D., Kerr, E.A.: The impact of comorbid chronic conditions on diabetes
care. Diabetes Care 29(3), 725–731 (2006)
250
T. Ongwere et al.
24. Princy, R.J.P., Parthasarathy, S., Jose, P.H., Lakshminarayanan, A.R.,
Jeganathan, S.: Prediction of cardiac disease using supervised machine learning
algorithms. In: 2020 4th International Conference on Intelligent Computing and
Control Systems (ICICCS), pp. 570–575. IEEE (2020)
25. Rallapalli, S., Suryakanthi, T.: Predicting the risk of diabetes in big data electronic health records by using scalable random forest classification algorithm. In:
2016 International Conference on Advances in Computing and Communication
Engineering (ICACCE), pp. 281–284. IEEE (2016)
26. Saberi-Karimian, M., et al.: Data mining approaches for type 2 diabetes mellitus
prediction using anthropometric measurements. J. Clin. Lab. Anal. 37(1), e24798
(2023)
27. Senan, E.M., Jadhav, M.E.: Diagnosis of dermoscopy images for the detection of
skin lesions using SVM and KNN. In: Poonia, R.C., Singh, V., Singh Jat, D.,
Diván, M.J., Khan, M.S. (eds.) Proceedings of Third International Conference on
Sustainable Computing. AISC, vol. 1404, pp. 125–134. Springer, Singapore (2022).
https://doi.org/10.1007/978-981-16-4538-9 13
28. Shankar, S., Bhandari, I., Okou, D.T., Srinivasa, G., Athri, P.: Predicting adverse
drug reactions of two-drug combinations using structural and transcriptomic drug
representations to train an artificial neural network. Chem. Biol. Drug Des. 97(3),
665–673 (2021)
29. Shanthi, D., Sahoo, G., Saravanan, N.: Designing an artificial neural network model
for the prediction of thrombo-embolic stroke. Int. J. Biometric Bioinform. (IJBB)
3(1), 10–18 (2009)
30. Silva, P.A.B., Bernstam, E.V., Markowitz, E., Johnson, T.R., Zhang, J., Herskovic,
J.R.: Automated medication reconciliation and complexity of care transitions. In:
AMIA Annual Symposium Proceedings, vol. 2011, p. 1252. American Medical
Informatics Association (2011)
31. Tran, T.N.T., Felfernig, A., Trattner, C., Holzinger, A.: Recommender systems
in the healthcare domain: state-of-the-art and research issues. J. Intell. Inf. Syst.
57(1), 171–201 (2021)
32. Vincent, P., Bengio, Y.: K-local hyperplane and convex distance nearest neighbor
algorithms. Adv. Neural Inf. Process. Syst. 14 (2001)
33. Wong, J.D., et al.: Medication reconciliation at hospital discharge: evaluating discrepancies. Ann. Pharmacother. 42(10), 1373–1379 (2008)
34. Ye, C., et al.: Prediction of incident hypertension within the next year: prospective
study using statewide electronic health records and machine learning. J. Med.
Internet Res. 20(1), e22 (2018)
Sentiment Analysis for Predicting
the Variation Trend of Stocks: A Case
Study of Vanke Co., Ltd.
Qiuyi Jin and Jin Zheng(B)
University of Bristol, Bristol, UK
jin.zheng@bristol.ac.uk
Abstract. Stock prices constantly fluctuate due to changes in supply
and demand. This relationship between supply and demand is highly
sensitive to investor sentiment and current news. For example, when a
company announces a new product, it suggests the potential for increased
future earnings, typically leading to more people buying stocks and
driving up stock prices. In addition to official information like government economic reports and company and industry news, the attitudes of
traders are significant, as they can reflect a direct intention to sell or purchase stocks. Furthermore, interconnected companies, such as investors
and investees, may share revenue and exhibit similar stock trends, making it necessary to consider relationships between listed companies. Our
primary focus is on China Vanke Co., Ltd. (referred to as Vanke) and
its related companies, executives, industries, and concepts. We establish
information about companies and their relationships using a knowledge
graph. China Securities Journal and Sina Finance are chosen as data
sources for their credibility and extensive information. We compare different text classification algorithms and select the Convolutional Neural
Network (CNN) model to categorize news into different industries and
label them as either ‘positive’ or ‘negative.’ Additionally, we adopt the
Bidirectional Long Short-Term Memory (Bi-LSTM) model integrated
with a Conditional Random Field layer (CRF) to extract organizations,
individuals, and locations relevant to the news to uncover key information. The organizations and individuals identified are associated with the
companies and executives in the knowledge graph, linking the news to
the relevant entities. This research contributes to the field of sentiment
analysis in financial news, particularly with respect to stock price prediction and the identification of key factors influencing market trends.
Keywords: Named entity recognition · Text classification ·
Knowledge graph · Sentiment analysis
1
Introduction
Stock markets in China have witnessed significant growth in recent years,
although they are relatively young compared to their counterparts in Europe
c The Author(s), under exclusive license to Springer Nature Switzerland AG 2024
K. Arai (Ed.): SAI 2024, LNNS 1017, pp. 251–270, 2024.
https://doi.org/10.1007/978-3-031-62277-9_15
252
Q. Jin and J. Zheng
and the United States. However, the structure and characteristics of China’s
stock markets differ from those of other countries. While institutional investors
dominate stock markets in the United States, retail investors play a crucial role
in China’s stock markets, resulting in a highly dispersed ownership structure.
According to the Statistical Yearbook of the Shanghai Stock Exchange in 2019,
retail investors accounted for 99.68% of all accounts, with a staggering 213.79
million accounts opened as of December 31, 2018.
While the large number of retail investors brings a more abundant source of
funding, their limited investment capacity and the excessive influence of mainstream media can undermine the stability of the stock market. Previous research
conducted by [5] examined the historical stock prices in the Shanghai Composite Index from July 2nd, 1997 to January 7th, 2005. The study revealed that
retail investors in China tend to react excessively to negative news and events
while displaying an insufficient response to positive news and events. This behavior often leads to overbought or oversold market conditions. This suggests that
retail investors’ investment decisions are significantly influenced by news and
public opinion, making sentiment analysis a potential indicator of stock indices.
Although numerous studies have focused on stock prediction, they have primarily concentrated on analyzing the relationship between news or social media
discussions and the price of specific stocks, without considering the influence of
related companies and industries. [9] discovered that listed companies within
the same industry are similarly affected by industry prospects, highlighting
the improvement in stock price prediction when considering the relationships
between stocks, particularly when associated companies invest in the same product. Therefore, it is important to analyze the sentiment of news related to companies and industries when predicting stock variation trends.
This paper aims to extend the analysis of news and public discussions beyond
the scope of a company’s stocks. It incorporates news related to executives,
associated companies, and projects. To accomplish this, we employ a knowledge
graph that represents entities such as companies and executives, along with their
relationships. Additionally, sentiment analysis is performed, and key information
is extracted, including the identification of organizations and individuals mentioned in the news. This project highlights the consideration of sentiment indicators from related enterprises and executives, and introduces a knowledge graph
to establish connections between different entities. To enhance text classification
accuracy and efficiency, we combine the TextCNN model with a Lexicon-based
classifier. To illustrate the application of our approach, we conduct a case study
focusing on China Vanke Co., Ltd., a prominent company in the real estate
industry, and its affiliated companies.
An outline of the paper follows. Section 2 reviews the literature. Section 3
describes the system design, explaining the workflow of the entire pipeline and
the techniques involved. Section 4 describes the implementation of the models.
Section 5 demonstrates the results and evaluations, and Sect. 6 concludes.
Case Study of Vanke Co., Ltd.
2
Background
2.1
News Sentiment Analysis for Stock Prediction
253
Various techniques have been developed for sentiment analysis to classify news
into different sentiment categories. These techniques primarily fall into two categories: machine learning-based approaches and lexicon-based approaches. For
instance, Kalra et al. (see [13]) employed a Naive Bayes classifier to categorize
news articles based on positive or negative sentiment. Additionally, Fazlija and
Harder (2022) (see [8]) used the BERT model to generate a sentiment score for
each news article and predict the direction of stock price movements. Bhardwaj et al. (see [2]) demonstrated the feasibility of using sentiment signals from
text to predict volatility trends in the stock market using a corpus-based approach. Furthermore, researchers have explored the application of topic modeling
(see [20]) and event extraction (see [26]) in the task of stock price prediction.
Apart from news articles, the analysis of sentiment expressed in social media
discussions (see [3]) has also been employed to examine the influence of public
sentiment on stock market fluctuations. However, social media discussions are
more suitable for analyzing short-term situations rather than predicting longterm fluctuations.
Moreover, researchers have combined knowledge graphs with sentiment analysis to predict stock price variations (see [17]), taking into account the correlation
between different stocks. Additionally, through community detection on a news
co-occurrence network, Wan et al. (see [25]) discovered that news pertaining to
a particular company not only correlated with its own market performance but
also indicated the performance of other related companies and the industry as
a whole. To explore the association between the stock price trends of related
companies, knowledge graphs (see [10,18]), where entities are interconnected by
different relationships, have been used to uncover the relevance between news
and stocks interconnected by different relationships, have been used to uncover
the relevance between news and stocks.
2.2
Natural Language Processing Approaches
Text Classification. At the earliest stage, the text classification task is solved
using probabilistic and statistical models, such as the naive Bayes classifier.
With the development of supervised learning algorithms, the decision tree classifier (see [16]), Support Vector Machines (SVM) (see [12]), K-nearest neighbors
(KNN) classifier (see [15]), and neural networks, such as CNN, Recurrent Neural Network (RNN), and LSTM, have also been applied to the text classification
task.
In recent years, self-attention mechanisms have become a vital part of
sequence modeling, and an attention layer was added to many neural networks,
such as LSTM-Attention (see [1]), to improve their performance. Also, the Transformer model (see [23]), which entirely relies on the attention mechanism, was
254
Q. Jin and J. Zheng
introduced to solve the text classification task. To further optimize the performance of the Transformer model, Bidirectional Encoder Representations from
Transformers (BERT) (see [6]) introduced a bidirectional encoder and removed
the decoder to improve feature extraction capacity.
Based on empirical comparisons of CNN with other supervised learning algorithms, including SVM, logistic regression, and random forest (see [14]), no algorithm always performs the best in all datasets, but CNN outperforms the other
three algorithms in precision in most cases (9 out of 16 experiments). This means
that the performance of the model depends on the characteristics of the dataset
and the proportion of the training dataset. Based on a comparative study of
long document classification (see [24]), whose datasets were similar to the news
dataset used in this project, BERT outperforms other models, including Naı̈ve
Bayes and Bi-LSTM with attention, in all cases.
To determine which model should be used in the text classification section,
an experiment was conducted to compare the accuracy of LSTM, BERT, and
TextCNN models. TextCNN works slightly better than BERT, but it trains
significantly faster than BERT. The TextCNN model spends roughly the same
amount of time training but performs much better than LSTM. Thus, TextCNN
is selected as the model to classify financial news in this project.
Name Entity Recognition (NER). Named Entity Recognition (NER) is
used to identify named entities in sentences. Currently, there are primarily three
types of methods for recognizing named entities: methods based on dictionaries
and rules (Rau, Lisa F. 1991; Collins, Michael et al. 1999) [18,19], methods
based on statistics (Takeuchi K. et al. 2003) [20], and methods based on neural
networks.
The earliest method used in NER tasks is the rule-based method, which relies
on manually constructed corpora to identify entities through rules and dictionary
matching. While the rule-based method accurately identifies named entities, it
requires a substantial effort to define handcrafted rules, and the construction
period is quite lengthy.
Statistical-based methods were introduced in the late 1990s, and common
statistical methods used for NER include Maximum Entropy (see [4]), Support
Vector Machine (see [27]), Hidden Markov Model (see [29]), CRF (see [19]), etc.
However, these methods also suffer from long training times, and they heavily
depend on feature selection.
In recent years, with the emergence and development of word embeddings,
neural networks, including CNN, RNN, and LSTM, have been selected to address
NER tasks. Based on experiments in [7], methods based on neural networks outperform statistical-based methods. In those experiments, the Bi-LSTM model
performed the best among neural network models. To further enhance the performance of neural network models in NER tasks, some statistical methods have
been integrated with neural networks, such as Bi-LSTM-CRF and LSTM-CRF,
resulting in clear improvements (see [11]) in performance. As a result, Bi-LSTMCRF has been selected as the model for recognizing named entities in the project.
Case Study of Vanke Co., Ltd.
3
System Design
3.1
Overview of the System
255
Fig. 1. System design
Our system consists of four main sections: data collection, text classification,
named entity recognition, and visualization In addition to using a relational
database to store news and discussion-related information, we have incorporated
a non-relational database called Neo4j, which is a graph database. This allows
us to establish connections between different news articles and discussions based
on the organizations and individuals mentioned within them. By doing so, we
can predict the tendency of stock prices not only based on news directly related
to a listed company but also considering the impact of its affiliated companies
and top managers.
The first step involves gathering news articles and discussions from various
sources. We then translate the article titles and discussions from Chinese to
English to enhance reader comprehension. Subsequently, we classify the news
and discussions as either positive or negative for the stock market and extract the
organizations and individuals mentioned within the content. Finally, we display
all the results of the text analysis on our website, including the articles, their
classification results, and the connections between the news and their related
companies.
In the text classification section, we have introduced a lexicon-based sentiment analysis layer into the TextCNN model to enhance classification accuracy
and improve efficiency. The workflow could be found in Fig. 1. When analyzing
a news article or discussion, we first pass it through the lexicon-based layer. If
256
Q. Jin and J. Zheng
the generated sentiment score clearly indicates a positive or negative tendency,
we directly save the result from the lexicon-based layer in the database without
further classification by the TextCNN model.
However, if the lexicon-based layer provides an ambiguous response, such as
when neither positive nor negative words are prominent in the text or when
the count of positive and negative words is close (less than or equal to 5), we
employ a pre-trained TextCNN model to perform further analysis. To classify
the text using the TextCNN model, we initially encode the text into a sequence
of word IDs using a pre-built bag-of-words approach. The encoded sequence is
then padded to a length of 800, discarding any excess part if the article’s length
exceeds 800. The output of the TextCNN model is either 0, indicating a negative
sentiment in the news or discussion, or 1, indicating a positive sentiment.
3.2
Named Entity Recognition
In addition to the text classification section, we will extract organizations and
persons mentioned in the news using a pre-trained Named Entity Recognition
(NER) model.
The pre-trained model, vocabularies, and tag dictionaries are initially loaded.
We add < Start > and < End > tags at the beginning and end of an article to
comply with the requirements of the Conditional Random Field (CRF) layer.
Next, we encode the article with the start and end tags, creating a sequence
of word IDs. The NER model will generate a sequence of tag IDs that match the
length of the word ID sequence. We then decode these tag IDs into tag names.
Once we have the tags for each character, we check for the presence of
‘S-ORG’/‘M-ORG’/‘E-ORG’ (indicating the start character, middle characters,
and end character of an organization) or ‘S-PER’/‘M-PER’/‘E-PER’ (indicating
the start character, middle characters, and end character of a person) in the tag
sequence. We concatenate the characters from the start tag to the end tag to
form a complete name of the organization or person. To present multiple names
more clearly, we add a semicolon at the end of each name.
3.3
Database Structure
Non-relational Database - Neo4j. As displayed in Fig. 2, our company information graph contains five different node types and seven relationship types,
including most of the related entities and relationships that could influence a
specific company. The properties of each node and each relationship are displayed in Tables 1 and 2.
The direction of the relationship represents that the source node has an influence on the target node. For example, the scandal of the chief executive officer
has a significant impact on the reputation of the company, and the negative effect
on the company is ultimately reflected in the stock price, which is represented
by the “ListedInfo” node in the graph. The impact could be bidirectional. For
instance, the debt problem of shareholders influences the stock price, and the
company’s performance also affects the income of shareholders.
Case Study of Vanke Co., Ltd.
Fig. 2. Graph database structure
Table 1. Properties of each node
Node type
Attributes
Person
name, sex, yearOfBirth
Company
name, englishName, industry, unifySocialCreditCode
Project
name, englishName, foundingRound
ListedInfoGroup aShareAbbreviation, aShareCode, bShareAbbreviation,
bShareCode, industry, name, englishName
Table 2. Properties of each relationship
Source node type Target node type Relationship type
Company
Company
stockholder
Company
invested
Project
holds
ListedInfo
shanghai stock exchange
shengzhen stock exchange
beijing stock exchange
hongkong stock exchange
Project
Company
belongs to
Company
invested
Group
Company
member enterprise
Person
Company
works in
257
258
Q. Jin and J. Zheng
Because there are four stock exchanges in China, including the Beijing stock
exchange, Shanghai stock exchange, Shenzhen stock exchange, and Hong Kong
stock exchange, there are four types of relationships between the “Company”
node and the “ListedInfo” node. In the stock market, there are primarily two
types: A shares (paid in Renminbi and traded on the Shanghai stock exchange)
and B shares (paid in U.S. dollars and traded on the Shenzhen stock exchange).
Both stock codes and abbreviations are set as attributes of the “ListedInfo”
node.
When conducting sentiment analysis, we primarily consider the impact of
one-hop and two-hop neighbors, and we take the direction of the relationship
into account.
Relational Database - MySQL. We primarily have two tables in the MySQL
database: one for discussion analysis and the other for news analysis. When
fetching news from various sources, our first step is to classify them as either
‘favorable for the stock market’ (represented by 1) or ‘unfavorable for the stock
market’ (represented by 0). We then proceed to extract the organizations and
individuals mentioned in the news. Each organization or person is separated
by a semicolon. These entities correspond to the names of Company nodes and
Person nodes in the graph database, and they are used to connect the news
with the nodes. The final step in news analysis involves translation, where we
translate the news title from Chinese into English. Following these steps, we save
the analysis results in the text news table.
The structure of the table and the workflow for processing discussion-related
data are similar to that of news analysis. The only difference is that we do
not perform named entity recognition on them. This is because we can use
stock codes to link discussions with specific companies. The discussions from
investors are retrieved from stock forums, and only companies with connections
to ListedInfo nodes, indicating their presence in the stock market, have discussion
information.
3.4
Translation
To help more users understand the news and discussions, we will translate news
titles and discussion from Chinese to English. We chose Translator in googletrans
package to help us translate.
4
Implementation
4.1
Text Classification: Lexicon-Based Sentiment Analysis
and TextCNN Model
Dataset Description. We employed a pre-trained word embedding corpus for
Chinese words and phrases, developed by Tencent AI Lab, in the TextCNN model
(see [22]). This corpus provides 100-dimensional vector representations for over
Case Study of Vanke Co., Ltd.
259
12 million Chinese words and phrases. We chose Tencent AI Lab’s embedding
because of its extensive coverage of domain-specific and recently popular words,
as well as its ability to effectively capture the semantic meaning of Chinese words
and phrases.
Next, we collected over 3,000 news articles and 5,000 discussions from sources
such as Sina Finance, Hexun Real Estate, Hexun Technology, and the Guba stock
forum. We subsequently categorized them manually into positive and negative
corpora. As shown in the table, the number of positive and negative data samples
is similar for both news and discussions (see Table 3).
Table 3. Words for Lexicon-based sentiment analysis
Positive Negative
News
1848
1486
Discussions 2687
2318
Lexicon-Based Sentiment Analysis. We collected word frequencies in both
the negative and positive datasets and selected the top 15 words from each type
to create a lexicon (see Fig. 3). To calculate the sentiment score, we count the
occurrences of words from the positive and negative lexicons. The sentiment
score is determined by taking the difference between the count of positive words
and the count of negative words. If the sentiment score is greater than 5, it
indicates that the news tends to be positive. Conversely, if the sentiment score is
less than −5, it suggests that the news tends to be negative, and in these cases,
we conclude the text classification section. When the sentiment score falls within
the range of −5 to 5, it becomes uncertain for us to determine the sentiment
tendency. Therefore, we need to employ the TextCNN model for a more accurate
classification of the news or discussions.
4.2
TextCNN
To implement the TextCNN model, we employ a model that uses four convolution layers with kernel sizes of 6, 8, 10, and 12 for news, and 2, 4, 6, and 8 for
discussions. Following the convolution layers, we apply a ReLU activation function to introduce nonlinearity into the model. A max-pooling layer is used with
a kernel size determined by subtracting the kernel size of the convolution layers
from the maximum sequence length, which is 800. The choice of different kernel
sizes for news and discussions is based on their respective characteristics. Discussions among investors are relatively short, typically consisting of only 2–20 words,
making shorter kernel sizes suitable. In contrast, news articles tend to be much
longer, requiring larger kernel sizes to capture word relationships. After the convolution blocks, we concatenate the output of the max-pooling layer and apply
260
Q. Jin and J. Zheng
Fig. 3. Words for Lexicon-based sentiment analysis
dropout. We then flatten the multidimensional layer into a lower-dimensional
layer with two neurons for each of the classes (positive and negative) in our
dataset.
To build the TextCNN model in PyTorch, we create a TextCNN class that
inherits from the torch.nn.Module class. We use the torch.nn.Sequential container to combine our layers sequentially. The Conv1D(), ReLU(), and MaxPool1D() layers handle the convolution, activation, and pooling operations.
We apply torch.nn.functional.dropout to introduce dropout to the convolution
results. Finally, the Linear() layer is used as the linear layer in the final step.
4.3
Named Entity Recognition: Bi-LSTM and CRF Model
We used the Resume dataset collected by Yue Zhang et al. (see [28]). This
dataset is derived from Sina Finance and includes the resumes of senior executives from listed companies in the Chinese stock market. These resumes are vital
for establishing connections between news and nodes in the graph.
The dataset has numerous categories of entities, including organization
(ORG), person (PER), title (TITLE), location (LOC), education (EDU), race
(RACE), country (CONT), and others (O). Each character is labeled with its
corresponding category, with the first character of a word, middle characters,
and the last character of a word prefixed with ‘B-’, ‘M-’, and ‘E-’, respectively.
Each character and its label are presented on a single line, separated by whitespace, and sentences are separated by a blank line. Figure 4 is an example of NER
results.
To implement the Bi-LSTM model, we employ a model that includes a single
Bi-LSTM layer with 100 hidden units. This model has the number of tags, which
is 22, as its output size. Additionally, a transition matrix is incorporated after the
Bi-LSTM layer. To build the Bi-LSTM and CRF model in PyTorch, we create
a BiLSTM CRF class that inherits from the torch.nn.Module class, similar to
the TextCNN model. We use the nn.LSTM cell with bidirectional set to True
for the Bi-LSTM model. A linear layer, implemented with the Linear() cell, is
used to convert the output of the nn.LSTM cell into the emission matrix. We
Case Study of Vanke Co., Ltd.
261
Fig. 4. Structure of NER training dataset. Translation: “Currently, he is the general
manager of China Datang Group”
use nn.Parameter() to represent the transition matrix, initializing all entries in
the transition matrix to 1.
5
Evaluation
5.1
An Empirical Analysis of the Correlation Between Sentiment
in News/Discussion and Stock Price
To investigate the correlation between sentiment in news/discussions and stock
prices, We used the positive ratio of individual investors’ discussions in stock
forums on each business day and the positive ratio of industry-related news as
sentiment indicators. Our analysis extended beyond assessing the relationship
between stock prices and Vanke’s own sentiment indicators. We also incorporated
the sentiment indicator of an adjacent node, Tahoe Group Co., Ltd., in our
knowledge graph to explore the impact of related listed companies.
We began by conducting an Augmented Dickey-Fuller (ADF) test to determine whether the time series data were stationary. The results indicated that all
sequences were non-stationary and integrated of order one. We then employed
the first-order difference of sentiment indicators and stock prices as input for
Granger causality tests to examine whether the sentiment indicators caused
changes in stock prices. For a comprehensive analysis of the relationship, we
retrieved related discussions and news from April 14, 2022, to July 14, 2022.
These were classified as positive or negative sentiment. To ensure the consistency of data length, we combined the counts of discussions and news from
weekends or holidays with the most recent business day. Additionally, we calculated the daily positive discussion ratio to provide a more accurate reflection
of sentiment. To assess the impact of sentiment indicators from related companies and individuals on Vanke, we collected news and discussions from entities
and individuals within one-hop or two-hops in the knowledge graph. Using the
same methodology described earlier, we calculated positive ratios, considering
the average of the daily positive discussion ratio and the positive news ratio from
10 days prior.
262
Q. Jin and J. Zheng
Stationarity Unit Root Test – Dick-Fuller (DF) Test and Augmented
Dickey-Fuller (ADF) Test. According to the Tables 4 and 5 below, we can
deduce that all of the time series are non-stationary, but their first-order differences are stationary.
Table 4. Unit root test
Series name
Test statistic P-value
1%
Critical value
5%
10%
Stock price
−2.16
0.84
−3.55 −2.91 −2.59
Positive ratio of Vanke’s discussion
−1.85
0.21
−3.56 −2.91 −2.59
Positive ratio of Vanke’s news
0.34
0.97
−3.56 −2.91 −2.59
Sentiment indicator of one-hop neighbors
−1.86
0.34
−3.55 −2.91 −2.59
Sentiment indicator of two-hops neighbors −1.52
0.18
−3.54 −2.91 −2.59
Table 5. Unit root test of first order difference
Series name
Test statistic P-value
Critical value
1%
5%
10%
Stock price
−3.42
0.01
−3.55 −2.91 −2.59
Positive ratio of Vanke’s discussion
−6.01
1.51e−07 −3.56 −2.91 −2.59
Positive ratio of Vanke’s news
−3.47
0.008
Sentiment indicator of one-hop neighbors
−8.66
4.69e−14 −3.55 −2.91 −2.59
Sentiment indicator of two-hops neighbors −7.53
3.42e−11 −3.54 −2.91 −2.59
−3.56 −2.91 −2.59
Granger Causality Test. The Granger causality test is adopted to determine
whether one time series is useful for forecasting another. A crucial premise of the
Granger causality test is that the time series should be stationary. Therefore,
we used the first-order difference of the sentiment indicators and stock prices as
input.
As shown in Fig. 5, both news and discussions have an influence on the change
in stock prices, underscoring the significance of our system’s implementation.
However, it’s worth noting that the lag in news is higher than the lag in discussions, indicating that the market responds slightly more slowly to sentiment in
news, while the sentiment in discussions more directly reflects investors’ sentiment.
Figure 6 illustrates the causality relationship between the sentiment indicators of Vanke’s neighboring entities and its stock price after the second-order difference. We initially applied Granger Causality Tests to the sentiment indicators
and stock price after the first-order difference. However, we found no causality
relationship between them. Subsequently, we conducted tests on the time series
Case Study of Vanke Co., Ltd.
263
after the second-order difference. As depicted in Fig. 6, the null hypothesis could
be rejected, confirming the necessity of including the knowledge graph in our system. Although the lags in the sentiment indicators of adjacent nodes are higher
than the lags of Vanke itself, it is evident that related listed companies still have
a causal relationship with Vanke’s stock price.
Fig. 5. Causality test of discussion and news of enterprise itself
Fig. 6. Causality test of sentiment indicators of neighbors
5.2
Non-linear Time Series Modeling
The relationships among sentiment indicators and stock prices are exceedingly
intricate due to the presence of time-lagged effects and the existence of both
short-run and long-run equilibrium. Once the causality relationship between sentiment indicators and stock prices has been confirmed, we will employ a Vector
Error Correction Model (VECM) to delve deeper into the relationship between
stock prices and sentiment indicators in both news and discussions. This will
also enable us to forecast the movement of stock prices and sentiment indicators
in the near future.
264
Q. Jin and J. Zheng
To construct a VECM model, our initial step involves utilizing a Vector
Autoregressive (VAR) model to determine the optimal lag. Subsequently, we
use the Johansen Co-integration Test to identify the co-integration rank and to
assess whether a long-run relationship exists between the time series data.
Vector Autoregressive (VAR). The vector autoregressive model (VAR) is
raised by Sims in 1980 [21] to model the interrelationships among a set of macroeconomic variables, and he advocated that each variable as a linear function of
past lags of itself and past lags of other variables. V ar(p) is defined as:
yt = c + A1 yt−1 + ... + Ap yyt−p + et
(1)
where yt = (y1t , ..., yKt ) is a set of K time series variables, c is a K × 1 vector
of constant, A is a K × K coefficient matrices, and et is error term.
The VAR model necessitates that all time series variables possess the same
order of integration. Based on the ADF tests conducted in Sect. 4.1.1, we can
conclude that all the time series are integrated of order one, rendering them
suitable for the VAR model. Subsequently, we can determine the optimal lag,
which is 9, by employing the Akaike Information Criterion (AIC) based on the
VAR model’s results (see Table 6). The lag order for the Johansen cointegration
test is 1 less than the optimal order of the VAR model, resulting in an optimal
lag of 8 for the Johansen cointegration test.
Table 6. VAR order selection
Lag AIC
BIC
FPE
HQIC
0
−18.92
−18.73
6.086e−09
−18.84
1
−18.94
−17.80
5.979e−09
−18.51
2
−19.57
−17.49
3.276e−09
−18.77
3
−19.80
−16.77
2.798e−09
−18.65
4
−20.84
−16.86
1.157e−09
−19.32
5
−21.17
−16.25
1.081e−09
−19.29
6
−21.46
−15.59
1.275e−09
−19.21
7
−23.46
−16.64
3.647e−10
−20.86
8
−26.28
−18.51
8.240e−11
−23.31
9
−33.45∗ −24.74∗ 9.858e−13∗ −30.12∗
Johansen Co-integration Tests. For lag p > 1, VAR model could be written
as:
p−1
Γi Δyti + et
(2)
Δyt = Πyt−1 +
i=1
Case Study of Vanke Co., Ltd.
265
j=p
j=p
where Π = j=1 Aj − Ik and Γi = − j=i+1 Aj .
If the matrix Π is a matrix of 0, then it means that the set of variables are
not co-integrated. To test whether Π = 0, we could check the rank of Π. If the
variables are co-integrated, then rank(Π) = 0 and the rank of Π equals to the
number of co-integration vectors.
As can be seen in Table 7, the rank for sentiment indicators and stock price
is 5, and all test statistics are smaller than critical values which means that we
could reject null hypothesis.
Table 7. Johansen cointegration test using trace test statistic with 1% significance
level
r0 r1 test statistic critical value
0
5
160.7
77.82
1
5
106.5
54.68
2
5
65.37
35.46
3
5
31.22
19.93
4
5
9.750
6.635
Vector Error Correction Models (VECM). VECM is a VAR model specifically designed for co-integrated time series. In contrast to the Johansen cointegration test, VECM model not only considers the co-integrating relationships
between variables but also incorporates short-run adjustment dynamics.
Suppose that there are two variables which are co-integrated and no lagged
difference terms,
(3)
y2,t = βy1,t
then the corresponding VECM is
Δy1,t = α(y2,t−1 − βy1,t−1 ) + e1,t
(4)
where alpha measures the speed of adjustment of the i-th endogenous variable
towards the equilibrium, and the beta coefficients are the actual long-run relationship coefficients. The beta for the stock price is standardized at 1 for ease
of interpretation of the other beta coefficients. The ec1 (beta.1), ec2 (beta.2),
ec3 (beta.3), ec4 (beta.4) and ec5 (beta.5) correspond to stock price, positive
ratio of Vanke’s news, positive ratio of Vanke’s discussion, sentiment indicator of
one-hop neighbors and sentiment indicator of two-hops neighbors respectively.
As observed in Fig. 7, the long-term influences of all variables appear to be
weak. However, in terms of short-term deviations, investor discussions and sentiment indicators of one-hop neighbors demonstrate a clear positive effect on stock
price. Surprisingly, the alpha coefficients of the positive ratio of Vanke’s news and
the sentiment indicator of two-hop neighbors are negative, which contradicts our
266
Q. Jin and J. Zheng
expectations. We believe this discrepancy may be attributed to the differences
in lags among the variables, as indicated by the results of the Granger Causality
tests. Since we used the same lag in the VECM model, the alpha coefficients are
not necessarily positive for all variables. Notably, all alpha coefficients for the
variables are statistically significant at the 1% and 5% levels.
Fig. 7. Alpha and Beta of VECM model
5.3
Evaluation of Text Classification Algorithm
We compared four text classification models - TextCNN, TextCNN with a lexicon
layer, BERT, and LSTM - in terms of accuracy, training time, and validation
time based on the news dataset. As shown in Table 8, the TextCNN model with
a lexicon layer excelled in all three aspects, so we chose it as the model to be
used in our system.
Neural networks offer the advantage of high precision; however, the training
and validation times for these networks are relatively slow. The Lexicon Classifier, on the other hand, is the simplest and fastest model for sentiment classification, but its accuracy is quite low. This is because it relies solely on word
counts in the positive and negative corpora and doesn’t consider the mutual
relationships between words.
When integrating the lexicon layer with the TextCNN model, more than half
of the news and discussions are initially classified by the lexicon layer. These
don’t need to be input into the TextCNN model, resulting in reduced validation
Case Study of Vanke Co., Ltd.
267
time. Furthermore, the lexicon classifier provides a rough labeling of the text
based on manually built lexicons, and the TextCNN model can further refine
the labeling when the lexicon classifier’s response is unclear.
Table 8. Comparison of TextCNN, TextCNN with Lexicon layer, Bert and LSTM
model
Algorithm
5.4
Accuracy Training time Validation time
(min/epoch) (s/100 pieces of text)
TextCNN with lexicon 85.45
2.4
16
TextCNN
2.4
21
84.63
Bert
84.62
65
20
LSTM
74.25
2.5
18
Evaluation of Named Entity Recognition Algorithm
We evaluated four named entity recognition algorithms, which included HMM,
CRF, Bi-LSTM, and Bi-LSTM integrated with CRF. We compared their performance at the label level.
As depicted in Fig. 8, the HMM model performs the worst, especially when
the support is small. This is primarily due to the initialization of the emission
matrix of the HMM model before training, which is not optimized during the
training process. The Bi-LSTM model integrated with the CRF layer outperforms the plain Bi-LSTM, particularly for the beginning and end characters,
thanks to the introduction of the transition matrix.
While the Bi-LSTM model with the CRF layer performs slightly worse than
both the plain Bi-LSTM and the CRF model, it excels in recognizing organizations and names, which are the primary focus of our system. Consequently,
we have chosen the Bi-LSTM with the CRF layer as the model for extracting
mentioned enterprises and individuals.
268
Q. Jin and J. Zheng
Fig. 8. Heat maps of different NER models
6
Conclusion and Future Work
This paper primarily focuses on the relationship between sentiment indicators
in news and discussions and stock price fluctuations, as well as algorithms for
extracting entities from news and classifying news and discussions as either positive or negative in terms of their impact on stock prices.
We initiated our study by validating that the sentiment indicators and stock
prices share the same order of integration, a result confirmed through the ADF
test. Furthermore, we demonstrated that sentiment indicators play a crucial role
and provide valuable information for forecasting stock prices, as evidenced by
the Granger Causality Test.
Additionally, we introduced a text classification model that combines a
lexicon-based sentiment analysis layer with the TextCNN text classification
model. This hybrid model enhances model performance and reduces the time
required to classify the collected news and discussions. To further link news with
specific companies or executives, we used the Bi-LSTM model integrated with
the CRF layer for entity extraction from news. We also harnessed a knowledge
graph to analyze the influence of related companies and executives.
In the future, we should take the graph structure into account. To delve
deeper into the study of the influence of related enterprises and individuals, we
Case Study of Vanke Co., Ltd.
269
could use algorithms designed to reveal concealed connections among various
entities within the graph. Prominent examples of such algorithms include the
Graph Convolutional Network (GCN) and the ConvLSTM model.
References
1. Bai, X.: Text classification based on LSTM and attention. In: 2018 Thirteenth
International Conference on Digital Information Management (ICDIM), pp. 29–32
(2018)
2. Bhardwaj, A., Narayan, Y., Vanraj, P., Dutta, M.: Sentiment analysis for Indian
stock market prediction using sensex and nifty. Procedia Comput. Sci. 70, 85–91
(2015). Proceedings of the 4th International Conference on Eco-friendly Computing
and Communication Systems
3. Bollen, J., Mao, H., Zeng, X.J.: Twitter mood predicts the stock market (2010)
4. Borthwick, A.E.: A maximum entropy approach to named entity recognition. Ph.D.
thesis, USA (1999). AAI9945252
5. Chen, M.W., Zhu, J.: Do investors in Chinese stock market overreact? J. Account.
Financ. Res. 13(3), 17–25 (2005)
6. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: pretraining of deep
bidirectional transformers for language understanding (2019)
7. Dhumal Deshmukh, R., Kiwelekar, A.: Deep learning techniques for part of speech
tagging by natural language processing. In: 2020 2nd International Conference on
Innovative Mechanisms for Industry Applications (ICIMIA), pp. 76–81 (2020)
8. Fazlija, B., Harder, P.: Using financial news sentiment for stock price direction
prediction. Mathematics 10(13), 1–19 (2009)
9. Feng, F., He, X., Wang, X., Luo, C., Liu, Y., Chua, T.S.: Temporal relational
ranking for stock prediction. Papers (2019)
10. Fu, X., Ren, X., Mengshoel, O.J., Wu, X.: Stochastic optimization for market
return prediction using financial knowledge graph. In: 2018 IEEE International
Conference on Big Knowledge (ICBK) (2018)
11. Huang, Z., Xu, W., Yu, K.: Bidirectional LSTM-CRF models for sequence tagging
(2015)
12. Joachims, T.: Text categorization with Support Vector Machines: learning with
many relevant features. In: Nédellec, C., Rouveirol, C. (eds.) ECML 1998. LNCS,
vol. 1398, pp. 137–142. Springer, Heidelberg (1998). https://doi.org/10.1007/
BFb0026683
13. Kalra, S., Prasad, J.S.: Efficacy of news sentiment for stock market prediction. In:
2019 International Conference on Machine Learning, Big Data, Cloud and Parallel
Computing (COMITCon) (2019)
14. Keeling, R., et al.: Empirical comparisons of CNN with other learning algorithms
for text classification in legal document review (2019)
15. Kwon, O.-W., Lee, J.-H.: Text categorization based on k-nearest neighbor approach
for web site classification. Inf. Process. Manage. 39(1), 25–44 (2003)
16. Lewis, D.D., Ringuette, M.: A comparison of two learning algorithms for text
categorization, December 1994
17. Liu, J., Lu, Z., Du, W.P.: Combining enterprise knowledge graph and news sentiment analysis for stock price prediction. In: Hawaii International Conference on
System Sciences (2019)
270
Q. Jin and J. Zheng
18. Matsunaga, D., Suzumura, T., Takahashi, T.: Exploring graph neural networks for
stock market predictions with rolling window analysis. Papers (2019)
19. McCallum, A., Li, W.: Early results for named entity recognition with conditional
random fields, feature induction and web-enhanced lexicons. In: Proceedings of
the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pp.
188–191 (2003)
20. Nguyen, T.H., Shirai, K.: Topic modeling based sentiment analysis on social media
for stock market prediction. In: Proceedings of the 53rd Annual Meeting of the
Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Beijing, China,
July 2015, pp. 1354–1364. Association for Computational Linguistics (2015)
21. Sims, C.A.: Macroeconomics and reality. Econometrica 48(1), 1–48 (1980)
22. Song, Y., Shi, S., Li, J., Zhang, H.: Directional skip-gram: explicitly distinguishing
left and right context for word embeddings. In: Proceedings of the 2018 Conference
of the North American Chapter of the Association for Computational Linguistics:
Human Language Technologies, Volume 2 (Short Papers), New Orleans, Louisiana,
June 2018, pp. 175–180. Association for Computational Linguistics (2018)
23. Vaswani, A., et al.: Attention is all you need (2023)
24. Wagh, V., Khandve, S., Joshi, I., Wani, A., Kale, G., Joshi, R.: Comparative study
of long document classification. In: TENCON 2021 - 2021 IEEE Region 10 Conference (TENCON). IEEE, December 2021
25. Wan, X., Yang, J., Marinov, S., Calliess, J.P., Dong, X.: Sentiment diffusion in
financial news networks and associated market movements. Sci. Rep. 11(1) (2021)
26. Xiao, D., Yue, Z., Liu, T., Duan, J.: Using structured events to predict stock price
movement: an empirical investigation. In: Proceedings of the 2014 Conference on
Empirical Methods in Natural Language Processing (EMNLP) (2014)
27. Yamada, H., Kudo, T., Matsumoto, Y.: Japanese named entity extraction using
support vector machine (2002)
28. Zhang, Y., Yang, J.: Chinese NER using lattice LSTM. In: Proceedings of the
56th Annual Meeting of the Association for Computational Linguistics (Volume
1: Long Papers), Melbourne, Australia, July 2018, pp. 1554–1564. Association for
Computational Linguistics (2018)
29. Zhou, G., Su, J.: Named entity recognition using an HMM-based chunk tagger.
In: Proceedings of the 40th Annual Meeting of the Association for Computational
Linguistics, pp. 473–480, Philadelphia, Pennsylvania, USA, July 2002. Association
for Computational Linguistics (2002)
Suicide Ideation Prediction Through
Deep Learning: An Integration of CNN
and Bidirectional LSTM with Word
Embeddings
Christianah T. Oyewale1(B) , Ayodeji O. J. Ibitoye2 , Joseph D. Akinyemi3 ,
and Olufade F. W. Onifade4
1
Mohammed VI Polytechnic University, Rabat, Morocco
oyewalechristianahtitilope@gmail.com, christianah.oyewale@um6p.ma
2
University of Greenwich, London, UK
3
University of York, York YO10 5GH, UK
4
University of Ibadan, Ibadan, Nigeria
Abstract. Suicide ideation prediction received attention in public
health because it affects not only the victim but the general public. Identifying suicide ideation can be traditionally tricky and time-consuming,
and the victim might have harmed himself or herself before detection.
Easy access to social media provided textual data for Deep Learning (DL)
models like Convolutional Neural Networks (CNN) and Long Short-Term
Memory (LSTM) for the identification of suicide ideation. These models are more promising at identifying suicide ideation than traditional
Machine Learning models because they can be applied to every form
of datasets. The main challenge is combining the DL models and word
embeddings. This research employed the DL model by combining CNN
with bidirectional LSTM and two-word embedding techniques, FastText
and GloVe (global vectors for word representation). The research questions are as follows: Will there be a difference in F1 score between twoword embeddings when combined with CNN-BiLSTM? And will the
overfitting issue in the existing study be addressed? Using an existing
study as a baseline, experimentation was done using a secondary Reddit dataset of 232,074 posts, achieving a 94% F1-score for FastText and
GloVe. It was observed that GloVe could do better without approximating the F1-score and FastText when the running time is considered.
Keywords: Suicide ideation · FastText · GloVe · Word embedding ·
Public health · CNN-BiLSTM
1
Introduction
Suicide ideation is the process of thinking of harming oneself to find a solution to
one’s problem [12]. Attention is paid to suicide attempts by the public because
of the negative impact they have not only on the victim but on the general
c The Author(s), under exclusive license to Springer Nature Switzerland AG 2024
K. Arai (Ed.): SAI 2024, LNNS 1017, pp. 271–283, 2024.
https://doi.org/10.1007/978-3-031-62277-9_16
272
C. T. Oyewale et al.
public [14]. Suicide can be categorized into three categories, namely individual,
social, and situational factors. Individual factors can be, for example, caused by
mental illness and trauma [15]. A social factor, for example, is caused by stigma
and the use of social media. Situational factors can be, for example, caused by
different kinds of bullying and drugs or alcohol [15].
The suicide rate in different countries varies from country to country. From
Fig. 1, the six countries with the highest suicide rates were India, China, the
United States of America (USA), Russia, Japan, and Nigeria, respectively [13].
The suicide rates are evident, which shows the importance of identifying suicidal
thoughts in victims quickly before they attempt them. However, it is a complex
and time-consuming task for the first set of researchers, which is the clinicians
having to predict suicide ideation in patients [4,7]. Knowing that suicide is a
public health issue, it has received importance in identifying early. Internet connectivity brought easy access to social media, which people from their comfort
zones can pour out of their minds without fear of stigma [2]. This made many
social media datasets available for deep learning models, which can handle large
amounts of unstructured data.
Fig. 1. Six countries with world’s highest suicidal rates [13]
Deep Learning models coming into place to support combating suicide ideation
require combining them correctly with word embedding. This task is a way of
optimizing suicide ideation prediction in social media text.
This research is dedicated to optimizing suicide ideation prediction (SIP) by
exploring the correct combination of classifiers, word embeddings, and datasets.
Drawing two research questions, one of which is whether there will be a difference in the F1 score between two-word embeddings when combined with CNNBiLSTM. Extending from an existing study and using the CNN-BiLSTM model
Hamiltonian Mechanics
273
and a dataset authored by the same researcher, this study employed FastText and
GloVe word embeddings [2]. Filling the research gap in this researcher’s study,
which primarily used word embedding, which has a problem with identifying
out-of-vocabulary words and achieving a high F1 score due to the overfitting
observed in the study, the second research question was drawn, which is whether
the overfitting issue in the existing research will be addressed. This study aligns
to identify suicide ideation and reduce the suicide rate accurately.
The remaining parts of this study are structured, with Sect. 2 discussing
related works. Section 3 explains the methodology used in this research. Section 4
presents the result and an analysis of the result. Lastly, the concluding parts of
the research can be found in Sect. 5.
2
Related Woks
As established in the previous section, the first set of researchers that began
the task of predicting suicide ideation is the medical professionals [1,5,10]. An
example of such a task by these professionals is checking the correlation of suicide
ideation among a group of individuals with some other emotional conditions
like depression, psychiatric distress, and situational conditions [8,9]. There is a
limit to the time that a medical professional can invest in this kind of task daily
because it involves manual approaches like going through patients’ clinical notes,
self-surveys, and phone interviews. This traditional method is time-consuming
and challenging to access globally in real time. The dataset derived from this
kind of research can be small, but a lot of time would have to be invested for it to
scale higher. Such a dataset can be analyzed thematically by clustering related
themes, using Natural Language Processing (NLP), or carrying out a statistical
analysis like regression analysis [3,8,9].
In addition to the efforts of early researchers and the availability of many
datasets on the internet, machine learning tools became popular among individuals, irrespective of whether they came from medical backgrounds. Some of
these researchers looked into detecting different emotional contents from social
media to classify texts as suicidal or not suicidal [6,11]. They employed the use
of varying dataset sizes. However, the standard practice for dataset size is to use
a dataset that is less than 50,000. However, optimizing models like deep learning
models requires considerable datasets. Another common practice is word2Vec
word embedding, which suffers from handling out-of-vocabulary text. Another
word embedding commonly used in text classification is GloVe word embedding,
but FastText word embedding is rarely used. Trying out other word embeddings
with deep learning models is essential to classifying texts accurately. As mentioned, existing researchers have different sizes of datasets and types of word
embedding, and a high F1 score has been achieved in previous studies. The
analysis of such studies showed the style employed in splitting the dataset was
early research used 90% for training and 10% for testing [11]. However, having a
dataset of less than 50,000 requires a different data-splitting strategy. The performance of a SIP model relies on the correct combination of word embeddings,
classifiers, datasets, and other techniques.
274
3
C. T. Oyewale et al.
Methodology
This section of the research explains the methodology used. Discussion on the
Reddit dataset, the preprocessing of the dataset, the splitting of the dataset,
feature extraction, the word embedding, the classifier, and the evaluation metrics
used (see Fig. 2).
Fig. 2. Generic methodology
3.1
Dataset
This research used a dataset from existing research authored by Aldhyani et
al. [2]. The dataset consists of 232,074 Reddit posts from “SuicideWatch” and
is divided into two equal classes (see Fig. 3). The suicidal class consists of more
negative words and data points. It is also uniformly distributed. On the other
side, the second class, which is the non-suicidal class, consists of more positive
and neutral words and data points. The dataset is skewed to the right. Replicating what was done in the existing result, the data was divided into 70% for
training, 10% for validating the model, and 20% for testing the model.
The dataset was visualized using different visualization techniques to draw
out some more information. A word frequency bar chart was used to check the
first 20 most-used words and the number of times they appeared. This provided
insight into the most frequently used suicide and non-suicide words. Similar to
the word frequency bar chart is the word cloud, which was used to show the
most commonly used words in the dataset.
From Fig. 4, parts of the commonly used words are “die”, “end”, “thought”,
“suicidal”, “pain,” and so on. And from Fig. 5, part of the common words used
are “life”, “help”, “friend”, “school,” and so on. The frequency of how the words
appear in Fig. 4 and Fig. 5 is proportional to the number of times that the words
appear.
Hamiltonian Mechanics
Fig. 3. Sample of the dataset
Fig. 4. Suicidal text word cloud snapshot
Fig. 5. Non-Suicidal text word cloud snapshot
275
276
C. T. Oyewale et al.
3.2
Data Preprocessing
Preparing the dataset for the deep learning model involves vectorizing the data;
this stage is known as data preprocessing. This process involves turning all the
posts into lowercase, removing special characters, stopping words, lemmatization, and tokenization. The removal of the unique character consists of removing
punctuation marks, emojis, and numbers. This helped in the easy preprocessing of text. Stop words like “the”, “a”, “an” and “in” were also removed. They
were removed since they didn’t have any significant usage in the identification
of suicide ideation in text. The texts were turned into their based form using
lemmatization. Tokenization involves breaking down a text into smaller units.
This step involved turning the text into a unique numerical integer. Replicating
the existing method, the maximum vocabulary size was set to 30,000. Feeding
equal input size into the neural network, the padding technique was employed.
The maximum padding was set to 430; this was decided based on the existing
study that this research extended from. Inputs not up to 420 in size were padded
with zero, and those more than that were truncated. Finally, the target vector
in categorical form was encoded by turning each into its numerical form.
3.3
Word Embedding
Two-word embeddings were used separately with the deep learning classifier: a
pre-trained word embedding, which is GloVe [17], and a custom-trained word
embedding, which is FastText [16].
FastText extends from word2Vec, but it solves the issue word2Vec has with
handling out-of-vocabulary (OOV) words. Solving the OOV problem through the
implementation of training using n-gram, FastText can quickly identify words
that have not been seen before because it might have come across the smaller
unit of such words. FastText was created by Facebook and is a valuable resource
for natural language processing tasks, thanks to its distinctive method for dealing with out-of-vocabulary words and subword details. The training data for
the fastText used in this study includes sources like Wikipedia 2017, UMBC
webbase corpus, and the statmt.org news dataset, encompassing a vast vocabulary of approximately one million words. The dimensionality of FastText is 300,
which significantly enhances its ability to capture intricate language nuances.
These characteristics make FastText suitable for various applications, including
sentiment analysis and text classification.
GloVe, an abbreviation for Global Vectors for Word Representation, is a
widely employed pre-trained word embedding method that provides distinctive
insights into word connections and their occurrence frequencies within a given
text corpus. Unlike other techniques, GloVe focuses on capturing the semantic significance of words and emphasizes the statistical patterns of their cooccurrences in extensive language corpora. One of GloVe’s notable strengths lies
in its capacity to analyze word relationships based on the contexts in which they
are found. This capability allows GloVe to uncover intricate semantic connections and subtleties, which can be valuable for various natural language process-
Hamiltonian Mechanics
277
ing tasks. For this research, the GloVe implementation employed has a vocabulary size of approximately 840 billion words. It was trained on a large combined
dataset, specifically the Wikipedia 2014 and Gigaword 5th Edition corpora. With
a dimensionality of 300, the resulting word vectors produced by GloVe are exceptionally expressive, effectively capturing the nuances of word meanings and their
proximities within a given text. GloVe’s ability to explore semantic relationships
and frequency patterns makes it a potent instrument for comprehending word
semantics and their contextual usage. Additionally, its pre-trained nature on
extensive text data significantly enhances its suitability for a wide range of natural language understanding and generation tasks.
3.4
CNN-BiLSTM Model
Convolutional Neural Networks (CNNs) [18] are proficient in processing textual data by detecting localized patterns and facilitating extracting essential
features from sentences, even in extensive texts. Combining CNN with Bidirectional Long Short-Term Memory (BiLSTM) addresses the challenge of capturing
distant dependencies in sequential data [20]. While CNNs excel at recognizing
localized patterns, BiLSTM [19] handles text sequences in both forward and
backward directions, comprehensively comprehending dependencies across different textual segments. This fusion leverages the strengths of both architectural
components.
Fig. 6. CNN-BiLSTM model snapshot
The architectural representation in Fig. 6 comprises the following vital elements:
1. Embedding Layer and Random Initialization: Input text undergoes
encoding into a numerical format using pre-existing word embeddings such
as FastText or GloVe. The input layer, representing the dataset, is fed into the
embedding layer, which transforms the dataset into a numeric representation.
278
C. T. Oyewale et al.
2. Three Stacked One-Dimensional CNNs: These layers identify specific
patterns and characteristics in textual data, progressively acquiring hierarchical representations. The output of the embedding layer is transmitted into
a one-dimensional convolutional layer (1D CNN). Since three 1D CNNs are
employed, the output of the first 1D CNN serves as the input to the second
1D CNN, and the output of the second serves as the input to the third.
3. Bidirectional Long Short-Term Memory (BiLSTM): This layer captures the text’s extensive dependencies and temporal context. The output of
the third 1D CNN is utilized as input for the BiLSTM layer.
4. MaxPooling for Dimensionality Reduction: Max-pooling extracts the
most critical information from the BiLSTM’s outputs and reduces the number
of nodes in the BiLSTM’s output. The output is subsequently directed into
the fully connected layer.
5. Sigmoid Activation for Classification: The final layer reveals the text’s
likelihood of conveying thoughts related to suicide, assisting in categorizing
the text into either a suicide or non-suicide classification.
The training procedure involves monitoring training, validation loss, and accuracy to assess the learning process and potential overfitting. Test loss and accuracy are calculated to evaluate the model’s performance on new, unseen data.
The model makes predictions on the test dataset, enabling further analysis.
Detailed metrics, such as precision, recall, F1 score, and a visual confusion
matrix, offer insights into the model’s effectiveness and areas of error.
4
Results and Discussion
This research assessed the performance of the two CNN-BiLSTM models by
employing evaluation metrics such as accuracy, precision, recall, and F1-score.
Contrasting this study’s model with a baseline model accentuates the strengths
and enhancements in our approach. This section provides an all-encompassing
perspective on the model’s capacities and constraints, facilitating its practical
utility in real-world scenarios.
4.1
Result
This section briefly details the CNN-BiLSTM model with FastText word embedding and CNN-BiLSTM model with GloVe word embedding.
CNN-BiLSTM Model with FastText Word Embedding. The initial
model, CNN-BiLSTM with FastText as word embeddings, displayed signs of
overfitting, evident in its F1-score of 93%, indicating a need for enhancements. Various attempts, such as adjusting embedding dimensions and introducing dropout layers, had a limited impact on performance. Incorporating a
one-dimensional (1D) Convolutional layer and max-pooling yielded only slight
improvements. Reducing the training epoch to 3 displayed a trend toward
Hamiltonian Mechanics
279
improvement, albeit with a slight drop in the F1-score to 91%. Initial efforts
involving early stopping and regularization did not yield satisfactory results.
Further refinement of regularization techniques, including removing L1 regularization, stabilized the training process but introduced some oscillations.
Adding an extra BiLSTM layer increased complexity and led to fluctuations
in the accuracy curve, resulting in a 1% accuracy improvement. Adhering to
insights from prior research led to a substantial increase in the F1-score to 93%.
By closely adhering to the model architecture reported in the literature, we
successfully replicated their documented F1 score of 95% [2].
Exploring configurations with stacked 1D CNNs and a single Bi-LSTM layer
resulted in an F1 score of 92%. Fine-tuning the learning rate and eliminating
early stopping ultimately led to convergence and an impressive F1 score of 94%
(see Fig. 7).
Fig. 7. CNN-BiLSTM model with FastText word embedding snapshot
CNN-BiLSTM Model with GloVe Word Embedding. Using the FastText
architecture as the baseline, the GloVe model architecture includes an embedding
layer with non-trainable GloVe word embeddings, followed by three Conv1D
layers with ReLU activations and two dropout layers for overfitting prevention.
It also features a bidirectional LSTM layer, a global max-pooling layer, and
two dense layers with ReLU activation, followed by a final dense layer with
sigmoid activation for binary classification. The model was compiled with the
Adam optimizer, using binary cross-entropy loss and accuracy as the evaluation
metrics. It’s trained over five epochs with mini-batches of 64 instances and a 10%
validation split. Transferring the knowledge gained while building the FastText
model, the GloVe model didn’t require much fine-tuning to achieve 94% F1 Score
(see Fig. 8).
280
C. T. Oyewale et al.
Fig. 8. CNN-BiLSTM model with GloVe word embedding snapshot
Table 1. Model evaluation result
Model
4.2
Accuracy Precision Recall F1 Score
CNN (FastText) 0.9356
0.9356
0.9356 0.9356
CNN (GloVe)
0.9394
0.9393 0.9393
0.9329
Discussion
This section provides the interpretation of the result, relates the result to an
existing study, and addresses the study’s limitations.
Model Interpretation and Evaluation Metrics. Both models achieved an
approximate accuracy of 94%, but when considering precision beyond rounding,
the GloVe model outperformed FastText (see Table 1). However, it’s important
to note that if speed is a critical factor, FastText would be the preferred choice as
it runs more efficiently. The FastText model excels in various aspects, including
high true negatives and true positives with fewer false negatives and relatively
fewer false positives, showcasing its balanced and robust performance with an
accuracy of 0.9356. On the other hand, the GloVe model also performs well, with
high true positives and true negatives, but has a relatively higher count of false
positives when compared to FastText. The GloVe model effectively identifies
actual positive instances but might benefit from further optimization to reduce
false positives (see Fig. 9 and Fig. 10) with a precision of 0.9394 and recall of
0.9393, higher than FastText.
Comparison with an Existing Model. As reported by Aldhyani et al. [2],
achieved an impressive F1 score of 95%. However, it displayed overfitting, as
Hamiltonian Mechanics
281
Fig. 9. FastText confusion matrix
evidenced by the increasing training accuracy, while the validation accuracy
plateaued or declined. In contrast, the researchers’ second experiment resulted
in an F1 score of 87%, with the accuracy curve converging, indicating an absence
of overfitting. This served as the baseline for the comparison in this study.
Employing a sigmoid activation function for binary classification and finetuned hyperparameters, this study demonstrated convergence in the accuracy
curve, reaching a peak F1 score of 94%. Notably, this study model outperformed
the initial model [2]. It’s worth emphasizing the significance of addressing overfitting, as highlighted by the accuracy curve of the original model. With a higher
Fig. 10. GloVe confusion matrix
282
C. T. Oyewale et al.
minimum F1 score, this study model underscores efforts to optimize and create
a robust model for identifying suicidal content in text data.
This study used the same dataset as the existing research, which is not guaranteed to be generalized to other social media texts. Approximating the F1
score is also an additional limitation because the difference in decimal numbers
indicates a difference between the performances of both models.
5
Conclusion
Suicide ideation prediction is primarily the public health professional’s task.
The knowledge of the combination of word embedding and classifiers helps in
the enhancement of model’s accuracy. This journey in this research led to the
development of the text classification models, which provided 94% F1 score, a
percentage below the F1 score of the first experiment of the existing research,
which overfits. However, this result is higher than the second experiment’s, where
the same existing model result didn’t overfit. Through our comparative analysis,
we have highlighted its superiority over existing models. In future endeavors, this
research will optimize the comparison between both types of word embeddings,
using other social medial texts and other parameters to evaluate the models’
results.
This research underscores the importance of creating precise suicide detection systems and encourages collaboration between mental health organizations
and technology companies. It also emphasizes the need to keep ethical considerations and user privacy protection at the forefront of responsible and effective
technology implementation.
References
1. Ahn, S.Y., Yu, S., Kim, J.E., Song, I.H.: The relationship between suicide bereavement and suicide ideation: analysis of the mediating effect of complicated grief. J.
Affect. Disord. 331, 43–49 (2023). https://doi.org/10.1016/J.JAD.2023.03.008
2. Aldhyani, T.H.H., Alsubari, S.N., Alshebami, A.S., Alkahtani, H., Ahmed, Z.A.T.:
Detecting and analyzing suicidal ideation on social media using deep learning and
machine learning models. Int. J. Environ. Res. Public Health 19(19), 1–16 (2022).
https://doi.org/10.3390/ijerph191912635
3. Davidson, J.E., et al.: Job-Related problems prior to nurse suicide, 2003–2017: a
mixed methods analysis using natural language processing and thematic analysis. J.
Nurs. Regul. 12(1), 28–39 (2021). https://doi.org/10.1016/S2155-8256(21)00017X
4. Dong, L., Kalesnikava, V.A., Gonzalez, R., Mezuk, B.: Beyond depression: estimating 12-months prevalence of passive suicidal ideation in mid- and late-life in
the health and retirement study. Am. J. Geriatr. Psychiatry 27(12), 1399–1410
(2019). https://doi.org/10.1016/j.jagp.2019.06.015
5. Fartacek, C., Kunrath, S., Aichhorn, W., Plöderl, M.: Therapeutic alliance and
change in suicide ideation among psychiatric inpatients at risk for suicide. J. Affect.
Disord. 323, 793–798 (2023). https://doi.org/10.1016/J.JAD.2022.12.028
Hamiltonian Mechanics
283
6. Ghosh, S., Roy, S., Ekbal, A., Bhattacharyya, P.: CARES: CAuse recognition for
emotion in suicide notes. In: Hagen, M., et al. (eds.) ECIR 2022. LNCS, vol.
13186, pp. 128–136. Springer, Cham (2022). https://doi.org/10.1007/978-3-03099739-7 15
7. Huang, Y., et al.: Comparison of three machine learning models to predict suicidal
ideation and depression among Chinese adolescents: a cross-sectional study. J.
Affect. Disord. 319, 221–228 (2022). https://doi.org/10.1016/J.JAD.2022.08.123
8. Ji, S., Pan, S., Li, X., Cambria, E., Long, G., Huang, Z.: Suicidal ideation detection:
a review of machine learning methods and applications. IEEE Trans. Comput. Soc.
Syst. 8(1), 214–226 (2021). https://doi.org/10.1109/TCSS.2020.3021467
9. Kang, C., et al.: Prevalence, risk factors and clinical correlates of suicidal ideation
in adolescent patients with depression in a large sample of Chinese. J. Affect.
Disord. 290, 272–278 (2021). https://doi.org/10.1016/j.jad.2021.04.073
10. Lauderdale, S.A., Martin, K.J., Oakes, K.R., Moore, J.M., Balotti, R.J.: Pragmatic screening of anxiety, depression, suicidal ideation, and substance misuse in
older adults. Cogn. Behav. Pract. 29(1), 105–127 (2022). https://doi.org/10.1016/
J.CBPRA.2021.06.003
11. Malviya, K., Roy, B., Saritha, S.: A transformers approach to detect depression in
social media. In: 2021 International Conference on Artificial Intelligence and Smart
Systems (ICAIS), pp. 718–723 (2021). https://doi.org/10.1109/ICAIS50930.2021.
9395943
12. NIH: Frequently Asked Questions About Suicide (2023)
13. Rebecca, A.: Nigeria has highest suicide rate in Africa, sixth globally. International
Centre for Investigative Reporting (2019)
14. Sinha, P.P., Mahata, D., Mishra, R., Shah, R.R., Sawhney, R., Liu, H.: #suicidal
- a multipronged approach to identify and explore suicidal ideation in twitter. In:
International Conference on Information and Knowledge Management, Proceedings, pp. 941–950 (2019). https://doi.org/10.1145/3357384.3358060
15. WHO, et al.: World Health Organization. Department of Mental Health and Substance Abuse. Preventing Suicide: a Global Imperative. World Health Organization
(2014)
16. Joulin, A., Grave, E., Bojanowski, P., Douze, M., Jégou, H., Mikolov, T.: Fasttext.zip: compressing text classification models. arXiv preprint arXiv:1612.03651
(2016)
17. Pennington, J., Socher, R., Manning, C.D.: Glove: global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural
Language Processing (EMNLP), pp. 1532–1543, October 2014
18. O’Shea, K., Nash, R.: An introduction to convolutional neural networks (2015).
arXiv preprint arXiv:1511.08458
19. Huang, Z., Xu, W., Yu, K.: Bidirectional LSTM-CRF models for sequence tagging
(2015). arXiv preprint arXiv:1508.01991
20. Lin, L., Chen, X., Shen, Y., Zhang, L.: Towards automatic depression detection:
A BiLSTM/1D CNN-based model. Appl. Sci. 10(23), 8701 (2020)
A Multi-clustering Unbiased Relative
Prediction Recommendation Scheme
for Data with Hidden Multiple Overlaps
Avivit Levy(B) , Michal Chalamish, B. Riva Shalom, Guy Sharir,
Opal Peltzman, and Sivan Salzmann
Shenkar College of Engineering, Design and Art, Ramat Gan, Israel
{avivitlevy,michalh,rivash}@shenkar.ac.il
Abstract. This paper presents a recommendation system designed for
shortening and streamlining hiring processes on both candidate and
recruiting company parts, which is specialized in Hi-Tech companies.
In this case, the hidden multiple overlaps challenge arises, where the
trivial matching of data items to types does not necessarily reflect a
ground truth labelling and other hidden multiple matches may exist. To
this end, we propose a multi-clustering unbiased relative prediction recommendation scheme. Our mechanism enables to efficiently deal with
data containing hidden multiple overlaps. The multi-clustering allows a
many-to-many matching of items to types, enabling to expose hidden
item-user matches. The mechanism’s efficiency is vital for supporting
a dynamic modeling implementation, which is required for the Hi-Tech
recruiting application. Moreover, we design an unbiased relative prediction scheme for providing recommendations. Our prediction scheme is
two-sided (but asymmetric): ranking the multiple items for a given user
and (differently) ranking the multiple users for a given item. Though the
underlying problem of overlapping clustering and multi-label learning is
complicated, our scheme enables to bypass this difficulty in a conceptually exquisite and efficient way for recommender systems, in particular,
for the Hi-Tech recruiting process.
Keywords: Overlapping clustering · Recommender system · Unbiased
prediction · Hi-Tech recruiting process
1
Introduction
Hiring process is complicated and time-consuming. Recruiting automation can
significantly improve the process and liberate recruiters from manual tasks,
thereby saving companies time and money. From the job seeker perspective, the
process is time-consuming and exhausting. According to estimates, 75% of the
companies expect automation to play a significant role in their hiring process.
Market capitalization of automated recruiting companies is expected to reach
3 billion dollars by 2025 [32]. Therefore, a recommendation system supporting
c The Author(s), under exclusive license to Springer Nature Switzerland AG 2024
K. Arai (Ed.): SAI 2024, LNNS 1017, pp. 284–302, 2024.
https://doi.org/10.1007/978-3-031-62277-9_17
A Multi-clustering Scheme
285
hiring processes on both candidate and recruiting company parts is a very useful
tool.
Recommendation engines are machine learning (ML) tasks, which are
designed to predict items that users are most likely to be interested in. Users
and services providers have improved the quality and decision-making process
with such systems [22]. A recommender system filters important information out
of a large volume based on user-provided data and other factors, finding out a
user-item match and imputing their similarities. Recommendation methods are
reviewed in Subsect. 2.1.
Researchers proposed several recommendation systems for shortening and
streamlining hiring processes. Gutiérrez et al. [13] present an interactive dashboard that enables job seekers to explore the labor market in a personalized way
based on their skills and competencies. Frid-Nielsen’s system [10] generates occupational recommendations by utilizing a novel data set consisting of administrative records covering the entire Danish workforce. They explore how well can different models predict each users’ next occupation and find that gradient-boosted
decision tree models provide the best recommendations for future occupations
in terms of mean reciprocal ranking and recall. However, their study raises concerns regarding trade-offs between model accuracy and ethical issues, including
privacy and the social reinforcement of gender divides. Yang et al. [34] propose
a dual-perspective graph representation learning approach to model directed
interactions between candidates and jobs in order to model the two-way selection preference from the dual-perspective of job seekers and employers.
Clustering is an important common Artificial Intelligence (AI) mechanism
enabling to manipulate, track and analyze individual pieces of data. Clustering
can serve to segment users based on their user data and behavioral patterns, such
as browsing history, likes or reviews, in order to target them with personalized
product or content suggestions. Identifying non-disjoint clusters is an important
issue referred to as overlapping clustering [24]. Traditional clustering methods
ignore the possibility that an observation can be assigned to several groups and
lead to exhaustive and exclusive clusters representing the data. Overlapping
clustering methods offer a richer model for applications requiring a non-disjoint
partitioning. Dealing with clusters overlaps has long been studied leading to
several methods in the literature [5,8,15,25,30,35,37]. A review of overlapping
clustering methods appears in Subsect. 2.2.
Our recommendation system for the recruiting process is based on a prediction algorithm suggesting suitable candidates for a company’s job offer, as well
as suitable job offers for a candidate, where the ML based prediction algorithm
reasons about past data using a clustering algorithm.
1.1
The Recruiting Process Hidden Multiple Overlaps Challenge
The hidden multiple overlaps challenge refers to a situation where the straightforward matching of data items to assumed types does not necessarily reflect a
ground truth labelling and other hidden multiple matches may exist. Existing
methods for overlapping clustering either are inadequate to expose such hidden
286
A. Levy et al.
multiple overlaps, rely on key assumptions (e.g. a probabilistic model) or suffer
from high cost (see Subsect. 2.2). We present a multi-clustering mechanism in
order to efficiently cope with such overlaps within a recommendation engine.
Our database contains employees of several Hi-Tech companies (see Sect. 4).
This database is rather homogeneous regarding the properties among employees
in each of the companies as well as among the companies. Moreover, it is very
reasonable that an employee of one Hi-Tech company may also be suitable for a
job in another Hi-Tech company. Therefore, this database should not be treated
as representing a ground truth labelling regarding the matching of employees
to their hiring companies. Indeed, we may assume to a certain extent that an
employee is suited to the current employing company. Yet, we cannot assume
that it is not suitable for other companies as well. Nevertheless, we want to infer
the possible non-explicit many-to-many matching of candidates to companies
from this database.
Paper Contributions. In this paper we:
1. Present an efficient multi-clustering recommendation mechanism to deal with
data containing hidden multiple overlaps. The multi-clustering allows a manyto-many matching of items to types, enabling to expose hidden item-user
matches. The mechanism’s efficiency is vital for supporting a dynamic modeling implementation, which is required for the Hi-Tech recruiting application.
2. Design an unbiased relative prediction scheme for providing recommendations
based on such data. Our prediction scheme is two-sided (but asymmetric):
ranking the multiple items for a given user and (differently) ranking the multiple users for a given item.
3. Apply these mechanisms to a recommendation system specialized in Hi-Tech
companies supporting hiring processes on both candidate and recruiting company parts.
2
Previous Work
2.1
Recommendation Systems Methods
Various recommender system techniques have been proposed and many sorts of
software have been developed for a variety of applications [19,22]. Recommendation systems common approaches include collaborative filtering (CF) – making
choices based on the opinions of other people who share similar interests [29],
content-based (CB) – recommending items that are similar to items previously
preferred by a specific user [27] and knowledge-based techniques – offering items
to users based on knowledge about the users, items and/or their relationships [6].
Each approach has advantages and limitations; for example, CF has sparseness,
scalability and cold-start problems, while CB has over-specialized recommendations [2].
To solve these problems, many advanced recommendation approaches have
been proposed, such as social network-based recommender systems [14], fuzzy
A Multi-clustering Scheme
287
recommender systems [18], context awareness-based recommender systems [3]
and group recommender systems [21]. Hybrid filtering techniques, which combine several strategies, are also suggested [4,7]. Computational intelligence techniques to construct recommendation models include Bayesian techniques, artificial neural networks, clustering techniques, genetic algorithms and fuzzy set
techniques [19,28]. This paper uses clustering techniques to create a recommendation engine.
2.2
Coping with Overlapping Clusters
Overlapping clustering methods have been studied for the last four decades
leading to several methods in the literature. These methods adapt many usual
approaches such as hierarchical, generative, graphical and K-means based approach [24]. However, most of the known overlapping clustering algorithms suffer
from high cost [30]. A naive approach consisted of running well-known algorithms
(K-means, fuzzy-K-means, EM, etc.) and modifying the result obtained to produce overlapping clusters [8]. Modifications to such algorithms are performed
by a threshold deciding whether an object belongs to a cluster according to its
proximity with the cluster. This approach outlines two fundamental problems
according to Cleuziou [8]: first, the algorithm initially used aims at optimizing
an objective function under constraints that do not match the expected clustering; secondly, the choice of a threshold, known as the “thresholding problem”,
remains unsolved.
Cleuziou [8] proposed Overlapping K-Means (OKM), which is a center-based
method extending K-means. However, it is initialization-sensitive (similar to the
K-means algorithm), which means that the results heavily depend on the chosen
initial cluster centroids [30]. OKM extensions have been proposed over the years.
Cleuziou [9] suggested two algorithms: (1) OKMED, which is an extension of the
K-medoids method, and (2) Weighted-OKM (WOKM) having local weighting of
the clusters indicating the importance of the different features. N’Cir et al. [25]
proposed the Kernel Overlapping K-Means algorithm (KOKM) to detect non linearly separable clusters based on a distance measure. Since OKM is very sensitive
to its hyper-parameters, where a slight variation can lead to a large variation of
overlaps size, N’Cir et al. [23] proposed the Parameterized R-OKM method to
control overlapping boundaries between clusters. Sina et al. [30] suggested using
a method called K-harmonic means algorithm, which integrates the harmonic
average to the objective function of the K-means algorithm as an approach of
initialization in OKM. Jiyoung Whang et al. [15] proposed an intuitive objective
function called Non-Exhaustive, Overlapping K-Means (NEO-K-Means), that
captures the issues of overlap and non-exhaustiveness in a unified manner.
Another relevant research direction is learning with ambiguity, which has
become a hot topic among the machine learning communities [11]. Multi-label
learning (MLL) is a common choice to deal with label ambiguity, because an
instance can be annotated by multiple labels simultaneously [36]. By Zhanget
al. [35], multi-view label enhancement main challenge is integrating multiple
types of heterogeneous information. A stochastic sketch strategy for MLL is
288
A. Levy et al.
provided by Gong et al. [12], which proposed the construction of a sketch matrix
to reduce expensive matrix multiplications. Wang et al. [33] proposed the Discrete Multiple Kernel (DMKKM) clustering model, which is able to directly
obtain cluster indicator matrix avoiding information loss and over reliance on
extra discretization procedures, while being parameter-free and avoiding hyperparameter tuning. Existing MLL methods divide the label set into relevant and
irrelevant labels, which can not be extended to label distribution form [37]. Partial label learning (PLL) was suggested to learn a discriminative model under
incomplete supervision, where each instance is annotated with a candidate label
set. The principle of PLL is that the unknown correct label of an instance resides
in its candidate label set [20].
The Chosen Implemented Method. Our multi-clustering scheme is designed
for an unsupervised learning based recommendation system. Multi-labels
describing the user-item matches do not explicitly exist in our scenario. Moreover, they should be inferred while exposing hidden matches other than the
trivial ones of employees to their employing company. Hence, existing methods
that require multi-labels or pre-define the overlaps structure (e.g., only at clusters boundaries) are inadequate here. Moreover, we prefer to rely solely on the
observed data. We do not assume user active feedback or a probabilistic model as
these limit the system practicality and scope. We, therefore, chose to deal with
the hidden multiple overlaps challenge using an overlapping clustering method
enabling to avoid such assumptions as much as possible. In addition, dynamic
modelling is an important feature of the system, which is supported by periodical re-modeling when 5% of the data is updated. As a result, low computational
cost is a crucial requirement from the used overlapping clustering method. We,
therefore, developed a multi-clustering partitioning-based mechanism tailored
for a recommender system to efficiently cope with the hidden multiple overlaps
scenario.
3
A Multi-clustering Relative Prediction
Recommendation Scheme
In this section we present our multi-clustering relative prediction recommendation scheme, while applying its principles to the Hi-Tech recruiting recommendation system case. Our scheme relies solely on the observed data via clustering and avoids user active feedback or the need of labeling. In order to expose
the hidden user-item matches through the multi-clustering model computation,
the scheme exploits data records relationships using a distance measure without
directly referring to the explicit user-item matches. Nevertheless, a distance measure usage is not sufficient for exposing and utilizing hidden user-item matches.
The scheme consists of the following three major principles:
1. Unified User-Item Modeling – A unified intermediate data representation
modeling both users and items.
A Multi-clustering Scheme
289
2. Multi-Clustering – A generic clustering enabling a many-to-many matching
between users and items. Different multi-clusters are characterised by different
types of the intermediate model data representation. Our multi-clustering has
two replaceable components:
(a) Distance Measure – Used as an underling tool for both the multiclustering process and the two-sided prediction mechanism;
(b) Distance-Based Clustering Algorithm – Used as an underling tool for the
multi-clustering process.
The distance measure as well as the distance-based clustering algorithm can
be adapted according to the specific application data requirements and preferences, while retaining the other scheme components. The scheme is, therefore,
independent of these choices.
3. Two-Sided Relative Prediction – A mechanism enabling to generate a ranked
list of recommended items for a user on the one hand, and, on the other hand,
a ranked list of recommended users for an item.
Below we describe these principles and demonstrate them on the Hi-Tech recruiting recommendation system case.
3.1
Unified User-Item Modeling
The choice of an intermediate model suitable for a unified modeling of the relevant users and items data is a crucial design step for exposing hidden multiple
overlaps. An intermediate model should enable a rich representation of each data
entity, while highlighting the potential shared attributes. Items can be modeled
as their users profile or vise-versa – users can be modeled by their preferred
items profile. The choice is data/application depended.
A unified modeling can be applied to a Hi-Tech recruiting recommendation
system using a person professional profile as an intermediate data representation model highlighting their shared attributes. It naturally fits the existing
companies employees database records. In addition, it is capable of describing
candidates professional profiles (as a CV representation) or a required professional profile for a job-offer. Figure 1 demonstrates how a unified modeling can
be applied to a Hi-Tech recruiting recommendation system using an intermediate
data representation model.
3.2
Multi-clustering
For the sake of explanation, assume that the unified intermediate model is via
users profile. Database items are then only represented using their collection of
users profile1 . A distance-measure based clustering algorithm can find disjoint
clusters of data items having a similar user profile. Figure 2 visualizes how a
clustering of employees database to professional personality types is used to
classify a candidate job-seeker profile to a professional personality type.
1
Our employees database is represented as their professional profiles. Companies are
represented by their employees profiles.
290
A. Levy et al.
Fig. 1. A unified model for a company existing employees, a candidate job-seeker and
a company job-offer highlights shared attributes
Multi-clustering and Sub-clusters Structure. The term multi-clustering
reflects the inner-structure of each user profile cluster. The unified modeling
enables considering both database users and items through a common intermediate representation. Hence, each cluster may contain users of several items all
sharing the same user profile. Thus, the cluster is a multi-cluster regarding the
matching of users to items. The multi-cluster structure is refined to sub-clusters
in order to avoid a biased ranking, for which there are increasing concerns about
in machine learning [10,16,17]. Ranking items according to the portion of itemusers records within a cluster may be biased towards items having more records
in the database. Therefore, the different users of each item within a cluster are
grouped by item as a sub-cluster having a sub-cluster center, which is an item
representative in the cluster. This enables a bias-free ranking: a user classified
to a cluster matching its profile is recommended the cluster-items list ranked
according to the user-profile distance from each item sub-cluster center (see Subsect. 3.3). Figure 3 presents a refined multi-cluster view for the Hi-Tech recruiting
recommendation system case.
3.3
Two-Sided Relative Prediction
Given a user, the profile representation is created, then the matching user-profile
cluster in the multi-clustering is sought. Thus, the user can be recommended the
list of cluster items ranked by its relative distance from each sub-cluster center,
which represents the users profiles of an item. Given an item, a similar procedure
can be applied by finding the matching user-profile cluster and recommending
the list of cluster users ranked by their relative distance from the item’s profile.
However, another relative prediction procedure can be performed. A sub-group
A Multi-clustering Scheme
291
Fig. 2. (a) Candidate classification by professional personality, (b) Clustering employees database to professional personality types
Fig. 3. A refined view of a multi-cluster structure use for the Hi-Tech recruiting recommendation system
of users sharing a property can be selected via a database query with a specified
attribute value. After the item user-profile representation is created, a ranked
list of this users sub-group by their profiles relative distance from the item’s
profile can be computed. A two-sided asymmetric relative prediction scheme can
be applied to the Hi-Tech recruiting recommendation system, where the ranked
users sub-group for a specific job-offer are the open position applicants:
1. for a candidate user - ranking companies items in a multi-cluster by their
representing center-profile distance to the candidate,
2. for a job-offer item - ranking the job-offer applicants profiles by their distance
to the job offer profile.
292
4
A. Levy et al.
TALENT.AI Recommender System
Talent.AI Server, Client, clustering algorithm code and data links are:
1. https://github.com/sivansalzmann/Final-project-Talent.AI-server,
2. https://github.com/sivansalzmann/Final-project-Talent.AI-client,
3. https://github.com/GuySharir/Talent.AI.
Data. Talent.AI recommendation system data is based on real data bought
from People Data Labs [1], a B2B data provider that has a data-set of over 3
billion unique person profiles and 18 million company profiles. We bought part of
this data of engineering domain, by defining specific attributes that must appear
in the retrieved records. Among the defined required attributes are: ‘education’,
‘experience’, ‘interests’, ‘skills’, ‘age’ and ‘gender’. We used ‘People Data Labs’
API service in order to receive the required data. Our database has 919 Hi-Tech
employees records, each containing 18 attributes. In order to ensure anonymity,
identifying information was removed prior to records use. Table 1 describes HiTech companies database records distribution.
Table 1. Hi-tech companies employees distribution
Company Name No. Records
Apple
84
Amazon
87
Adobe
91
Facebook
90
Twitter
61
Tesla
37
Google
108
IBM
94
Microsoft
86
Nvidia
21
Oracle
81
Salesforce
45
Uber
34
Clustering Algorithm and Distance Measure. The underlying clustering algorithm used in our system implementation is K-means. Given a set
{v1 , . . . , vn } of n records, K-means clustering partitions the n records into
A Multi-clustering Scheme
293
K ≤ n sets S = {S1 , S2 , . . . , SK } minimizing the Within-Cluster Sum of Squares
K (WCSS) defining the variance of the clusters as: minS i=1 v∈Si dist(v, µi )2 ,
where µi is the center of the records in cluster Si and dist is the chosen distance measure. Since our data-set consists of both numerical and categorical
attributes, we used a mixed-data distance measure based on [26].
In order to choose the parameter K in the K-means algorithm we used the
Elbow method [31], which was validated using two data-sets that have a known
number of clusters (K value). The validation data-sets taken from SKlearn are
the Iris data-set, which consists of 3 classes and has 150 samples, and the Digits
data-set, which consists of 10 classes and has 1797 samples. The method applied
on these data-sets had an elbow at the value of 3 in the Iris data-set and 10 in the
Digit data-set, as expected (see Fig. 4). Using the Elbow method, with WCSS
metric K was chosen to be 8 clusters in our data-set. These clusters represent
Hi-Tech employees professional profiles (see Fig. 5). Note that in our case the
elbow does not appear in a clear and unequivocal point, for both 7 and 9 are
also reasonable values to choose as elbow points according to the graph in Fig. 5.
Nevertheless, we chose 8 as the desired number of clusters being the middle point
of these three options. We stress that the other two values gave similar results.
Fig. 4. Elbow validation on the (a) Digits and (b) Iris data-sets.
System Description. Talent.AI system was developed in VS code for client
and server sides with Node.JS, React and Typescript, while using Py-charm for
algorithms python scripts. The system life cycles for candidate/company HR
(Human Resources) manager users appear in Fig. 6 and 7, respectively.
294
A. Levy et al.
Fig. 5. Elbow method applied to our data-set
Fig. 6. Talent.AI candidate system life cycle
Figure 8 shows two candidate screens. Data collected in Fig. 8(b) creates the
candidate profile.
Figure 9 and 10 show two company screens. Figure 10 shows the company
screen for creating an open job offer. Data collected in this form creates the
job-offer profile. Both profiles are represented in a unified structure. Figure 11
shows the system companies-ranking for a candidate.
A Multi-clustering Scheme
295
Fig. 7. Talent.AI company HR system life cycle
5
Evaluation
In this section we report on experiments performed for evaluating the system.
System Responsiveness. This was tested in two scenarios:
1. Candidate: the average time for ranking companies list is approximately 1 s.
2. Company: the average time for ranking applicants for an open position (20–50
applicants) is approximately 1.5 s.
In both cases, measured time is from initiating a request until a result is provided.
Intra-cluster Variance. The system’s recommendations merit highly depends
on the multi-clustering quality. The intra-cluster variance of the resulting multiclustering model using the WCSS measure is: 14.096.
Stability – Mini-Batch Tests. The multi-clustering algorithm robustness was
evaluated via 10 test-runs with new synthetic data based on the actual data.
Each record in the new data-set was minorly changed to check whether records
classified to a cluster are still identically classified after minor values changes.
The mini-batch test run data-set consists of 10 records chosen for each cluster,
summing up to 80 new records taken from the original data-set. Each record
was mutated in 2.5% of the total attributes for each record. Since, periodical remodeling is performed in our system when 5% of the data is updated, a change
in 2.5% of the total attributes for each record is substantial. The mutation
296
A. Levy et al.
Fig. 8. Talent.AI candidate forms: (a) sign-up; (b) edit skills
Fig. 9. Talent.AI company profile
function used is as follows. Let f (x) = x · 0.98 and g(x) = x + 1. Then, the
following if-statement is used for generating a mutated synthetic value for a
given attribute numeric value x: if x = f (x), then x ← g(x); else x ← f (x).
Each mutated record classified to its prior-mutation multi-cluster is considered
as correctly classified, otherwise, it is considered as incorrectly classified. The
accuracy score is the number of correctly classified records divided by the total
number of tested records. The accuracy score of the mini-batch test run appears
A Multi-clustering Scheme
297
Fig. 10. Talent.AI company open job offer form
Fig. 11. Talent.AI ranking companies for a candidate
in Table 2. The results show that the model is preserved under minor values
changes in 95% on the average, therefore, can be considered stable.
Data Content Sensitivity Test. Our system enables a simple mean to test its
data content sensitivity via the reducing bias option offered to HR users. While
providing HR users with a ranked list of applicants to a job-offer open position,
the system offers the possibility to ignore age and gender attributes. This effort
to reduce gender/age bias in the hiring process is in line with increasing concerns
about discrimination and bias in machine learning [10,16,17]. An HR following
298
A. Levy et al.
Table 2. Mini-batch multi-clustering test results
Cluster records Changed values Tests runs Accuracy Score
Min Max Average
10
3
10
90
97.5 95
this suggested option receives a ranked list of position applicants that is generated by a different data model, in which the age and gender attributes were
removed prior to its computation.
The above feature is used to test the system data content sensitivity as
follows. Several applicants for a system job-offer open position are inserted. These
applicants are database records with varying gender and age attributes values
that were not used for the multi-clustering model training. Then, an HR request
is applied to get a ranked list of these applicants without using the options to
ignore age and gender attributes. An example of such request results presents
the ranking: 1) Jason Keung - Male of age 35, 2) Brian Kim - Male of age
36, 3) Rashid Goshtasbi - Male (see Fig. 12). Next, an additional HR request
is applied to get a ranked list of these applicants, while ignoring both age and
gender attributes. This additional request results presents the ranking: 1) Kelly
Keodara - Female of age 60, 2) Brian Kim - Male of age 36, 3) Jason Keung -
Fig. 12. Ranked job offer applicants considering age/gender
A Multi-clustering Scheme
299
Male (see Fig. 13). Observe that an age 60 female, which did not appear in the
ranked list of the first request, became ranked first in the second request. This
is an evidence for the system’s data content sensitivity.
Fig. 13. Ranked job offer applicants ignoring age/gender
Remark. In order to deploy TALENT.AI the system should be used by HR
Hi-tech companies and applicants, which are its end-users. Then, the in-use
experience can be collected, measured and reported, thus, supplying more insight
to the system abilities.
6
Conclusion
We present TALENT.AI recommender system for the Hi-Tech recruiting process.
The system is based on a new multi-clustering scheme designed for an unsupervised learning scenario, in which multi-labels describing the user-item matches
do not exist explicitly. Moreover, hidden multiple matches should be inferred
besides the trivial matches of employees to their employing company.
The developed TALENT.AI recommender system efficiently deals with this
hidden multiple overlaps challenge using a unified modeling technique enabling
an un-biased multi-clustering. The multi-clustering has a satisfying variance
300
A. Levy et al.
within clusters and is robust under minor value changes. In addition, our recommendations demonstrate data content sensitivity.
The underlying overlapping clustering and multi-label learning problem is
widespread and complicated, as the literature shows. Nevertheless, our multiclustering unbiased relative prediction recommendation scheme enables to bypass
this difficulty in a conceptually exquisite and efficient way.
References
1. People data labs: a company specializing in persons and companies’ data-set creation enabling data-driven AI solutions (2022). https://www.peopledatalabs.com
2. Adomavicius, G., Tuzhilin, A.: Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. IEEE Trans. Knowl.
Data Eng. 17(6), 734–749 (2005)
3. Adomavicius, G., Tuzhilin, A.: Context-aware recommender systems. In: Ricci,
F., Rokach, L., Shapira, B. (eds.) Recommender Systems Handbook, pp. 191–226.
Springer, Boston, MA (2015). https://doi.org/10.1007/978-1-4899-7637-6 6
4. Al-Hassan, M., Haiyan, L., Jie, L.: A semantic enhanced hybrid recommendation
approach: a case study of e-government tourism service recommendation system.
Decis. Support Syst. 72, 97–109 (2015)
5. Bonchi, F., Gionis, A., Ukkonen, A.: Overlapping correlation clustering. Knowl.
Inf. Syst. 35, 1–32 (2013)
6. Burke, R.: Knowledge-based recommender systems. Encycl. Libr. Inf. Syst.
69(Supplement 32), 175–186 (2000)
7. Çano, E., Morisio, M.: Hybrid recommender systems: a systematic literature
review. Intell. Data Anal. 21(6), 1487–1524 (2017)
8. Cleuziou, G.: An extended version of the K-means method for overlapping clustering. In: 19th International Conference on Pattern Recognition, ICPR 2008, pp.
1–4 (2008)
9. Cleuziou, G.: Two variants of the OKM for overlapping clustering. In: Guillet, F.,
Ritschard, G., Zighed, D.A., Briand, H. (eds.) Advances in Knowledge Discovery
and Management. Studies in Computational Intelligence, vol. 292, pp. 149–166.
Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-00580-0 9
10. Frid-Nielsen, S.S.: Find my next job: labor market recommendations using administrative big data. In: 13th ACM Conference on Recommender Systems (RecSys),
pp. 408–412 (2019)
11. Geng, X.: Label distribution learning. IEEE Trans. Knowl. Data Eng. 28(7), 1734–
1748 (2016)
12. Gong, X., Yuan, D., Bao, W.: Fast multi-label learning. In: Zhou, Z.-H. (ed.) 30th
International Joint Conference on Artificial Intelligence, IJCAI 2021, pp. 2432–
2438 (2021)
13. Gutiérrez, F., Charleer, S., De Croon, R., Htun, N.N., Goetschalckx, G., Verbert,
K.: Explaining and exploring job recommendations: a user-driven approach for
interacting with knowledge-based job recommender systems. In: 13th ACM Conference on Recommender Systems (RecSys), pp. 60–68 (2019)
14. He, J., Chu, W.W.: A social network-based recommender system (SNRs). In:
Memon, N., Xu, J., Hicks, D., Chen, H. (eds.) Data Mining for Social Network
Data. Annals of Information Systems, vol. 12, pp. 47–74. Springer, Boston (2010).
https://doi.org/10.1007/978-1-4419-6287-4 4
A Multi-clustering Scheme
301
15. Whang, J.J., Hou, Y., Gleich, D.F., Dhillon, I.S.: Non-exhaustive, overlapping
clustering. IEEE Trans. Pattern Anal. Mach. Intell. 41(11), 2644–2659 (2019)
16. Kaya, M., Bridge, D., Tintarev, N.: Ensuring fairness in group recommendations by
rank sensitive balancing of relevance. In: 14th ACM Conference on Recommender
Systems (RecSys 2020), pp. 101–110 (2020)
17. Liu, W., Guo, J., Sonboli, N., Burke, R., Zhang, S.: Personalized fairness-aware
re-ranking for microlending. In: 13th ACM Conference on Recommender Systems
(RecSys), pp. 467–471 (2019)
18. Jie, L., Shambour, Q., Yisi, X., Lin, Q., Zhang, G.: A web-based personalized
business partner recommendation system using fuzzy semantic techniques. Comput. Intell. 29(1), 37–69 (2013)
19. Jie, L., Dianshuang, W., Mao, M., Wang, W., Zhang, G.: Recommender system
application developments: a survey. Decis. Support Syst. 74, 12–32 (2015)
20. Luo, H., Zhang, Y., Zhao, S., Chen, H., Li, C.: Exploring binary classification
hidden within partial label learning. In: De Raedt, L. (ed.) 31st International Joint
Conference on Artificial Intelligence, IJCAI 2022, pp. 3285–3291 (2022)
21. Masthoff, J.: Group recommender systems: combining individual models. In: Ricci,
F., Rokach, L., Shapira, B., Kantor, P.B. (eds.) Recommender Systems Handbook, pp. 677–702. Springer, Boston, MA (2011). https://doi.org/10.1007/978-0387-85820-3 21
22. Najafabadi, M.K., Mohamed, A.H., Mahrin, M.N.: A survey on data mining techniques in recommender systems. Soft Comput. 23(2), 627–654 (2019)
23. Ben N’Cir, C.-E., Cleuziou, G., Essoussi, N.: Identification of non-disjoint clusters
with small and parameterizable overlaps. In: International Conference on Computer Applications Technology (ICCAT), pp. 1–6 (2013)
24. Ben N’Cir, C.-E., Cleuziou, G., Essoussi, N.: Overview of overlapping partitional
clustering methods. In: Celebi, M.E. (ed.) Partitional Clustering Algorithms, pp.
245–275. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-09259-1 8
25. Ben N’cir, C.-E., Essoussi, N., Bertrand, P.: Kernel overlapping K-means for clustering in feature space. In: International Conference on Knowledge Discovery and
Information Retrieval KDIR 2010, pp. 250–256 (2010)
26. Noorbehbahani, F., Mousavi, S.R., Mirzaei, A.: An incremental mixed data clustering method using a new distance measure. Soft Comput. 19(3), 731–743 (2015)
27. Pazzani, M.J., Billsus, D.: Content-based recommendation systems. In:
Brusilovsky, P., Kobsa, A., Nejdl, W. (eds.) The Adaptive Web. LNCS, vol.
4321, pp. 325–341. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3540-72079-9 10
28. Pellegrini, R., Zhao, W., Murray, I.: Don’t recommend the obvious: estimate probability ratios. In: 16th ACM Conference on Recommender Systems (RecSys) (2022)
29. Schafer, J.B., Frankowski, D., Herlocker, J., Sen, S.: Collaborative filtering recommender systems. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds.) The Adaptive
Web. LNCS, vol. 4321, pp. 291–324. Springer, Heidelberg (2007). https://doi.org/
10.1007/978-3-540-72079-9 9
30. Sina, K., Naiier, A., Samaneh, S.: An improved overlapping K-means clustering
method for medical applications. Expert Syst. Appl. 67, 12–18 (2017)
31. Syakur, M.A., Khotimah, B.K., Rochman, E.M.S., Satoto, B.D.: Integration Kmeans clustering method and elbow method for identification of the best customer
profile cluster. IOP Conf. Ser. Mater. Sci. Eng. 336(1), 012017 (2018)
32. Vincent, J. (2021). https://www.theverge.com/2021/9/6/22659225/automatedhiring-software-rejecting-viable-candidates-harvard-business-school
302
A. Levy et al.
33. Wang, R., Lu, J., Lu, Y., Nie, F., Li, X.: Discrete multiple Kernel K-means. In:
Zhou, Z.-H. (ed.) 30th International Joint Conference on Artificial Intelligence,
IJCAI 2021, pp. 3111–3117 (2021)
34. Yang, C., Hou, Y., Song, Y., Zhang, T., Wen, J.-R., Zhao, W.X.: Modeling two-way
selection preference for person-job fit. In: 16th ACM Conference on Recommender
Systems (RecSys) (2022)
35. Zhang, F., Jia, X., Li, W.: Tensor-based multi-view label enhancement for multilabel learning. In: Bessiere, C. (ed.) 29th International Joint Conference on Artificial Intelligence, IJCAI 2020, pp. 2369–2375 (2020)
36. Zhang, M.-L., Zhou, Z.-H.: A review on multi-label learning algorithms. IEEE
Trans. Knowl. Data Eng. 26(8), 1819–1837 (2014)
37. Zhao, X., An, Y., Xu, N., Geng, X.: Fusion label enhancement for multi-label
learning. In: De Raedt, L. (ed.) 31st International Joint Conference on Artificial
Intelligence, IJCAI 2022, pp. 3773–3779 (2022)
An Unsupervised Deep Learning Model
for Aspect Retrieving Using Transformer
Encoder
Atanu Dey1(B) , Mamata Jenamani1 , and Arijit De2
1
Indian Institute of Technology Kharagpur, Kharagpur, India
atanu.dey.cse@kgpian.iitkgp.ac.in, mj@iem.iitkgp.ac.in
2
The University of Manchester, Manchester, UK
arijit.de@manchester.ac.uk
Abstract. We introduce a deep-learning-based aspect extraction model
called RATE, which stands for Retrieving of Aspects using T ransformer
E ncoder. When doing unsupervised aspect-based sentiment analysis, the
process of retrieving aspects is both critical and challenging. Most prior
efforts use some kind of topic modeling to extract aspect only. Despite
their efficacy, these techniques seldom provide consistent outcomes for
highly coherent aspects. Even though some approaches address these
issues by employing an attention-based deep neural model, their performance is hindered by their single-headed attention mechanism. Thus,
RATE is designed to improve the performance of extracting coherent
aspects from the text by using multi-headed attention mechanism with
transformer encoder, negative samplings, and word embeddings. This
model promotes the proximity in the embedding space of words that
arise in similar contexts, as opposed to topic models and other techniques
that often presume independently created words. To further enhance the
coherence of the aspects, we use multi-headed attention technique in
the encoder of the RATE architecture to downplay unimportant words
during training. The RATE model outperforms the highest performing
unsupervised baseline ones in terms of precision (8.34%), recall (0.94%),
and f1-score (5.44%) on the ACOS-Laptop dataset. Again, on the ACOSRestaurant dataset, RATE enhances precision, recall, and f1-score by
1.4%, 8.87%, and 4.31%, respectively, for finding more significant and
coherent aspects.
Keywords: Aspect extraction · Word embedding · Attention
mechanism · Transformer · Encoder · Unsupervised deep neural
network
1
Introduction
Organizations have been helped by consumer reviews posted on the internet in
order to get valuable information for better decision making [6]. For this purpose,
one of the fine-grained trending application areas of natural language processing
c The Author(s), under exclusive license to Springer Nature Switzerland AG 2024
K. Arai (Ed.): SAI 2024, LNNS 1017, pp. 303–317, 2024.
https://doi.org/10.1007/978-3-031-62277-9_18
304
A. Dey et al.
is identified as Aspect Based Sentiment Analysis (ABSA) [2,12]. Its primary
objective is to extract entity aspects on which people have voiced opinions [2,5].
Two example sentences are shown in Fig. 1; in the first, “charge” and “deal” are
identified as aspect terms that fall under the category of “battery” and “price”
respectively, and in the second, “sushi” and “staff” are thus identified that fall
under the category of “food” and “staff” respectively.
Fig. 1. Example of aspects and respective category from two sentences
In the realm of extracting aspect words from literature, three distinct schools of
thought have emerged, each with its own unique approach: rule-based, featureengineering, and deep-learning. The rule-based methods delve into the structural
intricacies of sentences [13,19], utilizing their syntactic or dependency structures
for aspect phrase extraction, as highlighted by Qiu et al. in 2011 [13]. On the
other hand, feature-engineering-based strategies leverage handcrafted features
and a meticulously annotated corpus to train classifiers, as demonstrated by
F. Li et al. in 2010 [8]. In the domain of deep-learning, the majority of algorithms adopt a supervised path, where models are honed on manually curated
labeled training data [2,12,16]. However, it’s essential to acknowledge the substantial manual effort involved across all these methodologies, encompassing rule
or feature formulation and the labor-intensive task of tagging training data. To
address this labor-intensive nature, a breakthrough emerges in the form of unsupervised deep-learning-based algorithms, which forge high-level representations
to facilitate the automated extraction of aspects [3,5,14], thereby alleviating
the reliance on extensive human intervention. Despite its efficiency in time/cost
and surpassing other unsupervised or semi-supervised methods, this approach is
hindered by a single-headed attention mechanism in comparison with state-ofthe-art supervised models [15]. Thus, the research question of the current study
is formulated as follows:
Research Question: Can we provide a better unsupervised model for aspect
extraction using multi-headed attention mechanism with transformer encoder?
To address the above research question towards findings aspect-terms from
sentences, the following contributions are as follows for this work: 1) We cluster
aspect terms with similar meaning into classes, with each class representing a
unique category (e.g., “sushi”, “crawfish”, “curried” and “broccoli” into one category “Main Dishes”). 2) We propose an end-to-end architecture for deep-learning
An Unsupervised Deep Learning Model for Aspect Retrieving
305
to extract aspect terms using above clusters and a multi-headed attention mechanism with transformer encoder. 3) To ensure that our model provides the best
possible results when extracting aspect terms, we do extensive experiments on
two benchmark datasets.
Here is how the rest of the paper is laid out. A review of the relevant literature follows. The proposed procedure is outlined in Sect. 3. The results of the
experiment are presented in Sect. 4. The article’s conclusion and future relevance
are discussed in Sect. 5.
2
Related Work
Over the last decade, researchers have paid a lot of attention to the issue of
aspect-extraction. First, we discuss the literature on aspect-extraction from the
perspectives of different Methodologies. Later, we focus on the different types of
word embedding approaches to use in deep-learning model.
2.1
Methodologies of Aspect Extraction Task
Over time, the landscape of Aspect-Based Sentiment Analysis (ABSA) systems
have evolved dynamically from manual rule-based to automatic aspect extraction
methods. Hu & Liu (2004) pioneered association rule mining to reveal complicated aspects of texts, laying the framework for future advances [6]. Qiu et al.
(2011) enhanced this technique by carefully identifying nouns and noun phrases
to provide a framework for extracting important aspect words [13]. This is an
alternative methods for frequency-based product aspect extraction arose alongside these efforts. Researchers used Part-of-Speech (POS) tagging data from
dependency parsers to find relevant aspects [8,13]. However, as effective as these
methods were, their applicability remained tethered to the confines of predefined
rules. This intrinsic limitation inadvertently cast a spotlight on their inability to
capture the entirety of the intricate aspect landscape, ultimately restricting the
scope of the extracted aspect words to a relatively confined collection of nouns.
To overcome rule-based restrictions, researchers have turned to supervised
learning for aspect extraction [2]. Hidden Markov Models (HMM) and Conditional Random Fields (CRF) with properly picked features were popular early
on [8]. Deep Neural Network (DNN) techniques changed the landscape, achieving incredible results in this field [9,17]. In 2016 and 2017, Wang et al. used
DNNs and word embeddings to find subtle, higher-level aspects in textual data
[17]. Neural language models offered by luminaries like Devlin et al. (2019) and
Mikolov et al. (2013) helped grasp context via word embeddings [4,10]. Despite
substantial advances, supervised deep-learning models relied on properly labelled
training data, making them arduous and resource-intensive [1,5].
In response to the challenges posed by resource-intensive labeled training
data, a wave of innovative unsupervised deep-learning-based methods emerged
[5], seeking to reduce reliance on such data. These approaches pioneered novel
avenues of exploration, departing from the established norm. Early forays delved
306
A. Dey et al.
deeply into topic modeling-based strategies, yielding valuable insights through
word distributions and rankings for each discernible aspect [11,18]. The seminal
work of Yan et al. in 2013 introduced the concept of the Biterm Topic Model
(BTM), a pioneering endeavor that shifted focus towards the generation of cooccurring word pairs, augmenting the richness of aspect extraction [18]. Subsequently, in 2017, He et al. unveiled an attention-based Deep Neural Network
(DNN) model tailored for aspect extraction, displaying a remarkable capability
to function effectively without the crutch of labeled training data [5]. Nevertheless, while this groundbreaking approach harnessed the capabilities of word
embeddings to uncover aspects from sentences, it faced a constraint associated
with the attention model due to its single-headed mechanism [14,15].
2.2
Word Embedding Representations on Aspect Extraction Task
NLP research have relied on word embeddings for years [4,10]. Several
researchers use BERT-based embedding to train deep-learning models [2,4]. The
BERT embedding by Devlin et al., 2019 uses a pre-trained model of 30, 000
words but not domain-specific datasets [4]. Thus, aspect and non-aspect words
(like opinion terms) may have similar embedding vectors. Consider a sentence
“The quality of the food is awesome”, where the Google pre-trained BERT model
matches “food” and “awesome” embedding vectors at 67% based on cosine similarity. During aspect extraction, it may not differentiate aspect words from other
terms. Under these situations, the Google pre-trained Word2Vec model performs
better [10]. Cosine similarity analyses show that the preceding two-word pair’s
embedding vectors are just 4% similar. Another example: Word2Vec finds a
2% similarity between “software” (an aspect term) and “worse” (a non-aspect
term), but BERT finds 62%. The pre-trained Word2Vec model performs better,
because domain-specific datasets may increase embedding quality. The dataset
results encourage us to include Word2Vec embeddings into the proposed model.
3
Proposed Architecture Description for RATE
We describe a deep-learning-based aspect extraction utilizing multi-headed
attention with transformer encoder as shown in RATE architecture in Fig. 2.
Model’s ultimate objective is to learn aspect embeddings that can be understood by looking at the closest represented words in the embedding space.
The model begins by generating embedding using Word2Vec for each word (Ei ,
i = 1, 2, ..., n) in a sentence of n words. The transformed words are sent to the
model to extract aspect terms (Ai , i = 1, 2, ..., k). The model includes encoder,
aspect-based sentence embeddings, aspect-category weight representation, and
reconstructed sentence embeddings. Following subsections outline all steps.
3.1
Encoder
Encoder consists of seven layers of operations as shown in Fig. 2. All the critical
layers are explained in detail as follows:
An Unsupervised Deep Learning Model for Aspect Retrieving
307
Fig. 2. RATE Architecture
Self Attention Model. Attention mechanisms weigh words based on their
context. The model’s initial average of all word embedding vectors (Ei , i =
1, 2, ..., n) to represent the sentence’s global context embedding (ES ). Where, n
is sentence word count, and every element of ES vector is an input layer node
in this model. Forward propagation of a linear projection (hidden) layer maps
input nodes to the global context embedding. The concealed node’s result of
the hidden layer is multiplied by each target-word-embedding element Ei . The
ultimate attention score (ai ) is the total of all multiplications (Eq. 1). Since
there is no objective attention score, this model must be linked into another
neural network model to adapt it. Next, backpropagated derivative loss changes
the attention score to fine-tune hidden and input layer weights (Q). Where,
Q = R ∈ d × d, and ‘d’ is the embedding vector size.
308
A. Dey et al.
ai = ES .Q.EiT
(1)
Fig. 3. Attention-based embedding vector for each word of a given sentence
The next phase involves multiplying each word embedding vector (Ei ) by
its respective attention score (ai ) as shown in Fig. 3. The intuition here is to
preserve the significance of the word(s) that we wish to highlight, while obliterating irrelevant words. Then, we concatenate all vectors to produce the final
self-attention matrix Z.
Multi-headed Attention. The work added “multi-headed” attention to the
self-attention layer. This boosts attention layer performance in two ways: 1) The
model may concentrate on more words. 2) It offers attention layer for several
representation subspaces. We get eight Z matrices by repeating the self-attention
computation eight times with various weight matrices.
Fig. 4. Multi-headed Attention-based embedding vector for each word of a given sentence
An Unsupervised Deep Learning Model for Aspect Retrieving
309
It presents a difficulty because, the feed-forward layer expects one matrix (a
vector for each word). So, as shown in Fig. 4, we need to reduce these eight into a
single matrix using the following methods: 1) We concatenate and multiply the
matrices by a model-trained weights matrix W. 2) The Z matrix would include
information from all attention heads. This information is also transmitted to the
feed-forward neural networks (FFNN) after normalization has been performed
by adding the input word embedding matrix E.
Feed Forward Neural Network. Our encoder’s layers include attention layers and fully linked feed-forward neural networks (FFNN) applied to each
word individually and identically. It consists of linear transformations with SoftMax activation as shown in Eq. (2). Linear transformations within a given layer
are uniform across all words but vary across layers based on their parameters.
Finally, after normalization is performed by adding the normalized attention
matrix Z from the previous layer, this information is then transmitted to the
final attention layer.
FFNN(Z) = softmax(Z.W1 + b1 )
3.2
(2)
Attention-Based Sentence Embeddings
Each vector (Zi ) within the matrix Z derived from the output of the preceding
layer is regarded as a multi-headed attended word vector. These vectors are
used in self-attention model to compute final attention score a1 , a2 , ..., an as
elucidated in the Sect. 3.1. The normalized attention score are a probability of
choosing a word (Ei ) to adequately represent the aspect’s primary notion. Sum
of the sentence’s each word embeddings in matrix E with the corresponding
normalized attention score to get an attention-based word embedding matrix.
Again, sum of normalized rows of the matrix reflects the sentence embeddings
ZS , as seen in Fig. 2. This vector may capture key sentence aspect information.
3.3
Aspect-Category Weight Representation
Number of clusters depends on domain and dataset. Trial and error determines
how significant and connected the words are. He et al. (2017), Brody & Elhadad
(2010), and W. X. Zhao (2010) find 14 restaurant clusters [5,8,19]. Cai et al.
(2021) find 10 laptop clusters [2]. For both domains, we started trial and error
with 10 aspect clusters and increased to 20. The findings suggest no notable
difference in 14 cluster groups for both datasets, hence we set K = 14.
Weight vector (wA ) for the K aspect embeddings over the attention-based
sentence embedding ZS is computed using Eq. (3), the weight matrix WA and
bias vector bA are randomly initialized and learnt during training. For the sentence S, each weight of wA represents the probability of each aspect.
wA = softmax(WA .ZS + bA )
(3)
310
3.4
A. Dey et al.
Reconstructed Sentence Embedding
Aspects, like words, have their own embedding space, and we would want to learn
how to get around that space. In order to do this, one must have a prototype
aspect embedding matrix EA ∈ RK×d , where K is the total number of aspect
category. The average embedding of the most similar words across categories
may be used to calculate such a prototype aspect embedding.
The model reconstructs (RS ) the sentence embedding ZS as a linear combination of prototype aspect embeddings from its matrix EA and the vector of
aspects’ weights wA , as shown in Eq. (4).
RS = ETA .wA
3.5
(4)
Model Training
The RATE framework trains aspect extraction models to reduce reconstruction mistakes. In hinge loss optimization, we use negative sampling [5]. Randomly selecting m training sentence as negative samples against each input sentence. Each negative sample for aspect extractions is referred to as Nl , where
l = 1, 2, ..., m. The embedding vector of each negative sample was constructed
by averaging its word embeddings. The goal is to minimize the inner product
between the reconstructed embedding and negative sample embedding and maximize it with the target sentence embedding. The aspect loss function (LOSSA )
is illustrated in Eq. (5).
LOSSA =
m
max(0, 1 − (RST .ZS ) + (RST .Nl ))
(5)
l=1
We aim to learn diverse aspect representations from a review dataset. To address
potential redundancy in the aspect embedding matrix (EA ) during training, a
regularization term (UA ) is incorporated into the objective function (LOSSA )
to promote diversity among the resulting aspect embeddings as shown in Eq. (6).
UA = (EAn . ETAn + I)
(6)
where, I represents the identity matrix and EAn denotes EA with normalized
row. In the matrix EAn .ETAn , every non-diagonal element represents the dot
product of two distinct aspect embeddings. The minimal level of UA is achieved
when the dot product of any two distinct aspect embeddings equals zero, thereby
encouraging orthogonality and penalizing redundancy. The final objective function of loss FL A combines LOSSA and UA for the outcome as shown in Eq. (7),
where λ is a hyper-parameter that determines the regularization term’s weight.
FL A = LOSSA + λ . UA
(7)
We execute 15 epochs for each sentence in every batch (size of 50) to train this
model, unlike He et al. (2017) [5], who only run 15 epochs per batch. Another
An Unsupervised Deep Learning Model for Aspect Retrieving
311
model testing modification, unlike He et al. (2017) [5], rejects extracted aspect
keywords that are in the sentiment lexicon [7] and the most common frequent
word list (Table 4 in Appendix).
Table 1. Inferred aspects and respective categories for restaurant domain
Keys (Categories)
Values (Aspects)
Main Dishes
ham, crawfish, boiled, curried, smoked, broccoli, chutney, fava
Service
response, unapologetic, rudely, loudly, aggressive, unprofessional, service
Area
francisco, phoenix, piladelphia, london, bayside, harlem, greenpoint,
jamaica, atlanta, hometown
Staff
host, patient, apologetic, waitstaff, gracious, hovering, responsive,
apathetic, solicitous
Price
charged, additional, paid, minimum, eventually, receive, discount,
unreasonable
Dessert Drink
strawberry, blueberry, vanilla, caramel, banana, milk, raspberry, hazelnut,
cranberry, cannoli, juice
Food General
unusual, unique, surprising, affordable, homestyle, impressive, organic,
healthy, tasty, japanese, pizza, uniquely, exotic, imaginative, imaginative
Physical Ambience ornate, colorful, painted, chandelier, banquette, furniture, photograph,
wood, candlelit, abstract, decorative, ambience
Adjectives
comfy, spacious, lighting, waterfall, comfortably, chic, retro, intimacy
Ingredient
garlicky, cumin, raisin, mayonnaise, seed, celery, saffron, mustard
Anecdotes
celebrated, holiday, anniversary, engagement, bridal, meeting
Misc General
hurry, worry, realize, ready, understand, hungry, complaining, homeless
Misc Location
park, street, avenue, located, eatery, destination, mall, hotel
Other
buona, schimmel, glancing, undeserved, phenomenon, loren, Frederick,
minimalistic, shamelessly
4
Results and Discussions
We conduct experiments to compare the proposed method with state-of-the-art
three unsupervised, one semi-supervised and two supervised methods, including
LocLDA, K-Means, ABAE, Double-Propagation, Double-Propagation-ACOS,
TAS-BERT-ACOS using four real-world data sets, including Restaurant-ACOS,
Laptop-ACOS, Citysearch Corpus, and Laptops-2023. Due to their extensive use
in numerous studies, the Restaurant-ACOS (2021) and Laptop-ACOS (2021)
datasets have been chosen for testing purposes [2], while two other datasets
have been chosen for training. We conducted the experiment using both ACOS
testing sets by randomly selecting 500 reviews from both the datasets for appleto-apple performance comparison. Specifically, we use aspect-terms from the
original ACOS datasets, with some minor modifications. The Citysearch Corpus
dataset contains more than 50, 000 restaurant reviews from Citysearch New York
312
A. Dey et al.
Table 2. Inferred aspects and respective categories for laptop domain
Keys (Categories)
Values (Aspects)
Battery Power
charging, long, drained, overheats, life, last, charger, battery, heat, power,
short, bettry
I/O Devices
speaker, display, resolution, camera, loud, sound, hd, keyboard, output,
fhd, screen, audio, keypad, touchpad, picture
Web Video Games
editing, playing, watching, movie, web, played, classroom, youtube, film,
watch, photoshop, rendering, steam, fps, multimedia, studio
Memory Processor Graphics core, ram, gb, processor, graphic, ddr, storage, memory, nvidia, drive,
multitasking
Adjectives
pain, switching, crash, crashing, error, respond, sleeping, connectivity,
connection, connecting, connect, hell
Company
trust, retail, cheated, cashback, refurbished, suggestion, doubt, horrible,
faulty, manufacturing, customer, support, brand, acer, company
Laptop general
superb, looking, awesome, fantastic, beautiful, stylish, extraordinary,
gorgeous, lightweight, product, computer, chromebook, laptop, tablet,
notebook, machine, device, killer
Shipping
week, replacement, received, warehouse, ship, delivered, replace, return,
ordered
Price
offer, purchase, festival, purchased, buying, discount, totally, rupee,
worthless, paid, price
Software
installing, installation, download, application, software, reload, slowdown,
reinstall, microsoft
Webcam
bright, crisp, clear, led, webcam, picture, flimsy, retina, gorgeous, digital,
glossy, lookwise, cam, brightness
Miscellaneous
reinsalled, patience, uninstalled, flash, accessory, technology, incorporate,
mousepad
Others
technology, custom, serious, default, messy, dust, physical, noisy
Out of scope
english, ground, middle, darker, reading, understand, data
[19]. The Laptops-2023 is our scrapped dataset that contains 8, 373 reviews for
24 products sourced from Amazon and Flipkart. We selected models from Dell,
Lenovo, and HP within the same price range.
4.1
Aspects’ Quality Evaluation
Table 1 and 2 presents all 14 categories of aspects inferred by RATE for the
restaurant and laptop domains respectively. Compared to gold-standard labels,
the inferred aspects are more fine-grained. For example, it can distinguish main
dishes from desserts, and drinks from food for restaurant domain.
4.2
Main Results
Table 3 reports result for all models on two datasets (Restaurant-ACOS and
Laptop-ACOS). We use identical word embeddings models and the same corpus in all the baseline approaches in order to make fair comparisons. It may
be noted that RATE is an unsupervised deep-learning approach and does not
require labelled training data. Interestingly, RATE outperforms all the baseline
An Unsupervised Deep Learning Model for Aspect Retrieving
313
unsupervised, semi-supervised and supervised methods across aspect extraction
task in terms of F1-scores. Our methodology demonstrates significant improvement, achieving a 21% enhancement in F1-score compared to the best-performing
unsupervised and semi-supervised method (ABAE) among LocLDA, K-Means,
Double-Propagation, and ABAE listed in the table. Additionally, it attains a 4%
enhancement in F1-score compared to the best-performing supervised method
(TAS-BERT-ACOS) among Double-Propagation-ACOS and TAS-BERT-ACOS.
Moreover, our approach outperforms unsupervised and semi-supervised methods
in Recall and closely aligns with the performance of supervised methods.
Table 3. Comparison among baseline and proposed methods on two datasets
Approach
Models
Precision (P) Recall (R) F1-Score (F)
0.27
0.32
0.46
0.75
0.51
0.43
0.39
0.31
0.44
Semi-Supervised Double Propagation
0.42
0.41
0.41
Double-PropagationACOS
0.61
0.52
0.56
TAS-BERT-ACOS
0.64
0.58
0.61
RATE (Proposed)
0.59
0.73
0.65
Unsupervised
Supervised
Ours
(Unsupervised)
LocLDA
K-Means
ABAE
Figure 5 and 6 presents P, R, and F separately on two different domains of
ACOS datasets (Restaurant and Laptop respectively) using the proposed model
RATE and state-of-the-art baseline method ABAE and Modified ABAE (Ours).
We have modified the ABAE by adjusting the training method, implementing
a strategy of running 15 epochs for each sentence within every batch (with
a batch size of 50). In contrast, He et al. (2017) employed 15 epochs per
batch [5]. While our modified ABAE outperforms the original ABAE, it faces
challenges with a single-headed attention model. To address this, we introduce a novel multi-headed attention-based model named RATE. RATE significantly enhances aspect extraction performance compared to the modified
ABAE, achieving improvements in Precision, Recall, and F1-Score. Specifically,
in the restaurant domain, there’s an increase of 1.4%, 8.87%, and 4.31%, respectively. In the laptop domain, the improvements are 8.34%, 0.94%, and 5.4%,
respectively.
Table 3, Fig. 5 and Fig. 6 exhibit significant improvements in results achieved
by the proposed unsupervised method RATE when compared to existing unsupervised and semi-supervised methods, showcasing its advantageous applicability across domains. Additionally, Table 3 compares the proposed and existing
unsupervised methods with state-of-the-art supervised techniques, highlighting
the proximity between the proposed and supervised models. With comparable
314
A. Dey et al.
Fig. 5. Result comparison between proposed and state-of-the-art approaches on
Restaurant data
Fig. 6. Result comparison between proposed and state-of-the-art approaches on Laptop
data
Precision levels and superior Recall/F1-Scores to supervised models, the RATE
method proves advantageous, eliminating the necessity for labor-intensive manual annotation in training dataset preparation.
5
Conclusion
Introducing RATE, an innovative unsupervised deep-learning-based approach
to aspect extraction that harnesses the power of multi-headed neural attention
models alongside transformer encoder. In stark contrast to conventional rulebased, POS-Tag, LDA, and single-headed attention neural models, RATE distinguishes itself by not only learns remarkably higher quality aspects but also by
adeptly capturing the nuanced aspects present in reviews. In our humble opinion, we are the pioneers in presenting an unsupervised neural architecture that
An Unsupervised Deep Learning Model for Aspect Retrieving
315
integrates multi-headed neural attention models with transformer encoder, dedicated to the task of aspect extraction. The brilliance of RATE lies not just in
its effectiveness but also in its intuitive, straightforward structure, which boasts
scalability. Rigorous experiments conducted on two benchmark datasets serve to
validate the remarkable efficiency of RATE. Collectively, these advantages make
RATE as a compelling and promising alternative to prevailing state-of-the-art
methods, with the potential to yield substantial benefits in real-world applications. This study is limited to aspect extraction tasks, but it has the potential
for expansion into opinion and sentiment extraction tasks through the utilization
of multi-headed neural attention models with transformer encoders. While the
current study is limited in scope, future research can overcome this limitation
by exploring alternative frameworks.
Acknowledgment. We are grateful to access the facilities of IIT Kharagpur, and we
are also thankful to the annotators who have helped to refine the datasets.
Appendix
Table 4. Most common frequent words from the datasets
Most Common Frequent Words
dont, thing, ever, never, every, though, ended, know, one, bit, got,
must, time, many, given, oh, came, get, back, away, take, went,
go, back, none, absolutely, should, would, may, coming, done,
took, across, taken, asked, give, especially, always, gotten, later,
believe, right, give, nothing, decent, way, completely, getting,
next, keep, kept, try, tried, within, without, use, since, otherwise,
said, using, think, something, usually, trying, today, really, say,
much, made, used, enough, little, quick, anything, near, people,
put, thought, fully, antwhere, around, also, issue, see, etc., year,
already, twice, major, minor, going, start, starting, small, couple,
ago, worked, initial, except, almost, seems, although, instead,
still, found, make, available, however, others, anyone, someone,
along, taking, lot, comment, could, might, another, far, found
316
A. Dey et al.
References
1. Brody, S., Elhadad, N.: An unsupervised aspect-sentiment model for online reviews.
In: Human Language Technologies: The 2010 Annual Conference of the North
American Chapter of the Association for Computational Linguistics, pp. 804–812
(2010)
2. Cai, H., Xia, R., Yu, J.: Aspect-category-opinion-sentiment quadruple extraction
with implicit aspects and opinions. In: Proceedings of the 59th Annual Meeting
of the Association for Computational Linguistics and the 11th International Joint
Conference on Natural Language Processing (Volume 1: Long Papers), pp. 340–350
(2021)
3. Chauhan, G.S., Meena, Y.K., Gopalani, D., Nahta, R.: A two-step hybrid unsupervised model with attention mechanism for aspect extraction. Expert Syst. Appl.
161, 113673 (2020)
4. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: pre-training of
deep bidirectional transformers for language understanding. arXiv preprint
arXiv:1810.04805 (2018)
5. He, R., Lee, W.S., Ng, H.T., Dahlmeier, D.: An unsupervised neural attention
model for aspect extraction. In: Proceedings of the 55th Annual Meeting of the
Association for Computational Linguistics (Volume 1: Long Papers), pp. 388–397
(2017)
6. Hu, M., Liu, B.: Mining and summarizing customer reviews. In: Proceedings of the
Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data
Mining, pp. 168–177 (2004)
7. Hutto, C., Gilbert, E.: VADER: a parsimonious rule-based model for sentiment
analysis of social media text. In: Proceedings of the International AAAI Conference
on Web and Social Media, vol. 8, pp. 216–225 (2014)
8. Li, F.: Structure-aware review mining and summarization. In: Proceedings of the
23rd International Conference on Computational Linguistics (Coling 2010), pp.
653–661 (2010)
9. Li, X., Lam, W.: Deep multi-task learning for aspect term extraction with memory interaction. In: Proceedings of the 2017 Conference on Empirical Methods in
Natural Language Processing, pp. 2886–2892 (2017)
10. Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013)
11. Mukherjee, A., Liu, B.: Aspect extraction through semi-supervised modeling. In:
Proceedings of the 50th Annual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers), pp. 339–348 (2012)
12. Pham, D.-H., Le, A.-C.: Learning multiple layers of knowledge representation for
aspect based sentiment analysis. Data Knowl. Eng. 114, 26–39 (2018)
13. Qiu, G., Liu, B., Jiajun, B., Chen, C.: Opinion word expansion and target extraction through double propagation. Comput. Linguist. 37(1), 9–27 (2011)
14. Tulkens, S., van Cranenburgh, A.: Embarrassingly simple unsupervised aspect
extraction. arXiv preprint arXiv:2004.13580 (2020)
15. Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information
Processing Systems, vol. 30 (2017)
16. Wan, H., Yang, Y., Du, J., Liu, Y., Qi, K., Pan, J.Z.: Target-aspect-sentiment
joint detection for aspect-based sentiment analysis. In: Proceedings of the AAAI
Conference on Artificial Intelligence, vol. 34, pp. 9122–9129 (2020)
An Unsupervised Deep Learning Model for Aspect Retrieving
317
17. Wang, W., Pan, S.J., Dahlmeier, D., Xiao, X.: Recursive neural conditional random
fields for aspect-based sentiment analysis. arXiv preprint arXiv:1603.06679 (2016)
18. Yan, X., Guo, J., Lan, Y., Cheng, X.: A biterm topic model for short texts. In:
Proceedings of the 22nd International Conference on World Wide Web, pp. 1445–
1456 (2013)
19. Zhao, X., Jiang, J., Yan, H., Li, X.: Jointly modeling aspects and opinions with a
maxent-LDA hybrid. ACL (2010)
Personalized Student Performance Prediction
Modeling for Student Digital Twins
Sean Mondesire1(B) and Emmanuel Nsiye2
1 School of Modeling, Simulation, and Training, University of Central Florida, Orlando, FL,
USA
Sean.Mondesire@ucf.edu
2 Institute for Simulation and Training, University of Central Florida, Orlando, FL, USA
Emmanuel.Nsiye@ucf.edu
Abstract. Intelligent Tutoring Systems (ITSs) and online education provide convenient and controlled platforms for learning, increasing accessibility, and standardizing education in and out of the classroom. For instance, many online ITSs are
accessible to students 24/7, have multi-modality (desktop, mobile, virtual reality
systems), and provide consistent curricula across students, classes, and schools.
A growing design consideration of ITSs is the customization of the student experience. Modern systems tailor the curriculum to each student’s performance by
providing remediation when students perform poorly or accelerating a student
through lessons when strong proficiency is demonstrated. A challenge to these
systems is that this customization is traditionally reactionary and only adjusts
the student’s learning plan after the student has performed sub-optimally or overachieved. Proactive interventions often rely on a shared and generalized predictive
algorithm that is trained on a collection of students’ past performances but applied
to individual students. The presented work investigates the inconsistencies of these
generalized predictive models to demonstrate that one algorithm does not work
best for all students. This work sets the groundwork for the human digital twin,
which models the individual student progression through an ITS and customizes
the student performance predictive capabilities.
Keywords: Digital twin · Machine learning · Predictive analysis · Learning
analytics
1 Introduction
Intelligent Tutoring Systems (ITS) have opened new opportunities for the world of education. ITSs allow sections of courses in the same or different schools to share a standard curriculum that aims to provide students with guided, automated instructions [1].
With their online availability, ITSs also allow students to access resources from everywhere and at any time, allowing many students to progress at their own pace. Moreover,
advanced systems have taken steps to personalize the student experience by including
automated remediation when there are cases when a student is underperforming. This
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2024
K. Arai (Ed.): SAI 2024, LNNS 1017, pp. 318–329, 2024.
https://doi.org/10.1007/978-3-031-62277-9_19
Personalized Student Performance Prediction Modeling
319
automated remediation aims to improve the student’s understanding of the presented topics. ITSs also aim to reduce the number of redundant and unnecessary learning activities
for students who over-perform and clearly demonstrate a solid grasp of the presented
concepts. The automated and personalized learning plans ITSs have demonstrated an
ability to improve test scores, learning rates, and short- and long-term skills retention.
One way to implement ITS automated remediation capabilities is by combining it with
other technologies such as Digital Twin. Figure 1 is an example of a ITS integrated
with a Digital Twin. In that type of environment, the ITS and the Digital Twin interact
to model the student success, as the student submits answer to the ITS, the ITS sends
the answer and assessment to the digital twin. The Digital Twin sends the performance
history and assessments to the model, the model makes intervention recommendations
and sends it to the ITS, the ITS presents the recommendation back to the student.
Fig. 1. In the Student Digital Twin Model, students interact with an Intelligent Tutoring System and the twin’s predictive model identifies when a student needs an intervention to improve
assessment performance
Unfortunately, as much as those customizations of a student experience might be
helpful, many instances of ITSs are reactionary; many ITSs wait until student performance has fallen below a threshold before intervening. On the other hand, proactive
instances tend to rely on shared, generalized student performance predictive models to
foresee when a learning intervention is needed [2]. Contrary to student learning customization and personalization goals, these shared models are trained on collections of
student data that can span student populations across the globe in large-scale learning
platforms. For example, an ITS will train one Bayesian Knowledge Tracing model [3]
on a previous semester’s class of student data to predict each student’s performance in
the following semester. Many proactive ITSs rely on generalized models because the
time required to generate shared models is considerably faster than individual models.
Generalized models also overcome the cold start problem earlier than individual models.
A cold start is when a predictive model has too little data to make accurate predictions.
Therefore, with generalized models, ITSs can predict student performance significantly
320
S. Mondesire and E. Nsiye
faster when it is seeded with data from larger sources of data, realized from collections
of students versus waiting for an individual to complete a minimum number of exercises.
Additionally, often ITSs choose their one model’s algorithm arbitrarily or do not
disclose which predictive algorithm they deploy. In building these models, the type of
algorithm selected (linear regression, random forest, deep learning, etc.) is critical for
prediction performance because each algorithm behaves differently depending on the
type of data, the distribution, and the quantity of data. Importantly, algorithm selection
should not be arbitrary in education because their predictions determine when an intervention is required and can significantly impact a student’s ability to receive assistance
before it is too late.
Over the rest of the paper, this work will investigate the consistency of generalized
and individualized student performance prediction models to determine the benefit of
using one versus the other. This work is part of an upcoming human digital twin design
that models students in real-time as they progress through a curriculum, and it will assist
in determining if the student digital twin should incorporate generalized or individualized
prediction models to indicate when a student needs a learning intervention to improve
understanding.
2 Related Work
2.1 Predictive Models for Student Performance
One emerging application of machine learning has been its use in modern education
through student performance prediction in ITSs. Here, machine learning models predict
how well the student will perform on the next exercise (assessment). Traditionally,
suppose a student is predicted to perform poorly (i.e., below 70% on an assessment).
In that case, the ITS will provide an intervention to remediate the student on a concept
that the student is underperforming. Inputs into these models vary, but most contain
performance data leading up to the current exercise and a predictive algorithm. Though
the goal of many of these proactive ITSs is to use a shared, generalized model to apply
to individual students, the models do not always perform similarly despite the same
parameters being used, leading to the hypothesis that predictive algorithms can perform
inconsistently on different values of the same metric types.
One such example uses Decision Tree, Linear/Logistic Regression (LR), Random
Forest (RF), Support Vector Machines (SVM), and K-Nearest Neighbor (KNN) to predict
academic performance and study strategies based on student motivation [4]. The inputs
of the algorithms were students’ motivation, age, gender, autonomy, competence, selfesteem, study strategies, and year. As a metric of measurements, the approach used the
mean absolute error (MAE), and it was observed that Decision Tree had the best MAE
at 0.3777, followed by RF with 0.391 and SVM with 0.402. However, for classification,
Random Forest yielded the best accuracy of 0.949, followed by Decision Tree and KNN.
This work observed that by introducing gender and age as features, bias was introduced in
the predictive model, although their results yielded highly accurate results. The issue of
using gender was addressed in research that used improved Random Forest Classifiers to
predict student performance [5]. With their improved Random Forest, the work obtained
96% accuracy, and some of the results they obtained provided insights such that students
Personalized Student Performance Prediction Modeling
321
who have divorced parents performed worse than their counterparts and that although
gender plays a role, its impact does not carry on year to year implying that gender might
not be a scalable feature.
Further, certain models are prone to bias when there is less data. A research study
applied machine learning and oversampling with Synthetic Minority Over-sampling
Technique (SMOTE) to reduce the teacher’s workload [6]. On the categorical input
data, classification algorithms, such as Logistic Regression (LR), Linear Discriminant
Analysis (LDA), KNN, and SVM, were used and compared for accuracy and consistency.
LDA was found to be the best algorithm with 0.81 accuracy followed by SVM with 0.8
and KNN with 0.78, though it was observed how similar each algorithm was in their
overall accuracy performance.
Though most algorithms (LR, LDA, KNN, SVM, etc.) are traditional predictive
algorithms with widespread use and are commonly accepted for their analytics utility,
learning analytics and ITSs favor Knowledge Tracing for assessing student performance.
Knowledge Tracing is a learning analytics staple for measuring student mastery. Studies have investigated the combination of Knowledge Tracing with machine learning,
especially with Neural Networks. One proposed approach is Deep Knowledge Tracing
(DKT), a deep learning solution for Knowledge Tracing [7]. The study applied DKT with
tracing data from famous online learning platforms Khan Academy, EdX, and Coursera.
Two models, a Recurrent Neural Network (RNN) and a variation of it called LongShort-term Memory (LSTM), were used, and their data were converted into a chain
succession of exercises by encoding when there are few unique exercises. For input
with large input exercises, they assign the inputs to random vectors. Deep Knowledge
Tracing had 0.89 AUC. Building upon DKT, Transformer Bayesian Knowledge Tracing
(TBKT) was proposed [8]. The model accepts as inputs the questions, the skills, and the
response. The data are augmented by training a priori Bayesian regression model on a
training set sample. Bayesian inference is used to assess the student’s knowledge, and
predictions are then made based on the posterior probability computed using Markov
Chain Monte Carlo. The model was tested on the ASSISTment2009 dataset provided by
platform ASSISTment; the results were measured in terms of AUC and 0.759 accuracy
and 0.762 AUC.
The commonality of all of the approaches discussed in this section is that they are
shared, general models that are trained on a collection of student performance data and
applied to individuals. Though error rates and accuracies are reported in their studies,
information needs to be provided on how well the models perform for individual students
(not just entire groups) and their consistency.
2.2 Inconsistency in Individualized Student Modeling
Models used for performance prediction and knowledge tracing were presented; however, all those models have in common that they are general models, models that are built
using all student information. There has been criticism of those models. One criticism of
the current knowledge evaluation process argues that it creates assumptions that students
are identical. Instead, it argues that the focus should be on student parameter changes
and how each change impacts their performance [9]. A model that will track the student
takes for each skill was proposed, and the student’s performance in each skill is used as
322
S. Mondesire and E. Nsiye
input. Expectation Maximization is used to fit the parameters for each student model,
and a general model is used to compare how each input feature impacts the model in
both cases. In addition, to evaluate student mastery of each model, they developed an
algorithm called ExpOpp, which evaluates the probability of mastery by the student. It
was found that in individual models, there is much variation in the number of practice
opportunities to evaluate mastery compared to using a general model. In addition, it was
found that a general model estimates mastery of skill earlier than is the case compared
to the individual models. Other arguments against general population models were that
the performances of multiple students at the same period are not similar and that performance of a student change over exercises and should not be treated similarly if the
difference in time between the exercises is long enough (Zhou, 2021). Based on that,
Leveled Attentive Knowledge Tracing (LANA), a Knowledge Tracing model based on
a transformer, was proposed., LANA inserts positional embedding into the modules; it
uses a student-related feature extractor to abstract the input sequences of student information. It uses those features with a module as a pivot to build decoders for different
students. Evaluated with the EdNet and RAIEd2020 datasets, LANA achieved an AUC
of 0.8059 on the EdNet dataset, and on RAIEd2020, it achieved an AUC of 0.8056 [10].
Others have looked at the issue of individualization in different aspects. One hypothesis was that models based on specific courses outperformed all forms of models [11].
Models were developed focusing on specific courses and students’ courses tuple. Using
a dataset provided by the University of Minnesota and containing grades compiled over
twelve years for 2949 students, the researchers developed two different algorithms; the
first is a course-specific regression model built with features such as students’ previous
courses, grades, and GPA. The second model is the student-specific regression model
based on students’ grades. Using the Root Mean Squared Error was used as a measurement, and in their results, they found that the model based on GPA outperformed the
model based on students’ grades, achieving lower RMSE in all cases.
Moreover, another research argues that students’ performances can be tracked by
focusing on students’ study habits and are interested in answering whether models should
be course based or generalized and if these models can identify in-danger students [12].
Using features about Session information such as the number of sessions, actions total,
session action average, average session duration, total session duration, social features
such as the number of comments and responses the student receives or gives, how many
questions the students ask on courses’ forums, they divided performances into two groups
distinction (A and above) or non-distinction (B+ or less). Models such as Support Vector
Machine (SVM), Random Forests, and Logistic Regression were used. Performances
tests were run such that a model was built on each course but trained with early data from
a course and evaluated with later data from a course to understand how early prediction
can be done. A model was trained on one course and tested on another. Using F1 scores
as measurement, the highest F1 score achieved by taking a model from one class (Java)
to another (Discrete Math) was 0.709. However, the score was lowered than the 0.745
achieved by training a model on the Discrete Math dataset.
The literature has demonstrated multiple approaches to predicting students’ performance. However, based on the studies’ results and prediction accuracy, there are
Personalized Student Performance Prediction Modeling
323
inconsistencies in how each algorithm performs for individuals, as performances were
evaluated for overall populations of students.
3 Methodology
This work hypothesizes that predictive models can perform differently across different
distributions of student data when either generalized or individualized models are used
such that using a specific model can be to a detriment for students. This work examines
that potential inconsistency by analysing the accuracy of generalized and individualized
predictive models on individuals.
3.1 Algorithm Consistency
To examine the consistency of generalized predictive models on individual students,
multiple generalized instances of predictive machine learning models were created on
a training set of 106 students. Then, each student of the testing set was individually
evaluated by the model, predicting the student’s performance on the next exercise. For
each exercise prediction, student performance data from the last three exercises were
input into the model, including the assignment number, percentage score, number of
attempts, and duration of each attempt. This approach was followed to limit the model
from having a look-ahead bias and only having access to immediate data leading up to
the assessment to be predicted. The training procedure trained each algorithm’s model
on all of the training set’s students’, also where the inputs were the last three assessment
results. The predictor for the training and testing phases was the student’s score on the
assessment, which ranged between 0 and 100%.
To examine consistency in an individual model, which is a model, trained solely on
a student’s data; a second experiment was executed. As with generalized model, student
performance data from the last three exercises were input into the model, except in
this case the model is trained on individual student data that was captured before each
predicted exercise. For example, for a student working on exercise 5, a predictive model
is trained on data preceding that exercise, data from exercises 1–4. In exercise 6, the
model was scrapped for a new model trained on that student’s data from exercises 1–5.
The new models are created instead of refitted on new data to have the highest accuracy in
training the model on all available data at a time. Each model is unique to the student, each
student has a model fitted before each exercise. Though time-consuming, this approach
yielded regression models with the lowest error compared to refitted models.
Algorithms List. Algorithms List Seven models were trained, and each model had a
distinct algorithm. The first model is an instance of Linear Regression (LR1), Eq. (1),
which uses the ordinary Least Square Equation based on the Sci-kit Learn [13] Python
library. Next are three different Linear Regression equations, LR2 (1), LR3 (2), and LR4
(3), Eq. (1) to Eq. (3). These regression implementations followed the StatsModel [14]
implementation regression for the library’s regression formula customization.
y = a + bx
(1)
324
S. Mondesire and E. Nsiye
b
ln(x)
(2)
ln(y) = a + bx
(3)
y=a+
One other framework used in performance prediction is knowledge tracing. A python
implementation of Bayesian Knowledge Tracing called PyBKT introduced in 2021
(Badrinath, A., Wang, F. and Pardos, 2021) will also be used. PyBKT models can be
trained similarly to models from the Scikit-Learn library by calling ‘fit’. It also offers
prediction capabilities that will predict the student grade based on the level of mastery.
To train a model with PyBKT, the dataset must be mapped to the algorithm parameters,
in this case Assignment number is mapped to the skill name and the point possible to the
correct answer option. Moreover, this work will be also using Random Forest Regressor
with maximum tree growth of 3, and a Neural Network model with two layers, the first
having 200 units, and the second having 70 units.
3.2 Dataset
A dataset of student performances from an online ITS of an undergraduate introductory
Microsoft Office and Desktop Fundaments course was used to train and test the predictive
models. The course teaches students how to use different functionalities of Microsoft
Office and Windows Desktop apps; the data is from 2020–2022. The dataset contains
anonymized student data, where each student is given a unique (randomized) student ID
serving as an identifier for each student across the entire set. Each data entry contained the
resource number, which is the identifier for each exercise and through sections (exercise
5.9.2 represents Sect. 5), the points possible per exercise, the required score, their score
in different attempts up to 13 per exercise, when they took those attempts and the score
percentage they obtained in their first attempt. For this research, only the student ID, the
resource number, the attempt date, and the score percentage were kept. The dataset’s
primary purpose is to predict the student’s performance based on their previous three
first attempt performance. To accomplish the task, the independent variable data were
represented as three columns in the data set corresponding to ‘Previous Attempt 1’,
‘Previous Attempt 2’, and ‘Previous Attempt 3’. In addition, a feature column called
Section was added to be able to track student performances through different sections of
the course. The scoring percentage was used as the target/predictor column and ranged
from 0–100, though eventually scaled between 0 and 1. The dataset contains information
on 328 students and was split into three different sets. The first set is the training set,
which contains data from 106 students from the earliest semester. The second set is the
validation set, which contains data from 32 students. The final set was the testing set
of 190 students’ data from the latest semester. The data set splits were used to analyze
the effectiveness of each predictive model on the newest batch of students. Additionally,
190 were selected to have a large enough sample to satisfy the Power G Analysis for a
sample size of at least 189 students.
Personalized Student Performance Prediction Modeling
325
4 Results
To measure the performance of the algorithms, the Root Mean Square Error (RMSE)
was used to determine how close the model’s prediction is to the actual values; the
lower the RMSE, the closer the algorithm predictions are to the actual values. The result
will measure how often each algorithm produces the lowest RMSE for our test cases.
Additionally, Tukey’s Honest Significant Difference (Tukey HSD) Test was used to
measure the difference in the RMSE. In comparing Group A and Group B, a positive
mean difference implies that Group B’s mean is more significant than Group A’s. A
negative difference implies that Group B’s mean is smaller than group A’s. Tukey HSD
allows for a comparison of the average RMSE of each algorithm. The seven models were
evaluated in a testing set containing information on 190 students’ performances. The
results are presented as a head-to-head comparison of each algorithm in their respective
groups.
4.1 General Models
After comparing general models in terms of RMSE, it was found that Random Forest
delivers the lowest RMSE for 76 out of 190 cases, followed by BestLR3 with 63, and
Linear Regression (LR1) with 22; the full results can be observed in Fig. 2. When
comparing the mean RMSE of each algorithm, it can be observed that the RMSE average
favors Random Forest only once against the rest of the algorithms. The full results of
the Tukey Test can be observed in Table 1.
Fig. 2. Student count comparison of the RMSE of the six different general models
4.2 Individualized Models
Observations from the individual models (see Fig. 3) yielded that the Deep Learning
model was the best, with the lowest RMSE of 132 out of the 190 testing set students,
followed by Linear Regression with 26 and Random Forest with 19. Statistical analysis
326
S. Mondesire and E. Nsiye
with Tukey HSD observed that Deep Learning produced a smaller RMSE mean difference when compared with the six other algorithms. The mean RMSEs for the individual
models are displayed in Table 2.
Fig. 3. Student count comparison of the RMSE of the six different individualized models
Table 1. Comparing the mean RMSE of the six different general models head-to-head
Group A
Group B
BKT
LR2
M ean
Difference
0.0031
Group A
Group B
LR3
LR4
M ean
Difference
-0.004
BKT
LR3
0.0031
LR3
DL
0.0064
BKT
LR4
-0.001
LR3
LR1
0
BKT
DL
0.0095
LR3
RF
0.0005
BKT
LR1
0.0031
LR4
DL
0.0105
BKT
RF
0.0035
LR4
LR1
0.004
LR2
LR3
0
LR4
RF
0.0045
LR4
-0.004
DL
LR1
-0.0064
DL
0.0064
DL
RF
-0.006
LR1
0
LR1
RF
0.0005
RF
0.0005
LR2
LR2
LR2
LR2
5 Discussion
The experiments examine how well generalized and individual models perform based
on consistency. Because ITSs, traditionally using generalized models, are deploying
and applying one algorithm’s model to all students, algorithm consistency should show
Personalized Student Performance Prediction Modeling
327
apparent domination in the lowest RMSE for one algorithm versus all other algorithms.
In the generalized model experiments, no algorithm outperformed the other algorithms
by yielding the lowest RMSE for half of the students in the testing set. In other words, no
algorithm consistently dominated the other approaches to a point where most students
had a lower prediction error (RMSE) than the other approaches. For example, random
forest yielded the best prediction error for 76 students, and LR4 was the best algorithm
for 63 students. These results support the hypothesis that the algorithms can perform
differently for different students. In that case, the random forest was the better predictive
algorithm for 40.64% of the students, while LR4 was best for 33.69%.
Table 2. Comparison of the mean RMSE of the six different individualized models.
Group A
Group B
BKT
LR2
M ean
Difference
0.156
Group A
Group B
LR3
LR4
M ean
Difference
-0.0122
BKT
LR3
0.1555
LR3
DL
-0.2176
BKT
LR4
0.1433
LR3
LR1
-0.206
BKT
DL
-0.0621
LR3
RF
-0.2057
BKT
LR1
-0.0505
LR4
DL
-0.2054
BKT
RF
-0.0502
LR4
LR1
-0.1938
LR2
LR3
-0.0005
LR4
RF
-0.1935
LR2
LR4
-0.0127
DL
LR1
0.0116
LR2
DL
-0.2181
DL
RF
0.0119
LR2
LR1
-0.2065
LR1
RF
0.0003
LR2
RF
-0.2062
Similarly, the other algorithms, baring deep learning, performed best for smaller
percentages of the students but still demonstrated their prediction accuracy capability
for some students, supporting the inconsistency claim. From these generalized model
results and the fact that ITSs rely on these algorithms for their shared prediction models,
this work emphasizes that algorithm selection is critical in accurately identifying when
a student needs intervention. Additionally, selecting the wrong algorithm may delay a
student from receiving remediation early enough for positive change.
Similar inconsistency in algorithm performance is observed in the individual models. Overall, it is observed that some algorithms generated better predictive models for
some students. However, contrary to the generalized models, the deep learning model
dominated performance, where it is the best algorithm for 69.47% of the students, which
is over most of the student testing population.
When examining the Tukey results, the pattern presence is confirmed, with the produced deep learning algorithm mean RMSE being lower than six other algorithms and
higher than 0.01 for all six comparisons. Outside of Deep Learning, it is observable that
328
S. Mondesire and E. Nsiye
LR1, the second-best algorithm in terms of the count of lowest RMSE, produces a lower
mean difference when compared to the five other algorithms (excluding deep learning),
and that mean difference range is more significant than 0.01 in four out of the five
algorithms. Consequently, even if deep learning is removed from consideration, there
will still be a dominant algorithm, highlighting the lack of consistency in the individual
model’s performance.
6 Conclusion
ITSs provide automated ways to disseminate information to students, typically including
methods to identify when a student needs remediation. Many proactive ITSs employ generalized prediction data models that identify the probability of a student succeeding on
the next exercise and intervene if that prediction is below a threshold, where these models
are trained on data from a collection of a student population. This work demonstrates an
inconsistent performance with these generalized models, where some algorithms yield
fewer errors for some students than others. Notably, random forest and a variant of
linear regression yielded the best results for most students, but Bayesian Knowledge
Tracing and deep learning performed the poorest. Individual models demonstrated similar inconsistencies among all tested algorithms, except deep learning, which dominated
test algorithm tests by yielding the best prediction performance for 69% of the students.
These results are a milestone for designing an upcoming student digital twin, which models student performance as they progress through an academic program. Additionally,
this work continues the optimization of real-time predictive analytics for customization
and personalization in student and systems modeling [15–17]. The conclusions from
this work are steering the model design to provide personalized, predictive performance
analytics for intervention alerts through individualized prediction ensembles that will
include deep learning prediction.
References
1. Baker, R.S.: Stupid tutoring systems, intelligent humans. Int. J. Artif. Intell. Educ. 26, 600–614
(2016). https://doi.org/10.1007/s40593-016-0105-0
2. Polyzou, A., Karypis, G.: Grade prediction with models specific to students and courses. Int
J Data Sci Anal 2, 159–171 (2016)
3. Pardos, Z., Heffernan, N.T.: Modeling Individualization in a Bayesian Networks. In: De
Bra, P., Kobsa, A., Chin, D. (eds.) UMAP 2010. LNCS, vol. 6075, pp. 255–266. Springer,
Heidelberg (2010). https://doi.org/10.1007/978-3-642-13470-8_24
4. Orji, F.A., Vassileva, J.: Machine Learning Approach for predicting students academic performance and study strategies based on their motivation. arXiv.org (2022). https://arxiv.org/
abs/2210.08186v1. Accessed 27 Oct 2022
5. Jayaprakash, S., Krishnan, S., Jaiganesh, V.: Predicting students academic performance using
an improved random forest classifier. In: 2020 International Conference on Emerging Smart
Computing and Informatics (ESCI), pp. 238–243 (2020). https://doi.org/10.1109/ESCI48226.
2020.9167547
Personalized Student Performance Prediction Modeling
329
6. Gull, H., Saqib, M., Iqbal, S.Z., Saeed, S.: Improving learning experience of students by
early prediction of student performance using machine learning. In: 2020 IEEE International
Conference for Innovation in Technology (INOCON), pp. 1–4 (2020). https://doi.org/10.
1109/INOCON50539.2020.9298266
7. Piech, C., et al.: Deep knowledge tracing. arXiv.org (2015). https://arxiv.org/abs/1506.05908.
Accessed 27 Oct 2022
8. Wei, T., Hu, B., Ni, Q.: Bayesian knowledge tracing based on transformer. In: 2022 IEEE
5th International Conference on Electronic Information and Communication Technology
(ICEICT), pp. 449–452 (2022)
9. Lee, J.I., Brunskill, E. (n.d.). The impact on individualizing student models on necessary
practice opportunities. Accessed 5 Nov 2022
10. Zhou, Y., Li, X., Cao, Y., Zhao, X., Ye, Q., Lv, J.: Lana: Towards personalized deep knowledge
tracing through distinguishable interactive sequences. arXiv.org (2021). https://arxiv.org/abs/
2105.06266. Accessed 4 Nov 2022
11. Gitinabard, N., Xu, Y., Heckman, S., Barnes, T., Lynch, C.F.: How widely can prediction
models be generalized? performance prediction in blended courses. arXiv.org (2019). https://
arxiv.org/abs/1904.07328
12. Pedregosa, et al.: Scikit-learn: machine learning in python. JMLR 12, 2825–2830 (2011)
13. Seabold, S., & Perktold, J. (2010). statsmodels: Econometric and statistical modeling with
python. In 9th Python in Science Conference
14. Badrinath, A., Wang, F., Pardos, Z.: pyBKT: An Accessible Python Library of Bayesian
Knowledge Tracing Models (2021)
15. Nsiye, E., Tse, B., Wright, T., Sanchez-Arias, R., Tapanes-Castillo, A., Mondesire, S.: Examining student performance prediction model consistency for human digital twin personalizatio.
In: IEEE Digital Twin and Parallel Intelligence, Orlando, FL, Nov. 7–9, 2023 (2023)
16. Sharma, N., Rivera, K., Angelopoulou, A., Mondesire, S.: Predicting Software Container
Load for Scalable Cloud-based Virtual World Simulations. In: International Symposium on
Big Data and Data Science (2020)
17. Sharma, N., Rivera, K., Angelopoulou, A., Mondesire, S.: Predicting large-scaled, cloudhosted virtual world resource demands for automated load balancing. In: 2020 International
Conference on Computational Science and Computational Intelligence (CSCI) , pp. 338–343.
IEEE (2020)
A Social Profile-Based Recommendation
Architecture for E-Learning Systems
Xola Ntlangula and Wai Sze Leung(B)
University of Johannesburg, Johannesburg, South Africa
200713134@student.uj.ac.za, wsleung@uj.ac.za
Abstract. The increased reliance of education institutions on e-learning systems
to provide learning and teaching activities and resources has prompted a need for
innovative strategies that deliver effective learning to students. One approach being
considered involves enhancing personalized learning experiences through the
implementation of personalized learning paths. Personalized learning paths have
proven highly beneficial for students; however, they are predominantly guided by
learning objectives rather than the unique attributes of each student. Additionally,
the inflexibility observed in e-learning systems poses a challenge in integrating
customization options. This paper proposes an architecture for a recommendation
system that creates personalized learning paths from a student’s profile. The resulting architecture enhances an e-learning system’s personalization by using external
student data to create a more comprehensive student profile, thereby improving
the system’s ability to generate learning paths and deliver technology-enhanced
learning experiences.
Keywords: Personalized learning paths · Social profile · Recommendation
systems
1 Introduction
Higher Education Institutions (HEIs) worldwide have embraced online learning (elearning) through Learning Management Systems (LMSs) to offer education beyond
traditional geographic and time constraints [1]. The early designs of LMSs followed
a “one-size-fits-all” approach, overlooking the diversity of student needs [2, 3]. This
has spurred research growth in the integration of Recommendation Systems (RSs) for
personalized learning in LMSs [4]. However, with limited data from LMS activities,
RSs traditionally create course-centric personalized learning paths. To bridge this gap,
researchers recommend collecting more extensive datasets, including a student’s social
profile from external sources, to enhance the quality of recommendations [5–7]. In this
context, the paper makes a valuable contribution to the field of research by presenting
an architecture for a technology-enhanced learning model that enhances the personalization of existing e-learning platforms, focusing on a student-centered approach. The
paper is organized as follows: Sect. 2 presents a background discussion on the current
state of personalization in LMSs. Section 3 reviews related work and Sect. 4 presents
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2024
K. Arai (Ed.): SAI 2024, LNNS 1017, pp. 330–343, 2024.
https://doi.org/10.1007/978-3-031-62277-9_20
A Social Profile-Based Recommendation Architecture
331
the specifics of the proposed architecture. Section 5 provides an evaluation of the architecture, and the paper concludes by discussing the future direction of this research in
Sect. 6.
2 Personalization in Learning Management Systems
The inclusion of personalized e-learning, which offers students tailored recommendations for delivering suitable learning resources, is considered a valuable addition to
LMS offerings. By implementing this approach, HEIs can deliver learning experiences
to a diverse student population while addressing each student’s unique needs. Personalized e-learning tailors the learning experience to an individual’s profile and preferences.
Thus, enabling students to learn with greater ease and experience higher satisfaction as
the e-learning platform adapts its personalization to the student’s evolving profile over
time [8]. It transforms e-learning environments into student-centric spaces, ultimately
elevating the learning experience [9].
The argument in favor of a more student-centric approach is straightforward: students have diverse characteristics, so expecting everyone to follow the same learning
path is unreasonable and will not lead to the same level of success. Though, personalization in e-learning can be in various forms: personalizing the curriculums, learning
objects, learning materials, learning activities, learning paths, or learning interfaces.
Traditional e-learning environments typically provide limited personalization options.
Despite the potential for increased flexibility with personalized learning paths, most
e-learning platforms prioritize personalizing learning objects [10, 11]. Additionally,
personalization often relies on the available course offerings rather than considering the
distinctive characteristics of individual students. Efforts to establish a student-centric
LMS with personalization, however, come with challenges. The data needed for these
personalized recommendations not only involve analyzing a student’s interactions and
engagement within the LMS but also require access to the student’s social profile, which
encompasses beliefs, motivations, interests, enthusiasm, and attitude [12]. Researchers
have highlighted that gathering such information would empower LMSs to collect and
analyze more comprehensive data, resulting in the creation of recommendations that are
meaningful, precise, and pertinent [5–7].
Consequently, the pressing challenge lies in identifying and applying the right data to
determine the optimal approach to address each student’s unique needs. In recent years,
the exponential expansion of information resources has been facilitated by technological
advancements, including enhanced internet access, and reduced processing and storage
costs. The information explosion has had a dual impact on students. On one hand, it has
provided them with a wide variety of learning resources and opportunities, but on the
other hand, it has led to learning confusion and information overload [13]. In the absence
of effective guidance and scientific recommendations, numerous students struggle to
adapt to the learning processes [14]. As a result, Recommendation Systems have become
an integral component of personalized e-learning, and the design of personalized elearning platforms now often revolves around the central research topic of personalized
learning paths [15].
332
X. Ntlangula and W. S. Leung
2.1 Recommendation Systems
Recommendation Systems (RSs) have gained attention in the education domain in the
past few years [16]. In general, RSs enable adaptive learning on e-learning platforms,
addressing the challenge of data overload by comprehending users’ requirements and
providing suggestions or predictions regarding their interests and preferences [16–18].
The general framework of RSs typically encompasses user modeling, course modeling,
recommendation modeling, and the presentation of recommendation results [18]. Three
of the most used RSs are Collaborative Filtering (CF), Content-Based (CB), and Hybrid.
Collaborative filtering is a domain-independent recommendation technique which
offers suggestions by calculating similarities between items and users with similar interests and preferences [19]. Content-Based RS is a domain-dependent algorithm that predicts outcomes by examining item attributes like web pages, user profiles, news, or
publications [19]. Hybrid RS combines various recommendation techniques to optimize
systems and address limitations present in other RSs [19]. Each of these systems plays a
significant role in personalizing e-learning platforms [18–20]. On e-learning platforms,
RSs are applied to provide accurate, relevant information that covers a vast range of
diverse student profiles.
In contrast to e-commerce or e-government systems, e-learning environments pose
a more complex challenge for RSs because of the diversity of student needs. One contributing factor is that the needs, expectations, and preferences of students tend to change
over time [10]. Such changes are difficult for current LMSs to identify and respond to,
largely due to limitations such as data privacy, data scarcity, diversity, overspecialization,
and novelty of recommendations [20]. One suggested solution to address some of these
limitations is the implementation of a hybrid recommendation system, which combines
multiple approaches drawn from different RSs. Nonetheless, Hybrid RSs come with
their own challenges, primarily stemming from multi-modularity, data heterogeneity,
sparsity, and uneven data distribution [18]. Despite these inherent limitations, hybrid RS
techniques have been generally well-received compared to their alternatives.
Currently, prevailing RS approaches typically concentrate on data associated with
user age, knowledge level, learning styles, preferences, and performance. User preference information is derived from either explicit (student characteristics) or implicit
(behavior during LMS usage) data sources [16, 17]. The idea is that researchers in this
domain might explore broadening the variables to encompass elements identified in
educational psychology research, such as motivational factors and levels of engagement
[10]. These variables can be harnessed by recommendation systems that utilize a combination of artificial intelligence techniques, including Bayesian networks, ontologies, collaborative filtering, data mining, deep learning, factorization-based methods, K-nearest
neighbor (KNN), neural networks, and knowledge maps [10, 21, 22]. Recently, deep
learning techniques have become a favored alternative among recommendation system
researchers in the field of education [16]. This preference arises from their capacity to
improve prediction quality, generate more pertinent recommendations, and strengthen
the recommender model, particularly when dealing with larger datasets.
A Social Profile-Based Recommendation Architecture
333
2.2 Personalized Learning Paths
Learning paths are instrumental in personalizing e-learning platforms [11]. A learning
path constitutes a series of activities a student follows in their learning journey. The
arrangement of these activities can lead to confusion or progress in a student’s learning, depending on how they are structured [13]. Personalization aids in optimizing the
structure of learning paths, and RSs can be utilized to tailor these learning paths for individual students. Consequently, a personalized learning path is a sequential list of learning objectives organized to address the specific requirements of an individual student
[15]. Learning paths contribute to enhancing student performance, designing curricula,
detecting learning challenges, and offering feedback to both students and teachers [11].
Learning path recommendations are not as common as recommendations for learning
objects, but research in this area is steadily gaining momentum [10]. While personalized learning paths are being adopted, they are often constructed primarily using course
data rather than focusing on the individual student’s data. This limitation is attributed
to the restricted data access, which hampers the implementation of effective personalized learning paths, as recommended by researchers. Studies suggest that to create
truly beneficial personalized learning paths for students, e-learning platforms need to
access high-quality data [5, 6, 23]. Typically, personalized learning path recommendation models comprise two key components: data extraction and the actual generation of
the learning paths [18, 23]. Therefore, the quality of the extracted data and the choice of
recommendation technique both have a significant impact on the quality of the generated
recommendations.
2.3 Student’s Social Profile
In a personalized e-learning environment, a student is a primary unit of interaction
with a profile that provides attributes that uniquely identify the student [24]. These
attributes or characteristics are central to personalization, as they encompass a wide range
of factors that define an individual student, including social interactions, community
involvement, interests, preferences, goals(beyond academic goals), self-efficacy, socioeconomic background, demographics, cognitive abilities, and ethical beliefs [25, 26].
Depending on these attributes a student’s profile in an e-learning environment can be
static or dynamic. Static profiles are independent of temporal aspects while dynamic
depends on temporal behaviors [27]. Most personalized learning paths typically rely
on data related to the student’s learning styles, activities within the LMS, and course
content, with limited utilization of data from the student’s profile beyond the course
structure or LMS activities [11, 28].
A student’s social profile provides a comprehensive view of the student as a complete individual, extending beyond attributes confined to the e-learning environment. A
social profile can be enriched with external data encompassing psychological characteristics, socio-economic traits, social life, broader life goals (beyond learning objectives),
interests, and individual traits like learning styles [24]. Conversely, despite the enhanced
relevance and accuracy of recommendations when using diverse data sources, the challenge lies in collecting data from various origins and integrating it into a single model.
This challenge not only impacts data reliability and confidentiality but also presents
334
X. Ntlangula and W. S. Leung
difficulties in the analysis and interpretation of the combined data [27]. External factors
or life aspects that evolve over time introduce additional complexity in constructing a
comprehensive social profile [28, 29].
3 Related Work
There is a wealth of research on recommendation system models for e-learning, reflecting
the ongoing expansion of this field due to the numerous challenges it can address. This
section of the study provides an overview of some of the relevant studies in this area. One
research proposes a personalized recommendation system that suggests the most suitable
learning resources, learning sequences, and learning tasks to learners based on their personas [13]. The personas encompass various characteristics, including static attributes
such as age, gender, education, and professional position, and dynamic attributes like performance, content engagement, progress, time, efficiency, and ability level. The research
employs collaborative filtering RS and uses Root Mean Square Error (RMSE) to assess
the algorithm’s outcomes. While the study aims to address the diversity limitations
among students through collaborative filtering, it acknowledges potential limitations
of this approach, such as reduced performance with larger datasets, challenges related
to cold-start scenarios, data sparsity issues, and heavy reliance on either inter-user or
inter-item relationships [30].
Another study introduces the concept of generating personalized learning paths in
real time by adapting to changes in learning preferences and performance [15]. The
research continuously monitors students’ logs, and feedback from these logs is used
to adjust dynamic factors. It applies Felder-Silverman’s learning style to analyze the
student’s preferred learning style and utilizes an ontology-based RS to address coldstart challenges and data sparsity. Ontologies serve as a storage unit for saving activity
logs, learner-related parameters, and learning object metadata. The student’s ability and
the difficulty of learning objects are adapted based on their academic achievements, time
invested in learning, the number of attempts made, and ratings. However, a limitation
of this model is that learning objects are manually established by instructors, and the
recommendation process depends on student feedback. During the cold start phase, the
values are not personalized, and data generated from learning paths cannot be reused.
In a different study, the authors propose employing deep learning to address the challenges associated with traditional recommendation systems [18]. The authors assert that
deep learning enhances system performance, elevates the quality of recommendations,
and yields more precise results, particularly in addressing complex problems. Neural
networks have demonstrated great success in modeling these improvements. There are
neural network techniques such as Multilayer Perceptron’s (MLP), Recursive Neural
Networks (RNN), Convolutional Neural Networks (CNNs), Neural Attentional Modeling (NAM) and Deep Reinforcement Learning (DRL). Deep learning techniques excel
in handling large-scale data, as they can be trained to identify significant features within
extensive datasets, manage data efficiently, and process it intelligently and in real-time.
This capability can aid in comprehending student behavior and enhancing the quality
of teaching and learning. The authors suggest considering students’ goals and longterm preferences, but the fundamental challenge remains in accurately representing and
modeling user preferences based on user data.
A Social Profile-Based Recommendation Architecture
335
In [31], the authors explore personalized learning paths through the utilization of big
data knowledge maps. They address issues related to knowledge overload and knowledge
loss by integrating knowledge maps with learning paths. The learning paths are created
by analyzing knowledge maps, considering factors such as the student’s learning pace,
interests, knowledge point difficulty, study habits, importance, and goals. The authors
employ a combination of a genetic algorithm and an ant colony algorithm to construct
the recommendation system.
In one Knowledge Tracing-based Knowledge Demand Model (KT-KDM) study,
a learning path recommendation model based on knowledge tracing and reinforcement
Their model recommends knowledge tracing to reflect students’ learning demands rather
than the learning cost manually generated and applies the reinforcement tracing model
to obtain the learning demand. Therefore, they apply knowledge tracing to trace learning processes dynamically, and dynamically provide personalized learning paths more
aligned with the actual situation. The base of this architecture is connected to neural
networks.
A different approach with KT-KDM was explored in [23], with a learning path
recommendation model grounded in knowledge tracing and reinforcement learning. In
this study, the authors proposed a model that used knowledge tracing to adapt to students’
learning requirements instead of manually determining the learning cost. It employed
the reinforcement tracing model to derive the learning demand, consequently utilizing
knowledge tracing to dynamically track the learning process and offer personalized
learning paths that align more closely with the current circumstances. The underlying
framework of this architecture is linked to neural networks.
Following an ontology/knowledge-based learning path approach, the authors in the
study in [15] were also able to devise a recommendation system for adaptive learning.
Ontology-based recommendation systems are predominant techniques, as they effectively address challenges like cold-start issues, data sparsity, and over-specialization. The
authors observe that many researchers have combined ontology with other AI techniques,
including natural language processing, context dimension tree, data clustering, collaborative filtering, rule association, Bayesian probability, and multi-agent techniques. The
study demonstrates that learning path recommendation processes are influenced by learning activities and context. Among nine research studies on ontology-based learning paths,
only three incorporate a dynamic recommendation process.
Another study introduces a Decision Support Model (DSM) that emphasizes the
importance of student motivation [32]. The authors argue that motivation is a vital component alongside effective teaching and educational programs. The model constructs a
student’s profile using Personal Data (age, hometown, marital status, education, employment, hobbies, perspectives on motivation, and teamwork); Analytical Data (musical
and logical strengths scores); Introspective Data (intrapersonal, existential, and visual
strengths scores); and Interactive Data (interpersonal, verbal, and kinesthetic strengths
scores. The DSM is designed with two algorithms: univariate or multivariate descriptive
modeling to summarize student profiles, and clustering (K-means) modeling to group
students based on their data. However, this model can only be tested with a limited number of users due to the computational cost of clustering. Experiments conducted over
several weeks with English students showed an increase in motivational levels when the
336
X. Ntlangula and W. S. Leung
model was applied, confirming that considering learner traits positively impacts student
motivation. It’s worth noting that the study’s limitation to the English language restricts
its applicability to a diverse student population, but the model could be extended for the
development of personalized learning paths.
In the study in [21], a clustering hierarchical progressive improvement model (GHPI)
is introduced, utilizing an analysis and context-dependent data envelopment analysis
method. This model categorizes students into different classes and calculates metrics to
establish a reference path for individuals with inefficient learning processes. It gives rise
to new recommendation system (RS) models, including collaborative filtering-based, KNearest Neighbor-based, neural network-based, knowledge map-based, and ontologybased approaches. The study demonstrated its capability to successfully recommend
learning strategies tailored to individual needs by employing the most suitable RS as
required. Moreover, the research augmented learning path recommendations with strategies aimed at enhancing learning efficiency, student performance, and preferences. The
model assesses each student’s efficiency level, incorporates their learning habits and
those of their peers, and offers personalized learning suggestions.
Big Data is leveraged in the study in [33], with the authors implementing an adaptive
e-learning model aimed at delivering the most appropriate learning content to individual students. A MapReduce-based Ant Colony Optimization algorithm is utilized to
identify pertinent future learning goals and employ a MapReduce-based Social Network Analysis to gauge learner motivation and social involvement, allowing for the
customization of specific learning speeds for each student. MapReduce is a well-known
Big Data framework for processing, managing, and analyzing large datasets [33]. The
research integrates Social Network Analysis (SNA) to extract student motivation and
social engagement data from social media. Their architecture is adaptable, capable of
accessing diverse data sources, and easily scalable for future systems. However, the
inclusion of social media data in the dataset reduces data reliability, thereby limiting the
accuracy of the recommendations.
The last relevant research introduces a hybrid recommender system called
MoodleRec, which functions as a plug-in within the Moodle learning management system [34]. MoodleRec navigates through various learning object repositories and provides
a prioritized list of learning objects based on keyword queries. Similar to many other
e-learning recommendation systems, MoodleRec centers around learning objects and
aids instructors in developing new courses. This hybrid RS combines content-based and
collaborative filtering techniques, with clustering algorithms utilized to categorize students according to their learning styles. The paper advocates for machine learning and
ontologies as technical foundations and supports for data mining, despite their not being
fully mature for their model. The authors acknowledge that the cold-start problem limits
the effectiveness of the RS and recommend expanding their datasets.
From the mentioned related studies, the following insights can be extracted:
• Data diversity enhances recommendation accuracy, but incorporating data from
multiple sources can introduce complexity in data interpretation, analysis, and
verification.
• Deep learning techniques are particularly well-suited for large datasets and contribute
to improved system performance.
A Social Profile-Based Recommendation Architecture
337
• Learner behaviour within the LMS is readily available, and many recommendation
system models leverage this data for their algorithms.
4 Proposed Model
This paper presents a novel architecture for e-learning recommendation systems, specifically emphasizing the provision of personalized learning paths for students using their
social profile. The visual depiction of the proposed design is illustrated Fig. 1. Typically,
Recommendation System (RS) architectures comprise three common layers: the input
layer, process layer, and output layer [32]. In the proposed architecture, the input layer
incorporates student data from both the LMS and external data sources. The process
layer is tasked with extracting the data and generating recommendations. Finally, the
output layer entails presenting the recommended learning paths to the student through
the LMS user interface. The central concept is to extend beyond the stored data within
the LMSs, extracting social data to generate more effective and personalized learning
paths for students. The paper argues that relying solely on LMS data may not suffice
for achieving a truly personalized learning experience. Thus, the proposed architecture
integrates additional data sources, potentially from external platforms or various forms
of user interactions, to enhance the recommendation process.
Fig. 1. High-level view of the personalized learning path architecture
The architecture proposes an extension to existing LMSs in the form of a plugin.
This plugin introduces an additional feature to any LMS, aiming to construct a student’s
social profile, generate personalized learning paths, and present these paths to the user.
The student’s social profile is developed by extracting, analyzing, and preparing data
from multiple sources. Personalized learning paths are then generated based on the
classification of the social profile using RS techniques. The proposed model considers
expanding the variables of learning activities that contribute to building learning paths
338
X. Ntlangula and W. S. Leung
provided that datasets containing this information are available. The detailed explanation
of the components of the architecture is provided in the next section.
4.1 Data Extraction Model Components
This component is foundational and crucial to the architecture, as the quality of the generated learning paths relies on the type of data extracted. The model conducts preprocessing
on data derived from various sources within the LMS and external data sources. From
the LMS, the model extracts LMS logs indicating interaction frequency and behaviors,
user statistics, resource information like learning materials and course details, as well
as user-specific data such as names, age, race, language, qualification level, goals and
gender.
External data sources can supply social media data, including media, captions, locations, hashtags, and engagement metrics, through social media APIs. Additionally, psychological data and the financial background of students can be obtained, either through
surveys or integration with existing university internal systems. The extracted data undergoes analysis and processing to generate a comprehensive user profile, encompassing personal, behavioral, goals, socio-economic, knowledge level, and social dynamic
datasets as shown in Fig. 2, Extraction model referencing multiple data sources.
Deep learning techniques are recommended for preprocessing, extraction, and sorting, as they excel in managing and interpreting large datasets more efficiently. The data
extraction model stores the data necessary for generating recommended learning paths.
Recommendations are then generated using the formulated social profile, LMS data, and
previously generated learning paths.
Fig. 2. Extraction model referencing multiple data sources
4.2 User Profile Component
This architecture relies on a social profile incorporating personal, socio-economic,
knowledge level, behavioral, goals, and social dynamic data (Fig. 2, Extraction model referencing multiple data sources). The social profile includes static data like age, address,
A Social Profile-Based Recommendation Architecture
339
language, professional level, and learning goals. Socio-economic data, covering household income, number of siblings, and financial dependencies, is also static. Knowledge
level, indicating learning performance and computer literacy, is dynamic. Behavioral
data, such as course performance, LMS logs (clicks behavior, historic login time, watching time, viewed content, browsing behavior), cognitive style, psychological data, and
learning styles, are also dynamic. Goals, including academic, health, and social goals,
are semi-dynamic. Lastly, social dynamics, like cultural activities, club memberships,
social life, activities on social media, social agents, religion, and politics, are also semidynamic. Course data, specific to a learning object and covering related material, notes,
important dates, course description, and outcomes, is not part of the social profile but
will be used for constructing recommendations.
4.3 Recommendation Model Component
The recommendation model comprises three subcomponents: extracted data (social profile data and course data), recommendation technique, and learning path processing
(generation and storage). The data extraction component prepares the extracted data
for generating recommended learning paths. Different recommendation algorithms are
applied based on the type of data available. The learning path processing component
defines the rules for generating learning paths, applies the recommendation technique,
stores the generated learning paths, and displays recommendations on the LMS user
interface.
4.4 Execution of the Solution
The proposed plugin operates through an API integrated into LMS systems, requiring
the LMS to grant permission for data access. Students have control over the data they
provide, granting consent for the plugin’s access. Once installed, the plugin extracts data
from both the LMS and external sources, constructing a visible social profile for students.
This profile is then utilized to generate personalized learning paths, which are displayed
on the LMS user interface. Configuration settings allow the system to determine the
accessible data.
5 Proposed Architecture Analysis
5.1 Data Collections
By effectively collecting and preprocessing data, a Recommendation System (RS) framework can generate accurate and personalized user profiles, laying the foundation for
meaningful and effective recommendations [18]. Inadequate data quality will result in
suboptimal recommendations. The data is crucial for monitoring learning dynamics and
student behavior, enabling predictions of future actions. The proposed architecture advocates for data collection from both the LMS and external sources. Extracting data from
LMS systems is deemed straightforward, given the precedent set by other models, as
the data is internal. Inclusion of external data sources enhances the system’s appeal
340
X. Ntlangula and W. S. Leung
by elevating recommendation quality. Nevertheless, the authors recognize that this augmentation introduces complexity to the recommendation system, impacting data validity
and reliability. Additionally, the designers must be mindful of security considerations,
although this aspect falls beyond this paper’s scope.
5.2 Data Extraction Model
The suggested architecture accounts for both explicit and implicit data to enhance RS
performance, with sorting being crucial step for precise user preference estimation.
Various extraction tools, tailored to the available data, are proposed to improve data
quality, incorporating AI-powered techniques like data mining, neural networks, and
association rules for data extraction [18]. In this stage, the architecture outlines the
available techniques without specifying which tool to use for different types of data
and the decision-making process regarding extracted data. The paper does, however,
highlight the categorization of data as static, semi-dynamic, or dynamic, which is used
to sort the data collected from external and LMS sources to construct a social profile. It
is acknowledged that the constraints of this architecture include data privacy concerns
limiting access to certain types of data, and security considerations restricting integration
with potential systems. Additionally, potential challenges in the reliability and accuracy
of data extracted from social media or surveys poses the threat of skewed data.
5.3 Recommendation Model
Creating an adaptive e-learning system that can deliver suitable learning content for
each student is a challenging task [33]. The recommendation model serves as a crucial
output of the processing stage, providing essential evidence for the effectiveness of the
proposed architecture. It emphasizes that personalized recommendations, while datadriven, heavily rely on the choice of recommendation techniques. The model follows
the standard design of recommendation model architectures for e-learning systems but
advocates for expanding data sources beyond the LMS to address issues related to missing
data and cold start problems.
5.4 Execution of the Solution
To empirically validate the solution’s practical efficacy, a prototype showcasing the fundamental components of this architecture will be developed. The prototype will illustrate
the creation of a student’s profile, the application of the RS technique based on predefined
rules for generating recommendations, and the presentation of the generated learning
paths to the students. Various user profiles will be employed to assess and exhibit the
adaptability and diversity of the generated learning paths.
5.5 Presentation
The presentation of recommendations to the student is intricately linked to assessing
the student’s preferences, potentially impacting their overall attitude and influence. The
A Social Profile-Based Recommendation Architecture
341
architecture is designed to exhibit information relevant and beneficial to the student while
concealing secondary data. However, the manner of data display relies on the LMS user
interface, suggesting an avenue for future research.
6 Conclusions and Future Work
This paper introduces a comprehensive architecture for a social profile-based recommendation system, providing essential concept definitions and insights into the mandatory
components. By exploring related work, the challenges and limitations are identified,
forming the basis for the architecture’s design. Emphasis is placed on the architecture
being data-driven, and while challenges exist with external data sources, proven techniques can address these issues. The authors acknowledge constraints, such as data
privacy concerns, security restrictions, and potential challenges in the reliability and
accuracy of data extracted from social media or surveys.
In the continuation of this research, the authors plan to delve deeper into both the
extraction model and the recommendation model. This extension will provide a detailed
exploration of how data is extracted, interpreted, and analyzed to construct the social
profile, including an explanation of the extraction technology employed. Additionally,
the recommendation model will apply existing recommendation techniques to generate
personalized learning paths based on the social profile. Future studies may also explore
the security implications of such an architecture and investigate personalized display
methods for learning paths.
References
1. Gamage, S.H.P.W., Ayres, J.R., Behrend, M.B.: A systematic review on trends in using Moodle
for teaching and learning. Int. J. STEM Educ. 9(9), 1–24 (2022)
2. Yang, S., Tian, H., Sun, L., Yu, X.: From one-size-fits-all teaching to adaptive learning: the
crisis and solution of education in the era of AI. ICSP J. Phys. 1237, 1–10 (2019)
3. Fathema, N., Akanda, M.H.: Effects of instructors’ academic disciplines and prior experience
with learning management systems: a study about the use of Canvas. Australas. J. Educ.
Technol. 36(4), 113–125 (2020)
4. Ogbonna, C.G., Ibezim, N.E., Obi, C.A.: Synchronous versus asynchronous e-learning in
teaching word processing: an experimental approach. S. Afr. J. Educ. 39(2), 1–15 (2019)
5. Graf, S., Kinshuk: personalized learning. In: Encyclopedia of the Sciences of Learning,
pp. 2592–2593. SpringerLink, Edmonton (2012)
6. Syed, T.A., Palade, V., Iqbal, R., Nair, S.S.K.: A personalized learning recommendation
system architecture for learning management system. In: Proceedings of the 9th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge
Management (KDIR 2017), pp. 275–282, Science and Technology Publications, Coventry
(2017)
7. Rahayu, N.W., Ferdiana, R., Kusumawardani, S.S.: A systematic review of learning path
recommender systems. Educ. Inf. Technol. 28, 7437–7450 (2023)
8. Lumy, J., Abraham, S., Mani, B.P., Rajesh, N.: Exploring the effectiveness of learning
path recommendation based on Felder-Silverman learning style model: a learning analytics
intervention approach. J. Educ. Comput. Res. 60(6), 1464–1489 (2022)
342
X. Ntlangula and W. S. Leung
9. Salazar, J.C., Aguilar, J., Monsalve-Pulido, J., Montoya, E.: A generic architecture of an affective recommender system for e-learning environments. Universal Access in the Information
Society (2023)
10. Li, H., Peng, K., Ren, H., Shang, F.: Personalized learning paths recommendation based
on learner personas. In: 2022 3rd International Conference on Intelligent Design (ICID),
pp. 290–293. IEEE (2022)
11. Raj, N.S., Renumol, V.G.: An improved adaptive learning path recommendation model driven
by real-time learning analytics. J. Comput. Educ. (2022)
12. Mbodila, M., Mkabile, B., Ndebele, C.: Critical success factors for the effective implementation of e-learning in south African higher education institutions. J. Gender, Inf. Dev. Africa
(JGIDA), 8(3), 229–249 (2019)
13. Matarirano, O., Panicker, M., Jere, N.R., Maliwa, A.: External factors affecting blackboard
learning management system adoption by students: evidence from a historically disadvantaged
higher education institution in South Africa. South Afr. J. Higher Educ. 35(2), 188–206 (2021)
14. Mpungose, C.B., Khoza, S.B.: Postgraduate students’ experiences on the use of Moodle and
canvas learning management system. Technol. Knowl. Learn. 27, 1–16 (2022)
15. Tetzlaff, L., Schmiedek, F., Brod, G.: Developing personalized education: a dynamic
framework. Educ. Psychol. Rev. 33, 863–882 (2021)
16. Pal, N., Dahiya, O.: A systematic literature review on the implications of educational recommender system in teaching learning environment. Int. J. Eng. Trends Technol. 71(4), 203–212
(2023)
17. Klašnja-Milićević, A., Ivanovića, M., Vesin, B., Budimac, Z.: Enhancing e-learning systems
with personalized recommendation based on collaborative tagging techniques. Comput. Educ.
48, 1519–1535 (2018)
18. Liu, T., Wu, Q., Chang, L., Gu, T.: A review of deep learning-based recommender system in
e-learning environments. Artif. Intell. Rev. 55, 5953–5980 (2022)
19. Tarus, J.K., Niu, Z., Mustafa, G.: Knowledge-based recommendation: a review of ontologybased recommender systems for e-learning. Artif. Intell. Rev. 50, 21–48 (2018)
20. Zhong, J., Xie, H., Wang, F.L.: The research trends in recommender systems for e-learning:
a systematic review of SSCI journal articles from 2014 to 2018. Asian Assoc. Open Univ. J.
14(1), 12–27 (2019)
21. Mohamed, M.H., Khafagy, M.H., Ibrahim, M.H.: Recommender systems challenges and solutions survey. In: 2019 International Conference on Innovative Trends in Computer Engineering
(ITCE), pp. 149–155. IEEE (2019)
22. Zhao, L.-T., Wang, D.-S., Liang, F.-Y., Chen, J.: A recommendation system for effective
learning strategies: an integrated approach using context-dependent DEA. Expert Syst. Appl.
211, 1–12 (2023)
23. Santos, O.C., Boticario, J.G., Pérez-Marín, D.: Extending web-based educational systems with
personalised support through User Centred Designed recommendations along the e-learning
life cycle. Sci. Comput. Program. 88, 92–109 (2014)
24. Cai, D., Zhang, Y., Dai, B.: Learning path recommendation based on knowledge tracing model
and reinforcement learning. In: 2019 IEEE 5th International Conference on Computer and
Communications (ICCC), pp. 1881–1885. IEEE (2019)
25. Le, D.-L., Nguyen, A.-T., Nguyen, D.-T., Hunger, A.: Building learner profile in Adaptive eLearning Systems. In: 4th International Conference on e-Learning (ICEL 2009), pp. 294–304.
Curran Associates, New York (2009)
26. Jando, E., Meyliana, Hidayanto, A.N., Prabowo, H., Warnars, H.L.H.S., Sasmoko: personalized e-learning model: a systematic literature review. In: 2017 International Conference on
Information Management and Technology (ICIMTech), pp. 238–243, IEEE, (2017)
A Social Profile-Based Recommendation Architecture
343
27. Yasar, Y., Kalfa, S.M.: The effects of social, cultural and education dynamics on the
education-training environment in architectural education; the case study of the department
of architecture, KTU. Procedia. Soc. Behav. Sci. 182, 408–413 (2015)
28. Farseev, A., Akbari, M., Samborskii, I., Chua, T.-S.: 360z user profiling: past, future, and
applications. ACM SIGWEB Newslett. 2016(4), 1–11 (2016)
29. Buraya, K., Farseev, A., Filchenkov, A., Chua, T.-S.: Towards user personality profiling from
multiple social networks. In: Proceedings of the Thirty-First AAAI Conference on Artificial
Intelligence (AAAI-17), pp. 4909–4910 (2017)
30. On-at, S., Quirin, A., Péninou, A., Baptiste-Jessel, N., Canut, M-F., Sèdes, F.: Taking into
account the evolution of users social profile: experiments on Twitter and some learned lessons.
In: 2016 IEEE Tenth International Conference on Research Challenges in Information Science
(RCIS), pp. 1–12. IEEE (2016)
31. Ibrahim, T.S., Saleh, A.I., Elgaml, N., Abdelsalam, M.M.: A fog based recommendation
system for promoting the performance of E-learning environments. Comput. Electr. Eng. 87,
1–29 (2020)
32. Xiao, G.: A personalized learning path for French study in colleges based on a big data
knowledge map. Sci. Program. 2023, 1–10 (2023)
33. Dahman, M.R., Dahman, S.: Decision support model to help language teachers grouping adult
learners in a classroom. Educ. Inf. Technol. 25, 4329–4352 (2020)
34. Birjali, M., Beni-Hssane, A., Erritalim, M.: A novel adaptive e-learning model based on Big
Data by using competence-based knowledge and social learner activities. Appl. Soft Comput.
69, 14–32 (2018)
A Comparison of Student Engagement Across
Three Teaching Modalities in an Introductory
Statistics Course
Sher B. Chhetri1(B) , Mario Toussaint2 , Nonhle C. Mdziniso3 ,
and Rebecca A. Hillman4
1 Division of Science, Mathematics and Engineering, University of South Carolina Sumter, 200
Miller Road, Sumter, SC 29150, USA
schhetri@mailbox.sc.edu
2 Department of Mathematics, P. K. Yonge DRS, University of Florida, Gainesville, FL, USA
mtoussaint@pky.ufl.edu
3 Department of Applied Statistics and Data Analytics, Rochester Institute of Technology,
One Lomb Memorial Drive, Rochester, NY 14623, USA
ncmsma@rit.edu
4 Division of Science, Mathematics and Engineering, University of South Carolina Sumter, 200
Miller Road, Sumter, SC 29150, USA
hillmanr@uscsumter.edu
Abstract. Most educators agree there is a strong positive correlation between student engagement and student success. However, there is a dearth of scholarship on
the teaching modalities that best foster engagement. This study presents a comparison of students’ behavioral engagement in an introductory statistics course
taught with three distinct teaching modalities: traditional face-to-face, targeted
flipped, and fully online. The study found that the fully online group had statistically significantly lower levels of student engagement than the students in the
face-to-face group and the targeted flipped group, which had comparable levels
of engagement. Further, the results indicated a large effect in the successful completion of out-of-class assignments for both the face-to-face and targeted flipped
sections over the fully online section, and that a correlation exists between course
delivery methods and student engagement.
Keywords: Blended learning · Flipped learning · Distance learning · Traditional
instruction · Student success · Effect size
1 Introduction
Introductory statistics is one of the required courses to complete many degree programs
in most tertiary institutions. A substantial number of students view it as one of the
most challenging courses they must complete to satisfy the requirements for graduation. Consequently, many students fail to engage with the course content because they
view statistics as a subject completely unrelated to their career goals and everyday life.
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2024
K. Arai (Ed.): SAI 2024, LNNS 1017, pp. 344–357, 2024.
https://doi.org/10.1007/978-3-031-62277-9_21
A Comparison of Student Engagement Across Three Teaching Modalities
345
The literature suggests a strong association between student engagement and successful
completion of the course. In addition to contributing toward students’ success rates,
engagement leads to improved retention and student satisfaction, Lei et al. [18]. Several studies have been conducted to investigate the effects of teaching modalities on
students’ learning outcomes, Carrol and Burke [3], Guardiola et al. [15] and Verde and
Valero [31]. However, few studies offer a comparison of different teaching methodologies and their effect on student engagement. To address the latter, this study aims to
investigate which of the three teaching modalities: face-to-face, fully online, or targeted
flipped, best promoted student engagement in an introductory statistics class.
1.1 Literature Review
Definition of Student Engagement. The term engagement is widely used in education. However, there is a dearth of scholarship on the matter and the studies that address
the importance of engagement in the classroom do not provide a practical definition of
the term nor the best ways to observe it. Many educators, and particularly administrators equate engagement with students’ interests. Engagement can be defined as active
involvement, Garett [12]. However, measuring active involvement can be tricky. Just
mere observation may not yield an accurate measurement of engagement. While many
studies have been published on the importance of engagement, there is no consensus on
the definition of student engagement in literature. Student engagement is the first step in
the learning process, Shulman [25]. Engagement means commitment, Silver and Perini
[26]. Students who are engaged with the material will perform the actions necessary to
be successful in the course.
The literature identifies three types of engagement: behavioral, emotional, and cognitive, Fredricks et al. [10]. As mentioned in Fredricks et al. [10], behavioral engagement is based on participation, which includes involvement in academic and social or
extracurricular activities. Emotional engagement is centered around positive and negative interactions with teachers, classmates, academics, and the school. However, cognitive engagement includes thoughtfulness and willingness to exert the effort necessary
to comprehend complex ideas and master difficult skills, Fredricks et al. [10]. Different
teaching strategies produce different effects on those three components of engagement.
Teacher effectiveness and teacher characteristics as well as students’ beliefs about their
own abilities influence emotional engagement. Student attitudes, motivation, and selfefficacy influence cognitive engagement. However, all these constructs have an impact
on students’ behavioral engagement.
Many educators start the discussion about promoting engagement by talking about
best practices without first determining what they want to encourage the students to
do. Some important components of engagement include alertness, eye contact, active
listening, and note taking among others. However, while these outward signs of engagement are necessary for learning, some researchers argue that they may not be sufficient,
Shulman [25], Silver and Perini [26] and Garrett [12]. They contend that engagement
involves completing assignments, discussing the material, and active participation. In
an online setting, engagement may include student participation in online discussion
forums. Engagement involves devoting time and effort to a series of tasks. As a result,
346
S. B. Chhetri et al.
there is a link between engagement and course grades. Engagement is the ingredient that
differentiates deep learning from superficial learning.
There is a tendency in many educational circles to equate engagement with participation. Engagement goes beyond participation in the classroom. It involves active participation outside the classroom by developing new work habits that help with long-term
retention of the material. Shulman [25] states that learning must start with engagement,
which in turn leads to knowledge and understanding. As noted previously, it is difficult to
define engagement. It might be appropriate to enumerate signs of disengagement. They
may include excessive absences or tardiness, sleeping in class, talking about and reading
materials unrelated to the course, non-participation in class discussions, not completing
assignments, etc. In contrast, engaged students prepare for class, ask questions, review
the material, collaborate and discuss the material, solve problems, and complete assignments. As the detection of genuine engagement is nearly impossible, this study focuses
on behavioral engagement and defines it in terms of students putting forth the time and
effort necessary to complete assignments that have a moderate impact on their overall
grades in the course. The authors seek to ascertain which of three teaching methods
face-to-face (F2F), targeted flipped (TF), or fully online (FO) better promoted student
engagement in an introductory statistics course. Discerning which teaching method best
fosters student engagement is important because increased engagement leads to deeper
learning which in turn leads to successful outcomes, Fredericks et al. [10] and Marzano
[21].
Relationship Between Engagement and Learning Outcomes. It is quite challenging to define engagement in the mathematics classroom. It is even more challenging to
uncover the relationship between student engagement and student success. Most educators believe there is a direct link between student engagement and successful learning
outcomes, Skinner et al. [27], Connell and Wellborn [6], Connell et al. [7], Marks [20].
However, Lawton and Taylor [17] found no significant correlation between students’
perceptions of engagement and learning outcomes. Mere participation in an activity
does not guarantee cognitive engagement and improvement of students’ understanding
of the underlying statistical concepts, Pfaff and Weinberg [23]. Supplementary activities outside of the classroom such as the successful completion of homework and quiz
assignments may be necessary to aid students in achieving a conceptual understanding
of the concepts.
The greater the confidence and interest, the greater the engagement, and the greater
the likelihood of a successful outcome. Engaged learning is similar to active learning
where students must complete a series of tasks and reflect on the tasks they are completing
before, during, and after completion of the tasks. Some studies suggest that in-class
activities and projects intended to get students actively involved with data collection and
analysis improve student scores on exams, Froelich et al. [11], Carnell [2]. Conversely,
Pfaff and Winberg [23] found that hands-on activities did not help the students gain a
solid understanding of the statistical concepts. However, it is worth noting that while
the activities did not increase the students’ understanding of the concepts, the students
reported feeling that the activities engaged them.
A Comparison of Student Engagement Across Three Teaching Modalities
347
1.2 Site of the Study and Course
This research was conducted at one of the public universities in the United States. The
course used in this study was an undergraduate statistics course, covering content in
mean, median, standard deviation, range, pie and bar charts, basic data manipulation,
sampling procedures, probability, uniform, binomial, normal and sample distribution,
inference, hypothesis testing, one-sided and two-sided t-test, confidence interval, p-value
and its’ interpretation, basic regression, correlation coefficient, and standard error. The
faculty offered three different types of instructional design for this course; a face-toface class taught with the traditional lecture method (F2F), a fully online class taught
asynchronously (FO), and a novel teaching approach designed by some faculty dubbed
a targeted flipped class taught in a hybrid setting (TF).
For each teaching method, the instructors used the same textbook and the same
online learning management system. In each section, the faculty used the same forms of
assessment and designed very consistent syllabi. Each instructor assessed the students’
mastery of the learning outcomes through the following tools: three exams worth 60% of
the students’ grade, twelve quizzes worth 20% of their grade, fourteen weekly homework
assignments worth 5%, and a cumulative final exam worth 15% of the course grade. Every
assessment problem was assigned randomly from the online platform Limes; therefore
each student had the same type of question and level of difficulty, Limes [19].
1.3 Participants
The student population in the data comprised individuals from various backgrounds:
about half of the students were Caucasian, 17% were African American, 19% were Hispanic, 5% were Asian, and 0.3% were Native American. Another 3% were international
students from many countries around the world. The participants in this study were 242
undergraduate students enrolled in three different methods of teaching an introductory
statistics course, of which 84 were enrolled in face-to-face classes, 95 were enrolled in
fully online classes, and 63 students were enrolled in targeted flipped classes. In terms
of classification, the face-to-face class comprised 33.3% freshmen, 40% sophomores,
13.3% juniors, and no seniors; 13.4% of the respondents were non-degree seeking or did
not specify their status on the questionnaire. The fully online class had 50% freshmen,
25% sophomores, no juniors, and 25% seniors. The targeted flipped class included 12.5%
freshmen, 29.1% sophomores, 39.6% juniors, and 14.6% seniors; 4.2% of the students
in the targeted flipped class were non-degree seeking or did not specify their status. For
the statistical analyses, students who did not complete the course were omitted from this
study. This included 4 students who were enrolled in the face-to-face class, 13 students
in the fully online class, and 6 students in the targeted flipped class.
The traditional lecture method (F2F) class met twice a week, once for the lecture
and once for practice in the lab and to complete the quizzes. All quizzes and exams
were proctored. On the other hand, for the fully online (FO) class, the lecture videos and
notes were posted on the Learning Management System (LMS), Blackboard, and the
quizzes and exams were not proctored. For the target flipped class (TF), the instructor
posted lecture notes and videos online, and the instructor held one lab session (optional
for students) once a week throughout the semester. The quizzes were not proctored but
348
S. B. Chhetri et al.
students were required to complete the proctored exams in the lab. If a student did not do
well on a quiz, they were recommended to come to the lab to ask questions and receive
additional instruction, prior to retaking the quiz to improve their grade.
2 Materials and Methods
2.1 Research Questions
In this paper, we define student engagement in terms of behavior outside the classroom,
that each separately had a modest impact on the student’s overall grade. The researchers
used twelve quizzes and fourteen homework assignments to determine which of the
three teaching methods better promoted student engagement in the introductory statistics
course. Successful completion is defined as a student scoring at least a 75% average per
assessment.
The following questions will guide the study.
1) Does student engagement vary based on teaching method?
a) Does successful completion of homework vary based on teaching method?
b) Does successful completion of quizzes vary based on teaching method?
2) Is there an association between engagement and final exam grade in each teaching
methodology?
2.2 Methodology
To answer the first research question, the following tests were performed: single-factor
analysis of variance (ANOVA), Chi-Square test for independence, and Cohen’s h effect
size. The single-factor ANOVA was used to determine whether there was a statistically significant difference in performance on students’ quizzes and homework grades
among the different course delivery methods, Snedecor and Cochran [28]. We present
the ANOVA results in Tables 4 and 5.
Additionally, we performed the Chi-Square test to determine if student engagement
in homework and classification of teaching methods are independent. Similarly, we tested
if student engagement in quizzes and the method of teaching are independent using the
Chi-Square test. The following hypotheses were used to perform the Chi-square test for
independence:
H0 : Student engagement in homework/quiz and teaching method classification are
independent.
Versus,
Ha : Student engagement in homework/quiz and teaching method classification are
dependent.
The test statistic for independence is given by
2
nij − Êij
2
χc =
Êij
RC
where E ij = in j , such that Ri = total for row i, Cj = total for column j, nij = observed
value of two nominal variables and n is the sample size. Note that the rejection region
A Comparison of Student Engagement Across Three Teaching Modalities
349
2
2
in this case is χc2 > χ
α with χα has (r − 1)(c − 1) degrees of freedom, and p-value is
2
2
given by P χc > χα . The results for Chi-square test of independence are discussed in
the statistical analysis and results section.
Finally, we use Cohen’s h effect size to address research question 1. The Cohen’s h
effect size was used to measure the effect of the different teaching methods on student
engagement. It provides a measure of the size of the effect between two proportions, pi
and pj , Cohen [5]. The effect size between pi and pj is calculated using the following
formula,
√
√
h pi −pj = 2∗arcsin pi −2∗arcsin pj
We made pairwise comparisons using Cohen’s h effect size based on the proportion
of students from each teaching method who successfully completed the assignment.
The following hypotheses were used:
H0 : There is no effect between the teaching method and student engagement, pi −pj =
0.
Versus,
Ha : There exists an effect between the teaching method and student engagement,
pi − pj = 0.
The results from the statistical analysis are presented in Table 6. The following
guidelines were used for the size of the effect in this study:
• h ≈ 0.2 is a “small” difference (effect),
• h ≈ 0.5 is a “medium” difference, and
• h ≈ 0.8 is a “large” difference, Cohen [5] and Ellis [9].
To determine whether there is an association between engagement and the final exam
grade, we calculated Pearson’s correlation coefficient, r. This addresses research question 2. The correlation coefficient between average homework scores and final exams,
as well as between average quiz scores and final exams, for each teaching method were
calculated and the results are presented in Table 7. In addition, we considered the proportion of students who were unsuccessful on the final exam who were also unsuccessful
on the homework and quizzes. These results are presented in Table 8.
3 Statistical Analysis and Results
For initial analysis, Tables 1 and 2 below provide descriptive statistics based on homework grades and quiz grades for the three different teaching methods. Table 1 shows
that the TF group had the highest mean homework grade. The F2F group was a close
second, followed by the fully online group with a significantly lower mean homework
grade. However, Table 2 shows the F2F group had the highest mean quiz grade followed
by the TF group, and the FO group was third.
Figure 1 below presents Box plots that show the distribution of grades for homework
assignments and quizzes. The green vertical lines represent the respective means, and
the red vertical lines are medians. A student was said to have completed their homework
or quizzes with a passing grade if they received at least 75% average on the assessment.
The proportion of students from each teaching method who completed the assignments
with a passing grade is shown in Table 3.
350
S. B. Chhetri et al.
Table 1. Descriptive statistics for homework grades
F2F
FO
TF
Sample size (n)
84
95
63
Mean
68.64
20.01
72.82
Standard Deviation
25.80
23.04
25.79
Standard Error
2.81
2.36
3.25
Table 2. Descriptive statistics for quiz grades
F2F
FO
TF
Sample size (n)
84
95
63
Mean
95.29
86.71
94.39
Standard Deviation
14.12
18.82
17.23
Standard Error
1.54
1.93
2.17
Fig. 1. Boxplots for homework and quiz grades
Table 3. Proportion of students who completed assignments with average ≥ 75%
F2F
FO
TF
HW
0.5238
0.0421
0.5238
Quizzes
0.9167
0.7158
0.9206
A Comparison of Student Engagement Across Three Teaching Modalities
351
3.1 Anova
For homework grades, we obtain F(2,239) = 124.05 with p < 0.001. Similarly for
quizzes, we have F(2,239) = 6.83 with p = 0.001. The results show that there are
significant differences in grades between the three teaching methods for homework and
quizzes. T-tests followed by Bonferroni corrections were performed. The statistics are
presented in Tables 4 and 5.
Table 4. Post-hoc t-tests for homework grades
F2F
F2F
FO
FO
TF
*
*
*
TF
*
* p < 0.05
Table 5. Post-hoc t-tests for quiz grades
F2F
F2F
FO
TF
FO
TF
*
*
*
*
* p < 0.05
For homework grades, the post hoc t-tests revealed that the face-to-face (F2F) performed statistically significantly better than the fully online group (FO) at the 0.05
significance level, (p = 0.001). Similarly, the targeted flipped (TF) performed statistically significantly better than the fully online group (FO) at the 0.05 significance level,
(p = 0.001). But, for face-to-face (F2F) and targeted flipped (TF) groups, we do not have
enough evidence to reject the assumption of the equality of the variances for homework
grades. Similarly, for quiz grades, the post hoc t-tests revealed that the face-to-face (F2F)
performed statistically significantly better than the fully online group (FO) at the 0.05
significance level, (p = 0.001). Similarly, the targeted flipped (TF) performed statistically significantly better than the fully online group (FO) at the 0.05 significance level,
(p = 0.010). But, for face-to-face (F2F) and targeted flipped (TF) groups, we do not
have enough evidence to reject the assumption of the equality of the variances for quiz
grades.
3.2 Chi Square Test
The Chi-square test was used to determine whether student engagement in homework
is independent of the teaching method. Similarly, we performed the Chi-square test
352
S. B. Chhetri et al.
to determine whether student engagement in quiz grades is independent of teaching
method. The p-value for homework versus teaching method is less than 0.001, and the
p-value for quiz versus teaching method is less than 0.001. The p-values in both cases
are less than 0.05, indicating that teaching methods influence student engagement in
the course. In other words, there is a relationship between course delivery methods and
student engagement.
3.3 Cohen’s h Effect Size
We applied the guidelines for Cohen’s h effect size mentioned in the methodology section
which is based on Bakker et al. [1], factors to consider when interpreting effect sizes. The
results for Cohen’s h effect size are given in Table 6. Our results show that there is a large
effect between student engagement in the face-to-face classroom and the fully online
classroom, based on homework. A large effect is also observed between students from the
fully online classroom and targeted flipped classroom, based on homework. On the other
hand, analysis based on quizzes shows that there is a medium effect between student
engagement and the teaching method when students from the face-to-face classroom
were compared with students from the fully online classroom. This is also the case when
students from the fully online classroom are compared with students from the targeted
flipped classroom. However, there is no effect between student engagement and the
teaching method when comparing face-to-face and targeted flipped classrooms in both
homework and quizzes.
Table 6. Cohen’s h effect size for pairwise comparisons
(pi , pj )
HW
Quizzes
(F2F, FO)
1.2051
0.5390
(TF, F2F)
0.000
0.0143
(TF, FO)
1.2051
0.5532
3.4 Pearson’s Correlation
Pearson’s correlation coefficient was used to determine whether there is an association
between engagement and the final exam grade. The correlation coefficient between the
average homework and final exam scores, and between the average quiz and final exam
scores were computed for each teaching method. Our results show a moderate correlation
between students’ performance on quizzes and final exams in all teaching methods. There
is also a moderate correlation between students’ performance in homework and the final
exam for students from the targeted flipped classroom.
Although there were no strong linear correlations between engagement and final
exam grades, we then considered the proportion of students who earned less than 75%
on the final exam, who also scored an average less than 75% on their homework or
A Comparison of Student Engagement Across Three Teaching Modalities
353
Table 7. Pearson’s correlation coefficients
F2F Final exam
FO Final exam
TF Final exam
Average HW
0.2146
0.1748
0.4645
Average Quizzes
0.4583
0.4726
0.4950
quizzes. These proportions are given in Table 8. The most interesting of these results
compared final exam score and homework completion in the fully online section. Every
student who scored less than 75% on the final exam had a homework average of less than
75% as well. This confirms what many faculty preach: “if you do well on homework
then you will be successful on the exam”. However, this result is not so clear in the
targeted flipped or face-to-face sections when pertaining to homework and is rejected in
our data when comparing quiz average and final exam score.
Table 8. Proportion who scored < 75% on their final exam and scored < 75% on homework or
quizzes
F2F
FO
TF
Final < 75 with HW < 75
0.5909
1
0.7727
Final < 75 with Quizzes < 75
0.1818
0.5
0.1818
4 Discussion
This study aimed to investigate whether the teaching method used in introductory statistics affected student engagement in the course. A single-factor ANOVA, Chi-Square
test for independence, and Cohen’s h effect size were used to answer our first research
question, does student engagement vary based on teaching method? The results of the
Chi-Square test provided the simple answer of yes, the teaching method does influence
student engagement. Through further examination, the single-factor ANOVA found there
was a statistically significant difference between the mean score on both homework and
quizzes when comparing the face-to-face and the online groups, as well as a statistically
significant difference in comparing the targeted flipped and online classes in both homework and quizzes. However, there was not a statistically significant difference in student
engagement between the face-to-face and the targeted flipped group in either quizzes or
homework assignments. This suggests that students in both the face-to-face and targeted
flipped classes were more engaged with the course than the students in the fully online
group.
To ascertain which group had the strongest level of engagement, Cohen’s h effect
size compared the proportions of students who were successful (scoring an average of
at least 75%) among the three groups. The results indicated a large effect in the successful completion of homework for both the face-to-face and targeted flipped sections
354
S. B. Chhetri et al.
over the fully online section. In terms of successful completion of quizzes, a moderate
effect was observed in both the face-to-face and targeted flipped sections over the fully
online section. No effect was found between the face-to-face section and targeted flipped
section in either form of engagement. Both strong and medium effects confirm that student engagement was higher in both the face-to-face and the targeted flipped teaching
modality compared to the fully online model, although there is a stronger effect regarding
homework than quiz scores. This difference adds weight to the results observed from the
Chi-square and single-factor ANOVA tests, especially since it pertains to the difference
in the proportion of students whose engagement in the course is being affected by the
type of teaching modality, Ellis [9].
While there are significant advantages of online learning, limitations associated with
this learning approach may be the reason students from the fully online class were less
engaged compared to the traditional face-to-face and targeted flipped classes. Online
classes place a lot of responsibility on the student and require them to have self-motivation
and self-discipline for them to be engaged in class activities independently, ION Professional eLearning Program [16]. Thus, online classes are not appropriate for students
who have poor time management skills and are not self-motivated to complete their
assignments in a timely manner. As mentioned by Goundar [14], students have no peer
pressure to compete and review class materials or assignments in an online class since
they do not have face-to-face interaction with their classmates. Thus, they are more
likely to have deficient performance, miss assignment deadlines, or withdraw from the
course. Students in an online class are more likely to procrastinate and less likely to use
resources useful in completing assignments, such as class discussions, Goundar [14].
We recommend that instructors address these limitations throughout the semester for
students to be more engaged in an online class.
Neumann et al. [22] found the targeted flipped model provided students the opportunity to experience deeper learning, see the relevance of studying statistics, increase
interest in the subject, and improve motivation and engagement in the subject with reallife data, when compared to the traditional instruction model. Also, the targeted flip
model provided hands-on activities, cooperative learning, and student-centered assignments that improved students’ self-concept and interest in the subject and improved
student engagement Clark [4].
Pearson’s correlation coefficient was used to answer the second research question,
is there an association between engagement and final exam grade in each teaching
methodology? Our results found a moderate correlation between students’ performance
on quizzes and final exams in all teaching methods. There was also a moderate correlation between students’ performance in homework and the final exam for students from
the targeted flipped classroom. However, there was only a small correlation between
students’ performance in homework and the final exam for both the face-to-face and
fully online sections. These results are contrary to the findings of Lawton and Taylor [17] who found no relationship between end-of-semester grades and engagement
rating by students. Gopal et al. [13] asserted that students’ engagement in statistics is
positively correlated to attitudes and self-efficacy. The results of this study contradict
those of Strayer [29], who found that college students were less satisfied in introductory
statistics courses using the flipped model than the traditional method.
A Comparison of Student Engagement Across Three Teaching Modalities
355
5 Limitations and Future Directions
Due to logistical constraints in educational settings, the sampling method used for data
collection was not random, thus limiting the generalizability of this study. In addition,
each teaching methodology was utilized by a different instructor. Consequently, differences in teacher characteristics may have affected the results. Further investigations
using a randomized design while controlling for teacher effectiveness may be necessary
to confirm the findings. As we see in Gundlach et al. [8], it might be interesting to
compare performance and perceptions by having the same instructor teaching the same
course using the three different delivery methods. Additionally, the data was collected at
a single institution in the Southeastern United States during one semester. Further investigations involving multiple institutions located in different geographic regions over
multiple semesters may be warranted to validate the results of the current study.
Silver and Perini [26] defined engagement as a commitment to learning. This commitment ensures that students will take the actions necessary for learning to occur. The
results suggest that the targeted flipped classroom modality and traditional method of
teaching elicited a stronger commitment to learning from students. However, the students’ commitment to learning in the fully online modality was significantly weaker.
This research gives educators insight into three different modalities of teaching an introductory statistics course, especially partially flipped courses, and an incentive to redesign
them for improved student learning. Redesign is best conceptualized as an iterative process and requires instructors to work carefully and intentionally toward improving their
approach to instruction over time, Ramsden [24], Toohey [30] and Gundlach et al. [8].
In addition, it is recommended that instructors put more emphasis on the importance
of assignments and assessments for successful completion of the course. Instructors must
compensate for their lack of physical presence with more flexible virtual office hours to
accommodate their online students. Institutions should also provide resources for online
students to get help with formative assignments, such as tutoring services, both virtual
and face-to-face, with more flexible hours of operation to help students be more engaged
in the course.
6 Conclusion
In this paper, we used a data set collected from three different methods (face-to-face
(F2F), targeted flipped (TF), and fully online (FO)) of teaching an introductory statistics
course. This work adds to the discourse on student engagement in introductory statistics
as it has become increasingly important in many students’ academic and professional
careers. To check if student engagement varies based on teaching method, we compare
the successful completion rates of homework and quizzes. The t-test with Bonferroni
corrections revealed that the F2F and TF groups performed better than the FO group
for both homework and quizzes. It is important to note that we were not able to reject
the equality of variances in homework or quiz grades for F2F and TF groups. The Chi
Square test and Cohen’s h effect size confirmed that there is a relationship between course
delivery methods and student engagement. This is for quizzes and teaching methods as
well as homework and teaching methods. We also investigated whether the final exam
356
S. B. Chhetri et al.
grade was correlated with the average homework and quiz grades for each teaching
method. For all 3 teaching modalities, the correlation coefficient indicates that quiz
averages and final exam grade are moderately correlated. However, it is less evident
when comparing homework to final exam grade in the F2F or FO section.
This study confirms that the choice of teaching modality is an important one, and
students as well as other stakeholders should make that decision with care. Understanding
which teaching modality best promotes engagement will help different stakeholders plan
for the offering of various teaching modalities, as well as help advise students regarding
the teaching modality that fits their learning style. The results suggest that students in the
targeted flipped group and the traditional face-to-face group had a stronger commitment
to learning than the fully online group. It underscores the need for improvement in the
design and implementation of distance learning courses.
Acknowledgments. The authors are thankful to the editor and the anonymous reviewers for their
detailed reviews and insightful comments.
Funding. The authors did not receive funding to work on this project.
References
1. Bakker, A., English, L., et al.: Beyond small, medium, or large: points of consideration when
interpreting effect sizes. Educ. Stud. Math. 102, 1–8 (2019)
2. Carnell, L.J.: The effect of a student-designed data collection project on attitudes towards
statistics. J. Stat. Educ. 16, 1–16 (2008)
3. Carrol, N.E., Molly, B.: Learning effectiveness using different teaching modalities. Am. J.
Bus. Educ. 3(12), 65–76 (2010)
4. Clark, K.R.: The effects of the flipped model of instruction on student engagement and
performance in the secondary mathematics classroom. J. Educ. Online 12(1), 91–115 (2015)
5. Cohen, J.: Statistical Power Analysis for the Behavioral Sciences, 2nd edn (1988)
6. Connell, J.P., Wellborn, J.G.: Competence, autonomy, and relatedness: a motivational analysis
of self-system processes. In: Gunnar, M., Sroufe, L.A. (eds.) Minnesota Symposium on Child
Psychology, vol. 23. University of Chicago Press, Chicago (1991)
7. Connell, J.P., Spencer, M.B., Aber, J.L.: Educational risk and resilience in African American
youth: context, self, action, and outcomes in school. Child Dev. 65, 493–506 (1994)
8. Gundlach, E., Richards, K.A., Nelson, D., Levesque-Bristol, C.: A comparison of student
attitudes, statistical reasoning, performance, and perceptions for web-augmented traditional,
fully online, and flipped sections of a statistical literacy class. J. Stat. Educ. 23, 1 (2015)
9. Ellis, P.D.: The Essential Guide to Effect Sizes: Statistical Power, Meta-Analysis and the
Interpretation of Research Results. Cambridge University Press, Cambridge (2010)
10. Fredricks, J.A., Blumenfeld, P.C., Paris, A.H.: School engagement: potential of the concept,
state of the evidence. Rev. Educ. Res. 74, 59–109 (2004)
11. Froelich, A.G., Stephenson, W.R., Duckworth, W.M.: Assessment of materials for engaging
students in statistical discovery. J. Stat. Educ. 16, 1–29 (2008)
12. Garrett, C.: Defining, detecting, and promoting student engagement in college learning
environments. Transform. Dialogues Teach. Learn. J. 5(2), 1 (2011)
13. Gopal, K., Salim, N.R., Ayub, A.F.M.: Influence of self-efficacy and attitudes towards statistics
on undergraduates’ statistics engagement in a Malaysian public university. J. Phys. Conf. Ser.
1132(1), 012042 (2018)
A Comparison of Student Engagement Across Three Teaching Modalities
357
14. Goundar, S. (ed.) Massive Open Online Courses - Current Practice and Future Trends.
IntechOpen (2023). https://doi.org/10.5772/intechopen.1001518
15. Guardiola, J.H., Duran-Hutchings, N., Elsalloukh, H.: Are Statistics Labs Worth the Effort?—
Comparison of Introductory Statistics Courses Using Different Teaching Methods 3(1), 5
(2010)
16. ION Professional eLearning Program: Strengths and Weaknesses of Online Learning, University of Illinois Springfield (2023). https://www.uis.edu/ion/resources/tutorials/overview/
strengths-weaknesses
17. Lawton, S., Taylor, L.: Student perceptions of engagement in an introductory statistics course.
J. Stat. Educ. 28(1), 45–55 (2020)
18. Lei, H., Cui, Y., Zhou, W.: Relationships between student engagement and academic achievement: a meta-analysis. Soc. Behav. Personal. Int. J.Behav. Personal. Int. J. 46, 517–528
(2018)
19. Limes Learning Management System (n.d.). www.limes.one
20. Marks, H.M.: Student engagement in instructional activity: patterns in the elementary, middle,
and high school years. Am. Educ. Res. J. 37, 153–184 (2000)
21. Marzano, R. J.: The art and science of teaching: a comprehensive framework for effective
instruction (2007)
22. Neumann, D.L., Hood, M., Neumann, M.M.: Using real-life data when teaching statistics:
student perceptions of this strategy in an introductory statistics course. Stat. Educ. Res. J. 12,
59–70 (2013)
23. Pfaff, T.J., Weinberg, A.: Do hands-on activities increase student understanding? A case study.
J. Stat. Educ. 17, 1–34 (2009)
24. Ramsden, P.: Learning to Teach in Higher Education. Routledge, New York (1992)
25. Shulman, L.S.: Making differences: A table of learning. Change Mag. High. Learn. 34(6),
36–44 (2002)
26. Silver, H.F., Perini, M.J.: The eight Cs of engagement: How learning styles and instructional
design increase student commitment to learning. On Excellence Teaching, Solution Tree
Press, 319–344 (2010)
27. Skinner, E.A., Wellborn, J.G., Connell, J.P.: What it takes to do well in school and whether
I’ve got it: the role of perceived control in children’s engagement and school achievement. J.
Educ. Psychol. 82, 22–32 (1990)
28. Snedecor, G.W. Cochran, W.G.: Statistical Methods, Eighth Edition. Iowa State University
Press (1989)
29. Strayer, J.: The effects of the classroom flip on the learning environment: A comparison of
learning activity in a traditional classroom and a flip classroom that used an intelligent tutoring
system (Doctoral dissertation, The Ohio State University) (2007)
30. Toohey, S.: Designing Courses for Higher Education. The Society for Research into Higher
Education and Open University Press, Buckingham (1999)
31. Verde, A., Valero, J.M.: Teaching and learning modalities in higher education during the
pandemic: responses to Coronavirus Disease 2019 from Spain. Front. Psychol. 12, 648592
(2021)
Impact Factor Game Scoring Model for Positive
Emotion Game Design
V. Sithira Vadivel(B)
School of Information and Physical Sciences, University of Newcastle, Newcastle, Australia
vsithira.vadivel@newcastle.edu.au
Abstract. Positive emotional games can have a significant impact on gamers
because they not only provide entertainment but also have the potential to promote
learning and evoke positive emotions. This article explores the effects of positive
emotional games compared to harmful emotional games on gamers. The research
method involved analyzing current issues with game design and proposing new
perspectives to improve learning opportunities and encourage positive learning
experiences. The paper cites real-life examples of positive emotional games and
their impact. The analysis of positive emotional games led to the suggestion of a
Positive Game Scoring Model for future educational emotional games that have
value. The framework proposes a weighted approach with criteria for both negative
and positive scoring. It recommends lower scores for violent or negative emotional
games and higher scores for emotionally positive games. Game designers are
encouraged to consider the positive factors that inspire happiness and joy, which
can bring about positive change among gamers.
Keywords: Interactive games · Positive emotional games · Positive
gamification · Cultural activity metaphors
1 Introduction
What do emotions have to do with games? Studies of emotional games (emotions elicited
by playing games) tend to emphasize game mechanics and user experience. Learning
from neuroscience, however, tells us that the combination of concentration and surges of
neurotransmitters when playing games or engaging in activities helps strengthen neural
circuits, providing genuine brain training. The amygdala serves as the major processing
center for emotions in the brain. Studies on how children store sensory information
through input from the amygdala indicate that their emotional state greatly influences
this process upon receiving information. When the amygdala is in a secure state and
emotions are positive, the data is then passed on to the brain’s reflective memory storage
and thinking networks [12].
Emotional factors can positively affect a child’s play. The six universal innate emotions are anger, fear, surprise, disgust, sadness, and happiness [1]. These emotions are
said to be primary to humans because they are instinctive and often transient. They differ
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2024
K. Arai (Ed.): SAI 2024, LNNS 1017, pp. 358–364, 2024.
https://doi.org/10.1007/978-3-031-62277-9_22
Impact Factor Game Scoring Model for Positive Emotion Game Design
359
from secondary emotions in their appraisal and behavioural response and are incredibly adaptive in dealing with crucial life tasks [2]. Emotional game consideration often
illustrates the need for an emotional presence in the performances and functions where
users are enticed into the game’s setting through means such as music, environmental
conditions and the activity involved. Gaming is powerful: it trains the mind, stimulates
ideas, sources creative thinking, and directly affects emotions.
2 Methodology
This study utilizes a mixed research approach to evaluate the harmful effects of violent
game designs on gamers and the potential benefits of positive emotional game design.
This evaluation is conducted through a literature review. The analysis from this study
leads to the development of a game scoring model, which aims to enhance impactful
game design in the future. Furthermore, the study discusses the practicality and relevance
of different levels of scoring criteria (positive and negative) within the proposed model.
2.1 Current Problem
The current practice of emotional game design has arguably evolved in the wrong direction with the creation of emotionally dangerous games. While some games have educational content, many popular games emphasise negative themes and promote violence
[4]. Research indicates that the relationship between violent games and violent behaviour
is complex and involves many factors. Still, there is evidence that violent games can
lead to a desensitisation of violence, aggressive behaviour, social isolation, and even
alienation [3]. The addictive nature of games exacerbates the impact. Psychologists and
psychiatrists now view video game addiction with great seriousness. They have recently
contemplated including it as a diagnosable psychiatric disorder in the fifth edition of the
Diagnostic and Statistical Manual for Mental Disorders (DSM). Currently, it has been
placed in the manual’s appendix to inspire additional research on the subject [13].
The contagious effect is a well-known observation that aggression leads to more
aggression. Considering that playing violent video games increases aggression, it is
possible that this heightened aggression also affects individuals connected to the player
[15]. Immoral and violent game perspectives could lead to destructive outcomes [11]
and aggressive behavior [16]. Emotional video game design should not infuse violence.
Analyses of egocentric social networking have revealed that playing violent video games
is linked to a rise in aggression, which subsequently extends to connected individuals.
Even participants who do not engage in playing violent video games themselves have
reported higher levels of aggression when their social network includes individuals who
do play such games [14].
In recent years, however, game design preferences have seen gamers presented with
challenging tasks such as completing missions in a war zone (gunning down enemies) and
executing extreme attacks involving bloodshed and destruction in urban settings. These
violent game designers have apparently given minimal thought to the possible effects
on the gamers. Gamers are enticed to these challenges seeking the excitement the game
provides, but gamers need to comprehend the serious ramifications that could arise. These
360
V. S. Vadivel
games can stimulate significant harmful ideas which could translate to the real world. A
plausible next step for gamers is the desire to explore similarly challenging tasks in the
real world. The spike in violence among teenagers in recent years has affected many of us.
Technological advancements, such as using mixed and virtual reality with serious games,
could have more grave implications and adverse effects. Chaotic, violent games could
develop negative emotions in gamers and lead to undesirable outcomes. Technological
advancement must be embraced positively. Adherence to different perspectives and game
design approaches is necessary. We urge, for example, crime investigation gameplay
rather than crime gameplay.
Game Designers might instead pay attention to the emotional presence during gaming
that engages positive outcomes—facilitating exciting gameplay that enhances creativity
and critical thinking and that has meaning and a motive. More positive emotional games
are necessary to spur positivity among gamers. The design of emotional games can
do wonders for gamers with long-term benefits and sustainability. Experimenting with
elements related to such positive emotional games has many benefits for gamers [5].
Taking a different approach to game design means adopting perspectives that stimulate good habits and a non-violent mindset. Game designers need to bear in mind that
gamers learn from games, whether intended or not: games have a learning outcome even
though their purpose is entertainment. Encouraging harmonious and peaceful emotions
could also improve gamers’ experience. Emotional games, such as culturally oriented
game design, can foster peace, harmony, and unity and benefit both the individual and
society by promoting positive citizenry. A sustainable gaming industry could consider
developing harmonious games that lead to positive emotional attachments with a motive.
Game designs with harmonious guidelines and criteria are how to move forward with a
serious game perspective.
2.2 Positive Emotional Games
One aspect of positivity in games relates to cultural appropriateness as it includes elements of cooperation, respect and courtesy missing in more generic and often violent
action games. An emotional game using cultural activity metaphors was evaluated in
Brunei and Indonesian schools. A Science topic taught with a traditional/standard game
activity was compared with one taught using a Southeast Asian game activity [9]. The
results showed improved learning outcomes and experiences when appropriate, positive
cultural activity was incorporated into the learning.
Similarly, another study in Indonesian schools on the use of culturally themed festive metaphors in learning materials found that this led to improved learning outcomes
and experiences for students [8]. Students were more positively engaged and culturally
motivated during the cultural activity. Furthermore, these cultural metaphors not only
improve students’ level of interaction but also teach students about Asian cultural activities. The significant presence of cultural emotion states helps speed up digitalisation and
e-learning adoption to achieve digital transformation—positive results through students’
positive engagement in interactive learning and game design [10].
In virtual reality, young adults played low-body involvement such as the Fruit Ninja
video games. This study aimed to explore the ability of commercial virtual reality games
to induce positive emotions and decrease negative emotions and anxiety [7]. Even though
Impact Factor Game Scoring Model for Positive Emotion Game Design
361
this was for leisure, gamers also learned about different types of fruits. More studies are
needed on positive emotional games in immersive technologies.
3 Future Direction
New directives and guidelines for game design are essential for a sustainable and positive
game creation model with a motive—positive learning to benefit individual gamers and
our society. In particular, game designers need to consider the type, level and purpose
of the violence they build into games and the possible impact on gamers.
Setting up serious game best practice guidelines would help the industry head in the
right direction. One approach is to define good quality game design which has impactful,
positive emotions based on ranking and scores [6]. Setting the guidelines and criteria
for scoring systems through ranking impact factors would motivate the community of
game designers to adhere to strict guidelines and best practices. A scoring system with a
high ranking for positive impact factors would encourage the creation of more positive,
emotional games. In contrast, violent and crime content would lead to a negative ranking
and reduce the quality of harmful games. A scoring system with ranking impact factors
for positive emotional games could set the path right.
An impact factor game scoring system with six positive scoring criteria and three
negative scoring criteria is proposed. The nine criteria and their maximum scores are
listed in Table 1.
Table 1. Impact factor game scoring system version 1
Criteria
Points
Positive Scoring Criteria
Design Excellence
+10
Interactive Design
+10
Stimulates Positive Emotions (E.g., Happiness/Joy/Fun)
+10
Invoke Positive Creativity
+10
Consists of Motivational Elements (E.g., Motivational Metaphors)
+10
Negative Scoring Criteria
Invoke bloodshed actions/activity/themes (E.g., Crime Scenes, Fights, War)
−10%
Invoke violence (E.g., Fighting with weapons, endangering human lives/animals)
−10%
Invoke/stimulate Anger/Vengeance/Crime
−10%
The scoring system for game design awards a maximum of 50 points. A game that
evokes positive emotions is categorized as having 45 points or more classified as a
high-rank game. A game that involves violent actions falls into the 21 to 44-point range
classified as a mid-rank game, while a violent game (which is not recommended for
individuals under 18) earns a score below 20 classifieds as a low-rank game (see Fig. 1).
362
V. S. Vadivel
Fig. 1. Educational game classification version 1
3.1 Discussion
The Educational Game Classification version 1 serves as a framework for game designers
to promote positive emotions, community and social interaction, and general knowledge.
This framework assigns high scores to games that meet these criteria, specifically scoring
more than 45 points. It can also be expanded to include additional criteria that contribute
to positive scores. The main goal is to create games that are engaging and positive, while
minimizing violent content. Games with a score of less than 20 points are considered
low-ranking in terms of design and are identified as having a violent gaming factor. The
ranking system can be used as a guide to inform parents and gamers about violent games
and can also serve as an index for popular and recommended games compared to those
with a negative impact.
Other existing frameworks in the field of game design and education include the
Game value factors [17], which explore the relationships between different factors to
create a model for game-based learning design. Additionally, the EFM model for educational game design [18] describes the internal factors such as motivation, flow, effective
learning, and educational content that contribute to a game’s design. However, the Educational Game Classification model is unique in its focus on the impact of game design and
will further incorporate symbolic features and sub-criteria for positive design approaches
in the future.
3.2 Limitation
The suggested model for promoting positive and engaging gameplay was developed by
studying the negative impacts of violent video games. To evaluate its helpfulness and
practicality, the model must undergo thorough testing and improvement using various
prototypes at different levels. This testing should focus on students’ learning experiences
Impact Factor Game Scoring Model for Positive Emotion Game Design
363
and outcomes. Additionally, each positive and negative criterion can be expanded with
sub-criteria to address crucial areas of negative impact in game design. Each level of the
model can also be developed to incorporate violent actions in game development.
4 Conclusion
Game designers are encouraged to think about the positive factors that inspire happiness
and joy. These factors can help improve players’ emotions and have a positive impact
on society by promoting cooperation, non-violence, and cultural learning. There is a
chance to adopt a unique approach to attract, captivate, and thrill gamers, fostering
positive emotions. If the directions are correct, well-designed games can bring about
positive change, revolutionizing the gaming industry, gamers, and our world.
References
1. Ekman, P.: Universal facial expressions of emotions. Calif. Mental Health Res. Digest. 8,
151–158 (1970)
2. Ekman, P.: Expression and the nature of emotion. In: Scherer, K.E., Ekman, P. (eds.)
Approaches to Emotion, pp. 319–344. Lawrence Erlbaum Associates, Hillsdale (1984)
3. Funk, J.B., Baldacci, H.B., Pasold, T., Baumgardner, J.: Violence exposure in real-life, video
games, television, movies, and the internet: is there desensitization? J. Adolesc.Adolesc.
27(1), 23–39 (2004)
4. Gros, B.: Digital games in education: the design of games-based learning environments. J.
Res. Technol. Educ. 40(1), 23–38 (2007)
5. Lin, J.J., Mamykina, L., Lindtner, S., Delajoux, G., Strub, H.B.: Fish’n’Steps: encouraging
physical activity with an interactive computer game. In: Dourish, P., Friday, A. (eds.) UbiComp
2006: Ubiquitous Computing: 8th International Conference, UbiComp 2006 Orange County,
CA, USA, September 17–21, 2006 Proceedings 8, pp. 261–278. Springer Berlin Heidelberg
(2006). https://doi.org/10.1007/11853565_16
6. McKitrick, B., Rogerson, M., Gibbs, M., Nansen, B.: “What are you bringing to the table?”:
the something awful let’s play community as a serious leisure subculture. Game. Cult. 18(3),
402–421 (2023)
7. Pallavicini, F., Pepe, A.: Virtual reality games and the role of body involvement in enhancing
positive emotions and decreasing anxiety: within-subjects pilot study. JMIR Serious Game
8(2), e15635 (2020)
8. Sithira Vadivel, V., Song, I., Bhati, A.S.: Culturally themed educational tools for enhancing
learning in Southeast Asian secondary schools. In: Arai, K., Kapoor, S., Bhatia, R. (eds.) FTC
2020. AISC, vol. 1288, pp. 950–968. Springer, Cham (2021). https://doi.org/10.1007/978-3030-63128-4_71
9. Sithira Vadivel, V., Song, I., Bhati, A.S.: Cultural emotion games as trajectory learning in
Southeast Asia. In: Hauge, J.B., . Cardoso, J.C.S., Roque, L., Gonzalez-Calero, P.A. (eds.)
ICEC 2021. LNCS, vol. 13056, pp. 62–74. Springer, Cham (2021). https://doi.org/10.1007/
978-3-030-89394-1_5
10. Vadivel, V.S., Song, I., Bhati, A.S.: Culture-centered ICT guidelines for Southeast Asian
secondary schools. In: HCI International 2022-Late Breaking Papers. Interaction in New
Media, Learning and Games: 24th International Conference on Human-Computer Interaction,
HCII 2022, Virtual Event, June 26–July 1, Proceedings, pp. 437–454 (2022). Cham: Springer
Nature Switzerland (2022). https://doi.org/10.1007/978-3-031-22131-6_33
364
V. S. Vadivel
11. Wei, M., Liu, Y., Chen, S.: Violent video game exposure and problem behaviors among
children and adolescents: the mediating role of deviant peer affiliation for gender and grade
differences. Int. J. Environ. Res. Public Health 19(22), 15400 (2022)
12. Willis, J.: How Your Child Learns Best: Brain-Friendly Strategies you can use to Ignite Your
Child’s Learning and Increase School Success. Sourcebooks, Inc. (2008)
13. Anderson, C.A., Warburton, W.A.: The impact of violent video games: an overview. Growing
Furious Reviewing Impacts Violent Sexualised Media Child. 56–84 (2012)
14. Greitemeyer, T.: The spreading impact of playing violent video games on aggression. Comput.
Hum. Behav.. Hum. Behav. 80, 216–219 (2018)
15. Greitemeyer, T.: The contagious impact of playing violent video games on aggression:
longitudinal evidence. Aggressive Behav.Behav. 45(6), 635–642 (2019)
16. Calvert, S.L., et al.: The American psychological association task force assessment of violent
video games: science in the service of public interest. Am. Psychol. 72(2), 126 (2017)
17. Shi, Y.R., Shih, J.L.: Game factors and game-based learning design model. Int. J. Comput.
Game. Technol. 2015, 11 (2015)
18. Song, M., Zhang, S.: EFM: a model for educational game design. In: Technologies for ELearning and Digital Entertainment: Third International Conference, Edutainment 2008 Nanjing, China, June 25-27, 2008 Proceedings 3, pp. 509–517. Springer Berlin Heidelberg (2008).
https://doi.org/10.1007/978-3-540-69736-7_54
Cyber Safety Awareness Through a Massive
Open Online Course (MOOC): Community
Engagement Knowledge Transfer
Elmarie Kritzinger(B)
University of South Africa, UNISA, PO Box 392, Pretoria 0003, South Africa
kritze@unisa.ac.za
Abstract. The world is becoming increasingly digitised, and information communication technology (ICT) has become integrated in every part of our lives. Constant connectivity (internet access) has provided opportunities that have enriched
and changed the way we do business, education environments, socialise and participate in personal activities. An environment that is growing due to constant
ICT development is the education space. This impact of ICT on the educational
environment made it possible to provide educational opportunities to numerous
communities that were previously excluded. This exclusion of educational spaces
and lifelong learning opportunities are a result of inadequate funding, transport
and time constraints, access to higher education spaces and lack of prior knowledge and skills. This research investigates how electronic learning (e-learning) can
be used to facilitate knowledge sharing amongst disadvantage communities. The
research explores a Massive Open Online Course (MOOC) approach with the aim
to enhance knowledge sharing and cultivating an environment for lifelong learning among previous excluded communities. This research uses a case study of a
cyber safety awareness community engagements project to determine if a MOOC
can be used to provide quality education, knowledge and skills to committees that
otherwise had no access to higher educational or further learning spaces. This
research covered the design, presentation, and evaluation of a cyber safety awareness MOOC on a 0-rated (free data available) university platform in South Africa.
The MOOC was evaluated by means of a pre and post questionnaire to determine
its success and the extent of knowledge transfer to community members.
Keywords: MOOC · Cyber safety · Community engagement · Lifelong
learning · Knowledge transfer
1 Introduction
Technology is developing at a rapid rate and includes global connectivity for all users via
the internet [1]. Internet access has provided online advantages to users across the globe
for the purposes of business, education, and leisure. As technology becomes integrated
into the daily lives of individuals, cyber users (those using cyberspace) are confronted
with the challenge of protecting their personal information and being cyber safe. All cyber
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2024
K. Arai (Ed.): SAI 2024, LNNS 1017, pp. 365–376, 2024.
https://doi.org/10.1007/978-3-031-62277-9_23
366
E. Kritzinger
users are open and vulnerable to a wide range of cyber-related risks and threats, which
include identity theft, phishing, cyberbullying and access to inappropriate electronic
material [2]. Cyber users are accessing cyberspace, but lack the knowledge and skills to
identify, avoid and/or eliminate these cyber risks and threats [3, 4]. This results in direct
and indirect harm to cyber users. It is therefore vital that all cyber users, irrespective of
their age, race, and geographical location, have the necessary cyber safety awareness
(knowledge and skills) to protect themselves in cyberspace [5, 6].
A wide range of cyber awareness programs are available, but there is a cost associated with many (in the form of registration fees and the cost of travelling to the training
venue and purchasing of study material). Many educational courses are linked to formal learning that requires prior knowledge or pre-requirements. In many disadvantaged
communities (for example in South Africa), community members are excluded from
any other form of education after formal schooling. One of the reasons are a lack of cost
free opportunities to educate community members regarding knowledge growth towards
lifelong learning [7]. It is therefore vital to provide alternative educational pedagogy that
includes informal education for communities that do not possess the means (funding,
time available and means of transport) to access many education opportunities. One
option to provide access to higher quality learning content is to use the technology in
form of open, accessible, and electronic learning (e-learning). The research proposes
Massive Open Online courses (MOOCs) to assist with lifelong learning opportunities
for disadvantaged communities with no other means to educational spaces.
MOOCs can be defined as 1) Massive; large number of participants, 2) Open; offered
at no cost, with no/small cost of enrolment, 3) Online; hosted on an online site (Internet)
and 4) Course; full length [8]. The content of a MOOC should be tailored to the needs
of the participants and/or communities. However, it is vital the that the content of a
MOOC should be of the same quality as formal courses [9, 10]. The main reason for
considering a MOOC for the purpose of this research was the significant potential of
MOOCs to empower communities through equipping them with information on a wide
range of topics [11]. MOOCs can be classified as zero-rated sites (meaning that no data
is needed to access the site), with no registration required. A MOOC can offer a basic
introduction for participants with no prior knowledge of the topic. The MOOC can then
be staggered to included follow up courses to build on the introduction of a topics. With
the aim in mind of lifelong learning it is critical that if participants do enrol and complete
the MOOC, that the experience will lay the foundation for the participants to enrol in
further MOOCs.
This research consists of two sections. The first section of the research included a
thematic analysis method analysing 170 research articles relating to cyber safety awareness and curriculum development (course design). The second section of this research
includes a quantitative analysis in the form of a pre and post online questionnaire of 100
MOOC participants to determine whether knowledge and retention improved through
participating in a MOOC. Full ethical clearance was obtained for this research. These
two sections culminate to determine if a MOOC approach can be used to improved
knowledge transfer and create a lifelong learning platform. This research has a special
focus on community engaging that includes communities that have now other access to
higher/alternative education, knowledge, and skills programmes.
Cyber Safety Awareness Through a Massive Open Online Course (MOOC)
367
2 MOOCs and Community Knowledge Transfer
The growing employment of MOOCs across the globe indicates a significant change and
adoption in digital teaching and learning [12–14]. MOOCs have been presented since
2012, with the main aim of transforming education [15–17]. MOOCs focus specifically
on an educational approach for online distance education and e-learning [18]. Participants
registering for MOOCs are usually diverse in terms of learning experience, environment,
goals and motivation [13]. MOOCs are seen as part of the pathway of open electronic
resources (OER), making possible the provision of quality content at lower or no cost
[10].
MOOCs are open and accessible to any participant with web access, and are typically
offered free of charge, with no penalties for non-participation [12]. MOOCs contribute to
accessible and open education for communities and make valuable learning experiences
available to participants across the globe [10, 12, 19]. The main focus of MOOCs is on
providing a development path through the provision of high quality content [9, 20].
MOOCs provide participants with a self-regulated learning environment to allow
them to decide when and how to engage with the material supplied [13]. Participants in
MOOCs have different needs, motivation, expectations and prior knowledge in relation
to the topic covered [13, 21]. MOOCs can also be used within community engagement
projects with social inclusion [7]. MOOC is not only for educated students but must
be designed and presented for all communities. MOOCs offer convenience in the area
of community participation due to the no enrolment cost, no prior knowledge, and no
physical venue, and community-based learning environment. There are no timelines or
due dates set for MOOCs, as participants may complete the course work any time of
the day or night. On completion of the MOOC the participants may receive an e-mailed
certificate of completion or participation, again, involving no cost. The discussion forums
and online workshops provide an environment for community and interaction.
MOOCs are also a means to introduce lifelong learning to communities that previously had no means of becoming part of the learning space [22]. This research focus on
using online learning to establish lifelong learning with the focus on high quality and
impactful knowledge and skills for communities. The research conducted focused primarily on providing free (open), inclusive cyber safety awareness education to communities. The evaluate if an open, electronic learning environment (by means of MOOCs)
are a viable option within the South African an African content. The research will
investigate if MOOCs are a viable option to share knowledge with communities and if
knowledge transfer occurs through the MOOC process. The case study will include a
community engagement project focusing on a critical academic knowledge sphere that
addresses improving cyber safety awareness among community members. This research
will explore the academic space of cyber safety awareness to evaluate the use of MOOCs
for community education to grow a lifelong learning experience.
368
E. Kritzinger
3 The Cyber Safety Awareness MOOC
Technology is changing our world at both a rapid and constant pace by developing and
supplying information and knowledge technologies and related devices to users around
the globe [23]. Many aspects of our lives now involve information communication technology (ICT), and ICT devices such as mobile phones, laptops and tablets. Technology
and ICT devices are becoming smaller yet ever more powerful, with greater technological capabilities. Technology is no longer a luxury, but rather a necessity. According
to Roztocki, Soja and Weistroffer [24], ICT plays a “significant role in all aspects of
modern society”. However, with the rapid advancement of technology, large part of disadvantages communities is left out of the educational. The aim of this research is to
investigate if MOOCs can be used to bridge this knowledge/technology gap between
higher education and communities. The scope of this research is to investigate this matter by using designing and creating a cyber safety Awareness MOOC for community
participation.
The Cyber Safety Awareness MOOC was based on a cyber safety awareness toolkit
that was created for South Africa and African countries. The toolkit was created by the
University of South Africa, the British High Commission and the Department of Communication and Digital Technologies (DCDT) [25]. The Toolkit included a wide range
of educational materials (posters, workbooks, guidelines, videos and pledges) as tools to
improve the cyber safety awareness knowledge. The main purpose of the MOOC was to
equip cyber users with foundational concepts to improve overall cyber safety awareness
knowledge and skills. The MOOC also focused on equipping community members with
the necessary cyber safety awareness and knowledge to assist cyber users to become
cyber safe. Participants who completed the MOOC would gain a basic knowledge of
cyber safety and how to be cyber safe. As mentioned above, the MOOC was free with a
zero-rated platform on the university LMS. All the material forming part of the MOOC
was quality controlled in accordance with the university quality assurance process for
study material. In addition to the critical aspects of cost, access and quality, the university platform was used to provide all participants (irrespective of their income, job status
or background) with a sense of academic belonging and to encourage lifelong learning
within the higher educational sphere.
Another important achievement of the cyber safety MOOC was the translation of the
course material into several the official South African languages to afford participants
the opportunity to study the content in their home language. The material in the Toolkit
was translated from English into Afrikaans, IsiXhosa, IsiZulu, Sesotho and Venda. This
will provide the participants the opportunity to study within their own language. Upon
completion of the MOOC activities, each participant received an e-certificate confirming
their participation. The rest of the research focuses on the design of the MOOC and the
evaluation of the MOOC.
Cyber Safety Awareness Through a Massive Open Online Course (MOOC)
369
4 Design Approach Adopted
This section focuses on the development, content (Learning Units) of the cyber safety
awareness community engagement project within an academic department at a South
African University.
4.1 Development Approach
The ADKAR approach was followed in the design of the MOOC. The ADKAR model
was devised by Jeffery Hiatt in 2006 [26] and is a model widely used within the education
environment. The model focuses on individuals and change, cultivating an environment
for engaged learning that consists of enablement and engagement by the participant. The
ADKAR model [26] has the following goals: 1) Awareness; creating of awareness about a
certain topic, 2) Desire; factors which motivate individuals to participate, 3) Knowledge;
the items within the toolkit which will enable participants to gather knowledge, 4) Action;
task to achieve the artefacts and 5) Reinforcement; items within the toolkit which aim
to monitor. Mapping the MOOC design elements to the ADKAR approach is depicted
in Fig. 1.
Fig. 1. Mapping the MOOC design elements to the ADKAR approach
The mapping illustrated in Fig. 1 was used to create the content to be presented in
the MOOC. The first to levels of Bloom’s Taxonomy was used including 1) remember
and 2) understand for the course content [27]. The creation of the content for the MOOC
included design elements that focused on:
• Visual presentation (posters, videos)
• Colour mapping (a different colour mascot and layout for each unit.
• Context-driven workbook and guidelines (translated)
• Multimedia (videos, games)
• Assessment (MCQs)
• Interaction (discussion forums)
The alignment of the design elements was included to ensure that participants with
no formal education, prior knowledge, and limited access to the MOOC space, would
still be able to participate. The focus was that the MOOC also reflects inclusiveness of
all participants in South Africa through mascot design, South African examples, and
language specific content.
370
E. Kritzinger
4.2 MOOC and Content Approach
This research project was a collaboration with the University of South Africa, the British
High Commission and the Department of Communication and Digital Technologies
(DCDT) [25] with the focus on providing disadvantage communities with needed knowledge on cyber safety awareness. The collaboration focusses on an African approach to
ensure inclusiveness to all participants amongst disadvantage communities by creating
a cyber safety awareness toolkit. The first part of the collaboration included creating the
needed content of the Toolkit.
Topics within the Cyber Safety Awareness MOOC were distributed over five learning
units (LUs) The LUs were obtained through thematic analysis method that focused on
cyber safety awareness key threats, agents, vulnerabilities, and victims association that
relates to basic cyber risks and threats. Each of the LU covered three topics relating to
key aspects of cyber safety awareness within that LU and is depicted in Table 1.
Table 1. Cyber safety awareness toolkit learning units (LU)
LU
Name of LU
Topics within LU
1
A trip into cyberspace
Digital footprint
Cyber risks and threats
Online privacy
2
Protecting people
Cyberbullying
Family safety
Respect and ethics
3
Securing devices
Technology threats
Mobile devices
Malware protection
4
Smart applications
Social media
Safe web browsing
Gaming
5
Useful information
Offensive and inappropriate content
Cyber scams
Password management
The relating topic for each LU was evaluated with means of an expert panel committee
that evaluated, and quality checked the content to be presented in the MOOC. The
MOOC content also underwent an internal university quality assurance process. The LUs
were delivered by means of written text documents (online downloadable files), videos,
cartoons, learning activities and an assessment activity (multiple-choice questions) for
each LU. Upon completion of all the learning units (and the assessment for each LU), each
participant received an electronic certificate of participation. The MOOC was designed,
implemented and active for 23 months over a two-year cycle on the university learning
management system.
Cyber Safety Awareness Through a Massive Open Online Course (MOOC)
371
5 Evaluation of MOOC and Knowledge Transfer
Within any educational programme it is vital to obtain a baseline of participants’ knowledge before the start of the course. This will allow that the research can measure knowledge transfer upon completion of the course. Knowledge transfer was one of the key
research aspects that would assist in answering the research aim. As part of this research
study, participants were asked to complete a pre and post online questionnaire before
and on completion of the MOOC as well as an evaluation section.
The pre baseline questionnaire consisted of 34 questions, of which 12 covered biographical information and 22 covered the cyber safety awareness knowledge of the
participants before starting the MOOC. The following scale was used for the knowledgebased questions: 1) No knowledge, 2) Minimum knowledge, 3) Good knowledge, and
4) Excellent knowledge.
The post baseline questionnaire consisted of the same 22 knowledge-based questions
as were included in the pre baseline questionnaire, with the same scale as that used in
the pre baseline questionnaire. This was to assist in investigating whether successful
knowledge transfer had taken place. The evaluation section of the questionnaire also
included 14 questions evaluating the MOOC, the participants were requested to respond
according to the scale 1) Very poor, 2) Poor, 3) Average, 4) Good and 5) Very good.
Open ended questions were included to obtain personal feedback from each participant.
100 participants provided online consent before completing the questionnaire. Ethical
clearance was obtained for the research study.
6 Data Analysis
The data analysis focused on three aspects; 1) Participants’ biographical information,
2) Knowledge gain through the MOOC; 3) Participant evaluation and feedback after
completing the MOOC.
6.1 Bio
The biographical data analysed related to gender, age, working environment, prior studies, and prior MOOC experience. Of the participants 69% were female and 31% were
male. The age range of the participants was as follows: 50% were born between 1981
and 1996, 45% were born between 1997 and 2002, and 5% were born before 1980. 55%
indicating that they had enrolled to improve their cyber safety awareness. 80% of the
participants indicated that this was their first enrolment for a MOOC, and 90% of the
participants reported never having taken a cyber safety awareness course.
The analysis indicated that 61% of the participants only had a high school certificate
with no further higher qualifications and 42.2% was not working (at the point of the
data gathering). Analysis of the biographical information indicate that the participants
were diverse in age, work environment, prior knowledge, and reason for enrolment. The
knowledge transfer and participants’ knowledge gain are presented in the next section.
372
E. Kritzinger
6.2 Knowledge Gain
In any education courses, it is vital to ensure that there has been knowledge gain
among the participants. The study examined 22 knowledge areas within the MOOC.
The questions include the current knowledge level for each participant regarding:
1) cyber safety
2) cyber space
3) relationship between cyber-safety and cyberspace
4) cyber risks and threats
5) protecting yourself and your information in cyberspace
6) implementing cyber safety countermeasures to protect your information
7) identifying possible cyber risks, threats, and attacks
8) protecting your digital footprint
9) impact of cyber threats
10) your responsibility within cyber space
11) your rights as a cyber-user
12) support for cyber victims
13) online privacy and protecting your information
14) your responsibility within a cyber-community
15) the impact of cyber bulling on learners
16) offensive and inappropriate content amongst learners
17) identifying cyber risks and threats on social media platforms
18) the disadvantages of cyber space
19) the impact of cyberbullying on learners
20) respect among learners in cyberspace
21) age restrictions on social media platforms
22) how to address a cyber-incident when occurred within a school
The knowledge transfer was measured in a pre-questionnaire and again in a poquestionnaire from the participants. Figure 2 depicts the 22 knowledge areas (x-axis)
and the average of the pre and post knowledge transfer scale (y-axis). The scale comprised
the following: 1) No knowledge, 2) Minimum knowledge, 3) Good knowledge, and 4)
Excellent knowledge. The pre knowledge is indicated by means of the left bar for each
knowledge area and the post knowledge gain is indicated by means of the right bar for
each knowledge area.
The findings indicate that the knowledge level of all participants increased from
before their enrolment for the MOOC to after the completion of the MOOC. The average knowledge level before participating in the MOOC was measured at 2.0, and after
completion of the MOOC at 3.4. This is a positive indication that the knowledge transfer occurred among participants. One can derive that a MOOC is a viable option to
use in community engagement for education, awareness, knowledge transfer and skills
development.
The next section contains a discussion of the evaluation of the MOOC (content,
presentation of the material and overall satisfaction with the MOOC).
Cyber Safety Awareness Through a Massive Open Online Course (MOOC)
373
Fig. 2. Pre/Post knowledge transfer in 22 knowledge areas
6.3 Evaluation of the MOOC
The evaluation of the MOOC was done in two parts. The first part was a qualitative
section and consisted of 14 questions. Questions were rated according to the following
five-point scale: 1) Very poor, 2) Poor, 3) Average, 4) Good and 5) Very good. The
average rating for each of the questions is reflected in Fig. 3.
Fig. 3. Evaluation of the cyber safety awareness MOOC
Figure 3 depicts the overall average rating of the 14 questions. The rating of the
MOOC indicated a score of 4.4 out of a possible 5 rating. The second section of the
evaluation was an open section in which participants were provided the option to give
written feedback and recommendations of their experiences during the MOOC. Some
of the feedback included:
“The MOOC was extremely informative and important for me personally.”
“It was very easy to understand the MOOC.”
“The MOOC was a very clear course that gave me awareness of what is occurring on
the internet and how to protect myself if I come in contact with any danger.”
“No comment so far I’m very satisfied.”
“The MOOC was informative and educational.”
“I am honestly satisfied with MOOC, I don’t see any need for improvement.”
374
E. Kritzinger
“The MOOC is a good platform of teaching and introducing learners to online learning.”
“Create more exciting courses as well.”
“The visuals are very good and exciting.”
“Thank you for enhancing my skills. I truly learned a few things I did not know.”
“The MOOC has made me 100% aware of Cyberspace and Cyberbullying. I learned a
lot.”
“My knowledge captured the part of cyber security and risks very good and the content
was well made and their terminology.”
“The MOOC help me in understanding the differences between the cyberbullying and
cyberspace and that you must always be aware of such things, and you must talk about
it to someone who you feel safe to talk too.”
“The MOOC is a good platform of teaching and introducing learners to online learning.”
Some feedback on suggested changes to the MOOC included:
“Set time limits for the activities – a more structured approach.”
“More online workshops are needed”
“More South African examples would be a good idea”
“A more in-depth MOOC can be developed as a “next stage” in the learning process”
These suggestions will be incorporated when the MOOC is updated. The overall
feedback of the MOOC was positive and indicated that a majority of the participants
would enrol for other MOOCs as well. One aspect to note is that 91% of the participants
indicated that they would enrol for another MOOC on the university platform and 79%
indicated they will recommend the MOOC to others. This indicates that a foundation of
learning has been set and could be the start of a lifelong learning focus.
The analysis of study indicated that through the feedback of the participants that
MOOCs can be successfully used for knowledge sharing amongst communities, with no
other access to lifelong learning spaces. One observation from the analysis indicated that
51% of the participants were not working. This provides a clear indication of the need
among communities in South Africa for educational support and inclusion in lifelong
learning. Different MOOCs can be linked together to create a holistic educational path to
obtain needed knowledge and skills. Or follow up MOOCs on this topic can be introduces
to continue the knowledge transfer and for skills building. The feedback also indicated
that the MOOC did indeed achieve the goals by means of providing quality academic
knowledge (free of charge) to community members.
With the analysis of the pre- and post-questionnaire the main aim of the research was
addressed that knowledge transfer can occur through the use of MOOCs. MOOCs can
be used to uplift communities that have no other access to education and learning spaces.
MOOCs can provide quality high level education initiatives (free of cost) to participants
can form a basis for lifelong learning.
7 Conclusion
A MOOC approach was proposed to provide access to lifelong learning to communities that currently have no or limited access to educational opportunities. The research
investigated whether a MOOC approach presented via e-learning would help community
Cyber Safety Awareness Through a Massive Open Online Course (MOOC)
375
members to improve their cyber safety awareness knowledge and skills. This is vital,
as mobile and internet access is widely available, but knowledge and skills regarding
the protection of personal information in cyberspace is lacking. The development and
presentation of a cyber safety awareness MOOC indicated the increase in the level of
knowledge among participants. Enrolment for the Cyber Safety Awareness MOOC was
free. The MOOC offered a lifelong learning option irrespective of status, geographical location, and prior academic knowledge. The study concluded by confirming that
MOOCs can be used for knowledge transfer to communities that otherwise would have
been excluded from lifelong learning.
References
1. Alzubaidi, A.: Measuring the level of cyber-security awareness for cybercrime in Saudi
Arabia. Heliyon 7(1), 1–13 (2021)
2. Tabassum, L., Baker, S.: Cybersecurity and safety measures. Int. Res. J. Mod. Eng. Technol.
Sci. 2(6), 1357–1360 (2020)
3. Zwilling, M., Klien, G., Lesjak, D., Wiechetek, L., Cetin, F., Basim, H.N.: Cyber security
awareness, knowledge, and behaviour: a Comparative Study. J. Comput. Inf. Syst. 62(1),
82–97 (2020)
4. Giri, S.: Cyber crime, cyber threat, cyber security strategies and cyber law in Nepal. Pramana
Res. J. 9(3), 662–672 (2019)
5. Chang, L.Y.C., Coppel, N.: Building cyber security awareness in a developing country: lessons
from Myanmar. Comput. Secur. 97(1), 101959 (2020)
6. Aloul, F.A.: The need for effective information security awareness. J. Adv. Inf. Technol. 3(3),
176–183 (2012)
7. Lambert, S.R.: Do MOOCs contribute to student equity and social inclusion? A systematic
review 2014–18. Comput. Educ. 145, 10369 (2020)
8. Perna, L.W., et al.: Moving through MOOCs: understanding the progression of users in
massive open online courses. Educ. Res. 43(9), 421–432 (2014)
9. Zhang, X.: Design and analysis of music teaching system based on virtual reality technology.
IPPTA Q. J. Indian Pulp Pap. Tech. Assoc. 30(5), 196–202 (2018)
10. Czerniewicz, L., Deacon, A., Glover, M., Walji, S.: MOOC—making and open educational
practices. J. Comput. High. Educ. 29(1), 81–97 (2017)
11. Aparicio, M., Oliveira, T., Bacao, F., Painho, M.: Gamification: a key determinant of massive
open online course (MOOC) success. Inf. Manag. 56(1), 39–54 (2019)
12. Ebben, M., Murphy, J.S.: Unpacking MOOC scholarly discourse: a review of nascent MOOC
scholarship. Learn. Media Technol. 39(3), 328–345 (2014)
13. Littlejohn, A., Hood, N., Milligan, C., Mustain, P.: Learning in MOOCs: motivations and
self-regulated learning in MOOCs. Internet High. Educ. 29(1), 40–48 (2016)
14. Adam, T.: Open educational practices of MOOC designers: embodiment and epistemic
location. Distance Educ. 41(2), 171–185 (2020)
15. Deng, R., Benckendorff, P.: What are the key themes associated with the positive learning
experience in MOOCs? An empirical investigation of learners’ ratings and reviews. Int. J.
Educ. Technol. High. Educ. 18(1), 9 (2021)
16. Baggaley, J.: MOOC rampant. Distance Educ. 34(3), 368–378 (2013)
17. Ruipérez-Valiente, J., et al.: Large scale analytics of global and regional MOOC providers:
differences in learners’ demographics, preferences, and perceptions. Comput. Educ. 180,
104426 (2022)
376
E. Kritzinger
18. Zhou, M.: Chinese university students’ acceptance of MOOCs: a self-determination perspective. Comput. Educ. 92–93, 194–203 (2016)
19. Baek, J., Shore, J.: Promoting student engagement in MOOCs. In: Proceedings of the 3rd
2016 ACM Conference on Learning at Scale, L@S 2016, pp. 293−296 (2016)
20. Akoglu, K., Lee, H.S., Kellogg, S.: Participating in a MOOC and professional learning team:
how a blended approach to professional development makes a difference. J. Technol. Teach.
Educ. 27(2), 129–163 (2019)
21. Reparaz, C., Aznárez-Sanado, M., Mendoza, G.: Self-regulation of learning and MOOC
retention. Comput. Hum. Behav. 111, 106423 (2020)
22. Buhl, M., Andreasen, L.B.: Learning potentials and educational challenges of massive open
online courses (MOOCs) in lifelong learning. Int. Rev. Educ. 64(2), 151–160 (2018)
23. Barbosa, J.: How to educate to build an effective cyber resilient society. Int. J. Cyber Res.
Educ. 2(1), 55–72 (2020)
24. Roztocki, N., Soja, P., Weistroffer, H.R., Soja, P.: Information technology for development the
role of information and communication technologies in socioeconomic development: towards
a multi-dimensional framework. Inf. Technol. Dev. 1102, 170–183 (2019)
25. UNISA: British High Commission, and Department of Communication and Digital Technologies: Cyber Safety Awareness Toolkit. https://www.cyberaware.co.za. Accessed 02 Feb
2023
26. Prosci: ADKAR Model. https://www.prosci.com/methodology/adkar. Accessed 04 Jun 2021
27. Bloom, B.S: Taxonomy of Educational Objectives – The Classification of Educational Goal.
Handbook 1: Cognitive Domain. Longman, London (1956)
Immersive Environments at School: “Stop
Cyberbullying by Proximity”
Cristina Sánchez-Romero(B) and Eva María Muñoz-Jiménez
National University of Distance Education (UNED), Faculty of Education, Juan del Rosal Street,
14, 28040 Madrid, Spain
csanchez@edu.uned.es
Abstract. Nowadays, adolescents are heavy users of newer electronic communication forms to communicate and share moments with their peer, however increasingly electronic aggression is emerging through these communication tools, like
a cyberbullying or social exclusion. In addition, this educational problem has a
strong dimension in the closest contexts such as the social and/or family environment, “cyberbullying by proximity”. Gamification has frequently been applied
and incorporated in many different school-based activities, at different stages of
education system with positive results. Metaverse is a relatively new technology, but with this innovative technological tool an interactive experience of realworld environment can be created. The principal aim of this design proposal is to
develop innovative and effective 3D educational content regarding the “cyberbullying by proximity” social problem by using the Metaverse Gamification for students
and teachers. This proposal presents technological innovation 3D Metaverse Gamification which enables the creation of virtual computer-simulated environments
by combining virtual reality (VR) with gamification strategies facilitating use in
the educational field. The proposal design will be synthesised in different steps
that will allow us to analyse bullying indicators and their corresponding different
problems associated with students’ well-being, developing top-notch and effective training content for bullying and cyberbullying to raise awareness, prevent
and stop this social and health problem.
Keywords: Cyberbullying · Virtual reality · e-Learning · Cybersecurity · Digital
transformation
1 Introduction
The global social and health crisis created by the coronavirus disease (COVID-19) pandemic has had an impact on the prevalence of bullying in the cyber world, as some
students use to bully the digital space that is even more difficult to spot due to the network anonymity which has a huge influence on cyberbullying [10], and online bullying
through threatening messaging, also emerging as a new form of aggressive behaviour
that may threaten adolescent social and emotional development [13]. Bullying at school
can have long-lasting consequences for students [11], it is a social phenomenon where
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2024
K. Arai (Ed.): SAI 2024, LNNS 1017, pp. 377–383, 2024.
https://doi.org/10.1007/978-3-031-62277-9_24
378
C. Sánchez-Romero and E. M. Muñoz-Jiménez
the students are involved as victims, aggressors, or bullies [6], also interfering with students’ school performance, family and peer relationships, psychological well-being and
physical health [2].
Regarding peer victimization, according to the author [16] pointed out some features
of computer communication that may trigger electronic aggression, such as the user’s
anonymity, the use of online communication in everyday life resulting from easy and
permanent access to different applications, the continuous widening of the audience,
the prevalence and incidence of harassment, gender-based violence, and discrimination in cyberspace. Recent research in [15] has analysed the adolescent’s risk factors
using social networking and instant messaging, the findings have shown the following: sexting (57%), cyberbullying (48%), spam, grooming, malware (43%), addiction
problems (33%) and other risks such as theft of personal information and spam with
24%. Exposure to bullying and cyberbullying are associated with psychological distress
like anxiety, depressive symptoms, and suicidal thoughts, therefore, adolescents’ wellbeing is affected by this issue. In conclusion, violence behaviour and bullying among
adolescents are a global challenge [19]. What is more, this bullying emerges in the closest environment “cyberbullying by proximity”, this educational problem has a strong
dimension in other contexts such as the social and/or family environment. Educational
institutions have a central key role in identifying subtle sings of bullying. The institution is also defined these behaviours in school settings. Regarding to this [4], state that
teachers need training in bullying. In their study, they described different scenarios with
physical bullying, verbal bullying and relational bullying events.
In addition, our reality and our lives are increasingly game-like, not only have games
become a pervasive part of our lives, but activities, systems and services are also becoming more and more gamified nowadays [8]. According to this, a new approach is emerging
with the development of gamification in education [1], Teachers can include innovation
in the academic environment [3]. And gamification is also changing teachers’, students’,
and parents’ roles in the method of teaching and learning [12].
Currently, Gamification is an educational technique that apply Information Communication Technologies (ICTs) game design elements and mechanics. This trend and/or
fashion analysed by different authors has been already introduced in the educational sector. Accordingly [18], describe how this gamified learning approach could influences
learning, through a series of gamified learning activities, based on mobile gamification learning system. The results show that incorporating mobile and gamification in
their learning process could achieve a better learning performance and a higher degree
of motivation is achieved [17].
The research in [14] suggests that Metaverse is a pervasive expression of technological culture whose impact will be global. First, through knowledge, and then through
social and now through geo-spatial, AI (the basis of the Metaverse) will connect all
entities on Earth through digital means, thus creating a three-dimensional layer of information and experience around the world dubbed Metaverse. In this term [7], describe
metaverses embedded in our lives create virtual experiences inside of the physical world.
According to study [9] state that metaverse supports a continuous and persistent virtual
world where all individual users can wander around transparently without predefined
goals.
Immersive Environments at School
379
In this innovative project, 3D educational content regarding the “cyberbullying by
proximity” social problem will be developed, using the Metaverse Gamification. The
use of SIX 3D Metaverse Gamification which enables the creation of virtual computersimulated environments by combining virtual reality (VR) with gamification strategies
facilitates its use in the educational field. In this term, eye-catching gamified training
content for young adolescents through didactic strategies such as escape room and role
playing will be created to diagnose symptoms of violence and discover their corresponding forms among adolescents. In addition, top-notch gamified sessions and effective
training content will be designed to raise awareness and create preventing actions for
the development of community bullying.
2 Methodology
This proposal design is carried out from a point of view based mixed methods that
allows us to delve into the bullying identification and analysis objectives. The innovative
project will be synthesised in different phases that allow us to find out and analyse
bullying indicators and their corresponding different problems associated with students’
well-being. The phases to carry out these gamified sessions are as follows:
Step 1. Identify Signs of Cyberbullying by Proximity in Vulnerable Groups. This
stage aims to analyse the manifestations of violence and identify them among adolescents. The aim is to diagnose the different forms of violence among adolescents today.
Furthermore, not only are indicators of school and online violence analysed in this
research proposal, but a new methodology will be also generated that will allows us
to diagnose symptoms of violence and to discover their corresponding forms among
adolescents, to create preventing actions and to raise social awareness through ingenious gamified sessions to ensure safe use of social networks in line with the Spanish
Government’s National Security Strategy. Therefore, in achieving this strategy we will
focus on technological capacity building and international collaboration to contribute to
the improvement of cybersecurity and prevent cyberbullying, considering the European
Digital Competences Framework (DIGCOMP).
Step 2. Design Gamified Training Content for Students About Bullying
and Cyberbullying. Metaverse Gamification allows users to experience virtual environments by making the brain think that they are somewhere else. In this proposal design,
three different interactive environments “SIX 3D AGAINST BULLING” will be created and students will be able to experience first-hand different environments related to
bullying and cyberbullying.
Teachers, students, and users can create their own interactive experiences in Virtual
Reality scenes will be like role-playing games, when the users play; their actions will
have certain consequences, reproducing the real consequences that they have with this
type of behaviour. In these immersive three-dimensional virtual worlds, in which people
interact as avatars with each other and with software agents, using the metaphor of the
real world but without its physical limitations [5]. The These scenes are completely
380
C. Sánchez-Romero and E. M. Muñoz-Jiménez
photorealistic, and will be set in the usual environments of teenagers, such as a school,
leisure areas or inside home, as we can see in the following figures (Figs. 1, 2 and 3)1 :
Fig. 1. Scene of classroom. Source: SIX 3D Metaverse
Fig. 2. Scene of the school gym. Source: SIX 3D Metaverse
Fig. 3. Scene of the leisure areas. Source: SIX 3D Metaverse
The scenes in a video game form a great catalyst for the students’ attention, empathy,
and resilience, which will become aware of these violent manifestations that they or other
1 These figures were undertaken as part of pilot project (Alberto Lozano-Porral C.E..O “SIX 3D
Metaverse”).
Immersive Environments at School
381
classmates may suffer. Users will also be able to create their own interactive experiences
and work on different scenes that would be useful for tutorial support.
Step 3. Apply the Gamified Training Content to a Control Group. Experimental
research about cyberbullying with Virtual Reality with Metaverse technology creating
different contexts will be carried out to develop a platform as a tool for generated users’
content.
Step 4. Evaluate the Impact on Teachers and Students’ Gamified Experiences. In
this phase, the impact on teachers’ and student’s gamified experiences in the control
group will be analyzed, as well as the users’ digital skills in the face of cyberbullying
by proximity to know the effectiveness of the gamified scenarios.
With these steps, scientific knowledge, and development of disruptive technologies
for the adolescents’ cybersecurity from an inclusive perspective will be created. Moreover [5], their team pointed out the application of metaverse for virtual team collaboration
and highlighting a new technological platform for global virtual collaboration, creating
an interactive experience of real-world environments. According to study [9] it stated
that metaverse supports a continuous and persistent virtual world where all individual
users can wander around transparently without predefined goals. The development of
this innovative 3D educational content to be applied as a tool for students and teachers to
raise their awareness on bullying and cyberbullying, creating the immersive technology
with Metaverse.
3 Conclusion
In conclusion, the incessant bullying of electronic issues need to be addressed in the
digital context, where these risk behaviours are difficult to identify. So, an interdisciplinary perspective and a coordinated educational response for facing cyberbullying by
proximity situation. In addition to these strategies of collaboration between the agents
involved, we must also monitor and control this social problem. The great contribution
of this project lies in the triangulation of innovation, gamification and metaverse technology for the awareness and prevention of cyberbullying by proximity through digital
competence.
Not only do we want to have an impact on improving the health of students, but we
also want to provide inclusive and equitable quality educational spaces where victims can
feel protected and safe, creating spaces of healthy coexistence for vulnerable groups. For
that reason, the process of digital literacy skills is necessary from an early age to build and
generate a better knowledge of the use and safety of mobile devices and social networks
that allows them to report bullying, communicate situations of sexting or grooming.
In this sense, it is necessary from an interdisciplinary perspective to develop joint and
cross-disciplinary training on digital security.
Different studies describe the positive results of gamification in different fields,
especially in education [7]. This current proposal design presents the novelty of the
application of the SIX 3D Metaverse in the educational community. The innovative 3D
educational content is a very powerful tool for generating disruptive content, opening
the possibility of making content in virtual reality, this takes the advantage of the proven
retention capacity that this type of content has.
382
C. Sánchez-Romero and E. M. Muñoz-Jiménez
4 Fundings
Grant TED2021-129132B-I00 Ecological and Digital Transition Projects 2021. Educational Sciences. Proximity Cyberbullying. How to Avoid it Through Gamification in
the Metaverse? Funded by MCIN/AEI/https://doi.org/10.13039/501100011033 and, as
appropriate, by “ERDF A way of making Europe”, by the “European Union” or by the
“European Union NextGenerationEU/PRTR”.
References
1. Aini, Q., Hariguna, T., Putra, P.O.H., Rahardja, U.: Understanding how gamification influences behaviour in education. Int. J. Adv. Trends in Comp. Sci. Eng. 8(1.5 Special Issue),
269–274 (2019)
2. Prummer, M., Hasmüller, A.: Organ. Superv. Coach. 22(2), 149–164 (2015). https://doi.org/
10.1007/s11613-015-0418-0
3. Bryan, C., Clegg, K.: Innovative assessment in higher education. In Innovative Assessment
in Higher Education (2006)
4. Bauman, S., Del Rio, A.: Preservice teachers’ responses to bullying scenarios: comparing
physical, verbal, and relational bullying. J. Educ. Psychol. 98(1), 219–231 (2006)
5. Davis, J., Green, J., Reed, A.: Interdependence with the environment: Commitment, interconnectedness, and environmental behavior. Journal of Environmental Psychology 29, 173–180
(2009)
6. Evans, C.B.R., Fraser, M.W., Cotter, K.L.: The effectiveness of school-based bullying
prevention programs: A systematic review. Aggress. Violent. Beh. 19(5), 532–544 (2014)
7. Hamari, J., Koivisto, J., Sarsa, H.: Does Gamification Work? — A Literature Review of Empirical Studies on Gamification. Proceedings of the Annual Hawaii International Conference on
System Sciences (2014)
8. Koivisto, J., Hamari, J.: Gamification of physical activity: a systematic literature review of
comparison studies. In: CEUR Workshop Proceedings, vol. 2359, pp. 106–117. CEUR-WS
(2019)
9. Kumar, S., et al.: Second life and the new generation of virtual worlds. Computer 41(9) (2008)
10. Mishna, F., Saini, M., Solomon, S.: Ongoing and online: Children and youth’s perceptions of
cyber bullying. Child Youth Serv. Rev. 31(12), 1222–1228 (2009)
11. Modecki, K.L., Minchin, J., Harbaugh, A.G., Guerra, N.G., Runions, K.C.: Bullying prevalence across contexts: A meta-analysis measuring cyber and traditional bullying. J. Adolesc.
Heal. 55(5), 602–611 (2014)
12. Quintana, J.G., Jurado, E.P.: Juego y gamificación: Innovación educativa en una sociedad en
continuo cambio. Revista Ensayos Pedagógicos 14(1), 91–121 (2019)
13. Raskauskas, J., Stoltz, A.D.: Involvement in traditional and electronic bullying among
adolescents. Dev. Psychol. 43(3), 564–575 (2007)
14. Bolger, R.K.: Finding wholes in the metaverse: Posthuman mystics as agents of evolutionary
contextualization. Religions 12(9) (2021). https://doi.org/10.3390/rel12090768
15. Sánchez Romero, C., Álvarez González, E.: Actitudes nocivas y riesgos para los menores a
través de los dispositivos móviles. Revista de Estudios y Experiencias En Educación 2(3),
147–161 (2018)
16. Shariff, S.: Cyber-bullying: issues and solutions for the school, the classroom and the home. In:
Cyber-Bullying: Issues and Solutions for the School, the Classroom and the Home, pp. 1–299
(2008)
Immersive Environments at School
383
17. Siyaev, A., Jo, G.S.: Towards aircraft maintenance metaverse using speech interactions with
virtual objects in mixed reality. Sensors 21(6) (2021)
18. Su, C.H., Cheng, C.H.: A mobile gamification learning system for improving the learning
motivation and achievements. J. Comput. Assist. Learn. (2015). https://doi.org/10.1111/jcal.
12088
19. UNESCO: Behind the numbers: Ending school violence and bullying. UNESCO, Paris (2019)
Simulation and Analysis of Cyber-Attack
on Modbus Protocol for Smart Grids
in Virtual Environment
Shampa Banik1(B) , Rajesh Manicavasagam1 , Trapa Banik1 ,
and Shudipta Banik2
1
Tennessee Technological University, 1 William L Jones Dr, Cookeville, TN, USA
{sbanik42,rmanicavasagam,tbanik42}@tntech.edu
2
East Delta University, Abdullah Al Noman Road, Noman Society,
Chattogram 4209, Bangladesh
Abstract. Smart grid capabilities have grown significantly in recent
years. The smart grid provides advanced real-time handling of faults,
advanced automatic control for efficient electricity transmission, monitoring and collection of the electrical system’s capacity, and communication for information sharing. Unfortunately, its exposure to public networks makes it increasingly vulnerable to privacy breaches, vulnerabilities, and cyber-attacks. Cyber security threats and vulnerabilities in smart
grid networks have become a primary concern that needs to be addressed
before deploying a smart grid. Furthermore, the wide range of protocols
increases the attack surface of a smart grid. This study focuses on the vulnerability of Modbus, which is regarded as one of the most prevalent control protocols in smart grid communication networks. This paper presents
preliminary findings of analyzing cyber-attacks against the Modbus protocol using a virtual testbed to investigate its effects on the smart grid
network protocol. The concept incorporates an emulated Modbus/TCP
network environment built from open-source software components that
imitate fundamental industrial control features of the smart grid. Finally,
we analyze the cycle of a cyber-attack leading through Reconnaissance to
a DoS attack on the Modbus/TCP protocol and propose improvements to
the test bed for protocol attack detection and mitigation.
Keywords: Smart grid (SG) · Control protocols · Modbus · Virtual
testbed · Cyber-attacks · Security vulnerabilities
1
Introduction
Smart Grid is a modern paradigm that has replaced the traditional electric grid
in the energy sector. The concept of a smart grid system, as depicted in Fig. 1, is
the heterogeneous interconnection of dependent domains such as energy production, transmission, and distribution. The smart grid can harness programmable
devices and networks to ensure two-way communication between the utility and
c The Author(s), under exclusive license to Springer Nature Switzerland AG 2024
K. Arai (Ed.): SAI 2024, LNNS 1017, pp. 384–401, 2024.
https://doi.org/10.1007/978-3-031-62277-9_25
Analyze Cyber-Attacks on Modbus Protocol
385
the grid and between the utility and the consumers. Smart grids are highly automated and widely dispersed systems with advanced features such as real-time
control, increased operational efficiency, grid resilience, and increased integration
of renewable technology. An important impact of these features is the capability
to minimize carbon footprint.
Fig. 1. Smart Grid’s conceptual model [1]
Smart grids combine disparate communication networks, such as those for the
Internet of Things (IoT), industrial devices, wireless components, and Wireless
Sensor Networks (WSN). Unfortunately, these protocols can be susceptible to
various security threats. In smart grid architectures, these vulnerable protocols
are typically connected to long-haul networks, such as the Internet, that expose
them to malicious attackers. As a result, they pose serious security risks [2]. Network vulnerabilities could significantly impact private citizens and businesses,
particularly if attackers target vital infrastructure like power plants, water distribution networks, and rail networks. Therefore, the government, businesses,
and academia focus on securing smart grids.
Modbus is one of the most prominent network protocols found in smart grids.
Traditional Modbus protocol cannot withstand cyberattacks as it lacks security
and encryption. Information Technology (IT) and Operational Technology (OT)
have seen exponential growth in security vulnerabilities for common network and
SG technologies in recent years. These vulnerabilities are listed in the National
Vulnerability Database (NVD) [4]. However, any vulnerability in the Modbus
communication systems can affect the whole SG network.
Recently the rise of cyber-attacks targeting various SG control protocols such
as Modbus, DNP3, Profinet, etc. is one of the primary reasons for multifaceted
anomalies in the grid’s operation ranging from power outages, transmission line
outages, unusual power consumption, and momentary and sustained outages [17].
386
S. Banik et al.
To delve into any security research for such an Industrial Control System
(ICS), particularly in the SG system, the testbed plays a pivotal role in modeling various cyber-physical system components that allow testing for security
vulnerabilities and efficiency before the system is deployed [23]. Vulnerability
analysis of the protocol based on testbed can mitigate the security risk before
deployment and save money and possibly (and most importantly) lives.
To address the security solution associated with the control protocol, explicitly design the SG testbed and analyze the vulnerabilities of the control protocols. However, testing such suggested solutions on real systems is impractical
because of the critical nature of SG systems. Our study presents a virtual SG
security testbed that is readily reconfigurable and scalable. It may be utilized
to develop and assess security solutions tailored to SCADA. We utilized attack
scenarios by performing some potential cyber attacks on Modbus protocol to
show how attackers may interrupt SG systems’ regular operations. According to
experimental results, the proposed testbed may be utilized successfully for other
SCADA system vulnerability investigations and cyber security assessments.
This paper’s contributions are as follows. First, we explore designing and
developing a test bed in a virtual environment to simulate the smart grid system by mimicking the network between server and client. Second, we exploit the
network vulnerabilities of Modbus/TCP by launching the Reconnaissance and
DoS attacks followed by the attack cycle to monitor the effects of the system.
We use Metasploit to simulate the attack on a simple test bed. We focus on
Modbus and TCP protocols because they are two of the most widely used network protocols in smart grids. With this method, we analyze the results of the
actions taken to compromise the smart grid’s security. The findings of our study
will be helpful for cyber forensics investigations into the Modbus protocol in the
virtual testbed for the smart grid and projections for the future by overcoming
the limitations.
In particular, the rest of this paper is organized as follows: Sect. 2 discusses the background literature. Section 3 outlined the Modbus/TCP protocol
overview. Section 4 provides the experimental setup, followed by Sect. 5with the
experimental results and analysis. The limitation and future work appears in
Sect. 6. Finally, Sect. 7 concludes this study.
2
Related Works
Recent, pertinent research on security on Modbus in the smart grid has been
studied in depth. We attempted to categorize our findings in this work from
many research viewpoints to explore the Modbus cyber attacks in the SG. The
most relevant work to our research is in the categories of Modbus cyber-attacks,
various detection techniques, including Intrusion Detection Techniques that can
be deployed upon Modbus, and testbed-based countermeasures to prevent cyber
attacks on Modbus.
Modbus Cyber Attacks. A complete theoretical analysis of the multiple
cyber-attacks against the Modbus protocol was published in [3]. The attacks’
Analyze Cyber-Attacks on Modbus Protocol
387
threat categories, targets, and control system asset impacts are summarized. The
attack taxonomies define Modbus’s control system and network security risks,
making formal risk analysis easier. In [5], the authors conducted four distinct
cyber-attacks on the Modbus protocol to compile pertinent data that machine
learning algorithms may use. The authors identified intrusions that share traits
with well-known assaults. They employed neural networks and decision trees to
categorize the traffic produced by this simulated environment. Another review
[9] presents different attack taxonomy as an intrusion on various protocols of the
SG communication protocol with a recommendation of mitigation technique.
The authors in [10] list 17 assaults on SCADA control systems. Attacks fall into
four categories: reconnaissance, response and measurement injection, command
injection, and denial of service.
Detection Techniques. T. Morris et al. published Modbus-related rules
in [8], and popular signature-based IDS like Snort and Suricata can use these.
Researchers developed the rule set for intrusion detection systems during a vulnerability study of the Modbus/TCP and Modbus over Serial Line networks.
A Modbus/TCP fuzzier called MTF was designed and implemented by Voyiatz et al. [12]. Reconnaissance is a step in the MTF testing process that helps
map the capabilities of the tested device and modify the attack vectors. In [14],
the authors describe a series of experiments demonstrating the effectiveness of
a signature-based Snort threshold module and an anomaly-based change detection method for spotting Modbus flooding attacks. In the research [19], the
authors analyzed 37 cases and then looked at the Implementation and Detection
of Modbus in the smart grid paradigm. The IDPS systems are an additional line
of defence that strengthens cryptographic operations by promptly identifying
and preventing potential security breaches. The paper’s authors [20] proposed
a two-fold IDS. They started by researching and improving the cyber-attacks
offered by the Smod pen-testing program. The second step is introducing an
intrusion detection system (IDS) based on anomalies that might spot intrusions
that result in Modbus/TCP denial of service (DoS). The authors in [28] suggested an intrusion detection technique that uses a honeypot for the Modbus
TCP protocol. The proposed solution uses an unsupervised clustering technique
based on similarities between log sequences and cluster attacking patterns.
Testbed Based Solution. A CPS testbed [15] was used to develop the Modbus/TCP Attack-Tree model to exploit several assaults. This model includes
attacks such as replay, man-in-the-middle, denial-of-service, and reconnaissance.
With the integration of assault time, detection time, and plant-hazard generating time, this serves as a formal risk assessment model. The approach in [18] is
the development of a test bed based on the Modbus Protocol of nuclear power
plants for cybernetic attacks analysis. The concept incorporates a simulated
Modbus/TCP network environment with fundamental industrial control features
created using free and open-source software. The paper [11] presented a virtual
testbed for implementing a Man-in-The-Middle (MITM) attack on the Modbus protocol to analyze Modbus vulnerabilities, practical limitations of modern
smart grid systems, and relevant prevention measures. The authors of [23,25]
388
S. Banik et al.
conducted an in-depth analysis of the known and reported network vulnerabilities in the Modbus protocol. They reviewed numerous attempts at exploiting
such network vulnerabilities in creating the virtual test bed. Since the main concept of the most deployed testbed is a cyber-physical system that incorporates
many simulation tools, Typhoon HIL, unfortunately, lacks many important features; for example, TYfoon is not worried about the network as to why it has
no packet analyzed. This is one of the crucial aspects of choosing a VM with all
the necessary tools to assist in conducting any network security-related study or
analysis.
Alternative Countermeasures. An alternative method is [16], which developed the MODLSTM Neural Network architecture, comprising input preprocessing, feature recording, and traffic classification to identify Modbus-related
DoS assaults. A case study presents various countermeasures that can enhance
security for the Modbus/TCP protocol in SCADA systems by studying and
testing the current security countermeasures specific to SCADA systems [21]. A
model for organizing information security in Modbus TCP was presented in [22].
The paper [26] surveys detection and mitigation techniques against denial-ofservice (DoS) attacks on the Modbus protocol in the smart grid. It evaluates
the effectiveness of the current approach and outlines the prospects for future
research. Different machine learning-based anomaly detection techniques were
applied to an industrial Modbus/TCP data set to detect fraudulent traffic in a
synthetically generated Modbus/TCP communication data set in a hypothetical
industrial setting. In [27], S. Anton et al. utilized and evaluated various machine
learning classification techniques for identifying Modbus threats. An approach
has been proposed to cluster cyber attacks targeting Modbus-enabled SCADA
and ICS devices. The network telescope sensors have been deployed to empirically evaluate the approach on one month of real network data. Fachkha et al.
[30] presents a thorough investigation to determine cyber scanning activities into
SCADA Modbus communication.
Different ways of testbed technologies in the SG system and the prominent
features and limitations from selected papers are presented in Table 1 to determine the research’s central concepts. This also highlights the research gap or
limitation which our proposed testbed aims to address.
Table 1. Categorization of the difference approaches
Different
SG testbed
Scalable Reconfigurable Hardware
Required
Vulnerability
Analysis
Vulnerability
Mitigation
Cyber-Physical Testbed [15]
Simulated Testbed [18]
Simulated Testbed [24]
Simulated Testbed [13]
Virtual Testbed [23]
Followed by
the
Attack Cycle
Virtual Testbed [25]
Proposed Virtual Testbed
Analyze Cyber-Attacks on Modbus Protocol
389
To our knowledge, there has not been enough published research on deploying a virtual environment as the SG testbed to carry out vulnerability analysis
on any crucial protocol such as Modbus followed by the attack cycle and in the
literature on virtual, simulating or cyber-physical testbed; nobody is concerned
about the network security. After digging deeper into the recent studies on Modbus security-related works, we focused on conducting the cyber investigation on
Modbus/TCP in a virtual testbed. Regarding network architecture and security
concerns, our work significantly differs from others, which could open a new
avenue to research in this domain.
The following aspects of a virtual machine (VM) have facilitated our work to
serve as a promising virtual testbed for Modbus/TCP that imitates the real-time
scenario.
– VM facilitates the real protocol setup into the Network architecture according
to the experimental requirements. This setup has many tools that allow us
immediate visualization and automation to carry on large-scale experiments.
– VMs allow automated experimental setup given dynamic parameters.
– VM limits the integration of hardware which makes it less complex to experiment with a particular control protocol in the SG paradigm.
– VMs allow analysis both inside the VM as well as outside. The attacker
can compromise analysis tools inside the VM but allow more specific and
detailed analysis. Analysis outside the VM protects the analysis tools from
the attacker.
– VMs isolate the network from dangerous attacks by simulating the network
instead of sending the packets over the actual network.
– The VM has many tools available to do analysis, compared to (examples:
packet analyzers, process image analyzers, memory analyzers, etc.).
– VM can be remotely operated, making the experiments more flexible.
This testbed is developed to resolve the following security issues in Modbus
protocol research.
– Establish a communication network between the Modbus components like the
real-time testbed
– Analysis of the level of vulnerability of Modbus/TCP protocol
– Estimating the security impact on the system due to this exploitation
– Discussing the defense mechanism
3
Modbus/TCP Protocol
Modbus was initially designed for communication in industrial control systems and was developed more than 40 years ago by PLC manufacturer
ModiconTM , which is now Schneider ElectricTM . It continues to be widely used
for SCADA/ICS applications such as smart grid systems. Its use is widespread for
several reasons: it is an open standard, simple to apply, and practically all commercial automation equipment comes with implementing it. Modbus is available
390
S. Banik et al.
in three formats: ASCII, RTU, and TCP. Hexadecimal is utilized in the first one
to encode messages. Despite being slow, it is excellent for radio links and phone
conversations. The Modbus TCP/IP protocol is frequently utilized in industrial
automation control systems. These control systems are crucial in industrial operations such as regulating refineries and gas turbines. It is simple to use and allows
the transmission of unencrypted raw data without encryption or authentication
[6]. This factor, in particular, led to the selection of Modbus/TCP. But years
ago, the protocol was developed without any security consideration. In the second, binary-coded messages are transmitted over RS232. Figure 2 illustrates the
stacks for the Modbus protocol application layer.
Fig. 2. Stacks for the application layer: Modbus [7]
Modbus is a master-slave protocol that enables one master, also known as
a remote terminal unit (RTU) or master terminal unit (MTU), and numerous
slave devices, including sensors, drivers, and PLCs, to communicate with one
another and share instructions in a SCADA system [8]. This OSI model level
seven application layer messaging protocol enables client/server communication
between devices connected to various buses or networks [33]. A master/slave
architecture is employed in Modbus to communicate with multi-drop devices.
Modbus TCP/IP is a request/reply protocol comprising protocol data units
(PDU) and application data units (ADU) [29]. Two-byte registers hold the data.
We read continuous blocks of registers using the Read Holding Register method
to request data from a device via Modbus TCP/IP. A Modbus protocol data
unit can have up to 253 bytes maximum. The Modbus TCP protocol enables
communication between a master and its slaves in a LAN-based Modbus network
and between multiple masters in IP-connected Modbus networks (each with
multiple slaves). By allowing a slave to communicate concurrently with many
masters and a master to have several transactions open simultaneously, Modbus
Analyze Cyber-Attacks on Modbus Protocol
391
TCP expands the capabilities of its serial counterpart [3]. Modbus/TCP masterslave, in other words, client-server communication, is illustrated in Fig. 3.
Fig. 3. Modbus/TCP client-server communication
4
Experimental Setup
In this section, we outline the specifications of the suggested testbed and discuss
the principle that was the foundation for validating our design. The simulation
strategy is software-based, and this virtual testbed component is described and
discussed in detail below.
This paper presents a real-time virtual test bed, as shown in Fig. 4. Our
testbed has been developed in a virtual environment integrating the Modbus
server, client, and attacker.
Our main network consists of three subnetworks which include Modbus Slave,
Modbus Master, and Attacker as shown in the Fig. 4. We employed a virtual
machine to run the Modbus Master Simulator for the master (VM2). First,
we create a simulated Modbus environment and test the connectivity between
the server and the client. The emulated Modbus Master can access and control
those Modbus Slaves through the Modbus Gateway as long as virtual machines
1 and 2 (VM1 and VM2) are connected. Numerous Modbus Slave Simulators are
available. ModbusPal, a Java-based Modbus Slave emulator that works with any
operating system that supports Java, is used in this project [31]. This software
provides a graphical user interface; therefore, creating many Modbus slaves is
simple.
392
S. Banik et al.
Fig. 4. The virtual testbed setup
We execute ModbusPal on Modbus Slave on one virtual machine (VM1)
(which accepts TCP connections on port 502). It emulates a Modbus Gateway
with the virtual machine’s IP address while running ModbusPal on just one
virtual machine (VM1) (accepts TCP connections on port 502). Each Slave we
build using ModbusPal is identifiable by its Unit ID (ranging from 1 to 247).
4.1
Modbus Master and Slave Communication Environment Setup
Modbus is a master-slave protocol where they communicate with each other
through IP addresses [14]. Therefore, attackers can use software such as Shodan
[34] to find and remotely target Modbus installations for Internet-connected
devices. Additionally, because various PLC manufacturers integrate the protocol
into their devices as an option, and those devices respond to external commands
without requiring authentication, they are vulnerable to injection attacks [8].
The Modbus packet has been captured to ensure the communication between
the master and slave employing wireshark in Fig. 5.
5
Results and Analysis
The attack cycle on the Modbus protocol is shown in Fig. 6. The malicious
hackers or attackers frequently follow four stages of the attack cycle to attack
and acquire control of the system through the Modbus protocol. Those stages
are unauthorized access, reconnaissance, scanning, and exploitation.
Analyze Cyber-Attacks on Modbus Protocol
393
Fig. 5. Wireshark captured modbus packets
Unauthorozed Access: In the first step, the attacker can get unauthorized
access to the system by several means, such as manipulating any user login,
temping user login, or installing covert and undetected software.
Reconnaissance Attacks and System Scanning: After obtaining unauthorized access to the system, reconnaissance is the second phase in which the
attacker learns everything about the target. In the third stage, scanning, the
attacker hunts for system network weaknesses. When an attacker targets a victim’s system to learn about its weaknesses, this is known as a reconnaissance
attack. These activities strive to discover the newly opened ports and learn
about the services provided by each port, as well as their shortcomings. During the exploitation phase, the attacker seeks to compromise and gain complete
control of the target by performing a wide range of potential cyber-attacks.
Once the attacker starts exploiting the victim, the fourth step begins. In this
394
S. Banik et al.
Fig. 6. Attacking cycle followed by attackers to exploit the system
virtual testbed, the Modbus/TCP has been the target of automated cyber
attacks scripted in a scripting language.
The two components of this analysis to plan and carry out a reconnaissance
attack on Modbus protocol are as follows:
– 1. Performing a network scan (IP address & MAC address) to discover our
virtual network’s IP addresses, MAC addresses, and open ports using NMAP.
– 2. To determine the registers, coils, and function codes available in the Modbus protocol using S-MOD.
In a reconnaissance assault, by using ’nmap’, the attacker can only get to
know the data and cannot alter or block packets, as shown in Fig. 7 and Fig. 8.
In the second attempt of reconnaissance assault, by using ‘smod’, the attacker
is only able to observe registers, coils, and function codes that are available in
the Modbus protocol as shown in both Fig. 9 and Fig. 10.
It is evident that function codes 1 through 4 allow the attacker to construct
the reconnaissance attack. Thus, it is possible to access the network, allowing
the attacker to see the resource data. However, in a reconnaissance attack, the
function codes 5 through 16 cannot be used by the attacker to alter the values
of registers or coils (because of write type data).
System Exploitation: The control center will receive delayed information from
IEDs during the DoS attack. If the DoS attack blocks the measurement, the
Analyze Cyber-Attacks on Modbus Protocol
395
Fig. 7. NMAP scanning from KALI VM
control center will not have an accurate image of the system’s status. Smart grid
applications in the control center that rely on IED data will malfunction.
In our denial-of-service attacks, the LAND assault and SYN Flood attacks
were developed in Part 1.
The LAND and SYN Flood assaults are two separate kinds of Denial of
Service (DoS) attacks that aim to block access to a system by authorized users.
Several attack strategies and weaknesses are used in each situation, which makes
them distinct. In Part 1 of the DoS attack, we employed the LAND attack by
the following command in the terminal of the Kali Linux machine.
Figure 11 illustrates the outcome of the LAND assault in Wireshark. When
the LAND attack was successful, we discovered that the PLC’s IP address was
faked and that both the source and destination showed the same IP address,
as shown in Fig. 2. Data transfer uses the TCP protocol and the 502 receiving
port. Additionally, it is demonstrated that repetitive network packets are sent
to establish a connection and cause a system crash and denial of service to
legitimate users.
In Part 2 of the DoS attack, we performed the SYN flood attack by writing
the following command in the terminal of the Kali Linux machine.
TCP SYN flood attack tactic was used to flood the Modbus Master to test the
delay-sensitive nature of Modbus TCP [32]. A TCP SYN Flood Attack uses the
TCP three-way connection handshake method to create a dependable session
between a sender and a receiver. The Modbus Master is subjected to a TCP
SYN flood attack, which floods it with TCP 0 connection requests from possible
Modbus clients with forged source IP addresses and arbitrary destination TCP
396
S. Banik et al.
Fig. 8. NMAP Port scanning from KALI VM
ports, according to Breaker Status. The Modbus master was targeted by fake
TCP SYN requests sent using the Hping tool [35].
Figure 12 shows the data collected in Wireshark after a successful SYN flood
attack by initializing the target IP address and port number.
Analyze Cyber-Attacks on Modbus Protocol
397
Fig. 9. Read the function code of Modbus protocol
Fig. 10. Reading holding register and coil status values of the Modbus protocol
5.1
Using Metasploit to Perform a DoS Attack
In reality, Metasploit is a specific type of DoS attack in which the attacker
takes advantage of flaws in the target system to make it crash or stop working.
Penetration testing and exploitation are the primary uses of this Metasploit
attack. It is distinguished by a unique traffic pattern shown in Fig. 13.
We carried out the LAND attack in Part 1 and then the SYN flood attack
in Part 2. They both represent distinct Denial of Service (DoS) attacks. The
LAND attack involves the attacker spoofing the victim’s IP address as the packet
source, sending packets to the victim’s system with the SYN flag set, and setting the destination port to the same port as the source port. As a result, the
victim’s system constantly tries to connect with itself, using up resources and
possibly resulting in system crashes. Furthermore, the SYN flood attack causes
the target system to allocate resources for each connection attempt. It keeps the
resources busy until they time out, which causes the system to become unresponsive or crash. The attacker floods the server with numerous SYN packets
with bogus IP addresses, making it challenging for the server to discriminate
between real and phony connection requests. Another observation is that the
SYN flood attack executes much more quickly than the LAND attack. The main
398
S. Banik et al.
Fig. 11. Packet captured in Wireshark after LAND command
Fig. 12. The captured packet in Wireshark after SYN flood attack
distinction between a LAND assault and a SYN flood attack is that the former
overloads the server with connection requests as traffic. At the same time, the
latter uses a flaw in the victim’s operating system by delivering packets with
a fake IP address. Lastly, the experiment has been accomplished, proving the
virtual simulation is good enough not to hamper any hardware in the loop.
6
Limitation and Future Work
Even though our strategy entails creating a test bed replicating a real-time
smart grid system and studying how the cyberattack on Modbus affects it, it
requires adding more hardware to the loop to identify the attack at a different
network layer. Another limitation is that it is not simulating full devices but
only simulating the mathematical properties of the respective devices. So, the
problem is scalability, as the VM is heavyweight. If we have an SG with many
devices and we spawn the VM for each device, my host will eventually be unable
to simulate all the VMs. It can be overcome by simulating multiple devices within
a single guest VM or having multiple network hosts running multiple guest VMs.
The assessment was only done on a limited scale; it must also be verified on a
larger size. We intend to extend our work performing with more potential cyberattacks, such as man-in-the-middle (MiTM) and false data injection (FDI) on
different protocols, for instance, on WiMax, DNP3 in the future, so that we
may look into the system how it responds to an attack or after the attack has
occurred. To validate, we can use some machine learning tools to verify our work
in the following stage.
Analyze Cyber-Attacks on Modbus Protocol
399
Fig. 13. The captured packet in Wireshark after Metasploit attack
7
Conclusion
This paper highlights the significance of employing a virtual environment for
examining the popular SG protocol, the Modbus, by identifying its vulnerabilities, exploiting them to launch different potential cyber-attacks, and analyzing
their influence on the SG system. Case studies demonstrate that the Modbus
protocol could be more trusting, as it lacks access control lists and a trusted
domain. The successful implementation has overcome the challenge of integrating hardware in the loop, vital in developing any CPS system, such as the SG
system. The effective deployment of the VM in the analysis can be considered
a pivotal contribution in this domain. One of the important findings during the
experiment was that none of the packets entered our campus network. This indicates that the virtual testbed can be considered a secure cyber-security-related
research and analysis platform. Aggregating or visualizing data after running the
experiment overnight is the nicest aspect of the virtual environmental setup. We
analyzed how the Reconnaissance and DoS attacks can severely affect the SG
system’s operation and stability. With the Reconnaissance attack, the attacker
can learn the system data and control commands into numerous smart grid
applications. As a result, the system’s security and stability are compromised,
allowing any attacker who gains knowledge of the IP address of the Modbus
master to target it for launching any malicious attack like DoS to cripple the
entire system.
References
1. Gopstein, A., Gopstein, A., Nguyen, C., Byrnett, D., Worthington, K., Villarreal,
C.: Framework and Roadmap for Smart Grid Interoperability Standards regional
Roundtables Summary Report. US Department of Commerce, National Institute
of Standards (2020)
2. Hahn, A., Ashok, A., Sridhar, S., Govindarasu, M.: Cyber-physical security
testbeds: architecture, application, and evaluation for smart grid. IEEE Trans
Smart Grid. 4, 847–855 (2013)
400
S. Banik et al.
3. Huitsing, P., Chandia, R., Papa, M., Shenoi, S.: Attack taxonomies for the Modbus
protocols. Int. J. Crit. Infrastruct. Prot. 1, 37–44 (2008)
4. Booth, H., Rike, D., Witte, G.: ITL bulletin for December 2013 the national vulnerability database (NVD): overview (2021)
5. Li, S., Huang, Y., Tai, B., Lin, C.: Using data mining methods to detect simulated
intrusions on a Modbus network. In: 2017 IEEE 7th International Symposium On
Cloud And Service Computing (SC2), pp. 143–148 (2017)
6. Goldenberg, N., Wool, A.: Accurate modeling of Modbus/TCP for intrusion detection in SCADA systems. Int. J. Crit. Infrastruct. Prot. 6, 63–75 (2013)
7. Boudier, U.: Design of a Prototype for Inverter Monitoring with SunSpec Modbus
Protocol (2018)
8. Morris, T., Jones, B., Vaughn, R., Dandass, Y.: Deterministic intrusion detection
rules for MODBUS protocols. In: 2013 46th Hawaii International Conference On
System Sciences, pp. 1773–1781 (2013)
9. Banik, S., Banik, T., Banik, S.: Intrusion Detection System in Smart Grid-A
Review (2023)
10. Morris, T., Gao, W.: Industrial control system cyber attacks. In: 1st International
Symposium For ICS and SCADA Cyber Security Research 2013 (ICS-CSR 2013)
1, pp. 22–29 (2013)
11. Banik, S., et al.: Implementing man-in-the-middle attack to investigate network
vulnerabilities in smart grid test-bed. In: 2023 IEEE World AI IoT Congress
(AIIoT). IEEE (2023)
12. Voyiatzis, A.G., Katsigiannis, K., Koubias, S.: A Modbus/TCP fuzzer for testing
internetworked industrial systems. In: 2015 IEEE 20th Conference on Emerging
Technologies & Factory Automation (ETFA). IEEE (2015)
13. Fritz, J.J., et al.: Simulation of man in the middle attack on smart grid testbed.
2019 SoutheastCon. IEEE (2019)
14. Bhatia, S., Kush, N., Djamaludin, C., Akande, A., Foo, E.: Simulation of man in
the middle attack on smart grid testbed. In: Proceedings Of The Twelfth Australasian Information Security Conference (AISC 2014) [Conferences In Research
And Practice In Information Technology, Volume 149], pp. 57–65 (2014)
15. Bashendy, M., Eltanbouly, S., Tantawy, A., Erradi, A.: Design and implementation of cyber-physical attacks on Modbus/TCP protocol. In: World Congress On
Industrial Control Systems Security (WCICSS-2020) (2020)
16. Zhang, H., Min, Y., Liu, S., Tong, H., Li, Y., Lv, Z.: Improve the security of
industrial control system: a fine-grained classification method for DoS attacks on
Modbus/TCP. Mobile Networks and Applications, pp. 1–14 (2023)
17. Banik, S., et al.: Anomaly detection techniques in smart grid systems: a review.
In: 2023 IEEE World AI IoT Congress (AIIoT). IEEE (2023)
18. Brito, I., Sousa, R., Jr.: Development of an open-source testbed based on the
Modbus protocol for cybersecurity analysis of nuclear power plants. Appl. Sci. 12,
7942 (2022)
19. Radoglou-Grammatikis, P., Siniosoglou, I., Liatifis, T., Kourouniadis, A., Rompolos, K., Sarigiannidis, P.: Implementation and detection of Modbus cyberattacks.
In: 2020 9th International Conference On Modern Circuits and Systems Technologies (MOCAST), pp. 1–4 (2020)
20. Radoglou-Grammatikis, P., Sarigiannidis, P.: Securing the smart grid: a comprehensive compilation of intrusion detection and prevention systems. IEEE Access.
7, 46595–46620 (2019)
Analyze Cyber-Attacks on Modbus Protocol
401
21. Luswata, J., Zavarsky, P., Swar, B., Zvabva, D.: Analysis of SCADA security using
penetration testing: A case study on modbus tcp protocol. In: 2018 29th Biennial
Symposium On Communications (BSC), pp. 1–5 (2018)
22. Ametov, F., Bekirov, E., Asanov, M.: Organizing the information security in Modbus TCP interfaces for use in the energy complex. IOP Conf. Ser. Mat. Sci. Eng.
1089, 012007 (2021)
23. Rahman, A., Mustafa, G., Khan, A., Abid, M., Durand, M.: Launch of denial of
service attacks on the Modbus/TCP protocol and development of its protection
mechanisms. Int. J. Crit. Infrastruct. Prot. 39, 100568 (2022)
24. Yang, Y., et al.: Man-in-the-middle attack test-bed investigating cyber-security
vulnerabilities in smart grid SCADA systems, pp. 138–138 (2012)
25. Tesfahun, A., Bhaskari, D.L.: A SCADA testbed for investigating cyber security
vulnerabilities in critical infrastructures. Autom. Control. Comput. Sci. 50, 54–62
(2016)
26. Ortega-Fernandez, I., Liberati, F.: A review of denial of service attack and mitigation in the smart grid using reinforcement learning. Energies 16, 635 (2023)
27. Anton, S., Kapoor, S., Fraunholz, D., Schotten, H.: Evaluation of machine learningbased anomaly detection algorithms on an industrial Modbus/TCP data set. In:
Proceedings of the 13th International Conference on Availability, Reliability And
Security, pp. 1–9 (2018)
28. Wang, P., Liao, I., Kao, K., Huang, J.: An intrusion detection method based on
log sequence clustering of honeypot for modbus TCP protocol. In: 2018 IEEE
International Conference on Applied System Invention (ICASI), pp. 255–258 (2018)
29. Kenner, S., Thaler, R., Kucera, M., Volbert, K., Waas, T.: Comparison of smart
grid architectures for monitoring and analyzing power grid data via Modbus and
REST. EURASIP J. Embed. Syst. 2017, 1–13 (2017)
30. Fachkha, C.: Cyber threat investigation of SCADA Modbus activities. In: 2019
10th IFIP International Conference On New Technologies, Mobility And Security
(NTMS), pp. 1–7 (2019)
31. Admin Modbuspal - a java Modbus simulator. http://modbuspal.sourceforge.net/
32. Chen, B., Pattanaik, N., Goulart, A., Butler-Purry, K., Kundur, D.: Implementing
attacks for Modbus/TCP protocol in a real-time cyber-physical system test bed.
In: 2015 IEEE International Workshop Technical Committee On Communications
Quality And Reliability (CQR), pp. 1–6 (2015)
33. Modbus, I.D.A.: Modbus application protocol specification v1. 1a. North Grafton,
Massachusetts (2004). www.modbus.org/specs.php
34. Shodan Shodan Search Engine (2023). https://www.shodan.io/. Accessed 15 Sept
2023
35. Hping3 hping3(8) - Linux man page (2023). https://linux.die.net/man/8/hping3.
Accessed 15 Sept 2023
Cybersecurity Governance in the Medical
Ecosystem: An Orientation Guide with Specific
Reference to the Merging of IT and OT Devices
Sebastiaan Von Solms(B)
and Jaco du Toit
University of Johannesburg, Johannesburg, South Africa
{Basievs,jacodt}@uj.ac.za
Abstract. Health institutions have started worldwide or are considering integrating the Internet of Medical Things (IoMT) into their hospital environments. However, it is well documented that the IoMT is introducing severe cyber risks into
the relevant environment. It is therefore the responsibility of the Board of Governors (BoGs), as part of their Corporate Governance oversight accountability,
to ensure the cyber resilience of their health institutions during such integration.
Furthermore, it is also well reported that BoGs are not always technically skilled,
and often do not know where and how to start such oversight responsibilities.
The purpose of this paper is to provide some basic readable background about the
IoMT, and then offers a number of questions which the Board can ask to get some
handle on and empower themselves on this oversight journey they are confronted
with.
Keywords: Internet of things · Internet of medical things · Cybersecurity
governance · Information technology · Operational technology
1 Background
The Internet of Things (IoT) is defined in many ways, but basically, it can be seen as a
giant network of connected devices [1].
The IoT can be visualised as an expansion of the well-known Internet. Originally
the Internet was seen as a global network of computers, allowing connectivity between
such computers and their supporting databases. In time, a wide range of different sensors
and devices were designed to capture data from, manage and control different devices,
sensors and machines. These sensors are then connected via network communication
structures to send the captured data to some central point for processing and storage.
This ‘sensor enhanced Internet’ became known as the Internet of Things (IoT) [1].
These sensors and network communication structures can broadly be divided into
two classes, based mainly on the type of data captured and the form of network communication used to transfer the captured data. The first class includes sensors which capture
digital data, and use standardised Internet protocols to transmit the data. This class is
broadly normal Information Technology (IT), digital computers and Internet transmissions. For that reason, this class is called the IT class with IT type devices. The second
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2024
K. Arai (Ed.): SAI 2024, LNNS 1017, pp. 402–417, 2024.
https://doi.org/10.1007/978-3-031-62277-9_26
Cybersecurity Governance in the Medical Ecosystem
403
class includes sensors which are used for managing and controlling operational systems,
generally in a factory setting. Because of the need for 100% uptime, this class often uses
specialised proprietary networks for the transmission of data and specialised industrial
control systems for management and control. For that reason, this class is called the
Operational Technology (OT) class with OT type devices [2]. Through such IT and OT
sensor environments in the medical ecosystem, masses of data are generated – data about
patients, data about possible treatments, data about the success or failure of treatments
and more. This constantly growing data collection cannot be analysed and interpreted
manually – it is too large. More intelligent analysis and interpretation are needed to get
value from this collected data. This is precisely where Artificial Intelligence (AI) comes
in to make sense of this data and provide possible actionable proposals.
As indicated, both classes of sensors (IT and OT) can be connected to communication
infrastructures to send the captured data to more centralised control stations where the
data could be monitored and analysed – as indicated above, possibly by AI systems.
Based on the AI analysis, some follow up actions could then be sent back to the relevant
sensors, allowing a sort of remote control or management of the device or machine to
which the sensor was connected. In the IT environment, this ‘sensor enhanced Internet’
became known as the Internet of Things (IoT). When an IoT is mostly rolled out in the
medical ecosystem, with most sensors related to the medical field, the environment is
known as the Internet of Medical Things (IoMT). In the OT environment, it is known
as the Industrial Internet of Things (IIoT). When AI gets involved, it is known as the
Artificial Intelligence Internet of Things (AIoT).
The IoT can be seen as ‘the connection of devices to the Internet and the connection
between them through sensors [1].
In many cases, the distinction between the IoT and the IIoT is very grey, and there
is a significant overlap between the IoT and the IIoT.
It is well known that using the Internet creates lots of cyber risks. Cyber criminals
also use the Internet as a platform to launch a wide range of cyber attacks on companies
and individuals to steal their money and personal data. ‘IoT devices, in recent times,
are increasingly subject to cyber attacks leading to revenue loss and data loss.’ [3].
Cyberattacks basically take place via the Internet, so the IoT and IoMT are presently
very prone to cyberattacks. With the IIoT now becoming more and more integrated with
corporate IT systems, cyberattacks in the IIoT and OT areas are growing [4, 5].
Cybersecurity is the discipline to help protect against cyber-attacks. Cybersecurity
Governance is the discipline responsible for overseeing the cyber risk of an enterprise, to
ensure the cyber resilience of the enterprise and to provide the strategic and policy framework for protecting an enterprise against cyber attacks [6, 7]. Cybersecurity Governance
is a core element of Corporate Governance, for which the Board of Governors, or Board
of Directors, (executive management) of a company is responsible and accountable. We
will refer to the Board of Governors (BoGs) [8].
Cybersecurity Governance has become established as Boards realised that ensuring
the cyber resilience of their enterprises, and ensuring that their enterprises are protected
against cyberattacks, is a part of their Corporate Governance oversight and accountability. Legal and regulatory developments have also helped to embed the aspect of
Cybersecurity Governance on Board agendas.
404
S. Von Solms and J. du Toit
However, reports indicate that Cybersecurity Governance in the IoT still has a long
way to go. ‘IoT governance is falling behind’ [9].
As the idea of the IoT is only a few years old, Cybersecurity Governance in this area
is still developing. As usual, the technical development is faster than the development
of Governance guidelines. ‘The term IoT governance is still in its early stages and there
are no definitive limits on what IoT governance should include, or which areas it should
cover. Hence, it is still not clear which areas should be addressed by any IoT governance
framework that will evolve’ [1].
When the devices connected to and controlled within the IoT are primarily related
to the medical field, it is called the Internet of Medical Things (IoMT). The IoMT can
therefore be seen as a subset of the IoT, and is exposed to all the cyber risks of the IoT.
‘Since medical devices are not designed with security in mind, they are often vulnerable
to cyberattacks, and have become a growing target of cybercrime.’ [10].
While, as stated above, Cybersecurity Governance in the wider IoT still has a long
way to go, it seems that Cybersecurity Governance in the IoMT still has an even longer
way to go. Most Boards of Governors are not up to date with the complexities and risks
related to the IoMT. There seems to be a lack of understanding and appreciation (a gap)
amongst healthcare executives (about) the (business and patient) risks of cyber attacks
and cyber breaches (within the IoMT) [11].
Cybersecurity Governance in the IoMT has a wide range of very complex aspects
which must be governed properly, as bad or no governance can result in risks to patients,
even resulting in loss of life. ‘Security is one of the top sticking points for healthcare
executives in implementing or expanding Io(M)T solutions.’ [12].
Two other developments closely related to the whole field of the IoMT, is the so
called convergence of Information Technology (IT) and Operational Technology (OT)
in the medical ecosystem, as well as the role of Artificial Intelligence (AI) and the impact
of AI in the IoMT. This makes the task of Governance in this area even more complex,
and members of the BoG often do not know where and how to start executing their
governance role in this area of converging technologies.
This paper intends to address specifically this converging process and tries to
empower members of a BoG to be able to start this journey in which they must play a
crucial role.
The paper is structured as follows:
• Section 2: The Objective of the Paper.
• Section 3: Related work.
• Section 4: IT in the IoT and IoMT.
• Section 5: OT in the IoT and IoMT.
• Section 6: AI in the IoMT – The AI-based IoT (AIoT).
• Section 7: The convergence of IT and OT in the IoT and IoMT.
• Section 8: Where to start.
• Section 9: Some positioning questions to be used by the Board.
• Section 10: Summary and Conclusion.
Cybersecurity Governance in the Medical Ecosystem
405
2 The Objective of the Paper
As stated above, Cybersecurity Governance in the IoT and therefore also in the IoMT, is
still in its initial stages. However, Boards of Governors (BoGs) need a good understanding
to get a proper handle on this matter. Although numerous technical publications are
available, a literature study could not identify a management-oriented guideline for such
Boards to explain what their governance oversight and accountability role as far as
operating in the IoMT includes, and to provide some guideline questions to ask. It was
especially the case in the development of the convergence of IT and OT in the medical
environment. This is an area which is regularly highlighted as having inherent cyber
risks. It is crucial that Boards understand this merging process, as it has the potential to
create serious cyber risks to the hospital.
‘The successful convergence of IT and OT is increasingly part of the Board’s agenda’
[1].
The objective of this paper is to be a starting point for Boards in the medical ecosystem in their journey to get a more comprehensive understanding of what the whole
development of the IoMT means, what the cyber risks are, and what questions to ask to
start this governance journey. The paper concentrates specifically on IT and OT convergence and the role of AI. The purpose is therefore to provide some form of an orientation
and guidance document:
• to help Boards to orientate themselves as far as cybersecurity governance in the
medical ecosystem is concerned
• to help Boards to understand what the whole IT/OT convergence process in the
medical ecosystem means,
• to make Boards aware of what the role of AI can be
• to provide some pertinent questions to start their understanding process
• to evaluate the risks of such a convergence process when they have to make decisions
on moving into an IoMT environment, or adding new sensors and devices to their
existing IoMT and
• to understand how and why the whole environment is actually part of their Governance
oversight and accountability responsibilities.
This orientation and guidance document will therefore eventually be a component
of a wider guidance document covering other aspects and dimensions of the IoMT, but
which is outside the scope of this paper. The part presented in this paper is primarily to
allow a Board that has very little background and experience in the Governance process
in the IoMT, to get started. It is not a technical document to convey technical details,
but more an orientation document for a Board to empower themselves in their job of
Cybersecurity Governance and oversight.
Because many of the problems and risks arising from moving into such an IoMT
environment are generic and the same as moving into a wider IoT environment, many
of the references in this paper will come from IoT publications. Another reason is that
there are not so many non-technical material specifically for Board members in the
IoMT environment specifically, presently available. However, specific aspects related to
the IoMT environment, not necessarily included in the general IoT environment, will be
addressed where relevant.
406
S. Von Solms and J. du Toit
Therefore, the paper intends to develop an orientation guideline that can help Boards
ask the right questions when confronted with creating or expanding an existing IoMT
environment and add to the Body of Knowledge in this area.
The next section provides an overview of some of the work related to this research
and highlights the gap that exists between the existing research and the objective of this
paper.
3 Related Work
This section aims to highlight the existing research related to cybersecurity governance
in the medical ecosystem, with the aim of highlighting the gap the research addressed
in this paper. The section specifically looks at cybersecurity governance-related work in
the medical environment, specifically emphasising governance activities related to the
convergence of Information Technology and Operational Technology.
Jalai, Rzaka, Gordon, Perakslis and Madnick [13] did an extensive bibliometric
analysis of health care and cyber security. One of the results from the study showed that
non-technical focused cybersecurity research is understudied. The authors specifically
mention non-technical aspects such as human-based and organisational aspects, strategy
and management.
A study by Bhagat [14] highlighted various cyber threats and vulnerabilities in existing IT and OT environments. The research highlights vulnerabilities that exist because
of the convergence of Industrial Control Systems into the corporate network, but does
not guide top management on how to address the identified threats and vulnerabilities.
Thomasian and Adashi [15] found that policies in medical organisations need to
be adapted to address IoT threats. Furthermore, they highlight that organisations must
include stakeholders from both the system and end users in their preparations. The
research makes various recommendations on how to address specific challenges. The
recommendations are, however, focused on middle management, whereas the research
objective in this paper focuses on aspects that can be consumed by top management.
Skierka [16] found that management safety and security risks in medical systems
require improvement in regulations and other industry standards. Improvement in regulations and industry standards will force cyber security risks on the agenda of top
management. The research in this paper will assist organisations in addressing the cyber
security risk agenda item.
As mentioned in Sect. 1, a literature review could not identify specific managerialrelated guidelines for Boards. The research in this paper addresses this requirement. The
next section builds upon the understanding that the Internet of Medical Things (IoMT)
is an Information Technology aspect, but later on, it becomes clear that there is also
Operational Technology (OT) aspects in IoMT.
4 IT in the IoT and IoMT
The use of Information Technology (IT) in the medical environment is well known
and well established. Basically IT consists of the use of computer hardware (desktops,
tablets, smartphones, servers and more), using software and connected via networks
Cybersecurity Governance in the Medical Ecosystem
407
to store, retrieve, share, process and organise information. The IT people operating
such a hospital IT environment are usually software programmers, system designers,
network technicians, operating system experts and more. A wide range of users use such
a system through workstations connected to the hospital’s communication network.
Information Security Governance in the field of Medical Informatics is well established.
As an example, a paper, ‘Security and confidentiality in healthcare informatics’ was
presented at a conference already in 1998 [17]. That was years before the concepts of
IoT and IoMT were defined.
In recent years, more and more devices and sensors were connected to the hospital’s
IT system via the Internet, allowing better patient care, administration and management.
5 OT in the IoT and IoMT
Again, the use of Operational Technology (OT) is also well established. Basically, the OT
environment is used in industrial environments where all types of machines are connected
to a wide variety of endpoint sensors. However, as more and more such machines are
integrated into the medical environment, the medical ecosystem historically has its share
of OT systems and sensors. Examples of these types of machines in the hospital’s OT
environment are usually on the medical side, like MRI equipment, monitoring tools,
ventilators, syringe pumps and more, or on the building side, like energy management,
elevators, air conditioning, water and oxygen pipes and more [18].
‘In a hospital scenario this (OT environment) includes many important technologies
and medical devices used for the continuation of healthcare operations. A basic example
of this is an insulin pump.’ [19].
The people active in this OT environment include engineers, bio-engineers and programmers who often program such hardware sensors directly. Traditionally, OT equipment is installed in physically secure environments, resulting in less emphasis on security. Such OT technical staff is not usually integrated with the business environment and
the business’s IT environment, and they use dedicated, very often proprietary forms of
network protocols to transmit data.
‘Historically, OT security was limited to protection of the physical plant because OT
systems were not connected to the internet. Strong perimeter gates, and human-based
access controls, such as security guards, were the standard and highly visible deterrents
to intrusions. The protection of the technology was highly conspicuous.’ [20].
Up until 2010, nobody was really worried about security for OT systems [21].
However, this is changing fast as the OT environment has become a specific target
for cyber attackers. In many cases, this happens via third party companies who manage
this OT environment.
‘Most attacks against the healthcare sector target third-party service providers, who
have extensive access rights to perform maintenance and upgrades to hospital (OT)
equipment.’ [18].
One reason for this attack avenue is that OT equipment was not originally designed
with security in mind, as mentioned above. ‘Furthermore, OT devices have not always
been designed with cyber security in mind or run by cyber security professionals, thus
making them even more vulnerable.’ [22].
408
S. Von Solms and J. du Toit
Originally the IoT was basically using the Internet, and using the standard Internet
communication protocols. As the OT environment was not based on the traditional
Internet protocol, it was not originally seen as part of the IoT development. The two
environments stayed separated.
Because of the growth of the Internet, and the financial benefits of standardising networking technologies, IT and OT systems started to converge by using Internet protocols
as a standardised communication structure. Apart from standardising the infrastructure
environment, hospital managements wanted to a getter overview over their OT environment and include them in their normal management system that was a main reason for
the IT/OT merger initiatives.
6 AI in the IoT and IoMT: The AI-Based IoT (AIoT)
Artificial Intelligence is one of the technologies which is seen as part of the 4th Industrial
Revolution. There are many definitions for AI, but basically, AI can be seen as ‘the
simulation of human intelligence processes by machines, especially computer systems.’
[23].
A big component of the AI environment is the aspect of Machine Learning (ML)
which allows AI systems to be ‘enhanced and enriched’ by adding new data to the
relevant AI system.
AI already play a major role in many systems and processes, but we will very briefly
look at some uses of AI in the IoMT in order to help a Board to evaluate the impact as
far as their Governance accountability is concerned.
Logically the main purpose of the IoMT can be seen as the collection of patient data,
as well as other types of data, from the medical environment. This collection process
creates massive amounts of data, making it humanly impossible to analyse the data to
extract any value from it. Other more sophisticated ways and technologies are needed
to do this. ‘You could face the problem that you are collecting massive amounts of data
that the traditional hospital cannot analyse. That is why it is important to support the
development of AI to improve the processing of the data for diagnosis purposes’ [24].
It is said that we can see ‘(The) Internet of things devices as the digital nervous
system while artificial intelligence is the brain of a system’ [25].
However, AI should not be seen as an ‘external’ component just analysing captured
data – many medical device sensors have built-in AI technology. ‘Artificial intelligence
(AI) is quickly being adopted in the medical field, meaning many medical devices are
starting to incorporate AI capabilities. These AI capabilities include enhanced imaging systems, smart robots, wearable technology, AI-based data analysis, simulation
platforms and more’ [26].
AI can, and is, also used by cybercriminals, meaning that the use of AI can result in
cyber risks [27].
It is therefore important that Boards are aware of the possible role of AI in their
medical environment and should be able to ask some sensible questions about that role.
The use of AI in IT environments has also been adopted to improve and enhance OT
environments. The cross-over from IT to OT or OT to IT and blurring boundaries are
discussed in the next paragraph.
Cybersecurity Governance in the Medical Ecosystem
409
7 The Convergence of IT and OT in the IoT and IoMT
Traditionally there had been a big divide between the IT and OT environments and the
two seldom met or overlapped [28].
In the last few years, the type of use of OT in the industrial environment has become
more and more useful in the medical environment. Different sensors which could help
to capture, measure and monitor critical patient data, were developed. As most of these
sensors still operate in the OT environment, separate IT and OT environments developed
in hospitals. In time it became clear that it would be operationally and financially beneficial to converge or merge these two environments, so that the OT sensor environment
can become part of the hospital’s management environment. Such convergences also had
some other benefits which will be discussed later. This convergence, however, increased
the cyber attack area, and allowed cyberattacks to increase.
‘Although the benefits of such integration are appealing, there are risks that need to
be managed if business are to protect their assets from cyberattacks’ [28].
This resulted in the so called ‘convergence of IT and OT’. Basically, it is nothing
more than ‘combining’ IT and OT. ‘IT and OT both increasingly depend on connections
to the internet or public networks’ [29].
In the IT field, communication between computing devices is and always has been
part of the development of IT – standardising on an Internet protocol. In OT, this inherent
connection and communication had not traditionally been part of the OT environment.
Where they were connected, it was mostly over closed proprietary networks not using the
IT Internet protocol. Furthermore, many OT devices had not really been computerised
[30]. As the IoT model developed, it also impacted the OT area, carrying many of the
IoT cyber risks to the OT environment.
‘Security is by far the biggest challenge to IoT deployments’ [31].
The same IoT risks are now affecting the IoMT. Cyber risks arising and resulting
from the IT/OT convergence and AI.
There are a number of risks in IT/OT environments. The use of AI may also increase
the complexities of the risks. This section first discusses the risks of IT/OT and then
discusses the risks of AI.
7.1 Cyber Risks Arising and Resulting from the IT/OT Convergence
IoT technology can basically be seen as providing the technical bridge to drive the
convergence of IT and OT. Basically OT devices collect data but were never designed
to transmit the captured data over large networks, specifically not the Internet. To get
the benefits of this data captured by OT devices, it must be integrated with existing
IT systems which have always been capable of processing data and providing in-depth
analysis of the captured and other data [30].
This ‘forced marriage’ between IT and OT does cause several risks. Some of the
cyber risks which can arise from this IT/OT convergence are:
• OT sensors and devices which has no or very little built-in security
• Insecure communications between OT devices and control stations
• Incompatibility between IT Internet network protocols and OT network protocols
410
S. Von Solms and J. du Toit
• Unmanaged and unpatched OT devices
Four principal security threats to IT/OT convergence, as highlighted by Bigelow and
Lutkevich [30] are as follows:
• Lack of collaboration between IT and OT teams. As these teams traditionally worked
separately, mutual insight could cause security oversights.
• Legacy OT systems. OT systems are traditionally very stable, and could operate
unchanged for years on end. IT systems had to be constantly patched and updated.
This caused problems in integrating older OT legacy systems
• Insufficient insight. Asset (device) visibility and management is core to the IT
environment, but was not so high-profiled in the OT environment.
• Mission-critical demands. While IT systems must often be interrupted for software
updates and other reasons, OT systems are designed to operate 24/7 and, in most
cases, are so critical that they cannot be interrupted.
All reports indicate that operating in an IoT, including an IoMT, will increasingly
result in growing cyber risks. The challenge for Board members and Governors of medical institutions, will therefore, amongst many other challenges, be to get some idea of
the cyber risks caused by this integration in the IoMT.
7.2 Cyber Risks Arising and Resulting from the Use of AI
‘As artificial intelligence (AI) becomes increasingly embedded worldwide, fresh
questions arise about how to safeguard countries and systems against attacks.’ [32].
AI as a technology can present several cyber risks to computer systems. Some of
these risks are discussed in ‘Top 5 Security Threats Facing Artificial Intelligence and
Machine Learning’ [33].
We will highlight one such risk, namely manipulating the system to make false and
wrong predictions and conclusions deliberately. By creating programs or algorithms
which are biased in some way, serious problems can arise.
It is by far not the purpose of this sub paragraph to convey the full complement of
cyber risks in using AI to Board members – this sub paragraph is just to make such
members aware that AI is already used in medical devices and systems. Cybersecurity
Governance in the IoMT must take note of this fact, and Boards must get an understanding
of how comprehensive this is already implemented in their area of governance and
accountability, and how to get a handle on it. ‘Specifically, boards need to understand
the implications of AI, as they relate to strategy, risk, ethics, compliance, social impact,
performance monitoring and public reporting’ [34].
8 Where to Start
As indicated above, ‘The successful convergence of IT and OT is increasingly part of
the Board’s agenda’ [1] and ‘Specifically, boards need to understand the implications
of AI, as they relate to strategy, risk, ethics, compliance, social impact, performance
monitoring and public reporting’ [34]. This is primarily because of the potential cyber
risks caused by the growing IoT and IoMT.
Cybersecurity Governance in the Medical Ecosystem
411
In many cases, such Board members have no expertise and knowledge of the whole
IT, OT, AI and IoMT environment – in many cases, they do not even have an idea of
what questions to ask. The normal procedure usually is that a Chief Cyber Security
Officer (CCSO) or Chief Information Security Officer (CISO) will brief the Board. It is
well reported that this approach is also often unsuccessful because of the inability of the
CCSO and CISO to explain technical aspects in a way understandable by non-technical
Board members [35].
In some cases, the Board may have a Cybersecurity expert on the Board, but in many
cases not. The challenge to such a Board is very often to just ask some questions to the
CCSO to get the discussion going. Furthermore, the whole OT environment was never
part of the job description of the normal CCSO or CISO, but that is now suddenly dumped
in his or her lap. This complicates the feedback to the Board even more. Notwithstanding
these complexities, it is still the Board’s oversight responsibility to empower themselves
to get a handle on this whole IT/OT/AI convergence road. For Boards in the Medical
ecosystem, it is even more important, as most probably some convergence has already
taken place in the hospital environment, which the Board is unaware of the resulting
risks.
The next paragraph tries to give such Board members some platform to start their
governance empowerment journey. Very few health institutions do not already have
some devices and sensors connected to the enterprise IT system, so the questions are
formulated for a Board that wants to get some handle on the whole situation by asking
their own questions.
There can be several ways in which the set of questions below can be used by the
Board. A few examples are given below.
• Example 1: In a Board meeting, let the Chief Cyber Security Officer (CCSO) or
Chief Information Security Officer (CISO) take the Board through every one of the
questions, and provide relevant answers to the questions. In cases where there are
not clear answers, the Board may request the CCSO/CISO to report back at the next
meeting
• Example 2: Create a sub-committee of the Board and let the CCS/CISO follow the
steps in Example 1. Report the outcome to the next Board meeting.
• Example 3: If the institution does not have a CCSO or CISO, get an outside consultant
to perform the actions mentioned in the two examples above.
The main purpose of the questions is to give an objective initiative to the Board and
not let another party, having hidden motives, provide input and feedback which may be
subjective.
9 Some Positioning Questions
This section formulates a list of questions that can be used by a hospital’s Board of Governors (BoG) when creating or expanding an existing IoMT environment. The section is
grouped into five different sub-sections. Each sub-section covers either some fundamental principles of understanding or is grouped into generally accepted governance topics.
The subsections are:
412
S. Von Solms and J. du Toit
• Section 9.1 (Fundamental Concerns) (Table 1) provides a group of questions that
allow the Board to position itself in the IoMT landscape.
• Section 9.2 (Risk Management) (Table 2) provides a group of questions that allow
the BoG to get a handle on some of the basic cyber security risk aspects relevant to
the governance level – basically some risk management information
• Section 9.3 (Organization) (Table 3) provides a group of questions that allow the
BoG to evaluate potential organisational structures to consider when addressing the
oversight and management of an environment that includes IoMT.
• Section 9.4 (Cybersecurity Culture) (Table 4) provides a group of questions that allow
the BoG to understand the important function of establishing a cyber security culture.
• Section 9.5 (Policies) (Table 5) provides a group of questions that remind the BoG
that organisation policies may need to be updated or created when adopting or using
IoMT, and that the BoG has a crucial role in this aspect.
9.1 Fundamental Concerns
The set of questions in this section is to determine whether IoMT is a fundamental
concern for the Board.
9.2 Risk Management
Risk management is an overall board responsibility. The Board must ensure that proper
Risk Management occurs in the organisation. Risk Management is usually an established
approach in the medical ecosystem to evaluate the risk resulting from equipment, staff,
finances and more arising in a typical hospital environment. Such Risk Analysis and
Management must be expanded to include the IoMT environment. The questions in the
following table may prove valuable to ensure that IoMT-specific risks are included in
the risk management strategy.
9.3 Organisation
The organisation must have the correct organisational structures and functions in place to
address IoMT concerns. The Board must ensure that there are organisational structures
to support cyber security. This is normally done by ensuring that all departments focus
on cyber security, that people are employed with the correct cyber security skills and
that there are proper oversight functions to ensure compliance. The IT/OT convergence
creates a complex environment where it may be unsure where the responsibility of IoMT
resides.
9.4 Cybersecurity Culture
A cyber security culture means that employees use information systems with high maturity and awareness. They know and understand their responsibilities related to cyber
security. However, employees may not normally associate IoMT as part of the cyber
security culture. The Board is responsible for ensuring that the cyber security culture
also includes using IoMT.
Cybersecurity Governance in the Medical Ecosystem
413
Table 1. Fundamental questions
Number Question
Reason
A1
Do we have devices and sensors
connected to our hospital’s enterprise IT
system?
This is the absolute first question to start
the journey
A2
Do we have an IT and OT based sensor
environment?
This is to identify if any convergence has
started already
A3
If YES on A1 or A2 or both above:
Do we have an Asset Register indicating
at least the
• location,
• type (IT or OT),
• responsible owner of every such device
and
• supplier/manufacturer of such a device
Visibility of all sensors/devices are
essential, as you cannot control
something if you do not know it exists,
and for Governance purposes, it is
essential to know precisely who part of
the hospital’s IT system is
A4
Does the Asset Register indicate if a
device has computing capacity, and has
built-in security features?
Devices and sensors with computing
power must have some operating system
that must be updated and patched
continuously. If the device has built-in
security features, it must be managed and
monitored to ensure compliance
A5
Is there a formal process of change
Such a process ensures the integrity of the
control (changing, registering and
enterprise IT system regarding connected
deregistering a sensor or device) when
and disconnected sensors and devices
devices are changed, connected and
disconnected from the enterprise system?
A6
Is AI integrated into any system,
This is the basic question to understand
equipment, or sensors in the hospital’s IT where AIis used
system?
A7
If ‘YES’ on A6, do we have a list of such This answer will provide the Board with
integration and where it occurs?
some platform to investigate this matter
at a later stage
9.5 Policies
Directives from the Board may be cast into specific organisational policies. The Board
may create directives because of internal factors, such as identified risks, or external
factors, such as compliance and legal regulations. In some cases, the use of IoMT
may fall directly under industry compliance or even certain medical or privacy regulations. The Board must consider the impact of risk, compliance and regulations on their
organisational policies.
414
S. Von Solms and J. du Toit
Table 2. Risk management questions
Number Question
Reason
B1
Does the relevant supplier have remote
Unmanaged remote access to an IT
access to the supplied sensors/devices for system is not good practice
maintenance purposes, and is there a
cybersecurity agreement governing this
access?
B2
Do the risk management processes
include the cyber security risks relevant
to IoMT environment?
B3
Do current risk mitigating factors include
controls based on industrial cyber
security best practices, such as zoning
and secure communication?
The relevant employees may only
identify risks in specific environments. It
is important to ensure that cyber security
risk management considers cyber
security risks identified by top, middle
and lower management
Table 3. Organisation based questions
Number Question
Reason
C1
Is there a current responsible person on
the Board that may expand their
responsibilities to become a ‘champion’
related to the cyber security of IoMT?
Cybersecurity governance is an important
priority for the BoG. The cyber security
governance responsibility will become
more complicated when IoMT is used in
an environment. The IoMT environment
must be part of the governance landscape,
and expertise is crucial
C2
Is there an IT/OT/IoMT Cooperation
Committee with hospital IT department
members and the OT sensor and devices
suppliers? Do this Committee report to
the Board in some way? Does the
Committee include experts in
Cybersecurity, Internet communications,
the safety aspect of connected devices,
and ethical and legal aspects mainly to
resolve conflicts that can arise?
Cooperation between IT and OT must
exist to prevent misunderstanding and
cyber risks. The BoG must stay informed
of the outcomes and decisions of this
Committee and ensure the integrity and
data security of information shared by all
IoT devices in the enterprise network.
‘Designing an integrated Io(M)T solution
requires input from internal and vendor
experts.’ [12]. Furthermore, some devices
may have life safety risks for patients,
and therefore ethical and legal aspects
must be taken into account
Cybersecurity Governance in the Medical Ecosystem
415
Table 4. Cyber security culture based questions
Number
Question
Reason
D1
Does the current cyber security
awareness programme include the
domain of IoMT?
Litigation against organisations not using
IT safely and securely is a risk normally
mitigated with awareness programmes. The
IoMT environment also expands the cover
of awareness programmes to improve the
cyber security culture in the IoMT
environment
Table 5. Policy related questions
Number Question
Reason
E1
Compliance and regulations may have a
different impact on different IoMT
devices. In some cases, IoMT devices
must comply with certain standards,
while in other cases, the data generated
by IoMT devices may be very personal.
The Board needs a basic understanding
of the exposure to non-compliance
Does the IoMT responsible person
understand compliance or regulations
related to the different IoMT devices and
the various policies affected by the
different regulations?
10 Summary and Conclusion
Boards of Governors in the medical ecosystem have a role of growing responsibility
and accountability in ensuring the resilience of their institutions against cyberattacks.
The convergence of IT and OT and the impact of AI only makes the responsibility more
complex. This paper tried to give such Boards some background of relevant developments
and provide some questions to get a foot in the door.
References
1. Morar B., Barkawie Y., Balakrishnan R., Khasawneh M., Bangara J., Abu Baker H.: IoT
governance, pp. 1–15 (2021). https://www2.deloitte.com/content/dam/Deloitte/xe/Docume
nts/technology/me_IoT-Governance.pdf
2. Kuusk, A.G., Gao, J.: Consolidating people, process and technology to bridge the great wall
of operational and information technologies, 1715–1726 (2015)
3. Kandasamy, K., Srinivas, S., Achuthan, K., Rangan, V.P.: IoT cyber risk: A holistic analysis
of cyber risk assessment frameworks, risk vectors, and risk ranking process. EURASIP J.
Info. Sec. 8 (2020). https://doi.org/10.1186/s13635-020-00111-0
4. KPMG: Industrial internet of things (IIoT) cyber security. https://home.kpmg/xx/en/home/ser
vices/advisory/risk-consulting/cyber-security-services/industrial-internet-of-things-cybersecurity.html (2023). Accessed 26 January 2023
416
S. Von Solms and J. du Toit
5. Santagate, J., Glaisner, R., Westervelt, R.: Operational cybersecurity for digitized manufacturing: Emerging approaches for the con-verged physical-virtual environment. (2019)
6. Von Solms, B., Von Solms, R.: Cybersecurity and information security–what goes where?
Info. Comp. Sec. 26, 2–9 (2018)
7. Yusif, S., Hafeez-Baig, A.: A conceptual model for cybersecurity governance. 16, 490–513
(2021). https://doi.org/10.1080/19361610.2021.1918995
8. World Economic Forum: Cyber risk and corporate governance. https://www.weforum.org/
projects/cyber-risk-leadership-and-corporate-governance (2023). Accessed 26 January 2023
9. Ismail, N.: IoT governance: How to deal with the compliance and security challenges. https://
www.information-age.com/iot-governance-compliance-security-challenges-16439/ (2022).
Accessed 26 January 2023
10. Millington, J.: Securing IT, OT, IoT, and IoMT in healthcare. https://www.frontier-enterprise.
com/securing-it-ot-iot-and-iomt-in-healthcare/ (2022). Accessed 26 January 2023
11. He, Y., Aliyu, A., Evans, M., Luo, C.: Health care cybersecurity challenges and solutions
under the climate of COVID-19: Scoping re-view. J. Med. Internet Res. 23, e21747 (2021).
https://doi.org/10.2196/21747
12. Lenovo: IoT in healthcare (2019)
13. Jalali, M.S., Razak, S., Gordon, W., Perakslis, E., Madnick, S.: Health care and cybersecurity:
Bibliometric analysis of the literature. J Med Internet Res 21, e12644 (2019). https://doi.org/
10.2196/12644
14. Bhagat, C.K.: Study of current cybersecurity threats to information & operational technology
(IOT) and their effect on e-governance in Nepal. J. UTEC Eng. Manage. 1, 41–50 (2023).
https://doi.org/10.36344/utecem.2023.v01i01.005
15. Thomasian, N.M., Adashi, E.Y.: Cybersecurity in the internet of medical things. Health Policy
and Technology 10, 100549 (2021). https://doi.org/10.1016/j.hlpt.2021.100549
16. Skierka, I.M.: The governance of safety and security risks in connected healthcare. Living in
the Internet of Things: Cybersecurity of the IoT 2018, 1–12 (2018)
17. Al-Salqan, Y.Y.: Security and confidentiality in healthcare informatics. In: Proceedings Seventh IEEE International Workshop on Enabling Technologies: Infrastructure for Collaborative
Enterprises (WET ICE ‘98) (Cat. No.98TB100253), pp. 371–375 (1998)
18. Wallix: OT in hospitals: The achilles heel of the healthcare industry. https://www.wallix.
com/blog/ot-in-hospitals-the-achilles-heel-of-the-healthcare-industry (2023). Accessed 26
January 2023
19. Lanner Electronics Canada Ltd: Need of proactive IT/OT security strategy in healthcare.
https://www.lanner-america.com/blog/healthcare-network-ot-security/ (2022). Accessed 26
January 2023
20. Pratap, G.: OT security: Risks, challenges and securing your environment. https://www.
tripwire.com/state-of-security/ot-security-risks-challenges-and-securing-your-environment
(2021). Accessed 26 January 2023
21. Lekidis, A.: Trends and security challenges from the IT/OT con-vergence. https://cyb
erstartupobservatory.com/trends-and-security-challenges-from-the-it-ot-convergence/ (ND).
Accessed 26 January 2023
22. RSM: Operational technology (OT) security in healthcare (2021)
23. Burns, E., Laskowski, N., Tucci, L.: What is artificial intelligence (AI)?. https://www.tec
htarget.com/searchenterpriseai/definition/AI-Artificial-Intelligence (2023). Accessed March
2023
24. Saltmarsh, A.: The “AIoT” and its role in the future of healthcare. https://emag.medicalexpo.
com/the-aiot-and-its-role-in-the-future-of-healthcare/ (2023). Accessed March 9 2023
25. Kumar Dhaka, R., Panda, S.: AIOT in Healthcare systems: a Thought on Approaches and
Criticism (2022)
Cybersecurity Governance in the Medical Ecosystem
417
26. GlobalData Healthcare: Regulating medical devices with AI capabilities. https://www.med
icaldevice-network.com/comment/medical-devices-ai/ (2022). Accessed 9 March 2023
27. Violino, B.: Artificial intelligence is playing a bigger role in cybersecurity, but the bad
guys may benefit the most. https://www.cnbc.com/2022/09/13/ai-has-bigger-role-in-cybers
ecurity-but-hackers-may-benefit-the-most.html (2022). Accessed March 9 2023
28. Katuruza, P.: IT-OT convergence: Managing the cybersecurity risks. https://gca.isa.org/blog/
it-ot-convergence-managing-the-cybersecurity-risks (ND). Accessed 27 January 2023
29. ZScaler Inc.: What is OT security?. https://www.zscaler.com/resources/security-terms-glo
ssary/what-is-operational-technology-ot-security (ND). Accessed 27 January 2023
30. Bigelow, S.J., Lutkevich, B.: What is IT/OT convergence? everything you need to
know. https://www.techtarget.com/searchitoperations/definition/IT-OT-convergence (2021).
Accessed 27 January 2023
31. Ismail, N.: Security is a great concern with IoT deployment, but what about edge
security?. https://www.information-age.com/security-iot-deployment-edge-security-14707/
(2019). Accessed 27 January 2023
32. ITU: Cybersecurity challenges in the AI age. https://aiforgood.itu.int/cybersecurity-challe
nges-in-the-ai-age/ (2022). Accessed 9 March 2023
33. Dror, N.: Top 5 security threats facing artificial intelligence and machine learning. https://
hubsecurity.com/blog/cyber-security/security-threats-for-ai-and-machine-learning/ (2021).
Accessed 9 March 2023
34. Hubel Labs: The top 5 AI resources for boards. https://hubel-labs.com/blog/the-top-5-ai-res
ources-for-boards (2022). Accessed 9 March 2023
35. Ackerman, R.J.: CISOs and their boards of directors: Viewing cyber risk differently. https://
www.tripwire.com/state-of-security/cisos-and-their-board-directors-viewing-cyber-risk-dif
ferently (2023). Accessed 27 January 2023
The Impact of Utilising the Amazon AWS
Hybrid Deployment Model on Assuring a Secure
Migration of a Commercial Web Application
into the Cloud
Khalied Koorowlay1 and Rafid Al-Khannak2(B)
1 High Wycombe, Buckinghamshire HP11 2UA, UK
khalied.koorowlay@cipherlogik.com
2 Computing Department, CDI, Buckinghamshire New University, High Wycombe, UK
Rafid.Al-Khannak@bnu.ac.uk
Abstract. Cloud computing has brought substantial changes to businesses, particularly small and medium-sized enterprises (SMEs), by streamlining IT resource
procurement and reshaping IT management responsibilities and security. Many
SMEs still rely on outdated physical data centres, servers, and hardware for their
IT infrastructure and applications. This reliance often stems from prior investments in infrastructure, software, and hardware, coupled with concerns about
upgradability and the high cost of adopting new technology. However, in today’s
competitive technological landscape, organisations must maintain an IT infrastructure that is scalable, highly available, cost-efficient, high-performing, adaptable, and secure. This research paper examines the case of Xcilent.com, a modest
taxi company, and its decision to migrate to the cloud, with a specific focus on
AWS as their chosen cloud service provider. This strategic migration is aimed
at bolstering Xcilent.com’s competitive stance within the ground transportation
sector, particularly when facing formidable competitors like Uber. The migration
plan emphasizes adherence to predefined business privacy and security standards
while striving for an elevated service level. Despite the initial challenges associated with cloud migration, this transition presents a valuable opportunity for
Xcilent.com to modernise its IT infrastructure, making it more secure, scalable,
and cost-effective.
Keywords: Cloud computing · Hybrid deployment model · Secure migration ·
AWS cloud · Cloud migration · Security
1 Introduction
Cloud computing, defined by NIST as a model centered on the provisioning of virtualized
computing resources, has garnered widespread attention in recent years [27], This model
offers remote services accessible over the Internet on a pay-as-you-go basis, aligning with
an enticing business strategy [8], features, including cost efficiency, high availability,
scalability, and various virtualization capabilities, have motivated enterprises to embrace
cloud computing and migrate their services [6].
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2024
K. Arai (Ed.): SAI 2024, LNNS 1017, pp. 418–431, 2024.
https://doi.org/10.1007/978-3-031-62277-9_27
The Impact of Utilising the Amazon AWS Hybrid Deployment Model
419
In the rapidly evolving transportation industry, companies face the dual challenge of
expanding operations and enhancing information security [20]. To achieve these objectives, many are adopting cloud-based infrastructure as a strategic approach. This research
paper focuses on the migration of Xcilent.com, a small to medium-sized Taxi Transportation Company from on-premises infrastructure to Amazon Web Services (AWS) cloud
using a hybrid deployment approach [7]. In this hybrid model, sensitive data remains onpremises to benefit from added security and control, while public-facing data is stored in
the cloud for accessibility, scalability, and high availability [28]. The specific data placement depends on the transportation company’s security and compliance requirements
[7].
2 Hybrid Deployment Model
The organisation has concluded that a hybrid deployment model, encompassing the
utilisation of software as a service (SaaS), infrastructure as a service (IaaS), and select
aspects of platform as a service (PaaS), represents the optimal approach to fulfilling
their requirements [31]. This strategic choice affords them the ability to leverage the
cloud’s scalability, high availability, security enhancements, and cost efficiency, all while
retaining governance over servers housing sensitive data that remain on-premises [24].
2.1 Background
The business operates with eight employees, three servers, and eight desktop computers,
utilising software such as Microsoft Office, Google Maps, Salesforce, Zoom, navigation
tools, and a bespoke Internet-based application platform. This platform serves both
clients for reservations and office administrators for job dispatch via GPRS technology.
According to Ivan, C. and Popa, R [9], this technology facilitates secure internetbased communication with drivers for scheduling, pickups, and drop-offs, maintaining
comprehensive audit trails of dates and times. It enables the instant transmission of
complete job details to drivers, including collection and delivery information, job references, and passenger details. Real-time updates with timestamps, wireless GPS tracking
through Google Maps and accessible navigation software on drivers’ and passengers’
smartphones are also part of this system.
Currently hosted on-premises at their Heathrow office in London, UK, hardware,
and software management have become challenging and cost-intensive as the business
expands. According to [33], the total cost of ownership (TCO) for SME IT comprises
capital expenditures (CAPEX) and operational expenditures (OPEX). CAPEX includes
server acquisitions, network devices, software licences, and routers, while OPEX encompasses electricity bills, software and hardware maintenance, employee salaries, and more
[10].
Recognising the potential benefits, the company aims to migrate to the cloud, as
suggested by Jamshidi, P. et al. [11]. This move will offer scalability, enhanced data
security monitoring, on-demand resource availability, storage flexibility, responsiveness, and cost-effectiveness, [16]. Additionally, it aligns with AWS’s improved security
420
K. Koorowlay and R. Al-Khannak
features, reinforcing the existing on-premises infrastructure. The goal is to enhance competitiveness in the transportation industry by effectively competing with market leaders
like Uber. The organisation seeks a streamlined, user-friendly cloud-based infrastructure
to optimise operations [22].
2.2 Project Scope
This project encompasses the migration of the company’s three servers, specifically
the web server, proxy server, and database server, to the AWS cloud infrastructure [15].
Additionally, it entails the transition of the eight on-premises desktop computers to cloudhosted virtualized desktops, retaining accessibility via existing on-premises desktop
devices. All resources will be hosted within the AWS cloud environment, necessitating
the following tasks: [17].
1) Assessment of Current IT Infrastructure: This phase will comprehensively analyse the
existing hardware, software, and data landscape to determine migration requirements
and strategies.
2) Design of Hybrid Cloud Architecture: The project will involve the creation of a hybrid
deployment model, enabling the organisation to retain sensitive data on-premises
while capitalising on cloud scalability and security [32].
3) Leveraging AWS IaaS Offerings: Key AWS services, including Amazon Elastic
Compute Cloud (EC2) and Amazon Simple Storage Service (S3), will be utilised
to facilitate seamless migration.
4) Integration of Software Applications: Integration of current software applications,
such as Microsoft Office and Google Maps, with the AWS cloud infrastructure (SaaS),
alongside the adoption of PaaS for web application development.
5) Migration Execution: This pivotal phase involves the actual migration of servers and
desktops to the AWS cloud environment.
6) Implementation of Security Measures: Robust security measures will be implemented to safeguard sensitive data, aligning with industry standards and compliance
requirements [2].
7) Secure On-Premises Data: Security measures like virtual private clouds (VPCs) and
security groups will be implemented to protect on-premises sensitive data, ensuring
adherence to industry standards, laws, and regulations.
The primary project objective, in accordance with insights from Chauhan and Babar’s
journal article [6], is to construct an optimally tailored hybrid cloud computing architecture meeting the organisation’s specific needs and also, to ensure the selection aligns
with project objectives and industry best practices [25].
3 Aim and Objectives
The main aim of this project is to migrate the business’s existing infrastructure to the AWS
cloud with the goals of enhancing scalability, reducing costs, and fortifying information
security. The specific objectives of the project are as follows:
1) Migrate the company’s three servers and five desktops to the AWS cloud environment.
The Impact of Utilising the Amazon AWS Hybrid Deployment Model
421
2) Implement Software as a Service (SaaS) solutions for the company’s existing software
applications, including Microsoft Office and Google Maps.
3) Utilise infrastructure as a service (IaaS) to dynamically scale computational resources
through virtualization, ensuring efficient resource allocation.
4) Support the company’s hybrid deployment strategy by preserving sensitive data onpremises, encompassing customer information, company trade secrets, and proprietary software technology. Consequently, servers housing this critical data will remain
on-premises [12].
5) Ensure a seamless migration to the AWS cloud, minimising disruptions to the
company’s operations and preserving customer satisfaction.
These objectives collectively constitute the framework for the successful execution of
the project, aligning with the organisation’s overarching goals and industry best practices
[19].
3.1 Ethical Considerations
Migrating to the AWS cloud necessitates the consideration of various ethical aspects.
Foremost among these concerns is the safeguarding of data privacy and security, particularly for sensitive client information, proprietary trade secrets, and bespoke software
technologies. Compliance with pertinent data privacy laws, regulations, and industry
standards, such as ITIL, HIPAA, SOC2, and PCI-DSS, is imperative throughout the
migration process to maintain data integrity and security, as suggested by Armstrong
[1]. Additionally, the ethical dimension extends to the use of cloud services from thirdparty providers like AWS. Ensuring that the provider upholds ethical and legal standards
in handling the company’s data is paramount to this consideration.
Access control, as highlighted by Murphy and Rocchi [21], necessitates stringent
measures to restrict access to sensitive on-premises data exclusively to authorised personnel. Implementation of security protocols, including multi-factor authentication, may
be required to bolster data access control.
Ethical deliberation encompasses cost considerations, as elucidated by Faragardi [7].
The financial implications of migrating to the cloud, encompassing data and application relocation costs as well as ongoing cloud service expenses, must be thoughtfully
managed.
Security, per Armstrong [1], mandates the establishment of robust measures to safeguard data and applications against cyber threats. Hybrid deployments introduce distinctive security challenges, particularly in securing communications between on-premises
and cloud-based systems. Ensuring the proper configuration and protection of cloud
infrastructure is essential to thwart unauthorised access and data breaches [34].
Workforce impact, as reiterated by Armstrong [1], entails shift in IT staff roles and
responsibilities and a demand for cloud-related skill development.
The environmental impact forms another ethical dimension. Minimising the environmental footprint requires the conscientious selection of energy-efficient servers, data
centres, and renewable energy sources.
Lastly, vendor lock-in, inherent to hybrid deployments, underscores the ethical
importance of ensuring flexibility in cloud vendor choices and preserving user authority
over cloud resources [35].
422
K. Koorowlay and R. Al-Khannak
Service Level Agreements (SLAs), according to Khanye et al. [13], emerge as an
ethical tool to maintain the balance of responsibilities between service providers and
customers. SLAs, which are comprehensible and reliable, instil trust and guarantee
compensation proportional to service disruptions, should they occur.
4 Methodology
AWS provides multiple migration methods for migrating cloud-based apps, data, and
infrastructure. Each approach, according to Pahl et al. [23], has its own pros and cons,
and the optimal plan for Xcilent.com will be determined by the organisation’s particular
needs and objectives.
The planned migration plan for Xcilent.com, as already determined, is based on
a hybrid deployment approach, in which sensitive data such as customer information,
company trade secrets, and custom software technologies will be preserved on-premises
while the remainder of the infrastructure is moved to the AWS cloud. The company will
utilise the Software as a Service (SaaS) and Infrastructure as a Service (IaaS) models
provided by Amazon Web Services (AWS).
Before beginning the migration, Jamshidi et al. [11], claim it is normally advisable
to have a well-defined strategy and a thorough understanding of the costs, risks, and
benefits associated with each approach. Consequently, as we have studied the hybrid
approach and concluded from the outset, it suits the organisation best. Then only can the
methods and techniques for using the optimal migration strategy be detailed as specified
below [36].
4.1 Lift and Shift
This methodology involves taking an existing on-premises application/server and moving it to the cloud without making any changes to the application. This approach is often
used when an organisation wants to quickly move an application to the cloud without
incurring the additional costs and time associated with re-architecting the application
[5].
4.2 Re-architecting
This methodology involves making changes to the application or its architecture in
order to optimise it for the cloud. This approach can include breaking down a monolithic application into microservices or using cloud-native services to improve scalability
and availability [18]. This approach is often used when an organisation wants to take
advantage of the benefits of the cloud, such as elasticity and scalability [1].
4.3 Refactoring
This methodology is similar to re-architecting, but it often involves making smaller
changes to the application to optimise it for the cloud. This approach can include replacing on-premises resources with cloud-based resources or modifying the application to
take advantage of cloud-specific features [5].
The Impact of Utilising the Amazon AWS Hybrid Deployment Model
423
4.4 Replatforming
This methodology involves replacing an existing application with a new platform that is
optimised for the cloud. This approach can include replacing an on-premises database
with a cloud-based database or replacing an on-premises application server with a cloudbased application server [5].
4.5 Data Migration
This type of migration involves moving data from one storage location to another, such
as from an on-premises data centre or server to an S3 bucket [5].
4.6 Hybrid: (Chosen Method)
This methodology uses a combination of the aforementioned approaches and is frequently used when an organisation wishes to migrate some resources to the cloud while
keeping other on-premises. This strategy will be selected based on the assessment of the
proposal, as it meets the company’s needs perfectly. It enables the company to use its
existing physical infrastructure and integrate all necessary cloud services [5].
We will be using the AWS Server Migration Service (SMS), which is recommended
as a migration tool by AWS for Hybrid scenarios, such as in our case. AWS SMS allows
us to automate the migration of our on-premises servers to the AWS cloud. It simplifies
the migration process by replicating our server’s data and configuration to an Amazon
Elastic Block Store (EBS) snapshot, which is then used to create an Amazon Machine
Image (AMI). We will then launch EC2 instances on AWS using these AMIs, effectively
migrating our servers to the cloud.
By using AWS SMS, you can maintain control over your sensitive data on-premises
while taking advantage of the scalability, flexibility, and cost-effectiveness of the cloud
for other non-sensitive components of your infrastructure. It provides a reliable and
efficient migration method that minimises downtime and ensures data integrity during
the migration process.
5 The Migration Plan Includes the Following Steps
Discovery and Assessment. The first step of the migration process, according to
jamshidi et al. [11], will be to conduct an inventory and evaluation of the existing
systems and data. This will require identifying all the systems and data to be migrated,
as well as their dependencies and interdependencies.
Planning. Once the discovery and assessment is complete, a detailed migration plan
will be developed. This will include identifying any necessary changes to the current
systems and data, as well as the resources required for the migration [5].
Execution. The actual migration of the systems and data will be done in phases, with
testing and validation done at each step to ensure a smooth transition. This will involve
moving the systems and data to the AWS cloud, configuring, and testing them, and then
switching over to the new systems and data.
424
K. Koorowlay and R. Al-Khannak
Optimization. After the migration is complete, the systems and data will be optimized
to take full advantage of the AWS cloud. This will involve monitoring and adjusting the
systems and data as needed to ensure optimal performance and cost savings [1].
Security. The company’s information security will be improved by utilizing the security
features of the AWS Cloud [3].
Monitoring. The Migration Will Be Monitored and Maintained to Ensure Smooth
Operation.
Strategy. Software as a Service (SaaS) will be used for applications such as Microsoft
Office and Google Maps and all other Google workspaces, navigation applications,
and all other administrative applications such as Salesforce, HR software, and Zoom for
online staff and customer meetings. Infrastructure as a Service (IaaS) will be used for the
company’s servers and desktop computers. A hybrid deployment model will be used,
with sensitive data remaining on-premises in the private cloud. cloud-native security
solutions will be implemented to protect the company’s data in the cloud. Backup and
disaster recovery plans must also be put in place, advises Fahmideh & Beydoun [8], to
make sure that the company’s data is always available and that its integrity is intact and
in good shape.
6 Detailed Implementation Plan
6.1 Phase 1: Analysis and Planning
This is the phase, during which the company will develop a detailed migration plan. This
plan, according to Fahmideh & Beydoun [8], will include a timeline for the migration,
a list of the data, infrastructure, and applications that will be moved to the cloud, and a
detailed list of the hardware infrastructure and software that will stay on premises [26].
The company will also need to plan for the training of its employees.
• Review current infrastructure and software applications.
• Develop migration plan.
• Design and implementation of a secure network connection between the on-premises
and cloud environments.
• Identify sensitive data that needs to be kept on-premises.
6.2 Phase 2: Execution
• Migrate non sensitive data servers and desktop computers to IaaS.
• Implement SaaS for applications such as Microsoft Office and Google Maps etc.
• Test and validate the integrity of data and functionality of the software applications.
6.3 Phase 3: Optimization and Security
• Utilize the features of the AWS cloud to optimize operations.
• Implement cloud-native security solutions.
• Implement backup and disaster recovery solutions.
The Impact of Utilising the Amazon AWS Hybrid Deployment Model
425
6.4 Phase 4: Monitoring and Maintenance
• Monitor the migration to ensure smooth operation.
• Maintain and update the infrastructure and software applications.
Four migration process steps and other migration tasks are included in the cloud
migration strategy. Feasibility research, needs analysis, vendor and service decisions,
and migration strategies are some of the tasks that are part of the migration planning
process. The outcome of this phase is a migration plan. The second process phase,
known as “migration execution,” entails duties that involve modifying the application’s
architecture as needed. Data from the local data store is retrieved and migrated to the
cloud data store when the application has been cloud-enabled and relocated to the cloud
platform. The migration evaluation step that comes after is when the application is put into
production and tested to see whether the migration was successful. The cloud migration
model, according to Pahl & Xiong [23], comprises all-encompassing themes and duties
that do not fall under any one process phase. Security, governance, staff training, effort
estimation, and business changes are some of the cross-cutting challenges expressed
Tekinerdogan & Ozcan [29, 30]. The cross-cutting issues’ associated tasks are classified
as umbrella activities that include the other process stages. The tasks include managing
change in a business, scalability and elasticity assessments, training, labour estimation,
security analysis, and governance. The cloud migration model is shown in Fig. 1 below.
Fig. 1. Cloud migration diagram model [14]
426
K. Koorowlay and R. Al-Khannak
7 Connectivity for Hybrid Architecture
To establish connectivity between the business’s on-premises infrastructure and our
Virtual Private Cloud (VPC) in AWS, a secure and dependable network connection is
essential.
AWS offers several options for achieving this, including:
1) AWS Direct Connect: This dedicated network connection establishes a private, highbandwidth link between the on-premises data centre and AWS. It ensures a consistent
network experience with lower latency and higher throughput compared to internetbased connections.
2) VPN (Virtual Private Network): AWS VPN enables the creation of an encrypted
connection over the internet, connecting your on-premises network with the VPC.
It utilises IPsec (Internet Protocol Security) to secure communication and maintain
connectivity even if the on-premises infrastructure employs dynamic IP addresses.
NB: In our scenario, the choice between these options depends on business size and
budget constraints. For our example, a VPN is a suitable choice as it offers a costeffective solution while meeting the necessary security and operational requirements for
a business of this scale.
The VPN option aligns with the business’s specific security prerequisites, compliance
obligations, and desired level of control and isolation for network traffic [4].
The below diagram in Fig. 2 give an idea of what the business hybrid architecture
will look like.
Fig. 2. On-premises infrastructure connected to the cloud system example (https://community.
netapp.com)
The Impact of Utilising the Amazon AWS Hybrid Deployment Model
427
8 Xcilent.Com’s Hybrid Architecture
The hybrid cloud system architecture designed for Xcilent.com is structured as follows:
The system employs a single-region architecture based in London, where the majority
of our customers are located and where our primary taxi services are offered. The web
server will be hosted on Amazon Elastic Compute Cloud (EC2) instances, supported by
an elastic load balancer (ELB). To manage traffic to the web server, an EC2 instance
will also serve as a proxy server. Comprehensive security tools and technologies will
oversee data and resource access, as well as inbound and outbound traffic, safeguarding
all Virtual Private Clouds (VPCs), subnets, zones, resources, and applications. Access
to the servers and specific resources will be restricted to only two IT security personnel
situated at our London headquarters (Admin1 and Admin2).
Several factors, including latency, compliance, and cost play a role in selecting
the optimal AWS region for hybrid system deployment. Given the business scenario’s
requirements, the choice of the EU (London) region is evident. This selection offers the
following advantages:
It is geographically located in London, aligning with the company’s operational base.
It ensures minimal latency to London, the primary service delivery city.
It adheres to multiple regulations, including the General Data Protection Regulation
(GDPR), ISO 27001, and the PSR Payment Systems Regulation (UK).
Fig. 3. Architecture diagram
Furthermore, AWS’s data centres (AZs) in London fully comply with UK and EU
GDPR regulations, delivering secure and compliant data storage solutions for Europeanbased enterprises. It is essential to emphasise that, regardless of the region chosen, strict
adherence to all applicable regulations and policies is paramount.
As depicted in the Architecture diagram in Fig. 3 below, an asynchronous data transfer mechanism is in place between the NAS device and the RDS MySQL database
engine, facilitating data synchronisation between these two systems. Typically, this synchronisation process operates at the application or database level and is responsible for
428
K. Koorowlay and R. Al-Khannak
transferring, updating, or replicating data between the on-premises storage device and
the RDS MySQL database engine.
9 Results and Analysis
9.1 Results
Implementing Amazon AWS’s hybrid deployment model for migrating a commercial
web application to the cloud has yielded significant outcomes:
1) Optimal AWS Region Selection: The choice of the EU (London) AWS region was
found to align seamlessly with the business scenario’s requirements. This region
provides low-latency access to the city where services are primarily offered and
complies with vital data protection regulations, including GDPR, ISO 27001, and
PSR Payment Systems Regulation (UK).
2) High Availability: Architecture employing Amazon EC2 instances, and an Elastic
Load Balancer ensures minimal downtime and uninterrupted customer service.
3) Scalability: Leveraging AWS’s IaaS model, Xcilent.com achieves dynamic resource
allocation, optimising performance.
4) Cost-Effective Migration: The hybrid approach balances data security with cost
efficiency, delivering savings compared to full migration.
5) Data Compliance Assurance: AWS’s London data centres are fully compliant with
GDPR, ensuring secure data storage.
6) Enhanced Security Measures: Robust security tools and strict access controls limit
system access to authorised IT security personnel.
7) Asynchronous Data Transfer: Efficient data synchronisation between the NAS device
and the RDS MySQL database maintains data integrity.
9.2 Analysis
Further analysis underscores the advantages of Amazon AWS’s hybrid deployment for
Xcilent.com’s web application migration:
1) Latency Reduction: By selecting the EU (London) region, the company can significantly reduce latency for its services, providing a smoother user experience.
This low-latency connection aligns with the location of its primary operations and
customers.
2) Compliance Confidence: Compliance with GDPR and other crucial data protection
regulations is vital for Xcilent.com, particularly when handling sensitive customer
data. The EU (London) region’s compliance assurances enhance data security and
regulatory adherence.
3) Stringent Access Control: Limited access to IT security personnel enhances security.
4) Efficient Data Synchronisation: Asynchronous data transfer supports data integrity
across on-premises and cloud systems.
The Impact of Utilising the Amazon AWS Hybrid Deployment Model
429
10 Conclusions
In conclusion, the adoption of Amazon AWS’s hybrid deployment model proves advantageous for Xcilent.com’s commercial web application migration to the cloud. The selection of the EU (London) region optimises latency, ensures compliance with data protection regulations, and aligns with the company’s operational base. Enhanced security
measures, including restricted access, protect sensitive data effectively. Additionally, the
asynchronous data transfer mechanism facilitates efficient data synchronisation.
These findings affirm the viability and effectiveness of Amazon AWS’s hybrid
deployment model for securely migrating a commercial web application to the cloud,
particularly for businesses with specific location and compliance requirements.
The selected architecture provides a secure, compliant, and efficient solution for Xcilent.com’s cloud migration endeavours. Furthermore, the adoption of Amazon AWS’s
hybrid deployment model for migrating Xcilent.com’s commercial web application to
the cloud has yielded positive outcomes in terms of high availability, scalability, and
cost-effectiveness. The architecture’s redundancy and failover mechanisms enhance
high availability, ensuring uninterrupted service for customers. Scalability, a critical
factor in handling varying workloads, is achieved through AWS’s IaaS model, enabling
Xcilent.com to adapt resources to demand dynamically.
Furthermore, the hybrid approach demonstrates its cost-effectiveness by balancing
the advantages of cloud scalability with cost savings attributed to the retention of sensitive
data on-premises.
These findings underscore the hybrid deployment model as a compelling solution for
businesses seeking to enhance availability, scalability, and cost efficiency during their
cloud migrations.
Xcilent.com’s experience serves as a reference point for organisations with similar
objectives and underscores the effectiveness of Amazon AWS’s hybrid cloud solution
in meeting these crucial criteria.
References
1. Armstrong, J.: Chapter 6 to 8. In: Migrating to AWS: A manager’s guide: How to foster
agility, reduce costs, and bring a competitive edge to your business, pp. 75–140. Oreilly &
Associates Inc. (2020)
2. AWS: Security best practices in AWS CloudTrail, AWS (2022a). Available at: https://
docs.aws.amazon.com/awscloudtrail/latest/userguide/best-practices-security.html Accessed:
21 March 2023
3. AWS: Security Group Rules - Amazon Elastic Compute Cloud (2022). Available at: https://
docs.aws.amazon.com/AWSEC2/latest/UserGuide/security-group-rules.html. Accessed: 21
March 2023
4. AWS cost calculator: Estimate AWS resource costs - aws.amazon.com (2022).
Available at: https://aws.amazon.com/premiumsupport/knowledge-center/estimating-awsresource-costs/. Accessed: 27 December 2022
5. AWS: AWS migration whitepaper, Amazon Web Services (2022). Available at: https://resour
ces.awscloud.com/migration-resources-whitepapers/aws-migration-whitepaper. Accessed: 1
January 2023
430
K. Koorowlay and R. Al-Khannak
6. Chauhan, M.A., Babar, M.A.: Towards process support for migrating applications to cloud
computing. In: 2012 International Conference on Cloud and Service Computing [Preprint]
(2012). Available at: https://doi.org/10.1109/csc.2012.20
7. Faragardi, H.R.: Ethical considerations in cloud computing systems. In: Proceedings of the
IS4SI 2017 Summit Digitalisation for a Sustainable Society, Gothenburg, Sweden, 12–16
June 2017 (2017). [Preprint]. Available at: https://doi.org/10.3390/is4si-2017-04016
8. Fahmideh, M., Beydoun, G.: Reusing empirical knowledge during cloud computing adoption.
J. Sys. Softw. 138, 124–157 (2018). Available at: https://doi.org/10.1016/j.jss.2017.12.011
9. Ivan, C., Popa, R.: A cloud based mobile dispatching system with built-in social CRM component: Design and implementation. Computers 4(3), 176–214 (2015). Available at: https://
doi.org/10.3390/computers4030176
10. Jamshidi, P., Pahl, C., Mendonça, N.C.: Pattern-based multi-cloud architecture migration.
Software: Practice and Experience 47(9), 1159–1184 (2016). Available at: https://doi.org/10.
1002/spe.2442
11. Jamshidi, P., et al.: Cloud migration patterns: a multi-cloud service architecture perspective.
In: Service-Oriented Computing - ICSOC 2014 Workshops, pp. 6–19 (2015). Available at:
https://doi.org/10.1007/978-3-319-22885-3_2
12. Juan-Verdejo, A., et al.: Moving business intelligence to cloud environments. In: 2014 IEEE
Conference on Computer Communications Workshops (INFOCOM WKSHPS) [Preprint]
(2014). Available at: https://doi.org/10.1109/infcomw.2014.6849166
13. Khanye, T., Ophoff, J., Johnston, K.: Issues in migrating legacy systems to the cloud. In: 2018
8th International Conference on Cloud Computing, Data Science & Engineering (Confluence)
[Preprint] (2018). Available at: https://doi.org/10.1109/confluence.2018.8443029
14. Lau, K.-K., Lamersdorf, W., Pimentel, E.: Service-oriented and cloud computing second
european conference, ESOCC 2013, Málaga, Spain, September 11–13, 2013, Proceedings,
pp. 200–226. Springer Berlin Heidelberg, Berlin, Heidelberg (2013)
15. McKenzie, C.: How to setup an nginx reverse proxy server example, TheServerSide (2022).
Available at: https://www.theserverside.com/blog/Coffee-Talk-Java-News-Stories-and-Opi
nions/How-to-setup-Nginx-reverse-proxy-servers-by-example. Accessed: 20 March 2023
16. Massey, J., Epstein, J.: Best practices for monitoring AWS CloudTrail logs (2020). Available
at: https://www.datadoghq.com/blog/monitoring-cloudtrail-logs/. Accessed: 21 January 2023
17. Mendonca, N.C.: Architectural options for cloud migration. Cloud Computing/Migration
47(8), 62–66 (2014). Available at: https://doi.org/10.1109/mc.2014.203
18. Mesbahi, M.R., Rahmani, A.M., Hosseinzadeh, M.: Reliability and high availability in cloud
computing environments: a reference roadmap. Human-centric Computing and Information
Sciences 8(1) (2018). Available at: https://doi.org/10.1186/s13673-018-0143-8
19. Microsoft: Microsoft Cloud Adoption Framework for azure - cloud adoption framework,
Cloud Adoption Framework | Microsoft Learn (2022). Available at: https://docs.micros
oft.com/en-us/azure/cloud-adoption-framework/opbuildpdf/toc.pdf. Accessed: 2 December
2022
20. Müthing, J., Jäschke, T., Friedrich, C.M.: Client-focused security assessment of mHealth apps
and recommended practices to prevent or mitigate transport security issues. JMIR mHealth
and uHealth 5(10) (2017). Available at: https://doi.org/10.2196/mhealth.7791
21. Murphy, B., Rocchi, M.: Ethics and cloud computing. Palgrave Studies in Digital Business &
Enabling Technologies, pp. 105–128 (2020). Available at: https://doi.org/10.1007/978-3-03054660-1_6
22. North, F.: Getting started, Amazon (2022). Available at: https://aws.amazon.com/getting-sta
rted/hands-on/create-mysql-db/. Accessed: 22 May 2023
The Impact of Utilising the Amazon AWS Hybrid Deployment Model
431
23. Pahl, C., Xiong, H.: Migration to paas clouds - migration process and architectural concerns.
In: 2013 IEEE 7th International Symposium on the Maintenance and Evolution of ServiceOriented and Cloud-Based Systems [Preprint] (2013). https://doi.org/10.1109/mesoca.2013.
6632740
24. Park, J., et al.: Approach for selecting and integrating cloud services to construct Hybrid
Cloud. Journal of Grid Computing 18(3), 441–469 (2020). https://doi.org/10.1007/s10723020-09519-x
25. Pahl, C., Xiong, H., Walshe, R.: A comparison of on-premise to cloud migration approaches.
Service-Oriented and Cloud Computing, 212–226 (2013). Available at: https://doi.org/10.
1007/978-3-642-40651-5_18
26. Rani, B.K., Rani, B.P., Babu, A.V.: Cloud computing and inter-clouds – types, topologies and
research issues. Procedia Computer Science 50, 24–29 (2015). Available at: https://doi.org/
10.1016/j.procs.2015.04.006
27. Shawish, A., Salama, M.: Cloud computing: paradigms and technologies. Inter-cooperative
Collective Intelligence: Techniques and Applications, pp. 39–67. Springer-Verlag Berlin
Heidelberg (2016). Available at: https://doi.org/10.1007/978-3-642-35016-0_2
28. Shbair, W., Cholez, T.: A Survey of HTTPS Traffic and Services Identification Approaches
[Preprint] (2018). Available at: https://www.researchgate.net/publication/343759266_A_S
urvey_of_HTTPS_Traffic_and_Services_Identification_Approaches
29. Tekinerdogan, B., Ozcan, O.: Architectural perspective for design and analysis of Scalable
Software as a service architecture. Managing Trade-Offs in Adaptable Software Architectures,
223–245 (2017). Available at: https://doi.org/10.1016/b978-0-12-802855-1.00010-1
30. Taherkordi, A., et al.: Future cloud systems design: challenges and research directions. IEEE
Access 6, 74120–74150 (2018). https://doi.org/10.1109/access.2018.2883149
31. Trinidad, P., Galán, J.G.: Migrating to the cloud - A software product line-based analysis. In:
Proceedings of the 3rd International Conference on Cloud Computing and Services Science
[Preprint] (2016). https://doi.org/10.5220/0004357104160426
32. Wettinger, J., et al.: Characterizing and evaluating different deployment approaches for cloud
applications. In: 2014 IEEE International Conference on Cloud Engineering [Preprint] (2014).
Available at: https://doi.org/10.1109/ic2e.2014.32
33. Yaseen, H., et al.: Factors influencing cloud computing adoption among smes: The Jordanian
context. Information Development/via Sage Journals, p. 026666692110479 (2022). Available
at: https://doi.org/10.1177/02666669211047916
34. Veenadhari, L., Srilakshmi, M., Pradeep, K.: International Journal of Advanced Research
in Computer Sciences - researchgate.net. Int. J. Adv. Res. Comp. Sci. (2018). Available
at: https://www.researchgate.net/profilesls-of-Cloud-Computing-Challenges.pdf?origin=pub
lication_detail. Accessed: 1 January 2023
35. Vance, S.: The hybrid, Amazon (2022). Available at: https://aws.amazon.com/hybrid/.
Accessed: 25 February 2023
36. Zhao, J.-F., Zhou, J.-T.: Strategies and methods for cloud migration. Int. J. Auto. Comp.
via Springer Link 11(2), 143–152 (2014). Available at: https://doi.org/10.1007/s11633-0140776-7
Mitigating Cache Pollution Attack Using Deep
Learning in Named Data Networking (NDN)
Mohd Maizan Fishol Hamdi1 , Zhiyuan Chen1(B) , and Milena Radenkovic2
1 University of Nottingham Malaysia, Jalan Broga, 43500 Semenyih, Malaysia
Zhiyuan.chen@nottingham.edu.my
2 University of Nottingham, Wollaton Road, Nottingham NG8 1BB, UK
Abstract. With the explosive growth of Internet traffic, Internet communication
pays more attention to the data itself rather than where it is physically located.
To better cope with Internet usage, a revolutionized shift from host-centric endto-end communication to receiver-driven content retrieval through InformationCentric Networking (ICN) with Named Data Networking (NDN) has emerged as
a big player in this new paradigm. New capabilities such as caching everywhere
have made it more challenging to protect all caches from new breed of security
issues especially Cache Pollution Attack (CPA). This paper presents a summary of
Caching security challenges in NDN but focuses mainly on CPA. The adaptation
of Machine Learning (ML) based technology is a promising emerging and proven
in experimental-based simulation using ndnSIM. We show that our Python-based
DL algorithm can improve the detection of CPA and outperforms some state-ofthe-art techniques that mostly works on numerical counters and probability-based
algorithm.
Keywords: Information-Centric Networking (ICN) · Data Networking (NDN) ·
Cache Pollution Attack (CPA) · Machine Learning (ML)
1 Introduction
This paper explores the open question of optimal content placement while users can
access it efficiently in large-scale dynamic networks. Based on reports by Cisco [1],
devices and connections in Machine-to-Machine (M2M) are accelerating growth, growing nearly 2.4-fold to 14.7 billion connections by 2023. Placement of information became
the biggest challenge because of distributed users and the need for high-access performance with less cost. The distributed system has been a prominent candidate to address
this issue with the introduction of Peer-to-peer networking (P2P) [2], Cloud Computing
[3], Content Delivery Networks (CDN) [4] and recently based on Information-Centric
Networking (ICN) known as Content-Centric Networking (CCN) [5]. The tremendous
growth of CCN has attracted a lot of research groups worldwide and it was Named Data
Networking (NDN) [6]. Effectiveness and efficiency become important in determining
the performance in networking while security is the most vital aspect to make sure it is
not just efficient but also secured at the same time.
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2024
K. Arai (Ed.): SAI 2024, LNNS 1017, pp. 432–442, 2024.
https://doi.org/10.1007/978-3-031-62277-9_28
Mitigating Cache Pollution Attack
433
1.1 From Host-Centric to Content-Centric
Internet was developed based on endpoint connections to get information from one node
to another. The idea was to focus on communication with high reliability and scalability.
The change from host-centric to content-centric gives some significant advantages such
as network load reduction, low dissemination latency, and scalability. From the beginning
of ICN’s architectural design, one of the key requirements is strengthening the security
aspect from all kinds of threats such as Denial of Service, Content Poisoning, Cache
Pollution, Naming & Routing, and Application [7]. Many researchers focus on solving
security issues in NDN due to its robustness and similarity to the current IP-based
Internet. Cooperating all components in the NDN architecture is the key to ultimate
security [8].
Content Store has always been recognized as the core component of NDN architecture since that is what the current Internet lacks without overlaying it with another
application layer such as in CDN [4]. Having the capability of storing chunks of data in
the network infrastructure itself with caching features has always been the missing link
to support the bulk growth of content-based data. Since caching will be done publicly in
NDN, a few types of security issues can be a threat to the content store and the caching
strategy itself. There are TWO (2) kind of Cache Pollution Attack (CPA) that has been
identified as locality disruption and false locality. The main idea is to mess around with
the popularity of cached data in the Content Store [7]. Caching involves a trade-off in
the middle of network efficiency and among some ways to abuse cache are:
i. An attacker can own the cache while messing with the locality.
ii. Using a Denial of Service (DoS) attack, cache capability will lessen.
iii. An attacker can take out the content in the cache, and risk privacy.
There is a need for new initiatives to address these issues and challenges. Previous
research focuses more on probability values [9] and popularity value [10] while recently
others are trying to adopt Machine Learning [11–13]. This project shows that a real
mechanism should consist of good detection and prediction as well without jeopardizing
the resources.
2 Related Work
2.1 Named Data Networking (NDN) Architecture
Content-Centric Networking (CCN) is considered the earliest idea of InformationCentric Networking (ICN) architecture. Initially, this idea was introduced in a Google
tech talk [14] in 2006 with the first paper explaining CCN’s published architecture
[5]. The NDN project was funded by the US Future Internet Architecture program to
continue the development of CCN architecture [15] and developed one of the most popular NDN simulators known as ndnSIM [16]. The NDN describes the reorganization of
the Internet Protocol (IP) arrangement by making a thin waist data exchange from the
current Internet’s hourglass architecture. Figure 1 shows the difference between both
IP packet-based and Content chunks-based hourglass. The top layer does not use IP to
communicate, meaning that the application can directly use the content’s name via NDN
data delivery.
434
M. M. F. Hamdi et al.
Fig. 1. Internet and NDN hourglass architectures[17]
NDN has three types of entities; (1) Consumer: an end user requests content by
sending interest messages, (2) Producer: an entity acting to produce and disseminate
data, (3) Routers: entities that forward interest and data to/from Consumer and Producer.
NDN consists of TWO (2) types of packets as in Fig. 2, known as Interest Packet sent
by Consumer and Data Packet.
Fig. 2. Packets in NDN architecture[18]
NDN communication begins with a request from a Consumer by issuing an Interest
Packet to another router. Next, the router will return with the Data Packet if it has it or
just forward the interest based on the Forwarding Information Base (FIB) until it reaches
a node that has the requested data. Then, the data is sent back with a Data Packet together
with a signature provided by following the reverse path taken to the Consumer by using
the Pending Interest Table (PIT). Figure 3 illustrates the way the NDN engine model
works in detail. During this transportation, routers in between will decide whether to
cache or not the Data based on the caching strategies chosen.
Content Store makes the whole router function differently not just passing packets
but also working as a storage server. However, the idea of caching everywhere seems
intimidating since it will expose all routers to security issues, especially locality and
privacy concerns. The data itself is protected by the digital signature provided by the
Producer but is still exposed to man-in-the-middle kinds of attacks while routers still
prune to dos-related attacks.
Mitigating Cache Pollution Attack
435
Fig. 3. NDN forwarding engine model[18]
2.2 Named Data Networking (NDN) Security Issues
To ensure the effectiveness of the security requirements duly met, additional evaluation is
required in the execution of the selected structure in a variety of settings. While the main
approach is to be accessed without the need for free access permissions, content delivery
is generally supported by encryption. In the context of security, many research projects
in ICN have concentrated on content authentication such as authorization and access
control, while the role of privacy and security threats is less extended and somewhat
limited. Secure communications on the Internet have become commonplace, and the
significant differences in safety features are tied to affect the use of ICN. An overview
of the security in ICN is generally discussed in a few angles [7, 19].
The most common approach to prevent cache pollution attacks is to optimize the use
of popular caches and prevent unpopular caching from becoming popular. The Taxonomy
of ICN [7] has classified attacks into four categories, namely naming, routing, caching,
and other miscellaneous related attacks. The classification depends on the primary target
of the attack. The main goal of the attackers was:
i. Increase the request timeout for a node ICN to contravene stability in the middle of
ICN asynchronous publication and the subscription process.
ii. Caching in ICN will store unpopular content, and subsequent users would request
the source rather than from sources nearby.
iii. An attacker can get private information about the requested content.
Naming. In this attack, the attacker attempts to stop the supply of content that is specific
to stop the transmission of content by tracking who requested the content.
Routing. The ICN infrastructure depends on the validity and integrity of the content
routing and is put at risk by poisonous injection name and route.
Caching. Caching is an important component of the due functioning of the infrastructure ICN is found on a recipient-driven caching to provide the closest to the user. The
operation ICN is susceptible to contamination and damage in the caching system.
436
M. M. F. Hamdi et al.
Miscellaneous. Possibility in this type is intended to weaken the ICN use and facilitate
the attacker to access information that is not allowed. This attack is more focused on the
distribution of the wrong data and inadequate.
Cache Pollution Attack [20] has been addressed in detail since the booming of cache
memories back in the 1980s. Caching at the endpoint level does not affect much since
it can be identified and controlled individually. However, the introduction of ICN with
NDN is especially really concern because caching will be done in the architecture itself
with a lot of users affected by any small vulnerabilities. CPA can be categorized into
TWO (2) groups known as false-locality and locality disruption.
False-Locality (FL). Messing with the locality of the content by repeatedly requesting
the same files unnecessarily. In the current Internet, the damage is not as big as in NDN
because this attack could be performed by not just malicious consumers but also malicious producers [7]. Malicious consumers’ objective is to exploit the content popularity
while malicious producers intend to store its content by making sure that in-between
routers cache it.
Locality Disruption (LD). Works by trying to change the caching efficiency by requesting new unpopular files until it is recognized as popular content by the cache [21]. This
will be done mostly by malicious producers that would like to make the content become
popular and cached all over the network but it could also act as malicious consumers to
activate this attack.
Even though LD and FL differ, there are some similarities in terms of objectives and
actors that could perform this attack. One of the earliest detection techniques known as
CacheShield [22] proposed using a probabilistic caching strategy and caching only the
content name until it confirms the popularity of that content.
2.3 Artificial Intelligence in Mitigating Cache Pollution Attack
Machine Learning (ML) has always been considered top of the list when involving
machine-to-machine communication. NDN research about CPA also has implemented
ML with the utilization of Multivariate Correlation Analysis to detect and mitigate CPA
in NDN [23]. By using a combination of node state and path information as the feature,
the Gradient Boost Decision Tree (GBDT) algorithm has been adopted to produce a
model of cache pollution detection [24] and some also integrate Gini impurity [25]. An
Artificial Neural Network (ANN) is an ML algorithm with inspiration from biological
neural networks containing nodes that communicate with each other via connections
that give birth to Deep Learning (DL). At first, most researchers use DL to improve the
caching strategies in all kinds of network architecture such as Mobile Social Community
Networks [26] and Multi-Access Edge Computing [27].
Anomaly detection has always been considered crucial in learning-based algorithms,
especially in DL. Among all the models in DL, LSTM has become one of the popular
ones for anomaly detection that has been used for multi-sensor [28], industrial Big Data
[29] and Interest Flood Attack (IFA) detection [30]. Some researchers have tried to adopt
DL in CPA with the usage of a Fuzzy system [31] and even extend it to Deep Reinforcement Learning (DRL) [13]. However, it is still not as efficient as Feed-forward Neural
Mitigating Cache Pollution Attack
437
Networks (FNN) as in LSTM, Recurrent Neural Networks (RNN), and the Sequential
Model itself. The basic idea of the Sequence Model as the core of FNN [32, 33] has
been tested and has already given promising results with a good balance of resources
and efficiency.
3 Method and Evaluation
3.1 Modelling and Performance Metrics
This research has been conducted by adopting an empirical and iterative approach that
concentrates on cache pollution attacks for the NDN. It consists of four phases namely
(i) Problem Formulation Phase, (ii) Development Phase, (iii) Evaluation and Verification
Phase, and (iv) Analysis and Findings phase.
Firstly, the focus is on reducing the attack on cache pollution because CPA ruins
the cache locality, indirectly it reduces the hit ratio and impacts the overall performance
of the network. Cache hits and cache misses that can also be interpreted as Cache Hit
Ratios have always become the priority and important metrics [21, 34, 35]. Meanwhile,
other things can be looked at by optimizing the infrastructure itself such as maintaining
good response time, size of the packet, and minimizing the hop counts. From our study,
we chose to adopt the two most common replacement techniques, Least Recently Used
(LRU) and Least Frequently Used (LFU). This model works by generating data from the
NDN traffic generator and emulating it in ndnSIM while passing all the emulated data
to the DL framework using TensorFlow (TF) with the best DL technique that produces
the highest accuracy. Then it will instruct the node in ndnSIM that has been identified
as a bad node and start preventing it from continuing to attack the other nodes. Figure 4
visualized the mechanism to detect and prevent CPA.
Fig. 4. Mechanism to detect and prevent CPA
Data generated from all requests and responses are collected and calculated based
on required variables. The cache hit ratio produced based on the value of each Cache
Hits and Cache Misses represents the number of cache hits associated with the number
of outgoing requests used to study the effect of caching attacks and solution capabilities.
The formula of this has been used since the early days of caching [20, 36] given by
Cache Hits
(1)
Cache Hit Ratio =
Cache Hits +
Cache Misses
438
M. M. F. Hamdi et al.
Request access time delay represents the delay between the first interest being transmitted and the data packets received. Hop count is the number of node hops to retrieve
the interest’s data and should be minimized as the requested content should be available
at the nearest node. Counting this metric will make sure that it keeps on reducing over
time if has prevented from cache-related attack. Throughput is the rate of message delivery and in this case contents. Download time was used as part of the user experience and
to evaluate the real effect of good cache hit ratios as the main purpose of protecting the
caches.
3.2 Deep Learning with Sequential Model
The sequential model is highly suitable because it allows us to specify a neural network,
precisely, sequential: from input to output, passing through a series of neural layers,
one after the other. Data flows sequentially through each layer and in this case, four
layers based on four inputs (nodes, cache hits, cache misses, packets) as in Fig. 5.
Fig. 5. How data flows through a sequential model [37]
Detection of anomalies worked straightaway as CPA suddenly happens mostly in
uniform random attacks and is different from normal traffic. The formula for this model
is given by:
y = f (x1 + x2 + x3 + x4 )
(2)
With x1 = node, x2 = cache hits, x3 = cache misses, x4 = packet.
3.3 Experiment Setup
We provide an experiment setup with a topology of a tree-like topology as a basic setup
with 4 leaves, 2 routers, and 1 root each linked with a 10Mbps speed of connection. This
likely is to further extend previous work [38] that used tree-based topology to represent
social networking. As part of the TF requirement to process only numerical value, all
of the nodes have been named in numbers 111,112,113,114 to represent leaf-1, leaf-2,
leaf-3, and leaf-4. While coding 211,212 were used to represent router-1 and router-2.
Lastly, the root of the tree has been presented by code 311 as in Fig. 6 below.
All leaves act as a consumer that requests content, routers will route the request and
work as to whether to cache or not cache the in-route content, and root act as a producer
that produces and responds to all interest. Parameter settings for each node in ndnSIM
as in Table 1 below while setup for TF as in Table 2.
Mitigating Cache Pollution Attack
439
Fig. 6. Tree-like topology for ndnSIM simulation setups
Table 1. Parameter Settings in ndnSIM
Nodes
Leaf (111,112,113,114)
Router (211,212)
Root (311)
Interest Frequency
100/second
Interest Distribution
Random Zipf Mandelbrot
Content Store Size
100000
Caching Policy
Least Recently Used (LRU), Least Frequently Used
(LFU)
Payload Size
1024 bps
Table 2. Setup for TF analysis
TensorFlow
Language-Based
DL Technique
Features
Python
Sequential Model with ANN
4 (Node, Cache Hit Ratio, Packet, Class)
4 Results and Discussion
Simulation from ndnSIM shows high accuracy of identifying traffic with attack and
no attack with Class 1 and Class 0. As soon as CPA started to work out in a uniform
random attack, all nodes started to behave strangely and all the performance metrics
kept on moving in the opposite way of normal traffic. This indicates that anomalies have
occurred and classify this traffic as Class 1. Both Class 0 and Class 1 were identified for
each traffic flow and kept updating in a flat file for analysis in TF. Further concept of DL
was used to train the sequential model with all the features identified earlier. Based on
1000 data collected from ndnSIM starting with normal and then anomalies traffic. That
data was separated by using 90% for training and 10% for testing purposes.
Figure 7 shows that this model works very fast to achieve high accuracy after just
10 epochs. The detection of anomalies combined with prediction using TF gives better
prediction and gives the network environment capabilities to learn each time CPA occurs
in the network.
440
M. M. F. Hamdi et al.
Fig. 7. Model accuracy for each epoch
Consistency of the prediction even when the traffic transforms from normal to anomalies can make sure routers focus on pure communication aspect regardless of traffic
behavior and only take action when needed as analyses were done by other parties using
TF from time to time. This is crucial as network latency plays a major role in any
networking environment and with caching embedded in the router itself, any security
measures with intensive resources should be done by other than routers. This kind of
implementation also can make sure NDN can scale to any size and environment by
running at its best performance without sacrificing security aspects like CPA.
5 Conclusion and Future Works
This project tries to look from a different angle of CPA attacks prevention. The basic idea
is to give good detection while minimizing the resources such as processor, memory,
and especially bandwidth. At the same time adopting DL that has so much to offer. FNN
has been contributing so much to NDN-based research especially. While most try to
focus on the accuracy prediction of the accuracy, this project takes a different path of
complexity. By still providing good detection rates, the consumption of resources also
has been addressed accordingly.
Since this project is considered a basic introduction with a small environment
involved. There is still room for future work, especially in different environments that
differ in terms of mobility and scalability aspects. Another thing is caching in NDN
always tries to decide based on the popularity of the locality. While it works in some
cases, there are still other things that work differently like in the military or even social
media.
References
1. Cisco Systems Inc, “Cisco Annual Internet Report (2018–2023) White Paper,” (2020).
https://www.cisco.com/c/en/us/solutions/collateral/executive-perspectives/annual-internetreport/white-paper-c11-741490.html. Accessed: 07 Jun 2023
Mitigating Cache Pollution Attack
441
2. Schoder, D., Fischbach, K., Schmitt, C.: Core Concepts in Peer-to-Peer. Peer-to-peer
computing, pp. 1–27. http://books.google.com/books?hl=en&lr=&id=hSeg14DTOd8C&oi=
fnd&pg=PA1&dq=Core+Concepts+in+Peer-to-Peer&ots=wePWMOsZQA&sig=27xeD1
0hIoNp-AlId2SSGqYajFs%5Cn (2005)
3. Hayes, B.: Cloud computing. Commun. ACM 51(7), 9–11 (2008). https://doi.org/10.1145/
1364782.1364786
4. Peng, G., Brook, S.: CDN: Content Distribution Network ∗, pp. 1–26 (2004)
5. PARC: Content Centric Networking project. http://blogs.parc.com/ccnx/
6. Zhang, L., et al.: Named Data Networking (NDN) Project Named Data Networking (NDN)
Project (2010)
7. Tourani, R., Misra, S., Mick, T., Panwar, G.: Security, Privacy, and Access Control in
Information-Centric Networking: A Survey
8. Saxena, D., Raychoudhury, V., Suri, N., Becker, C., Cao, J.: Named data networking: a survey.
Comput. Sci. Rev. 19, 15–55 (2016). https://doi.org/10.1016/j.cosrev.2016.01.001
9. Psaras, I., Chai, W.K., Pavlou, G.: Probabilistic In-Network Caching for Information-Centric
Networks, pp. 1–6. https://doi.org/10.1145/2342488.2342501
10. Silverston, T., Festor, O.: MPC : Popularity-based Caching Strategy for Content Centric
Networks, pp. 3619–3623 (2013)
11. Ndikumana, A., Ullah, S., Kim, D.H., Hong, C.S.: Deepauc: joint deep learning and auction for
congestion-aware caching in Named Data Networking. PLOS ONE 14(8), e0220813 (2019).
https://doi.org/10.1371/journal.pone.0220813
12. Shi, Z., Huang, X., Jain, A., Lin, C.: Applying deep learning to the cache replacement problem.
In: Proceedings of the Annual International Symposium on Microarchitecture, MICRO, IEEE
Computer Society, pp. 413–425 (2019). https://doi.org/10.1145/3352460.3358319
13. Zhou, J., Luo, J., Wang, J., Deng, L.: Cache Pollution Prevention Mechanism Based on Deep
Reinforcement Learning in NDN (2021)
14. Jacobson, V.: A new way to look at networking. In: Google Tech Talk (2006)
15. NSF: Named Data Networking project. https://named-data.net/
16. Mastorakis, S., Afanasyev, A., Moiseenko, I., Zhang, L.: ndnSIM 2.0 : A new version of the
NDN simulator for NS-3, pp. 1–8 (2015)
17. Named Data Networking: Executive Summary. https://named-data.net/project/execsummary/
18. NSF: Named Data Networking: Motivation & Details
19. Sakthivanitha, M., Saradha, S.: Survey based on security aware caching scheme for IoT based
information centric networking. EAI Endorsed Trans. Energy Web 8(32), 165960 (2018).
https://doi.org/10.4108/eai.1-7-2020.165960
20. Smith, A.J.: Cache Memories (1982)
21. Gao, Y., Deng, L., Kuzmanovic, A., Chen, Y.: Internet Cache Pollution Attacks and
Countermeasures.
22. Xie, M., Widjaja, I., Wang, H.: Institute of electrical and electronics engineers. In: Enhancing
Cache Robustness for Content-Centric Networking, vol. Proceedings IEEE. IEEE (2012)
23. Sohail, M., et al.: Triangle area based multivariate correlation analysis for detecting and
mitigating cache pollution attacks in named data networking. In: 2020 3rd International Conference on Hot Information-Centric Networking, HotICN 2020, Institute of Electrical and
Electronics Engineers Inc., pp. 114–121 (2020). https://doi.org/10.1109/HotICN50779.2020.
9350746
24. Man, D., Mu, Y., Guo, J., Yang, W., Lv, J., Wang, W.: Cache pollution detection method based
on GBDT in information-centric network. Secur. Commun. Netw. (2021). https://doi.org/10.
1155/2021/6658066
25. Singh, V.P., Ujjwal, R.L.: Gini impurity based NDN cache pollution attack defence mechanism. J. Inf. Optim. Sci. 41(6), 1353–1363 (2020). https://doi.org/10.1080/02522667.2020.
1809092
442
M. M. F. Hamdi et al.
26. Radenkovic, M., Huynh, V.S.H.: Cognitive caching at the edges for mobile social community
networks: a multi-agent deep reinforcement learning approach. IEEE Access 8, 179561–
179574 (2020). https://doi.org/10.1109/ACCESS.2020.3027707
27. Ndikumana, A., Tran, N.H., Kim, D.H., Kim, K.T., Hong, C.S.: Deep learning based caching
for self-driving cars in multi-access edge computing. IEEE Trans. Intell. Transport. Syst.
22(5), 2862–2877 (2021). https://doi.org/10.1109/TITS.2020.2976572
28. Malhotra, P., Ramakrishnan, A., Anand, G., Vig, L., Agarwal, P., Shroff, G.:“LSTM-based
Encoder-Decoder for Multi-sensor Anomaly Detection (2016). http://arxiv.org/abs/1607.
00148
29. Zhou, X., Hu, Y., Liang, W., Ma, J., Jin, Q.: Variational LSTM enhanced anomaly detection
for industrial big data. IEEE Trans Industr Inform 17(5), 3469–3477 (2021). https://doi.org/
10.1109/TII.2020.3022432
30. Zhang, X., Li, R., Hou, W.: Attention-based LSTM Model for IFA detection in named data
networking. Secur. Commun. Netw. 2022, 1–14 (2022). https://doi.org/10.1155/2022/181
2273
31. Karami, A., Guerrero-Zapata, M.: An ANFIS-based cache replacement method for mitigating
cache pollution attacks in Named Data Networking. Comput. Netw. 80, 51–65 (2015). https://
doi.org/10.1016/j.comnet.2015.01.020
32. Ashraf Zargar, S.: Introduction to Sequence Learning Models: RNN, LSTM, GRU. https://
doi.org/10.13140/RG.2.2.36370.99522
33. Sherstinsky, A.: Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network. Phys. D 404, 132306 (2020). https://doi.org/10.1016/j.physd.2019.
132306
34. Chen, J., Yue, L., Chen, J.: Mitigating cache pollution attack in information centric mobile
internet. KSII Trans. Internet Inf. Syst. 13(11), 5673–5691 (2019). https://doi.org/10.3837/
tiis.2019.11.022
35. Hidouri, A., Hadded, M., Hajlaoui, N., Bettahar, H., Lab, I., Muhlethaler, P.: Cache Pollution
Attacks in the NDN Architecture: Impact and Analysis Haifa Touati (2021)
36. Smith, A.J.: Disk cache—miss ratio analysis and design considerations. ACM Trans. Comput.
Syst. 3(3), 161–203 (1985)
37. Hoare, C.A.R.: A model for communicating sequential process A model for communicating
sequential process Recommended Citation Recommended Citation (1980). https://ro.uow.
edu.au/compsciwp/14
38. Maizan, M., Hamdi, F., Habbal, A., Zakaria, H., Hassan, S.: Evaluation of caching strategies
in content-centric networking for mobile and social networking environment. J. Telecommun.
Electron. Comput. Eng. 10, 24 (2018)
39. Huynh, V.S.H., Radenkovic, M.: Interdependent multi-layer spatial temporal-based caching
in heterogeneous mobile edge and fog networks. In: 9th International Conference on Pervasive
and Embedded Computing and Communication Systems (2019). https://orcid.org/0000-00034000-6143
40. Radenkovic, M., Huynh, V.S.H., Manzoni, P.: Adaptive real-time predictive collaborative
content discovery and retrieval in mobile disconnection prone networks. IEEE Access 6,
32188–32206 (2018). https://doi.org/10.1109/ACCESS.2018.2840040
41. Zakhary, S., Radenkovic, M., Benslimane, A.: Efficient location privacy-aware forwarding
in opportunistic mobile networks. IEEE Trans. Veh. Technol. 63(2), 893–906 (2014). https://
doi.org/10.1109/TVT.2013.2279671
A Community Security Operations Centre
(ComSOC) Model for SMMEs in Developing
Countries
Nombeko Ntingi , Jaco du Toit(B)
, and Sebastiaan von Solms
University of Johannesburg, Johannesburg, South Africa
201339901@student.uj.ac.za, {jacodt,basievs}@uj.ac.za
Abstract. Cybercriminals may target small, medium, and micro enterprises
(SMMEs) as entry points to large enterprises through violations of third-party
supply chain rules. Due to a lack of financial and human resources, SMMEs
are often compelled to outsource their cybersecurity needs to third-party cybersecurity service providers. To mitigate cyber-attacks or eliminate the possibility
of being the weakest link in the value chain, SMMEs need to collaborate and
share resources. Our central contention is that SMMEs in developing countries
lack resources, such as financial and human capital, to implement emerging or
advanced cybersecurity solutions. This paper explores the Community Security
Operations Centre (ComSOC) model for SMMEs in developing countries. The
ComSOC model is our proposed cybersecurity solution for SMMEs in developing countries. This model seeks to adapt or tailor-make the SOC services according
to the specific needs of the SMMEs that intend to utilise advanced SOC services
but lack the necessary resources to do so. Therefore, this paper’s primary objective
and outcome are to conceptualise the ComSOC model for SMMEs in developing
countries. This model will serve as a guideline for SMMEs in developing countries to implement emerging or advanced cybersecurity solutions adapted to their
specific needs, allowing SMMEs who could not otherwise afford to utilise the
large enterprise SOC services. The sharing of resources will enable the SMMEs
to implement these SOC services while minimising the cost of implementation.
Keywords: Community SOC · SMMEs · Sharing of resources · Homogeneous
1 Introduction
In a highly interconnected world, SMMEs are propelled to embrace digital transformation and compete in the digital world [1]. As noted by [2], through digital transformation,
SMMEs can expand their client base and offer services more efficiently and effectively.
However, the transition to digital transformation has provided cybercriminals with new
opportunities to introduce cyber threats [2], and cybercriminals have turned their efforts
to SMMEs who are under-resourced to strengthen their cyber defences [3]. SMMEs,
like large enterprises, face similar cybersecurity challenges and are equally vulnerable
to cyberattacks. The cybersecurity challenges confronting the SMMEs form the basis of
the research problem for this paper.
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2024
K. Arai (Ed.): SAI 2024, LNNS 1017, pp. 443–456, 2024.
https://doi.org/10.1007/978-3-031-62277-9_29
444
N. Ntingi et al.
SMMEs, particularly in developing countries, can no longer afford the “cut and
paste” approaches to defending against cyber threats as this can only escalate the cost
of implementing cyber defence solutions. The author in [4] notes that the cur