2/24/2023
Custom GPT3 Chatbot
RESEARCH PROPOSAL
THE UNIVERSITY OF LEEDS
MASTER OF SCIENCE: ARTIFICIAL INTELLIGENCE
MODULE: OCOM5204M DATA MINING AND TEXT ANALYTICS
Background
Artificial Intelligence has come of age, and there is an increasing demand for AI – powered applications
and solutions to enhance digital platforms. It is believed that great revolution in dialogue systems (Voice
User Interface, Chatbot, Conversational Agent, Digital Personal Assistant) came in 2011 when Siri an
Apple product was launched (McTear, 2021).
The emergence of advanced dialogue systems or programs is gaining popularity in the field of artificial
intelligence, particularly in natural language processing and deep learning, owing to the availability of
vast amounts of data and immense computing power (McTear, 2021).
A commonly used artificial intelligence (AI) solution is a customized bot, also known as a chatbot. As
described by (Mujeeb, et al., 2017), a chatbot refers to a software, program, or system that imitates
human-like response behavior in text-based communication. An effective chatbot is one that can provide
responses that closely resemble those of a human.
Chatbots can be grouped as either conversational agents, mimicking human chatting or task-based
agents that gets user input to complete a specified task such as robots or personal assistant (Siri,
Google Now, Cortana etc.). Conversational agents/Chatbots engages user in normal natural language
either in text or speech.
Chatbots or Conversational agents depends Natural Language Processing (NLP) which is a specialized
area of artificial intelligence that is focused on the understanding and manipulation of human language.
The various techniques used in NLP include tokenization, stemming, part-of-speech tagging, syntactic
analysis, and semantic analysis on data (text or speech). Two of the great models to change the
landscape in NLP are the BERT and GPT’s. During research, a decision had to be made between BERT
(Bidirectional Encoder Representation Transformer) and GPT (Generative Pre-Trained Transformer)
models, both of which work well with chatbots.
BERT is a bidirectional model that considers both the left and right context of a token when creating its
representation. It is pre-trained using two tasks: masked language modeling (MLM) and next sentence
prediction (NSP), as explained in (Devlin, et al., 2018). MLM involves masking a certain percentage of
input tokens, and the model is trained to predict the masked tokens based on the remaining context.
NSP, on the other hand, trains the model to predict whether two input sentences are consecutive or not.
GPT, and specifically GPT-3, is pre-trained using a single task: predicting the next token in a sequence.
It generates token representation based only on the left context. As described in (Brown, et al., n.d.),
the model consists of a stack of Transformer blocks, each of which contains multi-headed self-attention
mechanisms and feedforward neural networks. GPT-3 is one of the largest language models ever
developed, with 175 billion parameters. Nevertheless, NLP is an area that is getting tremendous
research with great strides in better efficient models.
When it comes to chatbots, BERT is better suited for extracting specific information from a user's input,
such as providing customer support or answering frequently asked questions, due to its ability to
understand the context of the input and extract relevant information. It can also classify the intent of the
user's input, which can be helpful in directing the conversation to the appropriate response (Devlin,
2019). GPT, on the other hand, is better suited for generating natural and coherent responses, which
can be useful for conversational chatbots or personal assistants. GPT can also generate multiple
responses based on different variations of the input, which can help make the chatbot more engaging
(Brown, et al., n.d.).
1
Based on this premise, the main goal of the chatbot is to provide a more natural and engaging response
to user inputs, rather than just extracting specific information from the Ashghal website. Both BERT and
GPT have demonstrated high levels of accuracy and prediction for information extraction, as seen in
(Devlin, et al., 2018). However, in this study, it is proposed to use GPT-3 for the chatbot, given its ability
to generate more human-like responses that are engaging and diverse.
Public Works Authority (Ashghal) is a government body of the state of Qatar responsible for designing,
constructing, and managing infrastructure (roads, drainage systems, public buildings, tunnels, and
bridges) projects. Ashghal Website https://www.ashghal.gov.qa/en is a digital platform that provide
information regarding projects, services and initiatives for the public to the citizens, resident, various
stakeholders like constructing companies and the general public.
A Custom Chatbot integrated to the website will enhance user experience, providing real-time feedback
24/7 and address issues in an efficient manner since its powered by GPT3 unlike customary means of
communication in emails and calls that are a bit slow and response restricted to specific working days
or hours (Daswani , et al., 2020).
This research paper explores and propose use of an embedded chatbot powered by GPT3 in Ashghal
website that will enable officials, engineers, professionals, citizens and the public to interact, access and
connect with Ashghal in a timely, efficient, fast, affordable and easy way, building user experience of
the website.
Contribution to knowledge.
The adoption of GPT models in conversational AI and bots in Qatar can stimulate further research on
similar projects in the Arabic language, enhancing the field of NLP and AI research in the Arab world.
These powerful technologies are revolutionizing the way business is conducted, and their use in
chatbots can awaken and encourage the wider community to embrace safe AI, NLP, and ML techniques
in various industries, including government agencies, e-commerce, healthcare, and others.
This development will strengthen any plans, frameworks, or policies initiated by other bodies in the
region, including Qatar.
Importantly to note, majority of the previous studies might have focused on general Chatbots such as
papers titled "A Survey of Chatbot Systems Through the Lens of Natural Language Processing" by
Serban et al. (2019) or "Designing Chatbots for Customer Service: A Literature Review" by Sahu and
Gupta (2020). These papers are very informative though gives generic overview of chatbots types, NLP
techniques, design principle and development techniques, challenges, and limitation of chatbot etc. This
research proposal is unique and cut itself apart for it is industry or domain centric.
Importance.
By embedding a chatbot named "PWA ChatbotGPT" directly into the Ashghal website, users can enjoy
a unique and seamless experience without having to navigate through multiple pages or applications.
Most chatbots are rule-based and rely on retrieving predetermined answers or matching patterns in text
sequences, which can often result in inaccuracies. While they are relatively easy to create, they lack the
ability to personalize the conversation to each individual user (Caldarini, et al., 2022). However, GPT
models are trained on large amounts of text using a vast number of parameters. By utilizing generative
transformers and training them on the content of the Ashghal website, the chatbot can provide
contextually appropriate and personalized answers that address the specific needs of each user,
resulting in an end-to-end solution.
2
Automating customer service operation will free up customer service representative In Ashghal, to other
important tasks. Website admins will also be freed to attend to other important tasks such website
maintenance and administration. This greatly increase efficiency and use of resources.
Accessing information through hotline numbers(calls) or emails is almost impossible due to the many
players involved. In some scenario, information is nested deep in web layers (Daswani , et al., 2020)
and administered by different departments or sections.
With all the benefits stated, several challenges must be overcome. GPT3 model requires great amount
of data just like any NLP or Machine language task. At our disposal we have a small dataset of merely
180,000 words. It might also require a team with different expertise to design a Chatbot that will be
embedded in Ashghal website that runs in GPT3, getting such a team with deep know-how of Natural
Language Processing and Machine Learning, software development and goodwill or corporation from a
government body by different players might be a challenge.
To overcome such challenges, Transfer Learning where Ashghal text is used to pre-train or fine tune an
already existing model can be employed. Also, a good concept highlighted in “topic-specific few-shot
learning” by (Sphere, 2022) would come handy, if need be, to overcome the dataset size issue. Expertise
and Corporation can be dealt with by collaboration between different Team Leads or Line Manager in
coming up with a very lean efficient team.
Research Hypothesis and Objectives.
We hypothesis a scenario where a user will engage a chatbot and gets response that is wide in insight,
informative, accurate and done in a quick efficient manner, saving on time enhancing ease of doing
business in digital government platforms. For this to become a reality a more advanced way of creating
robust efficient chatbot need to be embraced and pursued.
Transformers, encoder-decoder, Attention mechanism concept and the major strides done in Natural
Language processing evident by the ChatGPT need to be harnessed for betterment of service delivery.
This research proposal objective is:
 Training the text corpus from Ashghal website using state-of-the-art GPT3 language generation
model, an Open AI product that can retrieve content and generate with context accurate
response to a user satisfaction.
 Create a chatbot that is insightful, wide in scope of topics and coherent in response like a human
being.
 Increase different users experience that inwardly increase Ashghal effective communication.
The envisioned ramification of this approach is far-reaching, changing the scope of communication,
inducting, and sensitizing the entire region in AI solution and products.
Program and Methodology.
Chatbot will provide Information such as road closure and maintenance schedules, road opening,
construction progress updates, traffic diversion and updates.
In a study of Edubot for student survey (Abbas, et al., 2021), we see how appealing it is a one-on-one
conversation that saves a user time and effort.
Below is a Gantt chart of the task undertaken in this research proposal.
3
Figure 1: Gantt Chart of formulating this research proposal, indicating duration of time spent on each segment.
The process of designing a successful chatbot requires careful consideration of the conversation flow,
with a balance between information generation and question-and-answer responses. Additionally,
providing engaging prompts to encourage user participation is essential. A simple example of a
conversation flow as below could be used as a starting point.
User: Hi
Bot: Hello! How can I assist you today?
User: I want to know about road maintenance projects.
Bot: Sure, can you please specify which road in particular?
User: Al Waab Street
Bot: We have a scheduled diversion and maintenance on Al Waab Street on 10th March – 11th
March. The road will be closed from 11pm to 5am on Friday, night of 10th March. Traffic will be
diverted to Rasheeda Street to Mohammed Bin Hamad to Furousiya Street. Is there anything
else about maintenance project you would want to know?
User: Yes, is there an alternate route during the closure?
Bot: Yeah, we recommend taking the Al Saad Street or Rasheeda Street. Is there anything
else about maintenance project you would want to know?
User: Yes, I want to know the construction projects on Al Rayyan Road.
Bot: Sure, we have a new interchange construction on Al Rayyan Road. Hope this information
is helpful, is there anything else about maintenance project you would want to know?
User: Which company is undertaking the project.
Bot: Metito oversees Doha West and Al Rayyan. But you can visit Project Affairs in Ashghal
Tower 1 in Doha City Centre for more details such as the Engineer in charge and the different
stakeholders involved. Is there anything else about maintenance project you would want to
know?
User: No thank You.
Bot: You’re much welcomed, Qatar deserves the 4best and we are here to give you the best.
Don’t hesitate to reach out to us anytime 24/7 for any concerns, cheers!
The Sketch Engine tool, which is a powerful data mining tool designed for managing large text and text
analysis,
was used to extract English text data from the Ashghal website
(https://www.ashghal.gov.qa/en). Developed in 2013 by Lexical Computing, Sketch Engine provides
various analytical functionalities such as concordance, wordlist, frequency list, key word analysis, and
word sketches (Biasi, 2021). These functionalities help to enrich text understanding and gain insights
on the pattern of the data. An overview of the collated data from the Ashghal English website provides
information on the number of words, tokens, sentences, paragraphs, and documents.
Figure 2: Detail description of Ashghal website text corpus.
The extracted data from Sketch Engine is saved in two files: a structured txt file without part of speech
tags and a .vert file with part of speech tags. However, before these files can be used to train the GPT3
model, they need to be cleaned to remove unwanted special characters, HTML tags, and other
formatting that may interfere with learning.
Once the data has been cleaned, it should be tokenized using a simple python function or other libraries
such as NLTK and spaCy. Tokenization involves splitting the words into an input format that can be
understood by the GPT3 model.
To identify and deal with stop words, such as "a," "the," and "and" word frequency analysis can be
conducted. This analysis can also be used to classify the data into distinct categories, such as news,
events, projects, and resources, using WEKA. This insight can guide the splitting of the data into training,
testing, and validation sets, ensuring balance in each category.
Access to the OpenAI GPT3 model is available to the public for a nominal fee, and a unique user key is
issued upon sign-up or subscription. Using the unique key, the API can be called in Python to pass user
queries and generate text via a chatbot on the website. Below highlights calling of GPT3 API.
5
Figure 3: An example of calling Open AI GPT3 model of one’s choice. Assumption is openai library is not installed
(Pip install).
After the GPT3 model is developed, it can be integrated into the website to allow users to pass queries
to the model through the API, and receive generated text displayed via a Chatbot. Developing a Chatbot
that is efficient and insightful requires expertise in Conversational Software design and development,
and there are several platforms available for this task, such as Dialogflow, Botpress, and IBM Watson.
Incorporating frequently asked questions and answers from various departments' in-house surveys into
the training data can help fine-tune the GPT3 model to learn the nature of frequent questions and
answers. After integrating the chatbot into the Ashghal website using a chatbot platform's integration
tools, testing the bot before deployment is necessary to ensure its intended functionality are met.
6
References
Abbas, N., Pickard, T., Atwell, E. & Walker, A., 2021. University Student Surveys Using Chatbots: Artificial
Intelligence Conversational Agents. In: Learning Collaboration Technologies: Games and virtual Environments
for Learning. HCI International 2021. Washington DC, Springer, Cham.
Abbas, N., Pickard, T., Atwell, E. & Walker, A., n.d. University student surveys using chatbot. s.l., s.n.
Adel AL-Zebari, S. Z. K. J., 2020. Football Ontology Construction using Oriented Programming. JOURNAL OF
APPLIED SCIENCE AND TECHNOLOGY TRENDS, 01(2708-07 57), pp. 24-30.
Biasi, P., 2021. Using Sketch Engine - An analysis of nouns normally considered uncountable.Thesis for: Master's
Degree in Languages, Tourism and CulturesAdvisor: Dominic Stewart, s.l.: s.n.
Brown, T. B. et al., n.d. Language Models are Few-Shot Learners, s.l.: arxiv:2005.14165 40+32 pages .
Caldarini, G., Jaf, S. & Mcgarry, K., 2022. A Literature Survey of Recent Advances in Chatbots. Information 2022,
13, 41., Issue 2022, p. 44.
Cambria.E & White.B, 2014. Jumping NLP Curves: A Review of Natural Language Processing Research [Review
Article]. IEEE Computational Intelligence Magazine 9(2), pp. 48-57.
Daswani , M. et al., 2020. CollegeBot: A Conversational AI Approach to Help Students Navigate College. In:
Stephanidis, C., Kurosu, M., Degen, H., Reinerman-Jones, L. (eds) HCI International 2020 - Late Breaking Papers:
Multimodality and Intelligence. HCII 2020. Lecture Notes in Co. s.l., Springer, Cham.
Devlin, J., Chang, M.-W. & Lee, K. &. T. K., 2018. BERT: Pre-training of Deep Bidirectional Transformers for
Language Understanding. s.l., cite arxiv:1810.04805Comment: 13 pages.
Devlin, J. C. M.-W. L. K. &. T. K., 2019. BERT for Joint Intent Classification and Slot Filling. In Proceedings of the
2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human
Language Technologies (NAACL-HLT) (Vol. 1, pp. 4171-4186). Association for C. s.l., s.n.
McTear, M., 2021. Conversational AI: Dialogue Systems, Conversational Agents, and Chatbots. s.l.:Morgan &
Claypool.
Mujeeb, S., Arshad, T. & Javed, M., 2017. Aquabot: A Dialogue Chatbot for Achluophobia and Autism.
International Journal of Advanced Computer Science and Applications, Volume 8, No.9, p. 209.
RebecqCategories, A., 2016. Data analysis of the French football league players with R and FactoMineR.
[Online]
Available at: https://nc233.com/2016/10/data-analysis-of-the-french-football-league-players-with-r-andfactominer/
[Accessed 22 6 2022].
Sphere, L., 2022. Web-Based Chatbot Project, Module 2: GPT-3-generated responses assisted with a database
for topic-specific few-shot learning. [Online]
Available at: https://towardsdatascience.com/web-based-chatbot-project-module-2-gpt-3-generatedresponses-assisted-with-a-database-for-1ee2a0f4aad4
[Accessed 24 February 2023].
7