Generative Artificial Intelligence, or GenAI, is a transformative technology of the 21st
century that has revolutionized several industries such as healthcare, utilities, and transportation.
Its rapid expansion and adoption have yielded numerous practical benefits and uncovered a
multitude of security risks to information systems. At its core, GenAI refers to deep-learning
models that can take raw data inputs, “learn”, and generate meaningful outputs when prompted
(IBM). However, this capacity for generation, which makes it so powerful, also poses the
greatest security vulnerability to information systems. As the technology grows more accessible
and its outputs become harder to distinguish from human work, the risks to information
assurance grow more complex and urgent.
The earliest conception on generative artificial intelligence is owed to the development of
ELIZA, a conversational program created by Joseph Weizenbaum in 1966. This program utilizes
pattern matching to answer questions when prompted. ELIZA utilized custom templates that
flavored the response to prompts, the most famous of which – “the doctor”, mimicked a
psychotherapist. This simple advancement demonstrated that, given a simple set of rules, a
machine could simulate human conversation. It captured a spark of curiosity among developers
that fueled early wave of AI development. While ELIZA lacked any true understanding, it
showed that with a clever enough structure, computers could imitate intelligence.
From 1980-2000, language processing moved from rule-based programming featured in
ELIZA to statistical methods that utilized a probabilistic approach. This fundamental change
removed predetermined responses and freed programing to generate creative outputs. By
utilizing different models such as “n-gram” programs could guess at upcoming words in a
sentence. Hidden Markov Models (HMMs) and Naive Bayes classifiers further expanded this
capability by adding memory and classification, enabling more refined text predictions. This
shift unscored the march towards data-driven learning that examined large data sets to make
predictions and generate meaningful outputs. Though the responses were still limited in depth,
these models introduced the critical notion that machines could learn language patterns, not just
repeat them.
In the early 2000s, the development of deep learning was the revolution that catapulted
AI development. Despite meaningful advancements before, progress was slowed by poor
funding and insufficient computational power-a period often referred to as the AI winter. Neural
networks allowed large data sets to be analyzed over time and yielded improvements over
predictions n-grams had previously generated. Specifically, recurrent neural networks (RNNs)
had the capacity to store and contextualize previous information which guided their responses
when prompted. Furthermore, deep learning application extended beyond text and language into
voice and video. These breakthroughs produced speech recognition software such as SIRI and set
the stage for more generalized AI systems capable of understanding and generating diverse
forms of human data.
An interesting and significant development to applications of neural networks, conceived
by Ian Goodfellow, pitted two neural networks against each other. Coined Generative
Adversarial Networks (GANs), the competition between two antagonistic software drove output
improvements. Famously, StyleGAN used a neural network to produce fake photos and another
to detect fake photos. Over time, the generative photos became indistinguishable from real
photos. The key breakthrough realized from GANs is the ability to develop unique original
content and thus Generative AI was born. GANs pushed GenAI into the visual domain and
raised awareness of synthetic media’s potential risks, including identity fraud, deepfakes, and
disinformation.
Soon pretrained neural networks named generative pretrained transformers (GPTs)
became public and the movement exploded into the public sphere. These models utilized the
transformer architecture initially released in 2017, which significantly improved
context persistence and scalability. Transformers utilized the entire sequence of input at once,
rather than word by word, to facilitate more uniform and longer-length generation. GPTs were
trained on enormous and structured collections of text, allowing them to generate surprisingly
human-like output across virtually any topic. OpenAI launched GPT-2 in 2019 with the first
public acknowledgement of Generative AI’s capacity to be used for malicious intent.
Researchers initially withheld the trained model, citing concerns over the potential misuse in
phishing and misinformation campaigns. The program exhibited the ability to produce fluent,
seamless, natural, and context-aware text - perfectly suited phishing applications or
impersonating communications.
The launch of GPT-2 represented an important departure in how generative AI is viewed.
It wasn’t merely an advancement in research; it demonstrated real, present dangers. The very
traits that make GenAI useful for producing customer support scripts, educational pieces, or
business communications also made them suitable for exploitation. Now when it comes to
writing, threat actors don’t need any writing skill themselves, or even knowledge of ways to
manipulate. All that is required is minimal input and GenAI will produce competent,
grammatically perfect, believable output. This led to serious questions about situations in which
tools like GenAI could be exploited to bypass human agency, manipulate trust, and disrupt the
integrity of information systems. As GenAI continued to evolve, new avenues of misuse in
cybersecurity were realized and a pathway formed for a new class of digital threats. An age-old
example malicious content transformed by GenAI is phishing.
Traditional phishing is considered a common type of cyber-attack that targets people
through email, text, or phone calls. The message is aimed at deceiving a person into taking a
desired action. The focus of the attack is to uncover financial information, login credentials, or
other sensitive information. The nature of these attacks exploits humans and not technical
vulnerabilities. With the exploit focus on humans, the nature of the phishing attack will vary
drastically. Common attacks attempt to establish credibility by appearing to be a company or
reputable entity such as a government agency. The message will attempt to create a narrative and
introduce a sense of urgency; this could manifest as a large cash prize or a significant financial
penalty. Lastly, the prompt will provide an immediate means of remedy that ultimately will
exploit the user for the desired information.
The advent of generative AI has both enhanced the “quality” of targeted phishing attacks
and drastically increased the scale at which they are utilized. Traditional phishing attacks, while
deceiving to some, typically lacked the human element of realism and were readily identifiable.
Tail tale identifiers such as poor grammar, awkward wording, or downright obvious contractions
all remedied by generative artificial intelligence, making today’s attacks far more effective and
difficult to detect. This shift has introduced the hyper-personalized attack where attackers train
models to incorporate conversation style by a company and/or person. These personalized
attacks strengthen the phisher’s narrative which ultimately improves the perceived credibility of
the deception. With access to previous email threads or scraped public communications, an
attacker can exploit those data sets to train GenAI and tailor its responses to convincingly mimic
the tone, choice of words, and sentence structures of a trusted source.
Such a level of sophistication has transformed those broad psychosocial spam campaigns
into carefully targeted psychological operations. GenAI tools are now used to engineer scenarios
of spear phishing, BEC, or vendor impersonation and generate convincing and contextual
messages. Attackers no longer blindly guess what could work-they now deploy campaigns that
are painstakingly tailored to the victim environment and its vulnerabilities. Even with vishing,
generative models have found a way to clone a voice if given short snippets of audio.
Generative AI utilizing deep learning developed synthetic media that imitates a person’s
likeness through voice, video, or picture. This synthetic media coined a “deep fake” - has opened
several avenues for bypassing or subverting security systems to obtain sensitive information.
This is a bleeding-edge method in impersonation using a technology called "Deepfake." It is
used to attract victims with the bait to make it a successful phishing attack. These attacks mimic
entrusted individuals and target specific users with malicious instructions. An example may
include AI impersonating a vendor and instructing a purchasing manager to route payment to a
fraudulent account.
Deep Fakes have also exploited biometric security systems. These systems rely on
qualities unique to a person, such as fingerprints, facial features, or iris patterns to encrypt or
secure information. Generative AI has been successful at bypassing the “liveness detection”
protocols meant to prevent the use of static or artificial inputs. Attacks may permeate systems
through and during virtual camera injections, deep-fake video loops, or spoofed facial motion
overlays; all are often meant to trick systems into acknowledging as authentic erroneously
synthesized identities. The more the blurred line between the real and the generated becomes, the
less will biometric authentication stand as a reliable tool.
A strikingly concerning development of GenAI is polymorphic malware. Put simply, this
is malware that can adapt is code or structure and achieve the same malicious outcome. The
advantage being that traditional antivirus detection relied on code “signatures” to classify and
protect systems. Furthermore, this technology is being used to intentionally disguise the function
of simple code and make it more challenging to reverse engineers or analyze antivirus failures.
Generative AI has supercharged the automation of exploit development. AI can quickly
analyze massive amounts of software coding data, recognize patterns of code with
vulnerabilities, and produce exploits rapidly. The volume of these generated attacks overwhelms
incident response teams and significantly reduces the effectiveness of patch management. By the
time a team can identify the vulnerable code that led to an exploit, fix the code, and update the
software – generative AI has identified two more potential exploits.
The implications for information assurance are profound. Software attacks are adaptable,
single patches to recognize a signature won’t cut it. Attacks are obscured and challenging to
interpret reducing the effectiveness of a trained response. Generative AI brings scalability
drastically improving the ability to interpret source code leading to an avalanche of attacks. AI
generated code has changed the rules for security and companies must adapt to meaningfully
protect systems and data.
Companies have leaned on generative AI in security development to match the sheer
magnitude of attacks. This has led exploiters to develop methods of compromising AI security.
Data poisoning targets the “learning” portion of AI creation. In essence, this strategy is to inject
malicious data into a training set with the intention of compromising how the AI builds its
model. Targeted data poising can subtly teach AI to ignore malware or fail to recognize
vulnerabilities in code. Due to the nature of this attack making a causal relation from exploitation
to a poisoned data set is extremely difficult. With companies rushing to roll out generative ai
security the training set becomes an extremely important asset and target for bad actors.