Bot or not? How to tell when you’re
reading something written by AI
AI has gotten good at writing. Here’s how experts say
you can identify it.
By Clare Duffy and Kenneth Uzquiano
Illustration by Will Mullery
July 11, 2023
At some point this week, you may have read a sentence written by artificial
intelligence and not even known it.
In the wake of the viral success of ChatGPT late last year, a growing
number of tools have launched that make it cheap and easy for nearly
anyone to create convincing AI-generated text — from school
essays and work emails to news articles and real estate listings.
But the recent explosion of those tools creates a new challenge for internet
users: how to tell whether something was written by a human or a machine.
It’s more than just a thought exercise — these AI tools often confidently
assert false or misleading information. Learning to spot AI text will be an
increasingly vital skill for anyone who wants to be an informed consumer
and colleague.
The good news is that people can be trained to identify AI-generated text,
according to research published earlier this year by a group at the
University of Pennsylvania. Although AI models are rapidly improving,
“people can learn to spot them,” Chris Callison-Burch, a computer and
information science associate professor who led the research, told CNN.
“I think it’s an important enough issue that we’ll have to teach everyone
how to do it,” he said.
In many cases, the signs of computer-generated text have changed over
time, Callison-Burch said. While AI tools used to frequently make
grammatical errors, for example, computers are now pretty good at creating
human-sounding text. These days, people should instead be on the lookout
for oddly generic or repetitive writing, as well as factual errors.
The research group worked with CNN to create the following text samples
with ChatGPT to highlight some of the common signs of AI-generated text.
Is this economy news article written by a human or AI?
“What to expect from the Fed meeting as Powell faces a legacy-defining moment”
(Short)
Prompt
Please generate me a 3-4 paragraph economics article in the style of CNN
with the following title “What to expect from the Fed meeting as Powell
faces a legacy-defining moment”
According to the research team, this ChatGPT creation “is a pretty rough
one,” featuring a number of falsities.
To start, the CPI figure cited is incorrect. CPI rose by 0.1% in March 2023
and 1.2% in March 2022, so it’s unclear what the model was trying to say. It
turns out that CPI increased 2.6% for the 12 months ending in March 2021,
which appears to be the reason for the “hallucination” — a term used by AI
industry experts to describe a false statement generated by the technology.
Jerome Powell remains the Fed Chair, but the generated article falsely
suggests that Powell’s term came to an end in February 2022. The piece
also claims that the Fed is debating “tapering its bond-buying program,”
although it had already begun cutting back on buying bonds in September
2022.
As the US economy continues to recover from the pandemic, all eyes are
on the Federal Reserve's upcoming meeting on interest rates and
monetary policy. This meeting is particularly significant for Fed Chair
Jerome Powell, who faces a legacy-defining moment as his term comes to
an end in February 2022.
One key issue that the Fed will likely address is inflation, which has been
on the rise in recent months. In March, the Consumer Price Index (CPI)
rose by 2.6%, its largest increase in over eight years. While some experts
believe that this inflation is temporary and will level off as the economy
continues to recover, others are concerned that it could become a more
persistent problem.
Another area of focus for the Fed is the ongoing debate over whether the
central bank should begin tapering its bond-buying program. This program,
which was implemented in response to the pandemic, has helped keep
interest rates low and provided a boost to the economy. However, with the
economy showing signs of improvement, some policymakers argue that it's
time to start scaling back these purchases.
All in all, the Fed's upcoming meeting is expected to be a pivotal moment
for the US economy. Powell faces tough decisions on how to balance
inflation concerns with the need to continue supporting the recovery. As
always, the world will be watching to see how the Fed navigates these
challenges and what impact its decisions will have on the broader
economy.
“Surge in demand for toys causes unexpected ripples in the real estate market”
(Short)
Prompt
Please generate me a 3-4 paragraph economics article in the style of CNN
with the following title “Surge in demand for toys causes unexpected ripples
in the real estate market”
For this sample, the researchers asked ChatGPT to generate an
economics news article about an event that is not actually happening. They
then asked it to shorten its response.
The model wrote an article claiming that families have turned to external
storage solutions on the outskirts of urban areas in order to house their
ever-growing collection of toys — a rather unlikely scenario.
The article is also incredibly repetitive. The phrase “surge” or “increase in
demand” appears eight times, and “led to an increase in” also appears
repeatedly. This awkward repetition is a common issue with generative
models.
The toy industry has seen a significant surge in demand, leading to an
unexpected ripple effect in the real estate market. With more families
spending time at home due to the pandemic, the demand for toys and
games has skyrocketed. However, the increase in demand for toys has
also led to an increase in demand for storage space, particularly in the real
estate market.
The increase in demand for storage space has been particularly
pronounced in urban areas, where space is at a premium. Many families
are finding it difficult to accommodate their growing collection of toys and
are turning to storage solutions to help them free up space in their
homes. This surge in demand for storage space has led to an increase in
the demand for self-storage facilities, which are typically located on the
outskirts of cities.
The surge in demand for self-storage facilities has also had an impact on
the real estate market, particularly in areas where there is a shortage of
available land. Developers are finding it increasingly difficult to secure land
to build new self-storage facilities, and this has led to an increase in the
cost of land in certain areas. While the surge in demand for toys has had a
positive impact on the toy industry, it has also had unexpected
consequences in the real estate market, particularly in urban areas.
“Big Banks Experience Deposit Spike After Silicon Valley Bank Collapse”
Prompt
Please generate me an economics article in the style of CNN with the
following title “Big Banks Experience Deposit Spike After Silicon Valley
Bank Collapse”
For this sample, the researchers asked ChatGPT to generate an article
based on a real CNN headline. Since ChatGPT does not know about the
collapse of Silicon Valley Bank, it doesn't do a great job of explaining it.
The article that follows this headline is extremely generic to the point of
being suspicious, including the vague statement that the bank collapsed
due to "a series of bad loans.” It also quotes “John Doe” at the fictional
“XYZ Consulting,” and makes up a fake name for the CEO of Bank of
America.
New York (CNN) - Following the recent collapse of Silicon Valley Bank,
several big banks in the United States have reported a sudden surge in
deposits from both individual and corporate clients. This has come as a
surprise to many analysts who had predicted a slowdown in the banking
sector amid the ongoing economic uncertainty.
Silicon Valley Bank, which primarily served the tech and startup industries,
was one of the fastest-growing banks in the country until its recent
downfall. The bank was hit hard by the economic slowdown caused by the
COVID-19 pandemic, as well as a series of bad loans and mismanagement
of funds.
The sudden collapse of Silicon Valley Bank has led to a loss of confidence
in the banking sector, prompting many clients to shift their funds to more
established and stable institutions. Bank of America, JPMorgan Chase, and
Wells Fargo are among the major banks reporting an increase in deposits
in the weeks since the collapse.
Industry experts say that the surge in deposits is a reflection of the current
economic climate, as clients seek security and stability amid the ongoing
pandemic. The collapse of Silicon Valley Bank has also highlighted the
need for stricter regulation and oversight of the banking sector.
"The recent events have shown us that even the fastest-growing and most
innovative banks are not immune to economic downturns," said John Doe,
a banking analyst at XYZ Consulting. "This has put a spotlight on the need
for stronger regulations and risk management practices to ensure the
stability of the banking sector."
Despite the challenges faced by the banking industry, experts remain
optimistic about its long-term prospects. The recent surge in deposits is
seen as a positive sign that clients continue to have faith in the banking
system and its ability to weather the storm.
"We're confident that the banking sector will emerge from this stronger than
ever," said Jane Smith, CEO of Bank of America. "We're committed to
providing our clients with the stability and security they need during these
challenging times."
As the banking sector navigates the fallout from the collapse of Silicon
Valley Bank, industry leaders are calling for a renewed focus on risk
management, innovation, and collaboration to ensure the long-term stability
and growth of the industry.
Is this tech product review written by a human or AI?
Apple iPhone 14
Prompt
Write a technical product review in the style of CNN about the new iPhone
14
For this sample, the researchers asked ChatGPT to generate a review of
the iPhone 14, a product that was not out at the time of ChatGPT’s training
in September 2021. What follows is an article that gets a few details wrong,
although they may be hard to spot for all but the most savvy tech readers.
The article highlights the device’s “6.7-inch OLED screen,” although the
iPhone 14 actually has a 6.1-inch screen (the larger frame belongs to the
iPhone 14 Pro).
In the fourth paragraph, the model states that the phone has a 108megapixel camera, far more than its actual 12-megapixel camera. This is a
pretty major error. Other hallucinations include claiming the device has an
A18 chip (the real product uses the A15) and a triple camera lens (in reality
non-Pro iPhones tend to only have double lenses).
The article also fails to cite concrete details that readers would expect from
a human-written product review, such as the price, battery life and
comparisons to other products on the market.
The iPhone 14 is the latest addition to the Apple family of smartphones,
and it promises to be a game-changer in the industry. With a sleek design
and cutting-edge features, this device is sure to impress even the most
discerning tech enthusiasts.
One of the standout features of the iPhone 14 is its display. The device
boasts a stunning 6.7-inch OLED screen that offers incredibly vivid colors
and deep blacks. This makes it perfect for streaming videos or playing
games, as well as for everyday use.
In terms of performance, the iPhone 14 is powered by the latest A18 Bionic
chip, which is the fastest processor ever used in an iPhone. This means
that the device can handle even the most demanding tasks with ease, and
users can expect lightning-fast speeds and seamless performance.
Another key feature of the iPhone 14 is its camera system. The device
features a triple-lens setup that includes a 108-megapixel main camera, an
ultra-wide lens, and a telephoto lens. This allows users to capture stunning
photos and videos with incredible detail and clarity, and the device also
includes advanced features such as night mode and optical image
stabilization.
In addition to these features, the iPhone 14 also includes a number of other
upgrades that make it an excellent choice for anyone in the market for a
new smartphone. These include improved battery life, 5G connectivity, and
the latest version of Apple's iOS operating system.
Overall, the iPhone 14 is an impressive device that offers a range of
cutting-edge features and top-of-the-line performance. While it may come
with a higher price tag than some other smartphones on the market, the
device is well worth the investment for anyone who demands the best in
terms of technology and performance.
Apple AirPod Flex
Prompt
Write me a 3 to 4 paragraph tech product review for a fictional product
called the Apple AirPods Flex
For this sample, the researchers wanted to generate a review for a fictional
product that does not exist but was plausible, so they invented the AirPod
Flex.
The resulting article explains that the headphones are made entirely from a
“soft silicone material,” which is confusing given that it also states that the
headphones have “touch controls.”
This review is also relatively repetitive and generic, and fails to mention
basic details such as the price. It also makes no mention of any potential
downsides of the product, as is common in tech reviews.
The Apple AirPods Flex are the latest addition to Apple's audio accessory
lineup. The AirPods Flex are the first wireless earbuds from Apple
to feature a completely flexible design that allows them to comfortably fit
any ear shape or size. The earbuds feature Apple's signature sound quality
and are powered by the H1 chip for seamless connectivity with your Apple
devices. The AirPods Flex also come with an IPX4 water resistance rating,
which makes them ideal for use during workouts or rainy weather.
One of the most impressive features of the AirPods Flex is the flexibility of
their design. The earbuds are made from a soft, silicone material that
conforms to the shape of your ear, providing a comfortable and secure
fit. The earbuds also feature touch controls, which make it easy to pause,
skip, or play music with just a tap. The AirPods Flex also come with a
compact charging case that provides up to 24 hours of battery life, ensuring
that you'll never have to worry about running out of power.
Overall, the Apple AirPods Flex are a fantastic addition to Apple's audio
accessory lineup. The flexible design and touch controls make them
incredibly easy to use, and the sound quality is excellent. The IPX4 water
resistance rating and long battery life make them a great choice for anyone
who wants to use them during workouts or outdoor activities. If you're
looking for a high-quality pair of wireless earbuds, the Apple AirPods Flex
are definitely worth considering.
Is this tweet written by Elon Musk or AI?
Prompt
Please generate me a set of tweets from Elon Musk
The researchers asked the model to generate tweets from billionaire
Twitter owner Elon Musk, without giving it any examples. The resulting
tweets are very formulaic, each with three hashtags at the end. While the
Tesla CEO does frequently tweet about his companies, he is more often
dry, sarcastic and, occasionally, controversial. He’s also not much of a
hashtag guy.
Another signal that these tweets are AI-generated: Musk is known for his
controversial commentary, but these posts lack any potentially
inflammatory content. ChatGPT tends to stay away from that.
Just visited the SpaceX launch site - exciting progress on Starship! We're
on track to make space travel accessible to everyone. #SpaceX #Starship
#MarsBound
Cybertruck reservations are through the roof - thank you all for the
incredible support! This vehicle is going to be a game changer for both
utility and aesthetics. #Cybertruck #ElectricTruck #FutureIsHere
Just had a great meeting with Dogecoin developers. They're making
important updates to increase transaction speed and reduce
fees. Dogecoin has great potential as a global currency for the internet
age. #Dogecoin #Cryptocurrency #ToTheMoon
Prompt
Okay now please generate me a set of funny Elon Musk tweets
The next prompt specifically called for funny tweets inspired by Elon Musk
and while the results are a little more interesting, the tweets maintain the
formulaic use of hashtags.
Just realized that "Elon Musk" is an anagram for "Nuke Slum." Sorry, guys,
my bad. #AnagramFun #Oops
When I was a kid, my parents always said, "Elon, you can be anything you
want to be!" So I became an alien lizard overlord. #LifeGoals #JustKidding
Is this recipe written by a human or AI?
“Tommy John” Cocktail
Prompt
How do I make a "Tommy John" cocktail
For this sample, the researchers asked ChatGPT how to make a fictional
cocktail called a “Tommy John.” Although the drink name was made up,
ChatGPT produced a response that says the Tommy John is a “classic
cocktail.”
ChatGPT hallucinates something that is similar to a Bee’s Knees (2 oz gin,
3/4 oz lemon juice, 1/2 oz honey syrup) but with grapefruit, lime and bitters
instead of lemon. The researchers noticed that ChatGPT’s recipe calls for
double the citrus that the Bee’s Knees has and questioned whether the
drink’s ratios would work.
Reporter’s note: In the name of journalism, I tried out the AI’s “Tommy
John” creation and it was not half bad! This may just be one case when
ChatGPT makes up something useful…
The Tommy John is a classic cocktail that is not widely known, but it's easy
to make. Here's the recipe:
Ingredients:



2 oz. gin
1 oz. grapefruit juice
1/2 oz. lime juice

1/2 oz. honey syrup (1:1 honey and water) 2 dashes of Angostura
bitters
Instructions:
Fill a cocktail shaker with ice.
Add all of the ingredients into the shaker. Shake well for about 10-15
seconds.
Strain the mixture into a chilled cocktail glass. Garnish with a grapefruit
twist or a lime wheel. Enjoy your Tommy John cocktail!
CREDITS
WRITER
Clare Duffy
EDITORS
Seth Fiegerman and Matt Stiles
ILLUSTRATION
Will Mullery
DEVELOPER
Kenneth Uzquiano
UNIVERSITY OF PENNSYLVANIA RESEARCHERS
Chris Callison-Burch, Computer and Information Science Associate
Professor,
Liam Dugan, Computer and Information Science Ph.D. student,
Daphne Ippolito, Computer and Information Science Ph.D. student