ICSI
The Nemesysco Scandal – A Report
John J Ohala
Professor Emeritus, Linguistics,
UCB
1
Outline
1. Nemesysco, their beginnings, their products, and how
they promote them.
2. The Eriksson & Lacerda paper re Nemesysco
3. Press coverage of Nemesysco & of their threats to
the publisher of the Eriksson & Lacerda paper.
4. Coverage in the ‘Blogosphere’.
5. Who is the guy behind all this?
6. What can be done?
1. Nemesysco, their beginnings, their
products and how they promote them.
Starting around 1997 or 2000 (its not clear when) an Israeli firm,
Nemesysco, started promoting and selling devices that were said to be
able to detect emotion and stress in a speaker via an analysis of their
voice. It used an analysis techmique called ‘layered voice analysis’ or
LVA for short.
Their claim : “Nemesysco’s Layered Voice Analysis (LVA) technology detects
and measures the emotional content of human speech, captured live or
extracted from recorded audio. LVA identifies various types of stress, cognitive
processes and emotional reactions which together comprise the �emotional
signature� of an individual at a given moment, based solely on the properties
of his or her voice .
The technology detects minute, involuntary changes in the voice reflective of
various types of brain activity. By utilizing a wide range spectrum analysis to
detect minute changes in the speech waveform, LVA detects anomalies in
brain activity and classifies them in terms of stress, excitement, deception,
and varying emotional states, accordingly. This way, LVA detects what we call
�brain activity traces,� using the voice as a medium. The information that is
gathered is then processed and analyzed to reveal the speaker�s current
state of mind.
Segue to their professionally desgined website (live).
In their promotions they emphasize their technique is ‘patented’. Everyone
should know that it is not that difficult to get something patented; all you need to
do is follow the format dictated by the patenting agencies. (Aside: even I have
a patent; look it up!) As far as I can tell, there are two US patents awarded to
Amir Liberman. There are other patents filed in other countries.
This one will
become
important
later on in
this
presentation.
Further elaboration of their claims:
LVA Analysis Process
LVA has two basic formulas comprised of unique signal
processing algorithms that extract more than 120 emotional
parameters from each voice segment. These are further
classified into nine major categories of basic emotions.
Depending on the goal of the analysis, up to eight “final
analysis” formulas can be applied to the emotional
parameter data. These include: Lie stress analysis, Arousal
level, Attention level, Emotional level, Conflict level, Deception
patterns match, and additional methods for veracity
assessment
Just to note one of many inconsistencies: Elsewhere on their
website they insist that their products are NOT ‘lie detectors’.
Window dressing to make the customrer think they are experts in speech.
The Human Speaking Mechanism
The human speaking mechanism is one of the most complicated procedures
the human body is capable of, due to the number of muscles and physical
apparatus involved, and the ways in which they need to be synchronized in
perfect timing.
Initially, the brain apprehends a given situation and the possible implications of
whatever will be said. Then when a person decides to speak, air is pushed
upward from the lungs into the vocal cords. This causes the vocal cords to
vibrate at a specific frequency and produce sound. The vibrated air continues
to flow up toward the mouth where it is manipulated by the tongue, teeth and
lips to produce sound streams which we interpret as words or phrases.
The brain closely monitors all of these procedures, ensuring that the sound
emitted is what was intended, is intelligible, and is at a volume that can be
heard by the intended listener.
Due to this constant cerebral monitoring, every "event" that passes through the
brain will leave a trace on the speech flow. LVA technology ignores what your
subject is saying (i.e., the specific content) and focuses only on changes in
brain activity that are reflected in the voice. In other words, what is critical is
not “what” your subject is saying, but “how” he or she says it.
Nemesysco has spiffy computer interfaces:
A Figure from the patent of 17 Jan 2007.
To those unschooled in the basics of speech communication,
Nemesysco describes their methods in language that is
bound to impress them: it sounds scientific and authoritative.
SENSE Analysis Process
The SENSE technology is comprise of 4 sub-processes:
1. The vocal waveform is analyzed to measure the
presence of local micro-high frequencies, low frequencies
and changes in their presence within a single voice
sample.
2. A precise frequency spectrum of the vocal input is
sampled and analyzed.
3. The parameters gathered by the previous steps are
used to create a baseline profile for the subject.
4. The new voice segments to be tested are compared
with the subject's baseline profile, and the analysis is
generated.
Nemesysco cites a number of “studies” supposedly giving
evidence of the efficacy of their products.
As documented in the blogs about Nemesysco, a number of
these studies are worthless, being conference papers or
posters at conferences where there was no reviewing
process, or where the authors have some connection to or
financial interest in promoting Nemesysco’s products.
But let’s examine one study where the above
discounting factors are not involved.
One document cited on the Nemsysco website in support of one
of their early products called ‘Vericator’ (one thing Nemesysco
does well is coining clever names for itself and its products) is
said to report that
"A three-year study by Air Force Research Laboratory (AFRL)
Information Directorate engineers has concluded that several
features of voice stress analysis are effective for detecting
when a person is answering questions under stress..."
The report:
Investigation and Evaluation of Voice Stress Analysis
Technology
Final Report
February 13,2002
Darren Haddad, Sharon Walter AFRUIFEC Rome
Research Site
Roy Ratley Megan Smith ACS Defense Rome, NY
Parsing this study:
1. This (altered) quote is based on anecdotal evidence
from one law enforcement officer, not a proper controlled
test.
2. The whole report does not include results from any
other properly controlled test.
3. Moreover, the report contains factual errors.
It contains the
statement:
“VSA literature [9] points to a descriptor of the
physiological basis for the micro muscle tremor or
microtremor. This paper describes "a slight oscillation at
approximately 10cycles per second" (i.e. physiological
tremors) during the normal contraction of voluntary
muscle. All muscles in the body, including the vocal chords
[sic], vibrate in the 8 to 12 Hz range. It is these
microtremors that the VSA vendors claim to be the sole
source of detecting if an individual is lying. “
This statement,
in fact, is false.
Shipp & Izdebski
(JASA 1981)
studied this &
found no such
microtremor in
the laryngeal
muscles (but did
find it in limb
muscles;
moreover the
micro-tremor is
found when the
muscle is resting;
not when it is
active.
And, one might ask, why cite this study when on their own website,
Nemesysco declares:
“Layered Voice Analysis [LVA} is not a "Voice Stress" analysis
technology, nor does it use any previously known method for
detecting voice stress. As such, LVA does not perform "microtremors" analysis in the voice.”
Further suspicious omission:
In reviewing the literature on VSA, this very same report mentions:
The Nemesysco product evaluated.
My own take on this ‘study’: As for the officer quoted in the
report, Michael G. Adsit, Criminal Investigator, and the
authors of the report to the US Dept of Justice, Darren
Haddad, Sharon Walter, Roy Ratley, & Megan Smith, none of
them really know much about the motor control of the speech
apparatus. They don’t control the vast literature in this area.
(They may have expertise in other domains: speech
processing, speech compression.)
Finally, the Nemesysco list of ‘studies’ neglects to mention two studies
that did attempt to implement proper controls:
1. Hollien, Harry; James D. Harnsberger (2006-03-17), "Voice
Stress Analyzer Instrumentation Evaluation" (pdf), CIFA Contract –
FA 4814-04-0011,
http://www.clas.ufl.edu/users/jharns/Research%20Projects/UF_R
eport_03_17_2006.pdf]
“device showed significant sensitivity to the presence of stress or
deception in the speech samples tested. The true positive and
false positive rates were parallel to a great extent “
2. Damphousse, Kelly R. (March 2008). "Voice Stress Analysis: Only 15
Percent of Lies About Drug Use Detected in Field Test". NIJ Journal
(National Institute of Justice)
“Does VSA work? As our findings revealed, the two VSA programs
that we tested had approximately a 50-percent accuracy rate in
detecting deception about drug use in a field (i.e., jail)
environment; however, the mere presence of a VSA program during
an interrogation may deter a respondent from answering falsely.”.
2. The Eriksson & Lacerda paper re
Nemesysco (& one other similar
product named “Diogenes”)
Then came the paper by Anders Eriksson and
Francisco Lacerda:
“Charlatanry in forensic speech science: A problem to be
taken seriously”
The International Journal of Speech, Language and the Law
(Publisher: Equinox), IJSLL vol 14.2 2007 169–193.”
Papers published in IJSLL are peer-reviewed.
Francisco Lacerda,
Professor of
Phonetics at
Stockholm University
Anders Eriksson
Professor of Phonetics
Dept. of Philosophy,
Linguistics
and Theory of Science
University of Gothenburg
Eriksson, A. & Lacerda, F. (2007)"Charlantry in forensic speech science: A
problem to be taken seriously“Intl J. Speech Lang. Law 14, 169– 173.
ABSTRACT
A lie detector which can reveal lie and deception in some automatic
and perfectly reliable way is an old idea we have often met with in
science fiction books and comic strips. This is all very well. It is when
machines claimed to be lie detectors appear in the context of criminal
investigations or security applications that we need to be
concerned. In the present paper we will describe two types of
"deception" or "stress detectors" (euphemisms to refer to what quite
clearly is known as "lie detectors")." Both types of detection are
claimed to be based on voice analysis but we found no scientific
evidence to support the manufacturers’ claims. Indeed, our review of
scientific studies will show that these machines perform at chance
level when tested for reliability." Given such results and the absence of
scientific support for the underlying principles it is justified to view the
use of these machines as charlatanry and we argue that there are
serious ethical and security reasons to demand that responsible
authorities and institutions should not get involved in such practices.
Their paper reviewed existing literature but did not report the results
of a controlled study, unlike those of Hollien & Harnsberger (2006) or
Damphousse(2008). Rather, as concerned the Nemesysco product,
they they did something more fundamental and ultimately more
important – to inform the knowledgeable speech tech community:
they examined the publically accessible patent for the device. What
they found:
“[To say that] there is absolutely no scientific basis for the
claims made by the LVA proponents is an understatement. The
ideas on which the products are based are simply complete
nonsense.”
They also offer what we may characterize as a “sociological”
account of Nemesysco’s economic success:
“While, as we have seen, the voice stress detectors are not of any real
use as the lie or stress detectors they are claimed to be, they have
certainly not been without success in other areas. One such area is
making money for the vendors.”
The statistics are based upon what is
defined as thorns and plateaus which has
no relevance at all for voice analysis and is
moreover dependent on how the signal is
sampled.
Explanation:
This figure (from
the US Patent) is a
simple digitization
of the incoming
signal.
Data points whose
value is different from
flanking data points
(presumably at some
threshold level) are
“thorns”; dats points
that are not that
different from flanking
data points are called
“plateaus”.
The program computes statistics on the relative
incidence of “thorns” and “plateaus” and comes up
with a number that is said to show the speaker’s
emotional state.
The problem (as noted by Eriksson & Lacerda) is that these
statistics will vary with sampling rate, quantization of the
sampling, background noise, etc. Indeed, it could come up
with a “number” just by sampling traffic noise or bird song.
From the US patent:
This last statement, of course, is utter, unadulterated, nonsense.
A student incorporating this in an undergraduate
phonetics/speech science course would get an “F”!
From the US patent:
And yet the Nemesysco website declares their device is not
“lie detection” system.
“An examination of the description of the method in
the American patent documents confirms the
suspicion that the method is pure nonsense,
perhaps best described as statistics based on
digitization artefacts.” [Eriksson & Lacerda]
The legal controversy:
The use of words such as ‘charlatanry’, ‘fraud’ and,
perhaps, the inclusion of reports from a Swedish reporter
who interviewed Amir Liberman, the CEO of Nemesysco,
revealing that he had no scientific credentials in speech
analysis or in psychology or psychiatry, -- in fact no higher
education except in marketing -- were grounds for
Nemesysco threatening to sue Equinox (the IJSLL
publisher) and the authors for libel, defamation of
character, etc.
Charlantry in forensic speech science: A problem to be taken seriously
Anders Eriksson, Francisco Lacerda
NOTE FROM PUBLISHER December 4 2008
In the December 2007 Edition of the International Journal of Speech, Language and
the Law, an article was published which made serious allegations concerning Mr Amir
Liberman and Nemesysco Limited. We have received complaints from Mr Liberman
and Nemesysco Limited about the content of this article and particularly that the
allegations made against them in it were highly defamatory, containing many
inaccuracies and misleading statements. In addition, they complain that it was
prepared without reference to them and without giving them an opportunity to
comment upon it. The Journal accepts that Mr Liberman and Nemesysco Limited were
not asked to assist in the preparation of the article and further that they were not
invited to comment on the content of the article prior to its publication where, in view
of the content of the article, it would have been appropriate to invite them to do so. We
have agreed to publish a letter from Mr Liberman and Nemesysco Limited setting out
their objections to the article in more detail in a future issue of the journal. The article
will no longer be made available in electronic form through the Equinox website.
Janet Joyce
Managing Director
3. The response to this controversy in the
press and blogosphere.
BBC
14 August, 2003
[No author by line]
Truth test for insurance claims
Banking giant HBOS plans to use voice-sensitive lie detectors in an
effort to cut down on fraudulent insurance claims. The technology,
developed by Israeli firm Nemesysco Technology and distributed in the
UK by security software specialists DigiLog UK, will be used on HBOS'
claims hotline for a three-month trial period starting in September. …
Managing director Kerry Furber said: …
"We're … able to be fairly certain, not 100%, but fairly certain, that
there are risk problems within a claim that need further validation.
"The psychology of that persuades many claimants to withdraw from
the process altogether."
Wed December 17, 2008
Behavioral screening -- the future of airport security?
By Dana Rosenblatt
CNN
TEL AVIV, Israel (CNN) -- Keep your shoes and belts on: Waiting in long airport
security lines to pass through metal detectors may soon be a thing of the past.
Security experts say focus is shifting from analyzing the content of carry-ons to
analyzing the content of passengers' intentions and emotions.
Nemesysco, another Israeli-based technology company, believes the key to a
person's emotions and intentions lies in their voice. The company's patented LVA,
or Layered Voice Analysis, technology can pick up verbal cues from a passenger
who may pose a threat.
Unlike a polygraph test, which checks for lies, Nemesysco's systems work as an
"emotion detector," says Nemesysco CEO Amir Liberman. In other words, it's not
what passengers say, but how they say it.
Nemesysco's devices use a series of patented signal-processing algorithms that
can differentiate between a "normal" voice and a"'stressed" voice. If emotional
stress is detected, officials can determine if the passenger should be taken aside
for further questioning.
CNN (continued)
The system works on the premise that all voices have a certain frequency, and
any deviation of that baseline frequency can indicate stress.
Liberman says it takes approximately five to 10 seconds for their system to
capture a "normal" voice in casual conversation, which establishes a baseline.
Their system then measures changes from the baseline voice that signify an
increase in stress, excitement, anticipation, hesitation or other emotions that
can indicate a potential terrorism threat.
A computer processes the voice patterns and then flashes words such as "high
risk," "medium risk," "excited" and "highly stressed." Through his system,
Liberman says, he "can see what's going on in your brain."
Versions of Nemesysco's system already have been successfully tested at
Moscow Domodedovo International Airport, where officials used it to target
criminals and drug traffickers. A version was recently implemented at another
major international airport which Liberman declined to identify.
Layered Voice Analysis also has been used to test for insurance fraud and on
the TV program "Big Brother Australia."
From a New Yorker piece: HELLO, HAL. Will we ever get a computer we
can really talk to? by John Seabrook. JUNE 23, 2008:
There is a small market for voice-based lie detectors, which are
becoming a popular tool in police stations around the country. Many are
made by Nemesysco, an Israeli company, using a technique called
“layered voice analysis” to analyze some hundred and thirty
parameters in the voice to establish the speaker’s pyschological state.
The academic world is skeptical of voice-based lie detection, because
Nemesysco will not release the algorithms on which its program is
based; after all, they are proprietary. Layered voice analysis has failed
in two independent tests. Nemesysco’s American distributor says that’s
because the tests were poorly designed. (The company played Roger
Clemens’s recent congressional testimony for me through its software,
so that I could see for myself the Rocket’s stress levels leaping.)
Nevertheless, according to the distributor more than a thousand copies
of the software have been sold—at fourteen thousand five hundred
dollars each—to law-enforcement agencies and, more recently, to
insurance companies, which are using them in fraud detection.
http://www.guardian.co.uk/news/2007/sep/09/letters.theobserver
Letter sent to the on-line version of The Guardian re their
uncritical coverage of Nemesysco’s LVA voice stress detector:
The truth is out there
The technology referred to 'Lie detectors target benefit claim cheats'
(News last week) has been subject to considerable testing in the
academic research community. It has been found that the results
produced by such systems do not exceed the level of chance.
But these devices will probably save Harrow Council money because if
people are told their speech is being monitored by a gadget that
detects lies, they are more likely to tell the truth.
Anders Eriksson, Professor of Phonetics, University of Gothenburg
Paul Foulkes, Reader in Linguistics, University of York
Professor Peter French, forensic speech scientist, JP French Associates,
York
Francis Nolan, Professor of Phonetics, University of Cambridge
It is important to note that neither the publisher nor Nemesysco
disputed the scientific evaluation of the product.
The “sticking points” in the published paper were the words
‘charlatanry’ ‘fraud’ and inclusion of the statements from a
Swedish journalist who interviewed Amir Liberman and reported
that he had no academic or other scientific credentials which
would lend credence to the claims made for the Nemesysco
products.
Given the plaintiff-friendly libel laws in the UK (it has been
characterized as a country that invites “libel tourism”), the
publisher did what Francis Nolan characterized as what a person
facing a mugger with a gun would do: comply with the mugger’s
demands or face devastating consequences. One of the possibly
positive outcomes of the Nemesysco scandal is that some MPs
and the UK media are urging a change in these libel laws and have
cited the Nemesysco case as an example of why this is necessary.
4. Coverage in the ‘Blogosphere’.
The response in the “blogosphere” has been harsh –
toward Nemesysco – and toward the IJSLL publisher,
Equinox (but I don’t think they fully understand the
jeopardy faced by publishers given the lop-sided libel
laws in the UK). I append to this presentation a
number of links to blogs and other websites covering
and commenting on this scandal.
A purely personal judgment by me – take it or leave it -: the blogs did a far better job of investigating and
publicizing this scandal than the established press or
TV networks did: CNN, BBC, New Yorker, etc. I tried
to get the New York Times interested in this. Result,
so far: NOTHING. I also emailed our (Calif’s) senator,
Diane Feinstein. Result: NOTHING. If anyone wants
to research this scandal: go to the blogs.
5. Who is the guy behind all
this?
“This controversy, partly fought in a newspaper, caught the
interest of a journalist, Arne Lapidius, who was working in
Israel for the Swedish daily Expressen. After some research
he managed to locate Mr Liberman, a 32 year old (in 2004)
businessman in a small office in the town of Natania. The
business appeared to be a one-man operation. Mr Lapidus
interviewed Mr Liberman about his academic background
and was told that he basically had none. He has no degree
(never had time to get one, he explains) but has taken some
courses in marketing at an Israeli open university. As we have
explained above, the LVA is a simple program written in
rather amateurishly used Visual Basic.” [Eriksson & Lacerda]
Amir Liberman
6. What can be done?
Observation: this scandal demeans the speech tech
industry.
Question: How can the speech tech industry police
itself?
Suggested answer: the major speech tech
organization (ISCA, ASA, ASHA, etc.) have to issue
statements emphasizing that claims made about what
speech tech systems can do have to be backed up by
controlled scientific studies published in reputable
peer-reviewed journals.
Observation: in matters of food and drug safety and
efficacy, in the US, there is the FDA that is a watchdog and
gatekeeper on products and processes.
Can the speech tech industry lobby for a similar such body
to pass on the efficacy of products in their domain.
For that matter, why should such a body pass on ANY
technical product?
Consider, for example, that the NY Times recently reported on
a device marketed by a UK company, the ADE 651 “a handheld "remote portable substance detector" that is claimed to
be able to detect from a distance the presence and location of
various explosives, drugs, and other substances. The device
has been sold to a number of countries in the Middle and Far
East, including Iraq, for as much as $60,000 per unit. The
Iraqi government is said to have spent £52m ($85m) on the
devices. However, investigations by the BBC and other
organisations have reported that the device is little more than
a "glorified dowsing rod" with no ability to perform its claimed
functions. [from Wikipedia]
Useful Links.
The Nemesysco saga in the News and Blogosphere:
http://balneus.wordpress.com/2009/02/17/science-journals-commercialcensorship-law-and-social-security-benefits/
https://antipolygraph.org/cgi-bin/forums/YaBB.pl?num=1230840076/4
https://antipolygraph.org/blog/?paged=2
https://antipolygraph.org/blog/?p=245
Stockholm University account of the event:
http://www.su.se/english/about/news-and-events/scientists-threatened-withlegal-action-1.1149
A statement released by Nemesysco – with critical comments by Lacerda attached:
http://www.ling.su.se/staff/frasse/LVA_technology2009/NemesyscoResponsetoLa
cerda_commented.pdf
http://languagelog.ldc.upenn.edu/nll/?p=1390
The actual US patent:
http://www.freepatentsonline.com/6638217.pdf
A brief report by the “Science Insider” (AAAS website):
http://blogs.sciencemag.org/scienceinsider/2009/05/royal-swedisha.html#more
“Neuroskeptic” (scroll down to last article):
http://neuroskeptic.blogspot.com/2009_02_01_archive.html
An account in “Nature News” (web news from the prestigious British
science journal, Nature):
http://www.nature.com/news/2009/090216/full/news.2009.99.html
The Ministry of Truth blog – provides important evidence that the “scientific
studies” validating LVA have been done by individuals with a conflict of interest,
including financial ties, to firms promoting or distributing Nemesysco products:
http://www.ministryoftruth.me.uk/2009/03/18/purnells-lie-detector-indecentdisclosures/
http://www.ministryoftruth.me.uk/2009/03/13/purnells-lie-detector-somethings-the-guardian-didnt-mention/
Much overlap with previous links but adds discussion about legal and
ethical aspects of the use of Nemesysco’s device and of their threats to
quash scientific research which doesn’t boost their sales:
http://balneus.wordpress.com/2009/02/17/science-journals-commercialcensorship-law-and-social-security-benefits/
LanguageLog (by Mark Liberman)
http://languagelog.ldc.upenn.edu/nll/?p=1390
Language Log (by Mark Liberman) on the history of VSA:
http://itre.cis.upenn.edu/~myl/languagelog/archives/001149.html
I like this one, “Sceptiphrenia: an occasional blog on science,
scepticism, and philosophy”
http://sceptiphrenia.wordpress.com/category/voice-risk-analysis/
Thank you.
Slides not used
Controversial device analyzes passengers' voices
Tuesday, January 10, 2006;
CNN [no author by-line]
The GK-1 voice analyzer, created by Israeli firm Nemesysco, requires
passengers to don headphones at a console and answer "yes" or "no" into a
microphone to questions about their travel plans.
The manufacturers say the device, which will cost between $10,000 to
$30,000, will usually be able to pick up uncontrollable tremors in the voice that
give away liars or those with something to hide.
"When you are very stressed, very excited, very confused or you have some
hidden agenda, then different messages go to the voice which are not
controlled," Nemesysco CEO Amir Liberman told CNN. "Our software is capable
of extracting those out to build a profile and then make the decision.“ Those
that fail the screening are led away for more in-depth questioning and, if
necessary, searches. Liberman says the device has proved highly successful in
tests, but admits that the results can sometimes be difficult to interpret with
around 12 percent of passengers likely to show stress even when they have
nothing to hide. The detectors are also likely to raise objections from civil
liberties groups already upset over intrusions on privacy from current security
measures.