“Cheap” Tricks for NLP:
An “Invited” Talk
Craig Martell
Associate Professor
Naval Postgraduate School
Director, NLP Lab
Overview
• We’ve been doing work on microtext since before it was
“microtext”.
• About NPS
• NPS Chat Corpus (v1 and v2?)
– Overview
– Goal (Jane Lin, NSA)
– Age Detection Task (MAJ Jenny Tam, USA)
• POS and Dialogue Act tagging: using Treebank to bootstrap
(Lt. Col. Eric Forsyth, USAF)
• But do we really even need to POS-tag? (CAPT James Hitt,
USN)
– Getting by “on the cheap”
• Authorship detection in Twitter (LT Sarah Boutwell, USN)
• Good scientific goals for the community (??)
NPS NLP Lab
• NPS is both a university and part of the DoD
• As a university, we work on the same types of
sponsored research as civilian universities
–
–
–
–
DARPA*, IARPA, MURIs, NSF, etc.
Standard competitive process
Standard academia/industry expectations for results
Same tenure and promotion process
• As a part of the DoD, we do work more directly for
sponsors:
– DoD, DARPA*, NRO, NSA, etc.
– Depending on the type of money, results need to be more
operationally applicable
– We have had some cool results using “cheap” tricks that
could point to more “normal” academic research
Some Recent and Current Work
• IARPA SCIL
–
–
–
–
Persuasion Detection
Sub-group Detection
In forums, chat, etc. (“microtext”)
With UMD, UCSC, and Temple
• DoD, etc.
– Topic Detection in IRC Chat (Adams 2008)
– Authorship “signal boosting” with large author sets
• Any boost is remarkably useful to analysts
– Project away topic signal from documents for cleaner
authorship signal (topic does most of the work)
– L1 detection from English-L2 documents.
– “On-phone” NLP (above and more)
• Accuracy vs Computational Power
The NPS Chat Corpus, V1
• Gather 495,000 posts in age-based
rooms
– According to the terms of service of the
chat service
• To abide by the privacy act, we hand
anonymized 10,000 posts, tagged them
for dialogue act and part of speech
– Go to Web Page
The NPS Chat Corpus, V2?
• We have also gathered data to aid in doing
conversational thread extraction:
– Essentially, we want to cluster posts according to what
conversation they’re in
– Not necessarily mutually exclusive clusters
• We gathered data similar to that gathered by
Elsner and Charniak at Brown
– They gathered IRC data from Linux tech help
– We added iPhone and Python tech help, and Physics
Q&A
– It has all been hand “clustered” for conversations
– Working with UCSC CS (Lyn Walker) and Linguistics
(Pranav Anand) to augment the annotation to include
dialogue act and “attachment” instead of clusters.
First Use: Age Detection
• Second Youth Internet Safety Survey (2005) (YISS-2)
– Decrease in youths receiving solicitations
– Number of dangerous sexual overtures/aggressive
solicitations has not declined
– In 35% of the aggressive episodes, youths did not think the
solicitations were serious enough to tell anyone
– Only 7% of the aggressive solicitations were reported to
law enforcement, ISP, or other authority
• Need for an automated system that can recognize
adults conversing with teens to alert parents of
possible inappropriate conversations
Tam – Chat Classification
• NPS Chat Corpus (Talk City Chat Data)
– Teens, 20s, 30s, 40s, 50+
• Perverted Justice (IM chat logs)
– Pseudo Victims (adults posing as minors)
– Convicted criminals (solicitation of minors)
• Binary Classification
– teens vs. adults
– teens vs. specific age group
– teens vs. pseudo victims (similarity between actual teens to adults pretending to be
teens)
– criminals vs. teens (looking for minors soliciting minors)
– criminals vs. pseudo victims
• Classification Tool
– Linear Support Vector Machine with different slack variables
• Result: 80-90% success at detecting Teens from Adults. But the
most important is detecting Teens from 20s. >90%
• Current state of the art in the field!
Forsyth – Dialogue Act/POS Tagging
• An experiment in cross-domain NLP
• We wanted to POS tag chat (Lt Col Eric Forsyth, USAF)
• Lex. Bigrams → Bigrams → Lex. Unigrams → Unigrams → MLE from
training data
• WSJ train, chat test: 57.4% accuracy
– Not surprising. Chat is not like WSJ
• Treebank train, chat test: 65.8% accuracy
– Includes ATIS, Switchboard; chat is somewhat speech like
• Boot strapped/hand corrected POS tags for 10,000 posts
• Chat train, chat test: 73.7%
• But, add 10,000 chat posts to Treebank: 87.1%
– Using HMM tagger trained on combo: 90.8%
• Using these parts of speeches tags as part of the input, we can
dialogue act tag at 83.2% accuracy.
Hitt – Dialogue Act/POS Tagging
• But does POS tagging matter for dialogue act
tagging (our actual goal in chat, sms, etc).
• Sure, but it doesn’t have to be that good
Instead of using chat at all, we (CAPT James
Hitt, USN) simply generated the MLE for each
word string (no wsd) from pre-existing
resources (Treebank and Brown combined).
• Just using these “cheap” parts of speech we
get: 83.23%
L1 Language Identification
• Using International Corpus of Lerner English
• For each author L2 = English (except for native
speaker control group)
• Texts in English
• Task: Guess L1
• Using character 3-grams, we (LT Charles Ahn,
USN) got: 81.3%
L1 Language Identification
CPOS and L1 Identification
• Interestingly CPOS n-grams works very well here too:
• Cells contain average counts of documents over 26
trials
• ML: Multi-class Logistic Regression
Boutwell – Authorship Detection in
Twitter
•
•
•
•
Hot off the presses
Built a “social network” from the Twitter garden hose
Use it to simulate SMS messages within the group
If my phone is stolen, can it tell that it isn’t me writing
SMS?
• So, what do we need to do authorship detection over
“SMS”
– Doesn’t seem to be a lot of authorship signal in SMS
• Well, not in one, but in 23 there is
– If we have a stream of 23 messages, we got 90% accuracy over
10 authors.
– Authors are consistent in how they deal with the constraints?
– More error/success analysis needed
Research to be explored
• Can we build a better scientific understanding of
different domains of text and develop a theory of
what will be useful from pre-existing domains?
What will be needed from the new domain?
• How much can we actually do with as little as
possible?
– Do we need to parse?
• Should we expand (e.g., ur), or generate new grammars
• I argue we build new models sooner rather than later
– How do we get parallel corpora?
– How do we get best practices for mechanical turk?