Meeting Information Extraction
from Meeting Announcement in Korean
Kyoungryol Kim
Table of Contents
1. Introduction
 Motivation
 Goal
 Problem Definition
 Contribution
2. The Proposed Method
 Finding
3. Discussion
2
Introduction
3
Motivation (1/3) : Necessity
 Everyday we receive a lot of Meeting Announcement
 Conference, Seminar, Workshop, Meeting, Appointment…
 Meeting announcement accounts for 17%
(30,201 out of 183,022) of emails in Enron Email Dataset.
* Enron Email Dataset, August 21, 2009 version, http://www.cs.cmu.edu/~enron/
 Smartphone era
 Many people manage schedule using online-calendar via
smartphone
e.g. Google Calendar
 But, typing by touch screen keyboard make many errors and
even it’s difficult.
4
Goal
 Extracting schedule information from meeting announcement,
and update them to the calendar, automatically.
startTime
2011-07-19T14:00
isHeldAt
서울시 중구 명동 1-3 민들레영토
Latitude : 126.9797848,
Longitude : 37.5687868
location
Landmark
서울지하철 4호선 명동역 8번출구
Latitude : 126.9864660,
Longitude : 37.5609660
Meeting Announcement
무더운 날씨가 본격적으로 시작
되는 즈음하여 유니브캐스트의
상반기 평가와 하반기 운영을 위
한 정기팀장회의를 개최합니다.
날짜 : 7월 19일(토) 오후 2시
장소 : 민들레영토
민들레영토 오는길
지도와 같이 명동역 8번 출구로
나오셔서 쭉 상가 끼고 걸어가시
면 저기 YMCA빌딩 1층에 있습
니다.
Extract
Update
5
Problem Definition
To find Meeting Location, the problem divided into 3 parts :
1. Finding locations for each type of complexity.
2. Named entity disambiguation on found locations.
무더운 날씨가 본격적으로 시
작되는 즈음하여 유니브캐스트
의 상반기 평가와 하반기 운영
을 위한 정기팀장회의를 개최
합니다.
날짜 : 7월 19일(토) 오후 2시
장소 : 민들레영토
기본 안건
- 제작지원비 지급 지연에 대
한 설명
- 기금 조정 운영안
- 가을 워크샵 준비위 구성
- 기타(기타 안건으로 상정할
것이 있으면 각 팀장들은 제안
해 주시기 바랍니다)
민들레영토 오는길
지도와 같이 명동역 8번 츨구
로 나오셔서 쭉 상가 끼고 걸
어가시면 저기 YMCA빌딩 1층
에 있습니다.
참고하세요
1. Finding Locations
(Location-type NER)
무더운 날씨가 본격적으로 시
작되는 즈음하여 유니브캐스트
의 상반기 평가와 하반기 운영
을 위한 정기팀장회의를 개최
합니다.
날짜 : 7월 19일(토) 오후 2시
장소 : 민들레영토
기본 안건
- 제작지원비 지급 지연에 대
한 설명
- 기금 조정 운영안
- 가을 워크샵 준비위 구성
- 기타(기타 안건으로 상정할
것이 있으면 각 팀장들은 제안
해 주시기 바랍니다)
민들레영토 오는길
지도와 같이 명동역 8번 츨구
로 나오셔서 쭉 상가 끼고 걸
어가시면 저기 YMCA빌딩 1층
에 있습니다.
참고하세요
2. NE Disambiguation
Start/End Time Extraction
6
isHeldAt
민들레영토
민들레영토
YMCA빌딩 1층
location
Landmark
명동역 8번출구
3. Normalization &
Co-reference
startTime
2011-07-19T14:00
isHeldAt
서울시 중구 명동 1-3 민들레영토
Latitude : 126.9797848,
Longitude : 37.5687868
location
Landmark
서울지하철 4호선 명동역 8번출구
Latitude : 126.9864660,
Longitude : 37.5609660
7
Definition
 Definition 1. Location Named Entity
A particular point or place in physical space (Wiktionary).
 [Cyber Space] Exceptionally, If the cyber space is used as a place gathering people, then the cyber
space can be a location.
e.g. MSN에서 9시에 모입니다.
 [Road, Street, Transportation] cannot be a location, except if it points particular place or it is
necessary to describe the location.
e.g. 진천 I/C, 왼쪽에 석촌지하차도가 보임
 [Bridge] can be a location.
e.g. 납안교, 한강대교
 [Train/Subway Station, Bus-stop] can be a location.
e.g. 도곡역 1번출구, 뱅뱅사거리
 [Address] Full/partial address can be a location.
e.g. 전북 무주군 설천면 심곡리 43-15
 [Organization, Company, Heritage, Building] can be a location if it is used to represent the location.
 [Parenthesis] If the location is ambiguous when the string in the parenthesis is removed and separated
by the parenthesis, then the string including parenthesis are the part of the location.
e.g. COEX 컨퍼런스센터 4층 (402호), 건국대학교(서울) 의생명연구동 강당, 경인교육대학교 (경기캠퍼스),
부산벡스코(BEXCO) 컨벤션홀 201호, 생명과학관(녹지) 139호
 [Enumeration] The different representations for same location are recognized separately.
e.g. 장소 ? 가야 레스토랑. 전화/215-654-8900, 주소/1002 Skippack Pike, Blue Bell, PA 19422
전주 화산체육관 (전북 전주시 완산구 중화산동 1가 45번지), 2. 장소 : 늘푸름(오산시 은계동 91-8)
 Definition 2. Meeting Location
Meeting Location is the Location where the meeting will be held.
 Definition 3. Location Landmark
Location Landmark is the Location where can be used as a landmark to go to the meeting location.
8
Complexity of the problems
d
9
The Proposed Method
1) Location Named Entity Recognition
2) Relation Type Classification
3) Co-reference
4) Normalization
10
Overall Architecture
무더운 날씨가 본격적으로 시작
되는 즈음하여 유니브캐스트의
상반기 평가와 하반기 운영을 위
한 정기팀장회의를 개최합니다.
날짜 : 7월 19일(토) 오후 2시
장소 : 민들레영토
기본 안건
- 제작지원비 지급 지연에 대한
설명
- 기금 조정 운영안
- 가을 워크샵 준비위 구성
- 기타(기타 안건으로 상정할 것
이 있으면 각 팀장들은 제안해
주시기 바랍니다)
민들레영토 오는길
지도와 같이 명동역 8번 츨구로
나오셔서 쭉 상가 끼고 걸어가시
면 저기 YMCA빌딩 1층에 있습
니다.
참고하세요
Input
Document
무더운 날씨가 본격적으로 시작
되는 즈음하여 유니브캐스트의
상반기 평가와 하반기 운영을 위
한 정기팀장회의를 개최합니다.
날짜 : 7월 19일(토) 오후 2시
장소 : 민들레영토
기본 안건
- 제작지원비 지급 지연에 대한
설명
- 기금 조정 운영안
- 가을 워크샵 준비위 구성
- 기타(기타 안건으로 상정할 것
이 있으면 각 팀장들은 제안해
주시기 바랍니다)
민들레영토 오는길
지도와 같이 명동역 8번 츨구로
나오셔서 쭉 상가 끼고 걸어가시
면 저기 YMCA빌딩 1층에 있습
니다.
참고하세요
Named Entity
Recognition
(Location)
무더운 날씨가 본격적으로 시작
되는 즈음하여 유니브캐스트의
상반기 평가와 하반기 운영을 위
한 정기팀장회의를 개최합니다.
날짜 : 7월 19일(토) 오후 2시
장소 : 민들레영토
기본 안건
- 제작지원비 지급 지연에 대한
설명
- 기금 조정 운영안
- 가을 워크샵 준비위 구성
- 기타(기타 안건으로 상정할 것
이 있으면 각 팀장들은 제안해
주시기 바랍니다)
명동 민들레영토 오는길
지도와 같이 명동역 8번 츨구로
나오셔서 쭉 상가 끼고 걸어가시
면 저기 YMCA빌딩 1층에 있습
니다.
참고하세요
Relation Type
Classification
isHeldAt
민들레영토
민들레영토
YMCA빌딩 1층
location
Landmark
명동역 8번출구
Co-reference
서울시 중구 명동
YMCA빌딩 1층
민들레영토
Normalization
OUTPUT
11
1) Location Named Entity Recognition
12
Architecture of Location NER
Training the system
(supervised learning)
TF-IDF
Calculation
Testing the system
(actual use and evaluation)
Training
Corpus
Web
Input: Morpheme-level
tokenized sentence list
Tokenization
Gazetteer
Extraction
Feature
Extraction
Gazetteer
Boundary
Tagging (IOB2)
by CRFs Model
Boundary
Marking (IOB2)
Feature
Extraction
CRFs
Learning
TF-IDF
Score Data
CRFs
Model
Boundary
Merging
Output: NE Annotated Email
Document
13
NER - Boundary Detection
 Boundary Tagset : IOB2
 Features
 Linguistic
 {-2,-1,0,1,2} POS-level word, {-2,-1,0,1,2} POS-tag,
POS-tag + length of the word
 Orthographic : 18 types of the word
 isKorean, isAlpha, isAlnum, 2DigitNum, ...
 Gazetteer :
 Person/Location Pronoun dictionary (ETRI 99)
 from Training corpus :
 Heading words, Surrounding words, NE words
 External resources :
 Person : Chosun/Joins.com Person DB (64,042)
 Location :
Nate Local DB 35,335, Sigaji.com 8,193, Ofood 43,390
BusStop 19,431, Address,B/D 23,365, Subway 1,288,
Hotel (Auction accomodation, hotelnjoy) 884,
Country/Place name 11,946,
School(Elementary~University) 21,957
 Syntactic :




Position of the POS-level word in the chunk (relative:S/C/E, absolute)
Position of the chunk in the sentence (relative:S/SC/CE/E, absolute)
Position of the sentence in the document (relative:S/SC/CE/E, absolute)
TF-IDF
14
Features : Gazetteer data
 Location :
 Shop Name (80,436)
 Nate Local DB (3~10 chars.)
(http://localinfo.nate.com)
 Sigaji.com Shop DB (3~10 chars.)
(http://sigaji.com/location/)
 oFood
(http://ofood.co.kr)
 Hotel Name (884)
 Auction Accomodation
(http://accommodations.auction.co.kr)
 Hotelnjoy
(http://www.hotelnjoy.com)
 Public Transportation (20,719)
 Subway stations
 Bus-Stop names
 Address (from Zipcode DB) (23,365)
 Si/do, Gu/gun, Dong/myun/ri, B/D names
15
Evaluation Result (1/2) Baseline
 Boundary Detection
 Target : 13,076 sentences in 1,011 documents.
 CRFs Model, 10-fold cross validation, 3-order, Exact Matching
 Baseline is the case applying Word and POS-tag feature only
B-Location
I-Location
100.00%
100.00%
90.00%
90.00%
80.00%
80.00%
70.00%
70.00%
60.00%
60.00%
50.00%
50.00%
40.00%
40.00%
30.00%
30.00%
20.00%
20.00%
10.00%
10.00%
0.00%
0.00%
Precision
Recall
F-measure
baseline
B-Location
Precision
49.99%
47.93%
Recall
16.97%
F-measure
24.34%
Precision
Recall
F-measure
baseline
I-Location
Precision
24.94%
77.99%
64.84%
Recall
39.82%
69.58%
55.11%
F-measure
32.99%
73.54%
16
2) Relation Type Classification
17
Architecture of Relation Type Classifier
Training the system
(supervised learning)
Testing the system
(actual use and evaluation)
Training
Corpus
Web
Tokenization
Gazetteer
Extraction
Gazetteer
Feature
Extraction
Input: Location NE-tagged
Document
Feature
Extraction
Relation Type
Classification
By SVMs Model
Template
Generation
SVMs
Learning
SVMs
Model
Output: Extracted NE with
Meeting-NE Relation Type
18
Statistics of Relation Types
 Document-Location Relation Type Classification
 Target : 1,844 Location-type Terms
 848 isHeldAt (45.99%)
 161 locationLandmark (8.78%)
 835 generalLocation (45.28%)
161
835
General Location
isHeldAt
locationLandmark
848
19
Features
 Linguistic
1. Gazetteer
A. Named Entity Dictionary

Nate Local DB 35,335, Sigaji.com 8,193, Ofood 43,390
BusStop 19,431, Address,B/D 23,365, Subway 1,288,
Country/Place name 11,946,
B. from Training Corpus :
 Heading words in the current sentence
 Heading words in the previous sentence
 NE consisting words
2. Lexical Pattern
A.
B.
C.
D.
E.
POS-tag feature before and next to the NE
Is this NE the first location NE next to colon?
Is this term in the parenthesis?
Is parenthesis opened and closed next to the NE ?
Is direction word just next to the NE?
 Syntactic
3. Syntactic Features
A.
B.
C.
D.
Is the NE the first or the last Location-type of NE in the sentence?
Ratio of location NE in the current sentence to the document
Relative position of the NEs in the sentences
Is the NE the longest location NE in the sentence?
20
Experiment : Features (1/3)
1. Gazetteer
Feature 1A
A.Named Entity Dictionary


I.
II.
III.
IV.
V.
Collected from the web
Check if each morpheme, eojeol or term
matches the word in the dictionary.
Nate Local DB, Sigaji.com, Ofood
Address, Building name
Bus-stop, Subway station
Country name
Location-related Vocabulary
B.from Training Corpus :
I.
II.
isHeldAt
R (%) F (%)
locationLandmark
P (%) R (%) F (%)
Acc.
(%)
I
59.32 / 98.94 / 74.17
57.14 / 02.45 / 04.71
59.09
+II
60.45 / 93.99 / 73.58
47.62 / 06.13 / 10.87
58.39
+III
63.12 / 89.52 / 74.04
62.96 / 31.29 / 41.80
60.76
+IV
64.55 / 88.57 / 74.68
62.96 / 31.29 / 41.80
62.30
+V
70.64 / 86.45 / 77.75
70.91 / 47.85 / 57.14
67.20
locationLandmark
P (%) R (%) F (%)
Acc.
(%)
P (%)
Feature 1 (A+B)
Heading words in the current sentence.
Heading words in the previous sentence.
Heading word is the word before the colon in the sentence
e.g. 장소 : 피오레웨딩컨벤션 (봉계동 여수 세무서 옆)
III. Eojeol-level NE consisting words
P (%)
isHeldAt
R (%) F (%)
+I
81.45 / 85.87 / 83.60
71.09 /
55.83 / 62.54
75.17
+II
80.56 / 84.92 / 82.68
69.40 / 57.06 / 62.63
75.03
+III
84.86 / 87.16 / 86.00
77.44 / 63.19 / 69.59
79.93
21
Experiment : Features (2/3)
2. Lexical Patterns
A.
POS-tag feature just before and next to the NE
e.g. 장소 : 피오레웨딩컨벤션 (봉계동 여수 세무서 옆)
B.
Is this NE the first location NE next to colon?
Feature 1+2 (A~G)
isHeldAt
R (%) F (%)
locationLandmark
P (%) R (%) F (%)
Acc.
(%)
+A
86.52 / 86.93 / 86.72
77.10 / 61.96 / 68.71
80.77
+B
86.94 / 86.22 / 86.58
73.79 / 65.64 / 69.48
80.63
 34 direction words : 위, 아래, 밑, 옆, 앞, 내, 외, …
e.g. 장소 : 피오레웨딩컨벤션 (봉계동 여수 세무서 옆)
+C
87.98 / 86.22 / 87.09
76.19 / 68.71 / 72.26
81.05
Is the unit of length appeared in the next 3 eojeols
of the NE?
+D
88.38 / 86.93 / 87.65
75.33 / 69.33 / 72.20
80.63
+E
88.52 / 87.16 / 87.83
77.18 / 70.55 / 73.22
81.82
+F
88.76 / 87.40 / 88.07
77.70 / 70.55 / 73.95
82.10
+G
88.33 / 87.40 / 87.86
80.41 / 73.01 / 76.53
82.24
e.g. 장소 : 피오레웨딩컨벤션 (봉계동 여수 세무서 옆)
C.
Is this NE in the parenthesis?
e.g. 장소 : 피오레웨딩컨벤션 (봉계동 여수 세무서 옆)
D.
Is parenthesis opened and closed next to the NE ?
e.g. 장소 : 피오레웨딩컨벤션 (봉계동 여수 세무서 옆)
E.
F.
Is direction word just next to the NE?

G.
P (%)
[0-9]+(m|km|ft|yd|mile|미터|킬로미터|피트|야드|마일|리|초|분|시간)
Is transportation words contained in the left eojeol?
22
Experiment : Features (3/3)
3. Syntactic Features
A.
Is the NE the first or the last Location-type of NE in the sentence?
e.g. (1호선에서 갈아탈 경우 동묘역에서 6호선을 갈아타고 봉화산방향으로 타고 오시면 2번째 정거장이 보문역입니다.
B.
C.
D.
E.
Ratio of NEs in the current sentence to the document (<25%,<50%,<75%,<100%,=100%)
Relative position of the sentence to the document. (S / SC / CE / E)
Relative position of the eojeol to the sentence. (S / SC / CE / E)
Relative position of the NEs in the sentence (S / SC / CE / E)
e.g. (1호선에서 갈아탈 경우 동묘역에서 6호선을 갈아타고 봉화산방향으로 타고 오시면 2번째 정거장이 보문역입니다.
S
F.
G.
H.
I.
J.
K.
L.
M.
N.
O.
P.
Q.
R.
S.
T.
U.
V.
CE
E
Is the NE the longest location NE in the sentence?
Is this only location NE in the sentence?
Is the NE on the previous/next to the NE in the sentence?
Is same type of NE in the prev/next sentence ?
Is phone number on the left or right side of the NE?
Surrounding word (on the right side of the NE, n=1, pos=etc,p,j,m,x ) ?
Is Colon included in the curr/prev/next sentence?
Is the sentence starts/ends with the NE?
# of chunks of the NE (max:99)
Length of the NE (max:300)
Is location related word included in heading ?
Is location related word included in heading of prev. sentence?
Feature
is the NE which is appeared more than 2, is next to this NE?
Is the NE appeared more than 2?
isHeldAt
Transport Dic Feature (left / right)
P
(%)
R (%) F (%)
Order of the ne in the sentence.
does sentence starts with special char?
+GH
89.42 / 89.10 / 89.26
1+2+3 (G,H)
locationLandmark
P (%) R (%) F (%)
Acc.
(%)
79.22 / 75.78 / 77.46
82.23
23
Experiment: Relation Type Classification
 Meeting-Location Relation Type Classification
 Target : 1,844 Location-type NEs
 SVMs 3 classes (multi-class) classifier
 Total Accuracy : 82.23
100.00%
80.00%
60.00%
Precision
Recall
40.00%
F-measure
20.00%
0.00%
isHeldAt
locationLandmark
isHeldAt
locationLandmark
Precision
89.42%
79.22%
Recall
89.10%
75.78%
F-measure
89.26%
77.46%
24
3) Normalization & Co-reference
25
Architecture of Normalizer
Address Format
1
2
3
4
5
6
7
8
9
10
11
Country | State/City | City/Gu/Gun | Dong/Eup/Myeon | Ri | House no. | Org. | B/D | Floor | Shop Name | Room no.
Subway Format
1
2
3
4
City | Line no. | Station Name | Gate no.
isHeldAt
민들레영토
민들레영토
YMCA빌딩 1층
location
Landmark
명동역 8번출구
Input
Document
민들레영토
민들레영토
YMCA빌딩(A8) / 1층(A9)
명동역(S3) / 8번출구(S4)
민들레영토
민들레영토
YMCA빌딩(A8) / 1층(A9)
서울시(S1) 4호선(S2) 명동역(S3) / 8번출구(S4)
Verifying
Elements
Expansion
Subway
Addr.
Pattern
Open Map Services
(Google Maps, Yahoo! Maps,
Daum Map, Naver Map)
A1 : 대한민국
A2 : 서울시
A3 : 중구
A4 : 명동
A5 : A6 : A7 : A8 : YMCA빌딩
A9 : 1층
A10 : 민들레영토
A11 : -
Combine
OUTPUT
26
Discussion
1) Limitations
2) Applications
27
Limitations
1. Performance
 Both system should be refined more in detail, with sophisticated experiment.
2. Scaling Up
 For our corpus consist of 1,011 emails, the method to cover more data in the
real-world should be mentioned.
3. Feature Selection
 Since we use +165,000 word-gazetteer and many of these features always zero
in the training data. In order to save memory and to maximize the
performance, these unsupported features need to be removed.
28
Applications
1. Smartphone application
 Extracting start/end time, location from email and update them to
Google Calendar.
2. Contribution to OpenStreetMap community
 Update found locations automatically to openstreetmap.com
29