Ki-Jong Woo
Commissioner
Statistics Korea (KOSTAT)
Friday Seminar
“Big Data for Policy, Development and Official Statistics”
22 February 2013, New York
광업제조업 동향지수 빅데이터 활용 Pilot Project
UN(22
통계위원회(2013.2.22,
뉴욕)
44제th44차
UNSC
Feb. 2013, New York)
Contents
1. Global Responses to Big Data
2. Big Data Use in Public Sector of Korea
3. KOSTAT’s preparation in response to Big Data
4. KOSTAT’s pilot project on applicability of Big Data
5. Treats or Opportunities? What should we prepare?
광업제조업 동향지수 빅데이터 활용 Pilot Project
UN(22
통계위원회(2013.2.22,
뉴욕)
44제th44차
UNSC
Feb. 2013, New York)
“The Future Belongs to the One who Rules the Data”
US: ‘Big Data R&D Initiative’ - invest in R&D on big data technology (Mar 2012)
Japan : ‘Basic Strategy on Using Big Data’ as a major task for ‘Active Japan’ initiative (May 2012)
Public-Private-Academic Partnerships
- “Big Data Strategy Forum” for market analysis and future strategy (Mar 2012)
- “Korea Big Data Forum” for market growth and industrial transformation (Sep 2012)
National Master Plan on Big Data
- established by 5 concerned ministries including National Science & Technology Commission
and Ministry of Education, Science and Technology (Dec 2012)
- Identifying national agenda for big data use and its relevant infrastructure to ensure
applicability and benefits to the public
광업제조업 동향지수 빅데이터 활용 Pilot Project
1 / 15
UN(22
통계위원회(2013.2.22,
뉴욕)
44제th44차
UNSC
Feb. 2013, New York)
Using big data means handling ‘LARGE-SCALE’ & ‘REAL-TIME’ data only?
- For example, what if ‘administrative data’ in public sector ?
→ Could be… because it can provide greater INSIGHTS for reproduction of data
First Step is…
- to develop a system that can cross-link huge amount of administrative data
kept by different government agencies
3 Priority Tasks in 2013 are…
- Prevention of criminal incidents by predicting the location and time of crimes
- Early detection and prediction of natural disasters
- A traffic accident reduction scheme with a bottom-up method (Citizen Participation)
광업제조업 동향지수 빅데이터 활용 Pilot Project
2 / 15
UN(22
통계위원회(2013.2.22,
뉴욕)
44제th44차
UNSC
Feb. 2013, New York)
Youth at Risk Policy has been insufficient in fighting serious/life-threatening youth problems
Analyzing social data – buzz patterns related to suicide among youth – allows formulation of
policies to effectively prevent suicide by better understanding the circumstances and
psychological states of the youth
Harmful Contents
Suicide
risk factor
e.g.) Suicide Context Pool
e.g) Online spread of harmful content
e.g.) Influence by Major Players Online
(Source) National Information Society Agency Pilot Project (Jul-Sept, 2012)
광업제조업 동향지수 빅데이터 활용 Pilot Project
3 / 15
UN(22
통계위원회(2013.2.22,
뉴욕)
44제th44차
UNSC
Feb. 2013, New York)
Challenge!
scientifically collected data
huge amount of data
Quantity creates quality
Statistics can be also biased
MORE timely
Survey environment is challenging
On the basis of mathematical theories
Represent target population
Timely
“All Models are Wrong, but Some are Useful”
– George Box
“Will the Data Deluge Makes the Scientific Methods Obsolete?
- Chris Anderson, 『The End of Theory』
광업제조업 동향지수 빅데이터 활용 Pilot Project
4 / 15
UN(22
통계위원회(2013.2.22,
뉴욕)
44제th44차
UNSC
Feb. 2013, New York)
Since October 2012
A pilot project on the use of big data
in the process of editing existing
national statistics
Aims to raise awareness on big data
among KOSTAT staffs
Using media data for examining
outliers when producing the Index of
Industrial Production(IIP)
Organizes monthly seminars to
share up-to-date information and
trend of big data at global and
national level
광업제조업 동향지수 빅데이터 활용 Pilot Project
Completing the system set-up by
this February for a pilot trial run in
March and April 2013
5/ 15
UN(22
통계위원회(2013.2.22,
뉴욕)
44제th44차
UNSC
Feb. 2013, New York)
26 Industrial Types
(Division level)
4 Industrial Types
• C21 (Pharmaceuticals)
• C24 (Primary Metals)
• C26 (Electronic Components, …)
• C28 (Electric Equipment)
633 Products
83
162 Products
1,438 Establishments
8,300 Establishments
광업제조업 동향지수 빅데이터 활용 Pilot Project
6 / 15
UN(22
통계위원회(2013.2.22,
뉴욕)
44제th44차
UNSC
Feb. 2013, New York)
Work Flow
Visualization of Surveyed Data
Volume by product
and establishment
Data Edit
Index by industry
and product
Telephone
Inquiries
Analysis of Unstructured Data from Media
Check
related
information
on Internet
광업제조업 동향지수 빅데이터 활용 Pilot Project
Comprehensive
internet
information on
products and
establishments
7 / 15
UN(22
통계위원회(2013.2.22,
뉴욕)
44제th44차
UNSC
Feb. 2013, New York)
Crawling of related articles and web
documents online from 1st of previous
month to the present
- Collect internet data on products or establishments
indicating increases and decreases in trends
- Crawl internet news on real-time basis
- Download materials (i.e. PDF, DOC) on website and
upload to the server for analysis
IBM’ VIVISIMO
Analysis of unstructured data
Consolidate documents downloaded from
websites and internet news for
comprehensive analysis
- Enhance search accuracy by registering similar product
names in advance
- Provide website and internet news in order of search
accuracy
광업제조업 동향지수 빅데이터 활용 Pilot Project
8 / 15
UN(22
통계위원회(2013.2.22,
뉴욕)
44제th44차
UNSC
Feb. 2013, New York)
(Media Data) Information posted on websites and internet news articles
Industry
Source
Korea Pharma. Assoc.
C21
(Pharmaceuticals)
C24
(Primary Metals)
C26
(Electronic
Components,
media &
communication)
Jongkeundang and others
광업제조업 동향지수 빅데이터 활용 Pilot Project
www.kpma.or.kr / pharmaceutical news
Internet Newspaper
Major production, pharmaceutical products
Related articles
Korea Metal Journal
new.kmj.co.kr / news on products
Korea Iron & Steel Assoc.
Internet Newspaper
Ministry of Knowledge &
Economy
Korea Customs Service
Korea Semicon. Ind. Assoc.
Semiconductor Network
Korea Display Indus. Assoc.
Internet Newspaper
C28(Electric
equipment)
Contents
Internet Newspaper
www.kosa.or.kr / metal info / reports on survey data , steel news
Related articles
homepage / press release, announcement, policies
Homepage / breaking policy news, press release
Live semiconductor news
SN news
Industry news
Related articles
Related articles
9 / 15
UN(22
통계위원회(2013.2.22,
뉴욕)
44제th44차
UNSC
Feb. 2013, New York)
Direct access to the Industrial Production
System (IPS) database
Visualize data into graphs by industry,
establishment and product for time series
analysis on IIP and output volume
Visualization of
structured data
By
Industry/Product
- Graph of output by product/establishment
(volume/ Comparison with previous month/
By
Product/Establishment
comparison with same month of previous year)
- Graph of index by industry/product
(volume/ Comparison with previous month/
Index Data
광업제조업 동향지수 빅데이터 활용 Pilot Project
Volume Data
comparison with same month of previous year)
10 / 15
UN(22
통계위원회(2013.2.22,
뉴욕)
44제th44차
UNSC
Feb. 2013, New York)
광업제조업 동향지수 빅데이터 활용 Pilot Project
11 / 15
UN(22
통계위원회(2013.2.22,
뉴욕)
44제th44차
UNSC
Feb. 2013, New York)
Someday!
NSOs will have to decide on whether to produce
official statistics based on big data.
What would
users
prefer?
광업제조업 동향지수 빅데이터 활용 Pilot Project
12 / 15
UN(22
통계위원회(2013.2.22,
뉴욕)
44제th44차
UNSC
Feb. 2013, New York)
Encourage the research on the
comparison of the statistics produced
on the basis of big data and the
existing statistics.
i.e. 2-year pilot project on using big data for
price index in Europe
광업제조업 동향지수 빅데이터 활용 Pilot Project
13 / 15
UN(22
통계위원회(2013.2.22,
뉴욕)
44제th44차
UNSC
Feb. 2013, New York)
Openness .vs. Privacy
Legal framework and system to facilitate
data sharing and access should be
established in order to ensure effective use
of big data among agencies.
- KOSTAT provides 540 subjects of statistics for
free and plans to develop OpenAPI this year
There is a growing request for Microdata
in the advent of big data.
- Clever solutions are needed to the issue pertaining
to microdata, mainly the conflict between the need
for data sharing vs ensuring privacy protection
광업제조업 동향지수 빅데이터 활용 Pilot Project
14 / 15
UN(22
통계위원회(2013.2.22,
뉴욕)
44제th44차
UNSC
Feb. 2013, New York)
14,000 data scientists will be
needed by 2017 while there are
currently about 100 in Korea.
Data Scientists should be cultivated in
public and private sector.
- have to come up with strategies to train big
data specialists to cooperate with private,
academic and research sectors
- need to develop a curriculum to foster data
scientists by concentrating on business
analysis, statistics and IT studies
광업제조업 동향지수 빅데이터 활용 Pilot Project
15 / 15
UN(22
통계위원회(2013.2.22,
뉴욕)
44제th44차
UNSC
Feb. 2013, New York)
The future belongs to those
who rule the data
광업제조업 동향지수 빅데이터 활용 Pilot Project
UN(22
통계위원회(2013.2.22,
뉴욕)
44제th44차
UNSC
Feb. 2013, New York)
Thank You
감사합니다
광업제조업 동향지수 빅데이터 활용 Pilot Project
UN(22
통계위원회(2013.2.22,
뉴욕)
44제th44차
UNSC
Feb. 2013, New York)