Data analytics and mash-up
Real time analytics of employment data
Team Shadowfax
CMPE 272 - Class Project
7/27/2016
0
Agenda
•
•
•
•
•
What are data analytics and mash-up?
Architecture of web mash-up
Implementation
Implementation architecture
How is it going to be useful to the user ?
CMPE 272 - Class Project
7/27/2016
1
Data analytics and mashup
•
•
•
•
Data Analytics is examining of raw data from which we can retrieve the
information after work around with the data
In the data analytics there is no assumption of data unlike data mining where
little adjustments are made to draw conclusion
Includes data cleaning, transformation and documentation
Data from different data sources are analyzed and converted into
information after which all the information from different sources are clubbed
together into a single data source and presented to the user in a useful
fashion.
CMPE 272 - Class Project
7/27/2016
2
Data/Web mash-up
•
Data mashup is data integration of structured, web & unstructured
information available freely on different sources
•
More and more reusable structured data appears on the Web. Users want to
extracts related data from various resources and combine in a human
readable format, such as HTML, PDF, or text
•
Challenges:
Structured, unstructured data is available on multiple public resources in
various formats. There is a file format mismatch, schema mismatch, data
representation mismatch. Hence first challenge is to extract this data into
common data format.
CMPE 272 - Class Project
7/27/2016
3
Architecture of web mashup
•
The basic idea is to extract, analyze,
aggregate and visualize.
Divided into three layers:
CMPE 272 - Class Project
•
Presentation: This is the user interface
of mashups. The technologies used
are JSP, HTML, and JavaScript
•
Information Extraction and delivery:
The functionality to access data
through APIs (LinkedIn or any other
APIs)is provided using API service
•
Data: Data is retrieved from various
open data sources and is managed
in a database
7/27/2016
4
Implementation
Idea
• The basic concept – System facilitating user to search for labor stats and
related jobs
Data Scavenging
• Exemplifying application – Need for data.
•
Employment/Unemployment Data
o
o
o
Wdie range of data pools
Various granularity level of data
County specific data
•
Zip code – County connectedness
•
Job related data
CMPE 272 - Class Project
7/27/2016
5
Implementation
Data Transformation & mashing
•
•
•
Connecting 2 databases
Simple Java Parser Implementation
Loading final structured data into MySql.
Information staging
•
Shaped up data presentation to the user.
o
o
o
J2EE
MySQL
JS & REST API
CMPE 272 - Class Project
7/27/2016
6
Implementation architecture
CMPE 272 - Class Project
7/27/2016 7
How is it going to be useful to the user ?
•
The information is presented to the user using a web page and is available to
the user on the web any time from any place
•
•
The information available with Linked In requires the user to be a registered
user with them to share the job listing information with the user
User would be able to search for a job based on his preference of region
,climate and others
CMPE 272 - Class Project
7/27/2016
8
References
•
•
•
•
http://www.bls.gov
http://www.populardata.com
http://developer.linkedin.com
http://birming.jvmhost.net
CMPE 272 - Class Project
7/27/2016
9
Thank You!
Demo
CMPE 272 - Class Project
7/27/2016
10
Q&A
CMPE 272 - Class Project
7/27/2016
11