Massive Data Analysis Lab
(MassDAL)
S. Muthukrishnan
CS Dept
MassDAL
• Agenda: Gather, manage and process massive data logs---Web, IP/wireless traffic data, location trajectories of objects,
sensor readings of physical world.
• Key Challenges:
– Scale: Beyond the traditional “human” scale. Eg., IP data at a
single router interface for an hour exceeds total yearly worldwide
credit card transactions!
– Data Collection: probes/sensors with associated data quality and
communication problems.
• Need breakthroughs in Mathematics, Algorithms, Systems
and Engineering, to meet these challenges.
• Potential: Major impact in Homeland Security, Telecom,
Transportation and Society-at-large.
State of MassDAL
• Mathematics and Computer Science.
– Algorithmic tools for embedding vectors, strings, trees and
other objects for “compact” representation.
– Algorithmic tools for analyzing data summaries for heavy
hitters, deviants, clustering, decision trees, etc.
– Invited talks at ACM, SIAM, European conferences in
Algorithms, Databases, Statistics, and Data Mining on
novel models and algorithms.
– Over dozen research papers in last 2 years on experience
with massive data analysis.
– Supported by NSF grants. Partner: MIT, DIMACS.
State of MassDAL
• Science
– Developing wearable sensors for tracking location
of objects as well as “interactions” between
objects. Measuring behavioral data.
– Current partner: Telcordia. Their initial
investment: $300k/3 months (est). Potential parter
in works: Los Alamos National Lab.
– Potential: Analysis of social networks for
Epidemiology and Homeland Security, and health
industry.
State of MassDAL
• Engineering.
– Consulting in analysis of wireless network logs.
AT&T Wireless, 3rd largest in US, 20 Million
customers. Terabytes/month. Fully operational, telcograde!
– Incorporated novel algorithms in operational IP
network data analysis tools. Partner: Gigascope.
– Developed principled approach to data cleaning and
data quality monitoring for operational IP network.
Partner: PACMAN.
– Developed new burst-detection algorithms for text
streams. Partner: DIMACS, Monitoring message
streams.
Future
• See
http://cs.rutgers.edu/~muthu/massdal.html
Future of MassDAL
• Research: Need breakthrough research in mathematics,
systems, databases, algorithms, sensor networking.
• Expand data domains.
– Potential partners: Google, NJ auto insurance fraud data,
USPTO patent data, AWS location trajectories, etc.
• Build state-of-art facility at Rutgers.
– Secure, 24X7, data hosting and analysis infrastructure capable
of gathering and processing petabytes of data/month across
domains, data sources, etc. Unique in the world!
• Potential.
– Every wireless, telecom, internet service provider is looking to
farm out this crucial piece of their operations. Estimated
market for these services: 100’s of millions in US $ per year.
Crucial for NJ State. Interest from multiple VCs now.