Towards Automatic Structured Web Data Extraction System Tomas Grigalis, 2nd year PhD student Scientific supervisor: prof. habil. dr. Antanas Čenys Outline • Introduction • The ClustVX approach • Experiments • Conclusions Stuctured Web Data Stuctured Web Data Database Table with stuctured data Web server Title Model Price Fuji FinePix Z110EXR 14MP 562/6283 £119.99 Fujifilm XP30 14MP Waterproof 559/5101 £129.99 Samsung ST200F Smart 559/7635 £111.99 <...> Data Record <...> Rendered view in a web browser Browser The GOAL Unsupervised and domain independent stuctured web data extraction system Web pages with structured data Stuctured data Key Problems • Web pages with visually similar appearance usually have totally different underlying HTML source code • There are millions of web pages with different design and HTML source code • WEB 2.0 introduced asynchronous JavaScript HTTP requests (AJAX), that modifies HTML source code on-the-fly The ClustVX approach ClustVX is based on two fundamental observations: 1) Vast amount of information on the Web is presented using fixed templates and filled with data from underlying databases. 2) Although the templates and underlying data differ from site to site, humans understand it easily by analyzing repeating visual patterns on a given Web page HTML TREE Repeating patterns in HTML TREE (1st observation) Data which has the same semantic meaning is visualized using the same style (2nd observation) PRICE ClustVX: First, cluster visually similar web page elements ClustVX: Second, analyze clusters to identify data records Experiments: Data Sets • To evaluate ClustVX approach we use the following three publicly available benchmark datasets containing in total of 7098 data records: • These data sets contain web search result pages generated from databases Experiments: Evaluation • We use the precision and recall measures (which are widely used in information retrieval field) to evaluate the performance of ClustVX system Experiments: Results • We compare the evaluation results of ClustVX system to other state-of-the-art automatic structured web data extraction systems. • As shown in the following table, where the best results are marked in bold, ClustVX consistently outperforms other approaches. Conclusions • We presented ClustVX system, which, by exploiting visual and structural features of web page elements, extracts structured data. • The preliminary evaluation of ClustVX on three publicly available benchmark data sets demonstrated, that our method can achieve very high quality in terms of precision and recall. • Our future work will be concentrated on creating a new huge benchmark data set to test the applicability of this system in real world settings Thank you, Questions?