Project Title: Semantic Web Project Results Jordan Willis James Reid Submitted in partial fulfillment of The requirements for 03-60-569 Semantic Web Faculty of Computer Science University of Windsor Windsor, Ontario Dr. J. Lu December 15, 2013 Semantic Web Project Results Overview Our project goal was to create a scalable distributed web crawler with a search engine to query the pages crawled, which uses a Page Ranking to prioritize and sort search result while removing near duplicates before indexing and addressing similar search returns. Although we never achieved the distributed aspect because of resource limitations; we were able to test the individual components that comprise our remote severs and simulate the central sever search results. This document is a summary of our results and findings. To view our further information about our project you can review the project web site, and you can also test the search engine the Project Search Engine site. Over All Design: Divide and Concur Scheme The distributed severs are responsible for crawling and indexing their web space, calculating the local PageRank’s, and returning their k-best results to the Central Search Engine which in turn displays the k-best results from all the distributed servers to the searcher. This design is meant to reduce network traffic and search time by keeping the bulk of the information on the distributed severs and only return links between domain and search requests to the Central Search Engine. To address the possibility of duplicates being returned by the remote severs a similarity check is done before returning the results to the users Crawling The crawling package is currently a Custom Java Web Crawler threaded to use all 6 CPUs on our test machine and is able to crawl 20 pages/ second or 20796 pages in 18 minutes during time testing. To achieve the speed we had to modify our design to perform the crawling first and the duplicate detecting, indexing and page rank in a batch mode following crawling. We also had to avoid using a SQL data base to store parent child relationships which caused bottle necks. Below are three snap shots from the performance Monitor during the three phases of Crawling. The first phase is loading the URLs already seen into the URL visited pool to avoid visiting them again unless specified. To increase the speed here we kept the URL names in their MD5 Hash form to avoid recalculating the MD5 Hashes each time we restarted crawling. The second phase we populate the URL pool from the last few pages written to disk. Each page is a vector which contains the RAW HTML, and the children links from that page. And the third phase is the crawling function itself which crawls the web in a limited depth with a stay in Domain setting. When you examine the third phase you should notice that the CPUs are fully being utilized without causing network and disk bottle necks, therefore we believe that we can easily increase our crawling speed by moving to a processors with more CPUs. Cralwer Performance Monitor Loading already seen URLS Starting to crawl Middle of a crawl Page Rank In our project we chose to use a domain Rank as an approximation for the PageRank. As proposed by Wu and Aberer  and similar to Wang and DeWitt method of utilizing a Sever Rank combined with a local PageRank  to approximate a Global PageRank. Our PageRank algorithm is able to compute the PageRank for 80465 domains in under 1 second and is currently taking nine interactionsto converge at the Page Rank. In our sample set we found the top domains to have the following PageRank and compared them against Gephi’s PageRank. When we examine the results we find that our PageRank is very similar to Gephi’s PageRank which is sufficient enough to prove that our algorithm is correct enough for our requirements, since PageRank is only one of the weights we plan on using to prioritize search results. Our top 16 Page Rank Random teleports (beta=0.8) Gephi Default PageRank Random teleports (beta=0.8) 1 twitter.com 0.001567 twitter.com 2 facebook.com 0.001191 facebook.com 0.00599 facebook.com 3 akamaihd.net 0.001017 akamaihd.net 0.004807 akamaihd.net 4 google.com google.com 0.004713 google.com 5 apple.com apple.com 0.002376 apple.com 6 youtube.com 0.00041 youtube.com 0.002013 youtube.com 7 flickr.com 0.000284 linkedin.com 0.001496 linkedin.com 8 linkedin.com 0.000267 flickr.com 0.001353 flickr.com 9 t.co 0.000228 t.co 0.001065 t.co 10 export.gov 0.000202 export.gov 9.54E-04 export.gov 11 twimg.com 0.000197 pinterest.com 9.27E-04 vine.co 12 vine.co 0.000195 vine.co 9.20E-04 twimg.com 13 pinterest.com 0.000167 twimg.com 9.06E-04 pinterest.com 14 co.uk 0.000145 instagram.com 7.77E-04 instagram.com 15 instagram.com 0.000139 co.uk 7.20E-04 co.uk 16 tumblr.com tumblr.com 6.49E-04 tumblr.com 0.0009 0.000426 0.00012 Gephi Default Pag Random teleports 0.0078 twitter.com Near-Duplicate Detection In our project we modified the shingling algorithm discussed in class  to select the random k-grams in a document following stop words. The new algorithm SpotSigs  uses the fact stop words seldom occur in the unimportant template blocks such as navigation sidebar or links shown at the bottom of the page. By selecting the k-grams following stop words, the shingling algorithm focus on the documents true content ignoring the unimportant information. In experiments this method outperformed methods utilizing a purely random kgrams selection . Results The shingling algorithm detected 365 near duplicate documents out of a set of 1276 documents in 8 seconds with a Jaccard similarity threshold of .85. In additional testing we were able to process 9996 pages in under 6 minutes. Log Test set of 1276 0:00:00.000 Loading pages 0:00:00.009 Enumerating File List 0:00:00.126 Loading Files 0:00:00.128 Waiting for threads to work 0:00:06.320 1000 pages loaded 0:00:06.320 Constructing Shingles 0:00:08.101 Done Test set of 9996 0:00:00.000 0:00:00.008 0:00:02.625 0:00:02.637 0:04:30.231 0:04:30.231 0:05:38.123 Loading pages Enumerating File List Loading Files Waiting for (file loading) threads to work 9996 pages loaded Constructing Shingles Done Search Engine The search engine is based on an Apache Lucerne search engine which we modified to user our PageRank in conjunction with the key word weighting to prioritize search returns and we incorporated a final similarity check since our results are made up of the top results from the remote severs, which when combined could include duplicates. In addition to duplicate detection it also allows us to show more unique query returns to the user with an option to view similar returns. Results The indexing of the files into Lucerne is by far the most time consuming batch process which takes approximately 80 minutes to index 111068 pages. Findings The most glaring finding we discovered was the amount of processing power required to perform the crawling and indexing processes, and difficulties we encountered while looking for a unique novel fact in all the pages we crawled. At first we hoped to locate RDF information in the HTML pages we crawled; however as we learned in class the semantic web is for machines and HTML is for humans. Next we tried to use an FP-growth algorithm to hopefully strip out some useful facts, and in testing we discovered that text mining is a project unto itself and abandoned the idea when the FP- algorithm returned nothing when processing 2000 sports news pages. In an effort to discover a useful fact from all our data we adjusted near-duplicates algorithm to focus on the contents of the <script> tags in html documents to calculate a Domain Framework Similarity. Which allows us to group domains into similar styles to identify possible authors and allow for a focused mining effort for knowledge extractions on style rather than on simple words association. Below is the top ten domains similar to marislilly.de. Therefore once we able to extract meaningful information from marislilly.de we should be able to easily extract information from gepinsel.de. Domain Similarity marislilly.de 0.454545454545454 gepinsel.de 0.441176470588235 chocolate-bit.ch 0.435897435897436 feinschmeckerle.de 0.40625 dielorbeerkrone.com 0.4 oberstrifftsahne.com 0.4 depeu-japon.com 0.394736842105263 meinesuessewerkstatt.de 0.394736842105263 bearnerdette.de 0.393939393939394 midiamundo.com 0.382352941176471 fashionhippieloves.com 0.371428571428571 References  Hajishirzi, Hannaneh, Wen-tau Yih, and Aleksander Kolcz. "Adaptive near-duplicate detection via similarity learning." Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval. ACM, 2010.  Anand Rajaraman and Jeff Ullman, Mining of massive datasets , 2013.  Wu, Jie, and Karl Aberer. "Using siterank for decentralized computation of web document ranking." Adaptive Hypermedia and Adaptive Web-Based Systems. Springer Berlin Heidelberg, 2004.  Wang, Yuan, and David J. DeWitt. "Computing pagerank in a distributed internet search system." Proceedings of the Thirtieth international conference on Very large data bases-Volume 30. VLDB Endowment, 2004.