|
Saturday, 13th January 2007
New from Yahoo Research: Web Spam Detection using the Web Topology & Challenges in Distributed Information Retrieval
Two papers from Yahoo Research that might be of interest.
#1
Challenges in Distributed Information Retrieval
by Ricardo Baeza-Yates, Carlos Castillo, Flavio Junqueira, Vassilis Plachouras and Fabrizio Silvestri
This invited paper will be presented at In ICDE [International Conference on Data Engineering] in April, 2007.
From the abstract:
In the ocean of Web data, Web search engines are the primary way to access content. As the data is on the order of petabytes, current search engines are very large centralized systems based on replicated clusters. Web data, however, is always evolving. The number of Web sites continues to grow rapidly and there are currently more than 20 billion indexed pages. In the near future, centralized systems are likely to become ineffective against such a load, thus suggesting the need of fully distributed search engines. Such engines need to achieve the following goals: high quality answers, fast response time, high query throughput, and scalability. In this paper we survey and organize recent research results, outlining the main challenges of designing a distributed Web retrieval system.
Request the full text from YR or download (15 pages; PDF) via the web site of Carlos Castillo.
#2
Know your Neighbors: Web Spam Detection using the Web Topology
by Carlos Castillo and Debora Donato and Aristides Gionis and Vanessa Murdock and Fabrizio Silvestri (2006)
To access the full text of this report, either visit the web site of Carlos Castillo and download the paper (PDF; 10 pages) or request directly from Yahoo Research.
Abstract:
Web spam can significantly deteriorate the quality of search engine results. Thus there is a large incentive for commercial search engines to detect spam pages efficiently and accurately. In this paper we present a spam detection system that uses the topology of the Web graph by exploiting the link dependencies among the Web pages, and the content of the pages themselves. We find that linked hosts tend to belong to the same class: either both are spam or both are non-spam. We demonstrate three methods of incorporating the Web graph topology into the predictions obtained by our base classifier: (i) clustering the host graph, and assigning the label of all hosts in the cluster by majority vote, (ii) propagating the predicted labels to neighboring hosts, and (iii) using the predicted labels of neighboring hosts as new features and retraining the classifier. The result is an accurate system for detecting Web spam that can be applied in practice to large-scale Web data.
Source: Yahoo Research; Web Site of Carlos Castillo
See Also: SIGIR Forum: A Reference Collection for Web Spam, Open Source Information Retrieval Systems Other Articles
Category:
Views: 1232
|
Webinar Will Bring Big Data Down to Size
Law Librarians and Content Choices
