ResourceBlog Article: New from Yahoo Research: Web Spam Detection using the Web Topology & Challenges in Distributed Information Retrieval

contact us \| suggestion

Receive the weekly sampler of posts and "Resource of the Week".
Subscribe »

My Account »

If you find ResourceShelf useful, please supply a testimonial »

Home > ResourceBlog > Article

« All ResourceBlog Articles

Saturday, 13th January 2007

New from Yahoo Research: Web Spam Detection using the Web Topology & Challenges in Distributed Information Retrieval

Two papers from Yahoo Research that might be of interest.

#1
Challenges in Distributed Information Retrieval
by Ricardo Baeza-Yates, Carlos Castillo, Flavio Junqueira, Vassilis Plachouras and Fabrizio Silvestri
This invited paper will be presented at In ICDE [International Conference on Data Engineering] in April, 2007.
From the abstract:

In the ocean of Web data, Web search engines are the primary way to access content. As the data is on the order of petabytes, current search engines are very large centralized systems based on replicated clusters. Web data, however, is always evolving. The number of Web sites continues to grow rapidly and there are currently more than 20 billion indexed pages. In the near future, centralized systems are likely to become ineffective against such a load, thus suggesting the need of fully distributed search engines. Such engines need to achieve the following goals: high quality answers, fast response time, high query throughput, and scalability. In this paper we survey and organize recent research results, outlining the main challenges of designing a distributed Web retrieval system.

Request the full text from YR or download (15 pages; PDF) via the web site of Carlos Castillo.

#2
Know your Neighbors: Web Spam Detection using the Web Topology
by Carlos Castillo and Debora Donato and Aristides Gionis and Vanessa Murdock and Fabrizio Silvestri (2006)

To access the full text of this report, either visit the web site of Carlos Castillo and download the paper (PDF; 10 pages) or request directly from Yahoo Research.

Abstract:

Web spam can significantly deteriorate the quality of search engine results. Thus there is a large incentive for commercial search engines to detect spam pages efficiently and accurately. In this paper we present a spam detection system that uses the topology of the Web graph by exploiting the link dependencies among the Web pages, and the content of the pages themselves. We find that linked hosts tend to belong to the same class: either both are spam or both are non-spam. We demonstrate three methods of incorporating the Web graph topology into the predictions obtained by our base classifier: (i) clustering the host graph, and assigning the label of all hosts in the cluster by majority vote, (ii) propagating the predicted labels to neighboring hosts, and (iii) using the predicted labels of neighboring hosts as new features and retraining the classifier. The result is an accurate system for detecting Web spam that can be applied in practice to large-scale Web data.

Source: Yahoo Research; Web Site of Carlos Castillo

Category:

Search News (ResourceShelf)

blog comments powered by Disqus

« All ResourceBlog Articles

FreePint Family

A family of resources to help information workers be more effective, raise the value of information in their organisations and contribute to success. Read more »

Latest Family Articles:

Quilting big data threads
Thursday, 24th May 2012

Recently I have found myself cooing over visualisation maps (and heat maps) of health and well being resources. The content rich data is overlayed with mapping technologies, and some interesting themes and patterns are emerging.

The fallacy of information overload
Wednesday, 23rd May 2012

A lot of the talk around social media in the last year has been around information overload. Social media has provided us with new and exciting ways to create content. But it has also meant learning new ways to manage and engage with social media tools. Are we teetering on the edge of an information overload precipice?

Information overload: fact, fantasy or filter failure?
Wednesday, 23rd May 2012

Information overload is a figment of your imagination. Or a failure of your filter. Or a symptom of your technological submissiveness. Depends on who you ask.

Newsdesk: tracking millions of pieces of information a day
Tuesday, 22nd May 2012

What if you had to sort through 3.5 million articles and social media posts a day and try to pull out the most relevant items for your organisation? What if you then had to cobble it all together into something readable for your top groups and executives in your organisation?

Alacra Compliance adds managerial oversight
Tuesday, 22nd May 2012

Alacra Compliance saves time by aggregating information from both free and fee-based sources and enabling users to conduct an accurate federated search across these sources (coined “simultaneous search” by Alacra).

All Family Articles »
Family Articles by Category »

Tell us what you're working on,
and we'll talk to you about how FreePint can help »

FreePint Family Testimonials

"Fabulous resource to learn of unique tools and insights. Very useful." Manager, Futures and Forecasting, Virginia, USA

More testimonials »

Subscribe to the ResourceShelf Newsletter and receive the weekly sampler of posts and Resource of the Week.

Find out more »

ResourceShelf sponsored by:

Article Categories

Search News (9339)
Source File (9518)
Web 2.0 (2568)

All Article Categories »