Home > ResourceBlog > Article

« All ResourceBlog Articles

 

Bookmark and Share   Feed

Sunday, 29th July 2007

Grub, Distributed Crawling Will Be Used to Build Wikia While LookSmart Will Power Wikia Advertising; Other Open Source Crawling Tools

Grub distributed crawling technology is now being tested for use to to build Wikia after being acquired from Looksmart. This news came via comments from Wikia/Wikipedia co-founder, Jimmy Wales on Thursday.

Distributed crawling? Well, the concept has been around for web search for more than seven years. After a client is placed on your computer and during down time, you're computer will be one of many to crawl the web that will then build the overall database. See also the SETI@Home project.

More on that in a moment.

It's important to point out that advertising on the Wikia site will be delivered via a white-label ad serving platform from LookSmart.

This announcement was made two weeks ago.

It will handle both display and text-based ads and Wikia will be the first organization to use LookSmart's technology for the, "management/serving of display ad units utilizing CMP-based pricing."

"We did a lot of due diligence to find a flexible and intuitive ad serving technology that nets the highest revenue and yield," said Gil Penchina, CEO of Wikia. "We discovered in the process that LookSmart's platform and services not only provide dynamic optimization of both our advertisers and backfill networks, but the white label aspect of it fits perfectly with our brand strategy."

Now, back to the Grub crawling story.

Grub technology (the company's exec summary from 2001) was acquired by Looksmart in January, 2003) and is now in early testing for the Wikia project. LookSmart stopped using the technology in 2005 as mentioned in the annual report from April, 2006:

We discontinued the use and support of the Grub distributed crawling technology in 2005 in order to reallocate development and support resources to other revenue-generating initiatives in search technology.

Via a recent News.com item:

It's [Grub] meant to operate through open protocol and community collaborative added functions combined with the wiki.

+ Learn More About Grub ||| Monitor the Grub Wiki

History
"Help Grub Search the Past" by Chris Sherman, April 2003
&
LookSmart bets on distributed computing by Stefanie Olsen, News.com
See Also: Chris with more on Grub/Wikia in this Search Engine Land item posted the other day.

Blasts from the Past
Grub FAQ (12/09/2000). ||| Grub executive summary (April 2001)

Grub Home Pages (Back to 2000)
December 6, 2000 ||| January 30, 2003 ||| June 14, 2004

See Also: A Few Other Open Source Crawling/Search Tools

+ Nutch
Nutch (part of the Lucene project) is used several places including the massive U.S government web harvests containing terabytes of data. Another example is at UtilitySearch.info.

+ Heritrix

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler.

+ Avi Rappaport's essential SearchTools.com site lists many other open source crawlers and search engines.

Reading
Numerous projects are or have been tackling web search by building distributed and P2P tools
+ Emerging Semantic Communities in Peer Web Search

+ Scalable Hybrid Search on Distributed Databases

+ "Challenges in Distributed Information Retrieval" (PDF), From Yahoo Research

+ MINERVA: Collaborative P2P Search

+ Chora: Expert-based P2P Web Search

+ ODISSEA: A Peer-to-Peer Architecture for Scalable Web Search and Information Retrieval

+ Distributed Search in P2P Networks - Internet Computing, IEEE (PDF)

+ Evaluation of Peer Based Web Search

+ Webcast: Social Web Search (Part 2) ||| Slides
Held at Indiana University

Abstract: This talk will present two research projects under way in the Network and agents Network (NaN), which study ways of leveraging online social behavior for better Web search. GiveALink.org is a social bookmarking site where users donate their personal bookmarks. A search and recommendation engine is built from a similarity network derived from the hierarchical structure of bookmarks, aggregated across users. 6S is a distributed Web search engine based on an ad adaptive peer network. By learning about each other, peers can route queries through the network to efficiently reach knowledgeable nodes. The resulting peer network structures itself as a small world that uncovers semantic communities and outperforms centralized search engines.

See Also: Learn More and Demo Here


Category:

Views: 1854




« All ResourceBlog Articles

 

FreePint

FreePint supports the value of information in the enterprise. Read more »


FeedLatest FreePint Content:


  • Click to view the article Making Your Intranet Unusable
    Friday, 24th October 2014

    James Mullan has written a number of FreePint articles about some of the tools and techniques you can use to make your intranet more effective. Now in what some people might think is a cruel twist of fate, he's decided to look at some of the ways in which you can make your intranet unusable.

  • Click to view the article Assessing the Credibility of Crowdsourced Content
    Friday, 24th October 2014

    Libby Trudell investigates the role of crowdsourcing in enterprise information. She defines the term, provides an overview of quality control and gives some examples of crowdsourced research services in areas from company data to intellectual property. Finally she identifies key questions to ask if you are incorporating such resources into your research portfolio.

  • Click to view the article InfoDesk Eases the Push of Relevant Real-Time Content
    Friday, 24th October 2014

    InfoDesk is a content integration and information management solution company with an emphasis on current awareness content delivery. The company defines itself as content neutral, meaning it will integrate any electronic content to provide organisations with a custom search and content delivery system that makes the most of their information assets.

  • ... more ...

All FreePint Content »
FreePint Topics »


A FreePint Subscription delivers articles and reports that support your organisation's information practice, content and strategy.

Find out more and order a FreePint Subscription by visiting the
completing our online form: Subscription Order page.


FreePint Testimonials

"Loved it. Very impressed. It was great. A very high calibre event. The people participating were of a very high calibre. Felt that the topics ..."

Read more testimonials and supply yours »






 

 
 
 

Register

Register to receive the free ResourceShelf Newsletter, featuring highlighted posts.

Find out more »

Article Categories

All Article Categories »

Archive

All Archives »