Home > ResourceBlog > Article

« All ResourceBlog Articles

 

Feed

Sunday, 29th July 2007

Grub, Distributed Crawling Will Be Used to Build Wikia While LookSmart Will Power Wikia Advertising; Other Open Source Crawling Tools

Grub distributed crawling technology is now being tested for use to to build Wikia after being acquired from Looksmart. This news came via comments from Wikia/Wikipedia co-founder, Jimmy Wales on Thursday.

Distributed crawling? Well, the concept has been around for web search for more than seven years. After a client is placed on your computer and during down time, you're computer will be one of many to crawl the web that will then build the overall database. See also the SETI@Home project.

More on that in a moment.

It's important to point out that advertising on the Wikia site will be delivered via a white-label ad serving platform from LookSmart.

This announcement was made two weeks ago.

It will handle both display and text-based ads and Wikia will be the first organization to use LookSmart's technology for the, "management/serving of display ad units utilizing CMP-based pricing."

"We did a lot of due diligence to find a flexible and intuitive ad serving technology that nets the highest revenue and yield," said Gil Penchina, CEO of Wikia. "We discovered in the process that LookSmart's platform and services not only provide dynamic optimization of both our advertisers and backfill networks, but the white label aspect of it fits perfectly with our brand strategy."

Now, back to the Grub crawling story.

Grub technology (the company's exec summary from 2001) was acquired by Looksmart in January, 2003) and is now in early testing for the Wikia project. LookSmart stopped using the technology in 2005 as mentioned in the annual report from April, 2006:

We discontinued the use and support of the Grub distributed crawling technology in 2005 in order to reallocate development and support resources to other revenue-generating initiatives in search technology.

Via a recent News.com item:

It's [Grub] meant to operate through open protocol and community collaborative added functions combined with the wiki.

+ Learn More About Grub ||| Monitor the Grub Wiki

History
"Help Grub Search the Past" by Chris Sherman, April 2003
&
LookSmart bets on distributed computing by Stefanie Olsen, News.com
See Also: Chris with more on Grub/Wikia in this Search Engine Land item posted the other day.

Blasts from the Past
Grub FAQ (12/09/2000). ||| Grub executive summary (April 2001)

Grub Home Pages (Back to 2000)
December 6, 2000 ||| January 30, 2003 ||| June 14, 2004

See Also: A Few Other Open Source Crawling/Search Tools

+ Nutch
Nutch (part of the Lucene project) is used several places including the massive U.S government web harvests containing terabytes of data. Another example is at UtilitySearch.info.

+ Heritrix

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler.

+ Avi Rappaport's essential SearchTools.com site lists many other open source crawlers and search engines.

Reading
Numerous projects are or have been tackling web search by building distributed and P2P tools
+ Emerging Semantic Communities in Peer Web Search

+ Scalable Hybrid Search on Distributed Databases

+ "Challenges in Distributed Information Retrieval" (PDF), From Yahoo Research

+ MINERVA: Collaborative P2P Search

+ Chora: Expert-based P2P Web Search

+ ODISSEA: A Peer-to-Peer Architecture for Scalable Web Search and Information Retrieval

+ Distributed Search in P2P Networks - Internet Computing, IEEE (PDF)

+ Evaluation of Peer Based Web Search

+ Webcast: Social Web Search (Part 2) ||| Slides
Held at Indiana University

Abstract: This talk will present two research projects under way in the Network and agents Network (NaN), which study ways of leveraging online social behavior for better Web search. GiveALink.org is a social bookmarking site where users donate their personal bookmarks. A search and recommendation engine is built from a similarity network derived from the hierarchical structure of bookmarks, aggregated across users. 6S is a distributed Web search engine based on an ad adaptive peer network. By learning about each other, peers can route queries through the network to efficiently reach knowledgeable nodes. The resulting peer network structures itself as a small world that uncovers semantic communities and outperforms centralized search engines.

See Also: Learn More and Demo Here


Category:

Views: 1856




« All ResourceBlog Articles

 

FreePint

FreePint supports the value of information in the enterprise. Read more »


FeedLatest FreePint Content:


  • Click to view the article New Angles on Current Awareness, New Light on Research
    Thursday, 27th November 2014

    The FreePint Topic Series: News, and Beyond is now well-advanced, with a host of product reviews, articles and other resources exploring key themes relating to news for current awareness and for research. This update article helps you catch up with recent coverage and gives an update on what information is still to come.

  • Click to view the article Beating the Language Barrier in Research
    Thursday, 27th November 2014

    Andrew Lucas looks at why, despite the proliferation of translation tools, language remains a barrier to using local language content and why local language sources are hard to come by.

  • Click to view the article Why You Might Need a Media Asset Management System
    Wednesday, 26th November 2014

    This article describes why it is as important to manage our corporate audiovisual material as our written documents. Katharine Schopflin outlines what you need from a system to manage your media assets and what to do if you become responsible for creating order out of asset management chaos.

  • ... more ...

All FreePint Content »
FreePint Topics »


A FreePint Subscription delivers articles and reports that support your organisation's information practice, content and strategy.

Find out more and order a FreePint Subscription by visiting the
completing our online form: Subscription Order page.


FreePint Testimonials

"I was very impressed with the quality of the Webinar. I attend a lot of professional webinars and this was one of the best. It was well ..."

Read more testimonials and supply yours »






 

 
 
 

Register

Register to receive the free ResourceShelf Newsletter, featuring highlighted posts.

Find out more »

Article Categories

All Article Categories »

Archive

All Archives »