Home > ResourceBlog > Article

« All ResourceBlog Articles

 

Feed

Sunday, 29th July 2007

Grub, Distributed Crawling Will Be Used to Build Wikia While LookSmart Will Power Wikia Advertising; Other Open Source Crawling Tools

Grub distributed crawling technology is now being tested for use to to build Wikia after being acquired from Looksmart. This news came via comments from Wikia/Wikipedia co-founder, Jimmy Wales on Thursday.

Distributed crawling? Well, the concept has been around for web search for more than seven years. After a client is placed on your computer and during down time, you're computer will be one of many to crawl the web that will then build the overall database. See also the SETI@Home project.

More on that in a moment.

It's important to point out that advertising on the Wikia site will be delivered via a white-label ad serving platform from LookSmart.

This announcement was made two weeks ago.

It will handle both display and text-based ads and Wikia will be the first organization to use LookSmart's technology for the, "management/serving of display ad units utilizing CMP-based pricing."

"We did a lot of due diligence to find a flexible and intuitive ad serving technology that nets the highest revenue and yield," said Gil Penchina, CEO of Wikia. "We discovered in the process that LookSmart's platform and services not only provide dynamic optimization of both our advertisers and backfill networks, but the white label aspect of it fits perfectly with our brand strategy."

Now, back to the Grub crawling story.

Grub technology (the company's exec summary from 2001) was acquired by Looksmart in January, 2003) and is now in early testing for the Wikia project. LookSmart stopped using the technology in 2005 as mentioned in the annual report from April, 2006:

We discontinued the use and support of the Grub distributed crawling technology in 2005 in order to reallocate development and support resources to other revenue-generating initiatives in search technology.

Via a recent News.com item:

It's [Grub] meant to operate through open protocol and community collaborative added functions combined with the wiki.

+ Learn More About Grub ||| Monitor the Grub Wiki

History
"Help Grub Search the Past" by Chris Sherman, April 2003
&
LookSmart bets on distributed computing by Stefanie Olsen, News.com
See Also: Chris with more on Grub/Wikia in this Search Engine Land item posted the other day.

Blasts from the Past
Grub FAQ (12/09/2000). ||| Grub executive summary (April 2001)

Grub Home Pages (Back to 2000)
December 6, 2000 ||| January 30, 2003 ||| June 14, 2004

See Also: A Few Other Open Source Crawling/Search Tools

+ Nutch
Nutch (part of the Lucene project) is used several places including the massive U.S government web harvests containing terabytes of data. Another example is at UtilitySearch.info.

+ Heritrix

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler.

+ Avi Rappaport's essential SearchTools.com site lists many other open source crawlers and search engines.

Reading
Numerous projects are or have been tackling web search by building distributed and P2P tools
+ Emerging Semantic Communities in Peer Web Search

+ Scalable Hybrid Search on Distributed Databases

+ "Challenges in Distributed Information Retrieval" (PDF), From Yahoo Research

+ MINERVA: Collaborative P2P Search

+ Chora: Expert-based P2P Web Search

+ ODISSEA: A Peer-to-Peer Architecture for Scalable Web Search and Information Retrieval

+ Distributed Search in P2P Networks - Internet Computing, IEEE (PDF)

+ Evaluation of Peer Based Web Search

+ Webcast: Social Web Search (Part 2) ||| Slides
Held at Indiana University

Abstract: This talk will present two research projects under way in the Network and agents Network (NaN), which study ways of leveraging online social behavior for better Web search. GiveALink.org is a social bookmarking site where users donate their personal bookmarks. A search and recommendation engine is built from a similarity network derived from the hierarchical structure of bookmarks, aggregated across users. 6S is a distributed Web search engine based on an ad adaptive peer network. By learning about each other, peers can route queries through the network to efficiently reach knowledgeable nodes. The resulting peer network structures itself as a small world that uncovers semantic communities and outperforms centralized search engines.

See Also: Learn More and Demo Here


Category:

Views: 1856




« All ResourceBlog Articles

 

FreePint

FreePint supports the value of information in the enterprise. Read more »


FeedLatest FreePint Content:


  • Click to view the article Niche Data Visualisation Tools - FreePint for Biopharma
    Friday, 21st November 2014

    Ashish Shukla looks at three recent product reviews and how Cortellis, Innography and Digimind Social perform in terms of allowing information managers to slice and dice data and obtain intuitive visual outputs.

  • Click to view the article Sell Side of News Part 2 - Today's Competitive Landscape
    Friday, 21st November 2014

    Best options for current awareness when using licensed news content, plus other options such as workflow tools and DIY are under scrutiny in the second part of Robin Neidorf's article on "the sell side of news" with commentary on providers such as Cision, Meltwater and Moreover Newsdesk.

  • Click to view the article Is Facebook Moving Inside the Enterprise?
    Thursday, 20th November 2014

    For a long time there has been discussion around when and how Facebook will build a version of the social networking site that could be used within organisations. The benefits of this are clear, not just for Facebook, but also for organisations, many of whom are already taking advantage of enterprise social networks to enable content creation and collaboration. However, there are a number of issues that Facebook and organisations need to think about before they deploy Facebook At Work.

  • ... more ...

All FreePint Content »
FreePint Topics »


A FreePint Subscription delivers articles and reports that support your organisation's information practice, content and strategy.

Find out more and order a FreePint Subscription by visiting the
completing our online form: Subscription Order page.


FreePint Testimonials

"FreePint pulls together everything you need as an information professional. If there were no FreePint we would probably have to go to a number ..."

Read more testimonials and supply yours »






 

 
 
 

Register

Register to receive the free ResourceShelf Newsletter, featuring highlighted posts.

Find out more »

Article Categories

All Article Categories »

Archive

All Archives »