SharpSpider: Spidering the Web through Web services

K Moody; Marco Palomino

doi:10.1109/LAWEB.2003.1250304

SharpSpider: Spidering the Web through Web services

K Moody, Marco Palomino

University of Cambridge

Research output: Chapter in Book/Report/Conference proceeding › Published conference contribution

7 Citations (Scopus)

Abstract

Web search engines have become an indispensable utility for Internet users. In the near future, however, Web search engines will not only be expected to provide quality search results, but also to enable applications to search and exploit their index repositories directly. We present here SharpSpider, a distributed, C# spider designed to address the issues of scalability, decentralisation and continuity of a Web crawl. Fundamental to the design of SharpSpider is the publication of an API for use by other services on the network. Such an API grants access to a constantly refreshed index buiU after successive crawls of the Web.

Original language	English
Title of host publication	2003 First Latin American Web Congress
Publisher	IEEE Explore
Number of pages	3
ISBN (Print)	0-7695-2058-8
DOIs	https://doi.org/10.1109/LAWEB.2003.1250304
Publication status	Published - 12 Nov 2003
Externally published	Yes

Access to Document

10.1109/LAWEB.2003.1250304Licence: Unspecified

Cite this

@inproceedings{f7a32cce5b234627b4eee4072cad3861,

title = "SharpSpider: Spidering the Web through Web services",

abstract = "Web search engines have become an indispensable utility for Internet users. In the near future, however, Web search engines will not only be expected to provide quality search results, but also to enable applications to search and exploit their index repositories directly. We present here SharpSpider, a distributed, C# spider designed to address the issues of scalability, decentralisation and continuity of a Web crawl. Fundamental to the design of SharpSpider is the publication of an API for use by other services on the network. Such an API grants access to a constantly refreshed index buiU after successive crawls of the Web.",

author = "K Moody and Marco Palomino",

year = "2003",

month = nov,

day = "12",

doi = "10.1109/LAWEB.2003.1250304",

language = "English",

isbn = "0-7695-2058-8",

booktitle = "2003 First Latin American Web Congress",

publisher = "IEEE Explore",

}

TY - GEN

T1 - SharpSpider

T2 - Spidering the Web through Web services

AU - Moody, K

AU - Palomino, Marco

PY - 2003/11/12

Y1 - 2003/11/12

N2 - Web search engines have become an indispensable utility for Internet users. In the near future, however, Web search engines will not only be expected to provide quality search results, but also to enable applications to search and exploit their index repositories directly. We present here SharpSpider, a distributed, C# spider designed to address the issues of scalability, decentralisation and continuity of a Web crawl. Fundamental to the design of SharpSpider is the publication of an API for use by other services on the network. Such an API grants access to a constantly refreshed index buiU after successive crawls of the Web.

AB - Web search engines have become an indispensable utility for Internet users. In the near future, however, Web search engines will not only be expected to provide quality search results, but also to enable applications to search and exploit their index repositories directly. We present here SharpSpider, a distributed, C# spider designed to address the issues of scalability, decentralisation and continuity of a Web crawl. Fundamental to the design of SharpSpider is the publication of an API for use by other services on the network. Such an API grants access to a constantly refreshed index buiU after successive crawls of the Web.

U2 - 10.1109/LAWEB.2003.1250304

DO - 10.1109/LAWEB.2003.1250304

M3 - Published conference contribution

SN - 0-7695-2058-8

BT - 2003 First Latin American Web Congress

PB - IEEE Explore

ER -

SharpSpider: Spidering the Web through Web services

Abstract

Access to Document

Fingerprint

Cite this