Accessing hidden web documents by metasearching a directory of specialty search engines

Jacky K H Shiu, Stephen C F Chan, Fu Lai Korris Chung

Research output: Journal article publicationJournal articleAcademic researchpeer-review

Abstract

Many valuable Web documents have not been indexed by general search engines and are only accessible through specific search interfaces. Metasearching groups of specialty search engines is one possible way to gain access to large amount of such hidden Web resources. One of the key issues for returning quality metasearch results is how to select the most relevant specialty search engines for a given query. We introduce a method for categorizing specialty search engines automatically into a hierarchical directory for metasearching. By utilizing the directory, specialty search engines that have a high possibility of having relevant information and resources can be easily selected by a metasearch engine. We evaluate our algorithm by comparing the directory built by the proposed algorithm with another one that was built by human-judgments. In addition, we present a metasearch engine prototype, which demonstrates that such a specialty search engine directory can be beneficial in locating essential but hidden Web resources.
Original languageEnglish
Pages (from-to)27-41
Number of pages15
JournalLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume2822
Publication statusPublished - 1 Dec 2003

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Cite this