Accessing hidden web documents by metasearching a directory of specialty search engines

Many valuable Web documents have not been indexed by general search engines and are only accessible through specific search interfaces. Metasearching groups of specialty search engines is one possible way to gain access to large amount of such hidden Web resources. One of the key issues for returning quality metasearch results is how to select the most relevant specialty search engines for a given query. We introduce a method for categorizing specialty search engines automatically into a hierarchical directory for metasearching. By utilizing the directory, specialty search engines that have a high possibility of having relevant information and resources can be easily selected by a metasearch engine. We evaluate our algorithm by comparing the directory built by the proposed algorithm with another one that was built by human-judgments. In addition, we present a metasearch engine prototype, which demonstrates that such a specialty search engine directory can be beneficial in locating essential but hidden Web resources.