Hybrid term indexing for weighted boolean and vector space models

K.C.W. Chow, Wing Pong Robert Luk, K.F. Wong, K.L. Kwok

Research output: Journal article publicationJournal articleAcademic researchpeer-review

Abstract

Retrieval effectiveness depends on how terms are extracted and indexed. For Chinese text (and others like Japanese and Korean), there are no space to delimit words. Indexing using hybrid terms (i.e. words and bigrams) were able to achieve the best precision amongst homogenous terms at a lower storage cost than indexing with bigrams. However, this was tested with conjunctive queries. Here, we extended the weighted Boolean models using fuzzy and p-norm measures, as well as the vector space model using the cosine measure, for processing hybrid terms. Our evaluation shows that all IR models using hybrid terms achieve better average precision over those using words. Across different recall values, the weighted Boolean model using fuzzy measures with hybrid terms achieve consistently about 8% higher than those using words. The vector space model using the cosine measures with hybrid terms achieved the best improvement in the average recall and precision.
Original languageEnglish
Pages (from-to)133-151
Number of pages19
JournalInternational journal of computer processing of languages
Volume14
Issue number2
DOIs
Publication statusPublished - 2001

Keywords

  • Chinese Information Retrieval
  • Indexing
  • IR Models
  • Evaluation

Cite this