Measuring relevance between discrete and continuous features based on neighborhood mutual information

Qinghua Hu, Lei Zhang, Dapeng Zhang, Wei Pan, Shuang An, Witold Pedrycz

Research output: Journal article publicationJournal articleAcademic researchpeer-review

146 Citations (Scopus)


Measures of relevance between features play an important role in classification and regression analysis. Mutual information has been proved an effective measure for decision tree construction and feature selection. However, there is a limitation in computing relevance between numerical features with mutual information due to problems of estimating probability density functions in high-dimensional spaces. In this work, we generalize Shannon's information entropy to neighborhood information entropy and propose a measure of neighborhood mutual information. It is shown that the new measure is a natural extension of classical mutual information which reduces to the classical one if features are discrete; thus the new measure can also be used to compute the relevance between discrete variables. In addition, the new measure introduces a parameter delta to control the granularity in analyzing data. With numeric experiments, we show that neighborhood mutual information produces the nearly same outputs as mutual information. However, unlike mutual information, no discretization is required in computing relevance when used the proposed algorithm. We combine the proposed measure with four classes of evaluating strategies used for feature selection. Finally, the proposed algorithms are tested on several benchmark data sets. The results show that neighborhood mutual information based algorithms yield better performance than some classical ones.
Original languageEnglish
Pages (from-to)10737-10750
Number of pages14
JournalExpert Systems with Applications
Issue number9
Publication statusPublished - 1 Sept 2011


  • Continuous feature
  • Feature selection
  • Neighborhood entropy
  • Neighborhood mutual information
  • Relevance

ASJC Scopus subject areas

  • General Engineering
  • Computer Science Applications
  • Artificial Intelligence


Dive into the research topics of 'Measuring relevance between discrete and continuous features based on neighborhood mutual information'. Together they form a unique fingerprint.

Cite this