A robust correlation analysis framework for imbalanced and dichotomous data with uncertainty

Chun Sing Lai, Yingshan Tao, Fangyuan Xu, Wing W.Y. Ng, Youwei Jia, Haoliang Yuan, Chao Huang, Loi Lei Lai, Zhao Xu, Giorgio Locatelli

Research output: Journal article publicationJournal articleAcademic researchpeer-review

26 Citations (Scopus)

Abstract

Correlation analysis is one of the fundamental mathematical tools for identifying dependence between classes. However, the accuracy of the analysis could be jeopardized due to variance error in the data set. This paper provides a mathematical analysis of the impact of imbalanced data concerning Pearson Product Moment Correlation (PPMC) analysis. To alleviate this issue, the novel framework Robust Correlation Analysis Framework (RCAF) is proposed to improve the correlation analysis accuracy. A review of the issues due to imbalanced data and data uncertainty in machine learning is given. The proposed framework is tested with in-depth analysis of real-life solar irradiance and weather condition data from Johannesburg, South Africa. Additionally, comparisons of correlation analysis with prominent sampling techniques, i.e., Synthetic Minority Over-Sampling Technique (SMOTE) and Adaptive Synthetic (ADASYN) sampling techniques are conducted. Finally, K-Means and Wards Agglomerative hierarchical clustering are performed to study the correlation results. Compared to the traditional PPMC, RCAF can reduce the standard deviation of the correlation coefficient under imbalanced data in the range of 32.5%–93.02%.

Original languageEnglish
Pages (from-to)58-77
Number of pages20
JournalInformation Sciences
Volume470
DOIs
Publication statusPublished - Jan 2019

Keywords

  • Clearness index
  • Dichotomous variable
  • Imbalanced data
  • Pearson product-moment correlation

ASJC Scopus subject areas

  • Software
  • Control and Systems Engineering
  • Theoretical Computer Science
  • Computer Science Applications
  • Information Systems and Management
  • Artificial Intelligence

Cite this