Consistency of accuracy assessment indices for soft classification: Simulation analysis

Jin Chen, Xiaolin Zhu, Hidefumi Imura, Xuehong Chen

Research output: Journal article publicationJournal articleAcademic researchpeer-review

18 Citations (Scopus)

Abstract

Accuracy assessment plays a crucial role in the implementation of soft classification. Even though many indices of accuracy assessment for soft classification have been proposed, the consistencies among these indices are not clear, and the impact of sample size on these consistencies has not been investigated. This paper examines two kinds of indices: map-level indices, including root mean square error (rmse), kappa, and overall accuracy (oa) from the sub-pixel confusion matrix (SCM); and category-level indices, including crmse, user accuracy (ua) and producer accuracy (pa). A careful simulation was conducted to investigate the consistency of these indices and the effect of sample size. The major findings were as follows: (1) The map-level indices are highly consistent with each other, whereas the category-level indices are not. (2) The consistency among map-level and category-level indices becomes weaker when the sample size decreases. (3) The rmse is more affected by error distribution among classes than are kappa and oa. Based on these results, we recommend that rmse can be used for map-level accuracy due to its simplicity, although kappa and oa may be better alternatives when the sample size is limited because the two indices are affected less by the error distribution among classes. We also suggest that crmse should be provided when map users are not concerned about the error source, whereas ua and pa are more useful when the complete information about different errors is required. The results of this study will be of benefit to the development and application of soft classifiers. (ISPRS).
Original languageEnglish
Pages (from-to)156-164
Number of pages9
JournalISPRS Journal of Photogrammetry and Remote Sensing
Volume65
Issue number2
DOIs
Publication statusPublished - 1 Mar 2010
Externally publishedYes

Keywords

  • Accuracy assessment
  • Consistency
  • RMSE
  • Soft classification
  • Sub-pixel confusion matrix

ASJC Scopus subject areas

  • Geography, Planning and Development
  • Atomic and Molecular Physics, and Optics
  • Engineering (miscellaneous)
  • Computer Science Applications
  • Computers in Earth Sciences

Cite this