TY - JOUR
T1 - Nested logistic regression models and ΔAUC applications: Change-point analysis
AU - Lee, Chun Yin
N1 - Funding Information:
The author is grateful to the editor and two reviewers for their valuable comments and suggestions that greatly improved the quality of the paper. The author(s) received no financial support for the research, authorship, and/or publication of this article.
Publisher Copyright:
© The Author(s) 2021.
PY - 2021/7
Y1 - 2021/7
N2 - The area under the receiver operating characteristic curve (AUC) is one of the most popular measures for evaluating the performance of a predictive model. In nested models, the change in AUC (ΔAUC) can be a discriminatory measure of whether the newly added predictors provide significant improvement in terms of predictive accuracy. Recently, several authors have shown rigorously that ΔAUC can be degenerate and its asymptotic distribution is no longer normal when the reduced model is true, but it could be the distribution of a linear combination of some (Formula presented.) random variables [1,2]. Hence, the normality assumption and existing variance estimate cannot be applied directly for developing a statistical test under the nested models. In this paper, we first provide a brief review on the use of ΔAUC for comparing nested logistic models and the difficulty of retrieving the reference distribution behind. Then, we present a special case of the nested logistic regression models that the newly added predictor to the reduced model contains a change-point in its effects. A new test statistic based on ΔAUC is proposed in this setting. A simple resampling scheme is proposed to approximate the critical values for the test statistic. The inference of the change-point parameter is done via m-out-of-n bootstrap. Large-scale simulation is conducted to evaluate the finite-sample performance of the ΔAUC test for the change-point model. The proposed method is applied to two real-life datasets for illustration.
AB - The area under the receiver operating characteristic curve (AUC) is one of the most popular measures for evaluating the performance of a predictive model. In nested models, the change in AUC (ΔAUC) can be a discriminatory measure of whether the newly added predictors provide significant improvement in terms of predictive accuracy. Recently, several authors have shown rigorously that ΔAUC can be degenerate and its asymptotic distribution is no longer normal when the reduced model is true, but it could be the distribution of a linear combination of some (Formula presented.) random variables [1,2]. Hence, the normality assumption and existing variance estimate cannot be applied directly for developing a statistical test under the nested models. In this paper, we first provide a brief review on the use of ΔAUC for comparing nested logistic models and the difficulty of retrieving the reference distribution behind. Then, we present a special case of the nested logistic regression models that the newly added predictor to the reduced model contains a change-point in its effects. A new test statistic based on ΔAUC is proposed in this setting. A simple resampling scheme is proposed to approximate the critical values for the test statistic. The inference of the change-point parameter is done via m-out-of-n bootstrap. Large-scale simulation is conducted to evaluate the finite-sample performance of the ΔAUC test for the change-point model. The proposed method is applied to two real-life datasets for illustration.
KW - Area under the receiver operating characteristic curve
KW - change-points
KW - discriminatory measures
KW - m-out-of-n bootstrap
KW - nested models
UR - http://www.scopus.com/inward/record.url?scp=85107891207&partnerID=8YFLogxK
U2 - 10.1177/09622802211022377
DO - 10.1177/09622802211022377
M3 - Journal article
AN - SCOPUS:85107891207
SN - 0962-2802
VL - 30
SP - 1654
EP - 1666
JO - Statistical Methods in Medical Research
JF - Statistical Methods in Medical Research
IS - 7
ER -