The pitch-processing deficit associated with congenital amusia has been shown to be transferable to lexical tone processing. However, it remains unclear whether the tone perception difficulties of amusics are merely due to the domain-general deficit in acoustic processing or additionally caused by impaired higher-level phonological operations. Answers to this question can shed light on the influence of lower-level acoustic processing on higher-level phonological processing. Using a modified categorical perception paradigm, the present study indicates that the acoustic processing deficit systematically extends to higher-level phonological processing. These findings suggest that lower-level acoustics underlie higher-level phonological categories in lexical tone perception.