The development of collaborative intrusion detection networks (CIDNs) aims to enhance the performance of a single intrusion detection system (IDS), through communicating and collecting information from other IDS nodes. To defend CIDNs against insider attacks, trust-based mechanisms are crucial for evaluating the trustworthiness of a node. In the literature, challenge-based trust mechanisms are well established to identify malicious nodes by identifying the deviation between challenges and responses. However, such mechanisms rely on two major assumptions, which may result in a weak threat model and render CIDNs still vulnerable to advanced insider attacks in a practical deployment. In this paper, our motivation is to investigate the effect of On-Off attacks on challenge-based CIDNs. In particular, as a study, we explore a special On-Off attack (called SOOA), which can keep responding normally to one node while acting abnormally to another node. In the evaluation, we explore the attack performance under simulated CIDN environments. Experimental results indicate that our attack can interfere the effectiveness of trust computation for CIDN nodes.