Hyperspectral image (HSI) contains various spectral and spatial information, which is often used in remote sensing image analysis and widely used in areas of the people's daily life. Due to the advances of powerful feature representations, deep learning based methods are receiving increasing attention and getting acceptable classification results. As a representative of the deep learning methods, convolutional neural networks (CNNs) have shown their great ability in HSI classification tasks. However, the hyper-parameters of CNNs based HSI classification methods are often obtained through experience (e.g., the number of convolutional layers), and how to determine the number of convolutional layers (the model of convolutional layers connection) via data is seldom studied in existing CNNs based HSI classification methods. To deal with this problem, this paper proposes an effective approach to learn a structure of CNNs (e.g., a data-determined layers number of CNNs) in HSI classification tasks, where the CNNs structure can be learned via genetic algorithm (GA). with the learned adaptive CNNs structure can aquire better HSI classification result. Experimental results on two datasets demonstrate the effectiveness of the proposed method.