Hyper-Spectral Image (HSI) classification is an important task because of its wide range of applications. With the remarkable success from the Convolutional Neural Network (CNN), the performance of HSI classification has been significantly improved. However, two main challenges remained. One is that the samples of HSI have dramatic intra-class diversity and inter-class similarity, and the conventional cross-entropy loss is not good enough to learn discriminative features. The other is that the number of the training samples is so limited that the network is easy to overfit. To address the first challenge, we develop an improved triplet loss in order to make samples from the same class close to each other and make samples from different classes further apart. The proposed loss function considers all the possible positive pairs and negative pairs in a training batch, filters many trivial pairs, and prevents the impact of the outliers at the same time. To deal with the second challenge, we design an appropriate network architecture with less learnable parameters. We train the designed network based on the proposed loss with randomly initialized network weights using only hundreds of training samples, and attain quite good results. The experimental results show that the proposed method significantly surpasses other state-of-the-art methods, especially with less training samples. Furthermore, being less complex, the training process only takes a few minutes on a single GPU, which is faster than other state-of-the-art CNN-based methods.