Hyperspectral images (HSIs) classification is an important task in earth observation missions. Convolution Neural Networks (CNNs) with the powerful ability of feature extraction has shown prominence in HSIs classification tasks. However, existing CNNsbased approaches cannot sufficiently mine the sequence attributes of spectral features, hindering the further performance promotion of HSIs classification. This paper presents a Hyperspectral Image Transformer (HiT) classification network by embedding convolution operations into the transformer structure to capture the subtle spectral discrepancies and convey the local spatial context information. HiT consists of two key modules, i.e., spectraladaptive 3D convolution projection module and Convolution Permutator (ConV-Permutator) to retrieve the subtle spatial-spectral discrepancies. The spectral-adaptive 3D convolution projection module produces the local spatial-spectral information from HSIs using two spectral-adaptive 3D convolution layers instead of the linear projection layer. In addition, the Conv-Permutator module utilizes the depth-wise convolution operations to separately encode the spatial-spectral representations along the height, width, and spectral dimensions, respectively. Extensive experiments on four benchmarks HSIs datasets, including Indian Pines, Pavia University, Houston2013, and Xiongan datasets, show that the superiority of the proposed HiT over existing transformers and the state-of-the-art CNN-based methods. Our codes of this work are available at https://github.com/xiachangxue/DeepHyperX for the sake of reproducibility.