Lossless remote sensing image compression aims to reduce the storage size of images without any information loss, ensuring that the decompressed image is identical to the original one. Most existing methods focus on lossy image compression that reduce the storage cost with certain data loss. It is challenging to perform lossless compression due to the very high-resolution images, long encoding–decoding time, and low compression efficiency. In this article, we propose a lossless compression framework that compresses remote sensing images in a coarse-to-fine manner. Specifically, checkerboard segmentation is applied on each image to generate six subimages from the main diagonal and counterdiagonal of each channel to maximally preserve the detail and structural information. The subimages from the main diagonal are initially compressed by a traditional compression method, while the subimages from the counter-diagonal are compressed channel by channel using our proposed probability prediction network (P2Net) and arithmetic coding with the previously encoded subimages from both the main diagonal and counter-diagonal as prior knowledge. The proposed P2Net consists of a upsampling module, a feature enhancement module, a downsampling module, and a probability prediction module to learn the discrete probability distribution of pixels. Lossless compression is conducted with arithmetic coding on the discrete probability distribution. To the best of our knowledge, this is the first deep learning-based lossless compression framework for three-channel remote sensing images. Experiments demonstrate that our framework outperforms the state-of-the-art methods and requires about 3.4 s to compress a 1024 × 1024 × 3 image with 2.9% efficiency improvement compared to JPEG XL.