The local features have gained prestige as the powerful descriptors, however, when handling color images, most existing descriptors fail to find an efficient strategy to make full use of channel correlation. To tackle this problem, we propose the cross-channel similarity based histograms of oriented gradients (CCS-HOG) model for color images. Different from the existing methods, our model integrates the color correlation with the structure features together in a better way. To find out the inner connection between channels, the cross-channel similarity measure is developed as a suitable approach. Experimental results for face recognition and kinship verification illustrate the the performance of CCS-HOG superior to other state-of-the-arts descriptors.