The previous matrix regression based methods mainly focus on designing a robust error term to characterize the occlusion and illumination changes. In actually, it is very challenging to give a strong model for solving the original images directly since the images contains rich and complex structure information. To address this problem, we aim to simplify the complex images and propose a simple and robust matrix regression based classification model. In our method, we firstly employ the local gradient distribution to decompose the image into a series of gradient images (LID for short). Each gradient image reveals the local structure information in different gradient orientations. Subsequently, we consider each gradient image as the diagonal block element and construct the diagonal block matrix for image representation. Nuclear norm based matrix regression model (NMR) is then applied to complete the classification tasks. The proposed model can be called ID-NMR for short. We further design a fast ADMM optimization algorithm to solve the proposed ID-NMR due to the fact that the big diagonal block matrix will increase the computational load. Experimental results show that the proposed method performs favorably compared with state-of-the-art regression based classification methods.