Kernel ridge regression (KRR) is a kind of efficient supervised algorithm for multi-class analysis. However, limited by the implicit kernel space, current KRR methods have weak abilities to deal with redundant features and hidden local structures. Thus, they may get indifferent results when applied to analyze the data with complicated components. To overcome this weakness and obtain better multi-class regression performance, we propose a new method named low-rank kernel regression with preserved locality (RLRKRR). In this method, data are mapped into an explicit feature space by using the random Fourier feature technique to discover the non-linear relationship between data samples. In addition, during the training of the regression coefficient matrix, the low-rank components of this explicit feature space are simultaneously extracted for reducing the effect of the redundancy. Moreover, the graph regularization is performed on the extracted low-rank components to preserve local structures. Furthermore, the l norm is imposed on the regression error term for relieving the impact of outliers. Based on these strategies, RLRKRR is capable to achieve rewarding results in complicated multi-class data analysis. In the comprehensive experiments conducted on various types of datasets, RLRKRR outperforms several state-of-the-art regression methods in terms of classification accuracy (CA).