Noise estimation is crucial in many image processing tasks such as denoising. Most of the existing noise estimation methods are specially developed for grayscale images. For color images, these methods simply handle each color channel independently, without considering the correlation across channels. Moreover, these methods often assume a globally fixed noise model throughout the entire image, neglecting the adaptiveness to the local structures. In this work, we propose a locally adaptive multivariate Gaussian approach to model the noise in color images, in which both the content-dependence and inter-dependence among color channels are explicitly considered. We design an effective method for estimating the noise covariance matrices. Specifically, by exploiting the image self-similarity property, we could estimate a distinct noise covariance matrix for each local region via a linear shrinkage estimator. Experimental results show that our method can effectively estimate the noise covariance matrices. The usefulness is demonstrated with real color image denoising.