This article extends Fama and French (FF) models of observed factors by introducing latent factors (LFs) to further extract information from FF residual returns. A diagonally dominant (DD) rather than a diagonal or sparse matrix structure is adopted in this study to estimate remaining covariance between disturbance terms. Such an enhanced factor (EF) model provides a more comprehensive analysis for portfolio selection in high dimensions and also has certain advantages of estimation stability and computational efficiency. It is shown that the proposed EF-DD approach achieves overall better performance than competing models in terms of portfolio variance and the net Sharpe ratio.