Intelligent reflecting surface (IRS) is an emerging cost-efficient technology to enhance communication performance by implementing a large number of passive reflecting elements with tunable phases in wireless systems. In this paper, we propose a general framework for the IRS-aided MIMO system designs under both single-user and multi-user setups, in which the diverse performance metrics including weighted mutual information and weighted MSE, and the realistic multiple weighted power constraint are taken into consideration. Leveraging the alternating optimization approach, the optimal IRS phase shifts are obtained in semi-closed forms. Specifically, based on the matrix-monotonic optimization theory, it is found that optimizing IRS phase shifts is essentially equivalent to tuning the eigenvalues and the corresponding eigenvectors of the MSE matrix. Then the proposed general framework is extended to a multi-user system by introducing a majorization-minimization (MM)-based method for IRS phase shift optimization. Simulation results show that our proposed optimal design brings significant enhancement on the chosen performance metric compared to the traditional MIMO systems without the IRS, and also significantly outperforms various benchmark designs in both single-user and multi-user systems.