High penetration of renewable energy has become a prominent feature of the development of power systems in China. The uncertainty of intermittent renewable energy generation brings great challenges to the safe and economic operation of power systems. Accurate and reliable supply and demand forecasting is the basis of the analysis, operation, and control of the power systems with renewable energy sources. However, the forecasting error is difficult to be eliminated through the traditional deterministic forecasting. Probabilistic forecasting can effectively quantify the forecasting uncertainty and provide key information to the analysis, operation, and control of power systems. This paper systematically reviews the theories, methodologies and applications of probabilistic forecasting for the power systems with renewable energy sources. Firstly, the basic concepts of probabilistic forecasting for the power systems with renewable energy sources are introduced, including forecasting objects, time scales, probabilistic forecasting forms and performance evaluation indices. Secondly, the basic theories and methodologies of probabilistic forecasting for the power systems with renewable energy sources are reviewed. Then the multi-scenario applications of probabilistic forecasting in the power systems with renewable energy sources are summarized. Finally, the problems in the probabilistic forecasting of the power systems with renewable energy sources are summarized and the development trend is prospected.