To achieve carbon neutrality in the power system, renewable energies (RENs), such as wind and solar power, are being rapidly installed. However, due to their strong intermittency and high uncertainty, balancing power supply and demand is becoming increasingly difficult. Time-of-use and real-time pricing (RTP) are crucial methods to ensure effective consumption of RENs, encouraging flexible resources on the demand side to work with RENs through demand-side response. Air conditioning loads represent a significant portion of urban loads, with >50% of the peak load in China. They can use the building’s thermal inertia to provide regulation services, a current area of focus for demand-response research. To adapt to the RTP mechanism in the future electricity market, this study explores demand-side optimization control technology for an ice storage system in a large-scale district cooling system (DCS). An optimal control method for an ice storage system is proposed, based on compound second-sampling reinforcement learning (RL), which can effectively manage uncertainties from users’ cooling demands and real-time market prices. First, a Markov decision process (MDP) is constructed to address the operational control issue of the DCS. Second, a model-free RL algorithm is used to solve the MDP. Third, the compound second-sampling mechanism is proposed to improve training efficiency and convergence performance by combining immediate return and temporal-difference error to overcome the issue of low learning efficiency caused by uniform random sampling in the traditional RL algorithm. Finally, the experimental results confirm the effectiveness of the proposed method.

摘要 为推动电力系统实现碳中和, 我国近年来风、光等新能源发电装机规模不断增加. 由于风、光等新能源具 有强间歇性和高度不确定性, 电力系统供需实时平衡的难度不断增大. 通过分时电价或实时电价政策, 引导需求 侧灵活负荷与新能源发电协同(即需求响应)是未来保证高比例新能源有效消纳的重要技术路径. 空调负荷是城市 用电负荷占比最高的单一类型负荷之一(在我国部分城市夏季高峰负荷中占比超过50%), 且可利用建筑热惯性提 供较好的调控能力, 是近年来需求响应领域研究的热点. 针对未来电力市场实时电价机制场景, 本文研究大型区 域供冷系统中的冰蓄冷需求响应优化控制技术, 提出了基于复合两端采样机制的强化学习方法, 实现对冰蓄冷系 统的优化调度与控制, 可以有效应对来自用户用冷需求和市场实时电价的双重不确定性. 首先, 针对实时电价下 区域供冷系统运行控制问题的特性, 构建马尔可夫决策过程. 其次, 采用非模型的强化学习算法对马尔可夫决策 过程进行求解, 并针对传统强化学习算法中均匀随机采样导致的学习效率低下的问题, 利用结合立即回报和时序 误差的复合两端采样机制, 提高控制器的训练效率和收敛性能. 最后, 基于真实系统仿真模型开展实验, 验证了本 文所提需求响应优化控制方法的有效性.