5G radio access network (RAN) is consuming much more energy than legacy RAN due to the denser deployments of gNodeBs (gNBs) and higher single-gNB power consumption. In an effort to achieve an energy-conserving RAN, this paper develops a dynamic on-off switching paradigm, where the ON/OFF states of gNBs can be dynamically configured according to the evolvements of the associated users. We formulate the dynamic sleep control for a cluster of gNBs as a Markov decision process (MDP) and analyze various switching policies to reduce the energy expenditure. The optimal policy of the MDP that minimizes the energy expenditure can be derived from dynamic programming, but the computation is expensive. To circumvent this issue, this paper puts forth a greedy policy and an index policy for gNB sleep control. When there is no constraint on the number of gNBs that can be turned off, we prove the dual-threshold structure of the greedy policy and analyze its connections with the optimal policy. Inspired by the dual-threshold structure and Whittle index, we develop an index policy by decoupling the original MDP into multiple one-dimensional MDPs – the indexability of the decoupled MDP is proven and an algorithm to compute the index is proposed. Extensive simulation results verify that the index policy exhibits close-to-optimal performance in terms of the energy expenditure of the gNB cluster. As far as the computational complexity is concerned, the index policy is much more efficient than the optimal policy, which is computationally prohibitive when the number of gNBs is large.