Millimeter wave (mmWave) communication is one of the key technologies in 5G and beyond systems to address the tremendous growth in mobile data traffic owing to the abundant spectrum resources. Ultra-dense network deployment is a promising solution to combat the limited coverage, high propagation loss and attenuation of mmWave signals. This study investigates the beam management, with focus on beam configuration of mmWave base stations, in the ultra-dense mmWave network. To fulfill adaptive and intelligent beam management while protecting user privacy, we employ a double deep Q-network under a federated learning to tackle the beam management problem which is formulated to maximize the long-term system throughput. Simulation results demonstrate the performance gain of our proposed scheme.