Accurate extraction of three-dimensional (3D) vegetation is essential for monitoring urban ecological environments and carbon sinks. Two-dimensional vegetation data in cities has been widely researched. However, large-scale urban vegetation height inventories are lacking. This study proposes a novel framework for 3D extraction of urban vegetation, which can be widely applied based on remote sensing approaches. A multi-task convolutional neural network is established to extract the urban vegetation cover and estimate the vegetation height at the pixel level. The results indicate that this method can derive the complete urban vegetation cover and height from stereo satellite data. Compared with the traditional stereo-photogrammetry method, this method enables rapid inference of vegetation height in urban areas with a root mean square error (RMSE) of 3.16 m. This model is capable of accurately separating vegetation in complex urban environments and performs well despite shadow effects. Furthermore, in this study, the first vegetation height map with 1-m spatial resolution has been produced, covering six urban districts in Beijing (approximately 1,378 km). It only takes 2–3 min to process the imagery of the whole study area. The high-resolution map can display more urban vegetation details over the existing 10 m/30 m resolution vegetation height maps. Furthermore, the established framework and benchmark for urban vegetation 3D information offer unique insights and provide a basis for further research.