Timber has been widely utilized as a type of green material in the construction industry. However, the anisotropic and highly heterogeneous nature of timber increases the difficulty of damage identification, which is critical for maintaining structures in which it is used. In this paper, we propose a timber damage identification dynamic broad network, namely TimberNet, that can quickly realize damage identification via a one-shot calculation. Ultrasonic signals are fed into the dynamic network to automatically extract features for damage identification, avoiding excessive artificial involvement in feature selection. Furthermore, the proposed method allows incremental updating of the damage detection model and greatly reduces the updating time and computational cost. Comparison studies with some well-known algorithms demonstrated that the damage identification accuracy of TimberNet is about 30% higher than that of the Naïve Bayes classifier. Moreover, its training efficiency and inference speed are 12 times and 2.1 times greater than those of the one-dimensional convolutional neural network (1DCNN), respectively. Finally, a series of validation experiments indicates the robustness of the proposed method in timber damage identification.