Smartphone as a portable device can be used for public participatory structural seismic response acquisition. This data acquisition approach not only reduces the acquisition cost but also facilitates the possibility to obtain city-scale building seismic responses. In this paper, we propose a novel structural seismic response classification method using time–frequency fusion features-based incremental network, namely SeismicNet, that can automatically classify the signals acquired through smartphones into structural response under normal operation conditions (structural normal response) or structural response under strong ground motion (structural seismic response). In this method, the time-domain signals of structural response are converted into time–frequency maps, which will be fed into a lightweight convolutional network for the extraction of the initial fusion features. Subsequently, these features are mapped by random weights to dynamic nodes. This not only improves the capability of feature expression, but also allows incremental updating of the model. Finally, a simulated structural response dataset and some real structural seismic responses are used to verify the accuracy and practicability of the proposed method. Furthermore, the results showed that the proposed method is more than 84 times in training speed and 2 times in inference speed than some well-known deep learning networks (VGG16 and ResNet50).