In this study, a fusion convolutional neural network with random term (FCNN-R) model is proposed for driver injury severity analysis. The proposed model consists of a set of sub-neural networks (sub-NNs) and a multi-layer convolutional neural network (CNN). More specifically, the sub-NN structure is designed to deal with categorical variables in crash records; multi-layer CNN structure captures the potential nonlinear relationship between impact factors and driver injury severity outcomes. Seven-year (2010–2016) single-vehicle crash data is applied. Models with different CNN layers are tested using the validation set, as well as various model layouts with and without a dropout layer or regularization term in the objective function. It is found that different model layouts provide consistent predictive performance. With the limited training data, more CNN layers result in the prematurity of the training procedure. The dropout layer and the regularization technique help improve the stability of the effects of different variables. The proposed model outperformed other five typical approaches in the predictability comparison. Moreover, a marginal effect analysis was conducted to the proposed FCNN-R model, the FCNN model and the mixed multinomial logit model. It shows that the proposed FCNN-R model can be used to uncover the underlying correlations similar to the traditional statistical models. Additionally, the temporal stability of the proposed FCNN-R approach is discussed based on the model performance in different years. Future research is recommended to include more information for improving the universality of the proposed approach.