In order to calibrate camera with radial distortion model, a novel approach based on the hybrid neural network with rotational weight matrix and self-adaptive genetic-annealing algorithm is proposed. Firstly two sorts of neural networks are structured, whose weights are corresponding to the camera's extrinsic parameters and intrinsic parameters without and with radial distortion, so the structured neural networks coincide with the camera's pin hole model and radial distortion model respectively. And the performance index is obtained from the square of 2-norm of the difference between the vector consisted of network's outputs and the tested retinal coordinates of corresponding feature points projected in image planes. At the same time, a genetic-annealing algorithm is introduced into the solving-programming, where the probabilities of crossover and mutation are tuned according to the distance density of individuals, and unequal probability matching strategy is adopted. Thus while the system come to the equilibrium, the camera's parameters with radial distortion are obtained in the light of network's weights. The experimental results illustrate that the proposed approach is robust, and has the advantages over existing algorithms in calibration precision, and orthogonality of rotational matrix, in particular the precision of intrinsic and extrinsic parameters of camera, which provides a practical scheme for calibrating camera with radial distortion model.