A hand gesture recognition system using superpixel tracking and radial basis function (RBF) classifier is proposed in this paper. Firstly we employed the motion detection of superpixels and unsupervised image segmentation to detect the target hand in the first few video frames. Then the hand appearance model is constructed from its surrounding superpixels. By incorporating the failure recovery and template matching in the tracking process, the target hand is tracked by the proposed adaptive superpixel based tracking algorithm. Experimental results show that the extracted motion trajectories recognized with a trained RBF classifier can achieve better performance compared to the existing state of the art methods.