Human action understanding is one of the most attractive research areas in computer vision. In this paper, we focus on a subset of human action which is the gesture performed by hand motion. To track the trajectory of motion, we adopt efficient and robust object detection and tracking schemes, which used Randomized Forest and Online Appearance model. Multiple hand templates are leaned using positive and negative samples (P-N learning). According to robust hand tracking and trajectory enhancement, we recognize the gesture with the baseline SVM tool. The effectiveness of the approach is demonstrated by experiments on the dataset of hand signed digit gestures.