Parkinson’s disease (PD) is a nervous disorder that can cause motor impairment. PD severity assessment based on a series of motor movements illustrated in Unified Parkinson’s Disease Rating Scale (UPDRS) is an important part of clinical PD diagnosis. However, the current quantifying method heavily relies on human observation, which is time-consuming and subjective. Therefore, automatic severity estimation stemming from machine learning methods is receiving increasing amount of research attention. However, these advances are still limited by data availability and interpretability. In this paper, we release a large PD motor dataset of over 300 real PD patients collected under doctors’ instructions and propose a pipeline to automatically quantify the motor severity of PD in finger tapping and postural stability. These two selected movements are representative of local and global motor control, exhibiting great clinical importance. The pipeline contains three-stage: pose estimation, domain knowledge extraction, and classifification stage. The pose estimation uses deep-learning-based methods to extract 21 and 17 key points for fifinger tapping and postural stability respectively. The domain knowledge extraction stage extracts several explicit featurespre-defifined by experienced neuro-physicians. Finally, a classififier is trained to infer PD severity under MDS UPDRS. To combine deep-learning-based features from pose estimation and domain features from the expert, the pipeline achieves a better trade-off between the model effificiency and clinical interpretability. Experiments show that our method achieves a micro average f1-score of 88%, 84%, and 84%, respectively on left fifinger tapping, right fifinger tapping, and postural stability, outperforming previous methods by a large margin. In addition, involving expert knowledge in the feature extraction stage greatly improves our model’s interpretability, which is essential in automatic PD detection.