Partial label learning (PLL) is a weakly supervised methodology dealing with tasks that have annotation problems by replacing the single label with a collection of candidate labels. Compared to single labels, utilizing partial labels faces challenges: (1) The limited supervision and sensitivity to the false positive candidates; (2) Situations where the ground truth is not in the candidate label sets (noisy PLs). However, in the case that there exists a subset of samples that can be easily labeled (referred to as clean samples), the existing PLL paradigm needlessly assigns these instances with partial labels randomly. To better utilize the clean samples, and alleviate the obstacles of adopting partial labels, we proposed a specific Partial Label Knowledge Distillation (PLKD) framework to distill the knowledge from the samples with low annotating cost, further guiding partial label samples with limited supervision in these scenarios. The teacher model of PLKD was pre-trained on the clean samples with a single label, which can reduce the effect of noisy PLs when training on the remaining PLL samples. Additionally, recognizing that the existing candidate labels are sampled under the uniform distribution, which may not reflect real-life scenarios, we also proposed a label-specific candidate generation method. Correspondingly, a new loss function based on our generation method is presented to evaluate the distinction between partial labels and predictions. Furthermore, we also present a partial-label guided version, denoted as PLKD-pl, to alleviate the teacher's risk of over-confidence when the distribution between the clean set and partial label set varies widely. Extensive experimental evaluations have been conducted to demonstrate the superiority of PLKD over six state-of-the-art counterparts.