Partial Multi-Label Learning (PML) is a more practical learning paradigm, in which the labeling information is ambiguated. Most existing PML algorithms rely on assumptions to resolve ambiguity. However, these assumptions do not account for the origin of the noise labeling and therefore fail to address the impact of noise on the learner's performance at the root. In this paper, we will propose a PML method jointly granular ball-based robust feature selection and relevance fusion optimization (PML-GR). Specifically, in the first stage, we construct a granular ball to compute the core-set with weights and then design a feature importance evaluation function to assign weights to each feature in the core-set, resulting in a ranking of feature importance for the PML learner; in the second stage, based on the selected features, a fusion-based objective function is constructed to compute the label confidence by taking into account the joint effect of the global sample similarity and local label relevance. Finally, a multi-label prediction model is learned by fitting the multi-output regressor to the label confidence. The experimental results demonstrate that the proposed method achieves competitive generalization performance by effective feature selection and relevance fusion optimization, which can focus more on discriminative features and minimize the effect of noisy labels during training.