Label distribution learning (LDL), leveraging the label significance (LS), is more appropriate for solving label ambiguity problems than multi-label learning (MLL). However, directly obtaining the LS of LDL is extremely expensive and challenging. Thus, label enhancement (LE) algorithms are effectively proposed to acquire inherent LS from MLL for training the LDL models. Nevertheless, most existing LE models will suffer from low accuracy and low efficiency with following issues: i) ignoring mapping relationship between feature and label space, resulting in inaccurate enhanced data; ii) designing independently apart from LDL models, resulting in an inability of unified LE-LDL learning; iii) requiring to optimize numerous parameters iteratively, resulting in worse training efficiency. Consequently, a novel unified LE-LDL learning framework, namely Stacked Graph-regularized Polynomial-based Fuzzy Broad Learning System (SGP-FBLS), is proposed by following three innovations: 1) Polynomial-based fuzzy system is introduced to enhance feature mapping ability while improving learning performance effectively; 2) Graph regularized-based optimization objective function (GP-FBLS) is presented by considering inter-instance correlation and label correlation to mine potential LS, thereby improving the accuracy of subsequent LDL tasks; 3) A weight stacked strategy is innovatively proposed to directly transmit LS and weighted parameters from GP-FBLS to SGP-FBLS without retraining, achieving the most satisfactory performance while significantly improving training efficiency. Finally, comparative studies on 19 practical datasets demonstrate the effectiveness and superiority of proposed methods.