Remote sensing technique is beneficial for rainfall data retrievals, however, enhancing the accuracy remains a challenge. In this study, a novel framework based on a broad learning system (BLS) was proposed to assimilate multi-source data. The dataset includes six satellite-based rainfall products (3B42V7, 3B42RT, IMERG, CBLD, GSMaP, and PCDR), gauge-based rainfall, and environmental data (temperature, specific humidity, wind speed, and locations) from 1 March 2014 to 31 December 2017 over southeast China (SEC). Leave-one-year-out cross-validation (LOYOCV) and independent validation were used to evaluate the BLS assimilating model. The proposed BLS model outperformed six original satellite-based products on Pearson's correlation coefficient (CC), root-mean-square error (RMSE), and Nash-Sutcliffe coefficient of efficiency (NSE) in each test year of LOYOCV. BLS model considering the environmental factors performed better on CC, RMSE, and NSE compared to that without environmental factors. Seasonal variations of daily gauge-based precipitation were accurately captured by BLS-based estimates. BLS method outperformed satellites on CC, RMSE, and NSE at most validation sites at low altitudes (0–1000 m). According to the independent validation, more accurate daily precipitation estimates could be obtained at more than half of the validation sites using the proposed model compared to the source datasets. The BLS-based framework considering environmental factors has the potential to improve estimates over SEC and is expected to be applied to other regions.