Weakly supervised medical image analysis is of great significance for computer-aided diagnosis due to the difficulty in obtaining accurately labeled medical data. In this paper, we proposed a new Multi-instance Learning (MIL) framework called HybridMIL integrating CNN Convolutional Neural Networks (CNN) and Broad Learning Systems (BLS). Our HybridMIL can overcome several challenging issues over existing MIL methods based on either CNN or BLS alone: (i) Multiple levels (i.e., different resolutions) of feature information can be simultaneously extracted through a newly proposed instance-level feature enhancement (IFE) module; (ii) Global-level semantic information contained in the deep layers can be better represented under the global-level semantic enhancement (GSE) module; (iii) Hybrid feature fusion (HFF) module is newly designed to effectively fuse and align the multi-level outputs of IFE and global-level semantic information of GSE for subsequent classification and localization tasks. The proposed HybridMIL is evaluated on various public medical and MIL benchmark datasets. The results indicate that HybridMIL surpasses other recent MIL models in terms of classification and localization performance by up to 8.5% and 9.0%, respectively. Lastly, we demonstrate the highly competitive performance of HybridMIL in general MIL problems, going beyond weakly supervised medical image analysis.