There are three primary challenges in the automatic diagnosis of arrhythmias by electrocardiogram (ECG): the significant variation among individual patients, the multiple pathologies in the ECG signal and the high cost in annotating clinical ECG with the corresponding labels. Traditional ECG processing approaches rely heavily on prior knowledge, such as those from feature extraction and waveform analysis. The preprocessing for prior knowledge incurs computational overhead. Furthermore, standard deep learning methods do not fully consider the dynamic temporal, spatial and multi-labeling characteristics of ECG data. In clinical ECG waveforms, it is common to see multi-labeling in which a patient is labeled with multiple classes of arrhythmias. However, multiclass approaches in current research mainly solve the multi-label machine learning problem, ignoring the correlation between diseases, resulting in information loss. In this paper, an arrhythmia detection and classification scheme called multi-label fusion deep learning is proposed. The objective is to build a unified system with automatic feature learning which supports effective multi-label classification. First, a multi-label ECG-based feature selection method is combined with a matrix decomposition and sparse learning theory. The optimal feature subset is selected as a preprocessing algorithm for ECG data. A multi-label classifier is then constructed by fusing CNN and RNN networks to fully exploit the interactions and features of the time and space dimensions. The experimental result demonstrates that the proposed method can achieve a state-of-the-art performance compared to other algorithms in multi-label database experiments.