Cryptographic technology can effectively defend against malicious attackers to attack sensitive and private information. The core of cryptographic technology is cryptographic algorithm, and the cryptographic algorithm identification is the premise of in-depth analysis of cryptography. In the cryptanalysis of unknown cryptographic algorithm, the primary task is to identify the cryptographic algorithm used in the encryption and then carry out targeted analysis. With the rapid growth of Internet data, the increasing complexity of communication environment, and the increasing number of cryptographic algorithms, the single-layer identification scheme of cryptographic algorithm faces great challenges in terms of identification ability and stability. To solve these problems, on the basis of existing identification schemes, this paper proposes a new cluster division scheme CMSSBAM-cluster, and then proposes a multi-layer composite identification scheme of cryptographic algorithm using a composite structure. The scheme adopts the method of cluster division and single division to identify various cryptographic algorithms. Based on the idea of ensemble, the scheme uses the hybrid random forest and logistic regression (HRFLR) model for training, and conducts research on a data set consisting of 1700 ciphertext files encrypted by 17 cryptographic algorithms. In addition, two ensemble learning models, hybrid gradient boosting decision tree and logistic regression (HGBDTLR) model and hybrid k-neighbors and random forest (HKNNRF) model are used as controls to conduct controlled experiments in this paper. The experimental results show that multi-layer composite identification scheme of cryptographic algorithm based on HRFLR model has an accuracy rate close to 100% in the cluster division stage, and the identification results are higher than those of the other two models in both the cluster division and single division stages. In the last layer of cluster division, the identification accuracy of ECB and CBC encryption modes in block cryptosystem is significantly higher than that of the other two classification models by 35.2% and 36.1%. In single division, the identification accuracy is higher than HGBDTLR with a maximum of 9.8%, and higher than HKNNRF with a maximum of 7.5%. At the same time, the scheme proposed in this paper has significantly improved the identification effect compared with the single division identification accuracy of 17 cryptosystem directly and the 17 classification accuracy of 5.9% compared with random classification, which indicates that multi-layer composite identification scheme of cryptographic algorithm based on HRFLR model has significant advantages in the accuracy of identifying multiple cryptographic algorithms.