In current deep learning-based infrared and visible image fusion algorithms, the image processing step involves converting the RGB channels of visible image into luminance channels. While these approaches concentrate on the texture details of visible image, which tends to neglect the color information, which causes the fused image have the problem of color deviations which do not match human vision. Color information, a crucial role in human visual perception, is one of the most intuitive evaluation metrics for image fusion. In order to restore the color of fused images, researchers have made many attempts, such as enhancing brightness or contrast. but the fusion results are not satisfied. For tackle the problem of low color fidelity, Dif-Fusion was proposed. This approach establishes a multi-channel data distribution, directly generating fused images with RGB channels, resulting in exceptional fusion effects with high color fidelity. However, Information input mechanisms of Dif-Fusion may lead to an imbalance in the network's extraction of infrared and visible image features. In addition, there is still room for improvement in preserving texture and thermal radiation information. Consequently, we propose an enhanced algorithm based on diffusion and generative adversarial networks, named DGFusion. Firstly, we change the Information input mechanism to balance the weights of infrared image features and visible image, which can enhance the expression of infrared information. Meanwhile, an improved diffusion structure is applied to obtain deep-level features, which utilize UNet++ to reduce the loss of multi-scale feature information phenomenon during the propagation process in deep networks. Furthermore, we establishing the adversarial game between the fusion network and the discriminator, which can force the fusion network to achieve superior texture detail preservation. We conducted comparative experiments and ablation studies, which shows that the DGFusion yields superior fusion results. Ablation experiments show that DGFusion improves on most metrics compared to the unmodified method, validating the effectiveness of our approach. Comparison experiments show that our method outperforms several state-of-the-art fusion methods in terms of metrics and visual effects.