Based on the ever-growing emphasis on high security of biometrics recognition, finger vein recognition has captured more and more attention. However, due to the light scattering in human skin tissue during near-infrared light transmission imaging, the collected finger vein images are always degraded dramatically, which leads to the unreliability of vein features and the low accuracy of finger vein recognition. Although considerable traditional methods are dedicated to eliminating the effect of light scattering on imaging, the clearer images cannot be output end-to-end, the processes of restoring degraded finger vein images are laborious as well. Thereupon, with the aim at improving the visibility of finger vein features and generating clear finger vein images end-to-end, this effort represents a simple and effective method utilizing Convolutional Neural Network (CNN). First, in our previous work, the biological optical model used to settle the matter of skin scattering is modified to output restored finger vein images in an end-to-end manner. Second, a multi-scale CNN named E-Net is established to acquire credible estimation map of finger vein features, which is conducive to the acquisition of pleasurable restoration outcome. Finally, a scattering removal framework, addressed as Finger Vein Image Scattering Removal Network (FVSR-Net), is designed via integrating improved biological optical model with E-Net. Such a novel design facilitates the generation of clearer venous regions and increases computational efficiency and stability. Experiments accomplished on two finger vein datasets demonstrate the superiority of our proposed method in terms of visual quality and recognition performance.