In this paper, we show that the GPU (graphics processing unit) can be used not only for processing graphics, but also for high speed computing. We provide a comparison between the times taken on the CPU and GPU to perform the training and testing of a back-propagation artificial neural network. We implemented two neural networks for recognizing handwritten digits; one consists of serial code executed on the CPU, while the other is a GPU-based version of the same system which executes in parallel. As an experiment for performance evaluation, a system for neural network training on the GPU is developed to reduce training time. The programming environment that the system is based on is CUDA which stands for compute unified device architecture, which allows a programmer to write code that will run on an NVIDIA GPU card. Our results over an experiment of digital image recognition using neural network confirm the speed-up advantages by tapping on the resources of GPU. Our proposed model has an advantage of simplicity, while it shows on par performance with the state-of-the-arts algorithms.