The accuracy and hardware efficiency of a neural system depends critically on the choice of an activation function. Rectified linear unit (ReLU) is a contemporary activation function that yields high accuracy and allows the construction of efficient neural chips, but it results in a lot of dead neurons, especially at the output layer. This problem is more pronounced in case of multichannel, multiclass classification. This is due to the fact that ReLU cancels out negative values altogether, as a result of which the corresponding values cannot be successfully backpropagated. This phenomenon is referred to as the dying ReLU problem. In this article, we present a novel ‘surrogate gradient’ learning scheme in order to solve gradient vanishing and the dying ReLU problems. To the best of our knowledge, this is the first learning scheme that enables the use of ReLU for all network layers while solving the Dying ReLU problem. We also present a high-performance inference engine that uses ReLU-based actuators for all the network layers in order to achieve high hardware efficiency. The design is excellent for online learning as well, since the derivative of the activation is equal to a constant and can be implemented using low-complexity components. The proposed technique significantly outperforms various contemporary schemes for the CIFAR-10 dataset, and can successfully yield about 98.39% accuracy on MNIST dataset while using less than 159k synapses. Moreover, the proposed hardware implementation is able to perform about 218 giga operation per second (GOPS) while consuming only about 3.95 slice registers and 25.89 slice look-up tables per synapse on a low-end Virtex 6 FPGA. The system is able to operate at a clock frequency of 93.2 MHz.