Aquaponics is an emerging area of agricultural sciences that combines aquaculture and hydroponics in a symbiotic way to increase crop production. Though it offers a lot of advantages over traditional techniques, including chemical-free and soil-less farming, its commercial application suffers from some problems such as the lack of experienced manpower. To operate a stable smart aquaponic system, it is critical to estimate the fish size properly. In this context, the use of dedicated hardware for real-time aquaponic monitoring can greatly resolve the issue of inexperienced handlers. In this article, we present a complete methodology to train a deep neural network to perform fish size estimation in real time. To achieve high accuracy, a novel implementation of swish function is presented. This novel version is far more hardware efficient than the original one, while being extremely accurate. Moreover, we present a deep learning accelerator that can classify 40 million fish samples in a second. The dedicated real-time system is about 1600 times faster than the one based on general-purpose computers. The proposed neuromorphic accelerator consumes about 2600 slice registers on a low-end model of Virtex 6 FPGA series.