Integrating the energy-harvesting (EH) relay and non-orthogonal multiple access (NOMA) technologies into narrow band internet of things (NB-IoT) networks can efficiently improve the energy and spectrum efficiency of the network and the quality-of-service of edge users. Therefore, we consider an EH relay aided NOMA NB-IoT network in this letter. To reduce the rate variance among NB-IoT devices, we aim to maximize the proportional fairness of data rate across all NB-IoT devices through jointly optimizing the communication resource allocation and successive interference cancellation (SIC) ordering subject to the minimum data rate requirements. Considering the non-convexity of this optimization problem, we propose a deep reinforcement learning based online optimization algorithm to obtain the sub-optimal solution. Simulation results demonstrate that the proposed algorithm can efficiently improve the proportional fairness and the total throughput among NB-IoT devices, in comparison with orthogonal multiple access techniques.