With the maturity of 5G cellular communication systems and mobile edge computing (MEC), a large number of base stations (BSs) with edge-computing servers are densely deployed. There are extensive overlapping coverage areas among the BSs in which some heavy computational tasks from Internet of Things (IoT) devices can be divided and offloaded to multiple BSs via the Coordinated Multipoint (CoMP) technique for parallel processing. However, it is a challenging issue about how to make proper task offloading decisions among multiple connected BSs while satisfying delay requirements of multiple devices. To address this challenge, this paper presents an efficient multi-device and multi-BSs task offloading scheme with the goal of minimizing the delay for completing the tasks of the devices. Through conducting quantitative analysis of local delay and offloading delay, a nonlinear and non-convex delay optimization offloading problem, which is based on the theory of non-cooperative game, is formulated. We prove the existence of Nash equilibrium by analyzing the feature of the proposed offloading problem and further propose a distributed task offloading algorithm called DOLA. Finally, simulation experiments based on real-world dataset from the Melbourne CBD area of Australia are conducted to validate the efficacy of our DOLA algorithm. Comparison experiments are also carried out to demonstrate the superiority of DOLA in comparison with some existing schemes.