As an essential part of motion behavior modeling, pedestrian trajectory forecasting with social interactions has become an increasingly important problem in many applications, such as visual navigation and intelligent video surveillance. Most existing methods adopt autoregressive frameworks to forecast the future trajectory, where the trajectory is iteratively generated based on the preceding outputs. Such a process suffers from large accumulated errors over long-term forecasting. To address this issue, in this work, we propose a synchronous bi-directional framework for pedestrian trajectory prediction, where the predicting procedures for two opposite directions are forced to be synchronous through a shared motion characteristic. Unlike previous works, the mutual constraints inherent to our framework from the synchronous opposite predictions can significantly prevent error accumulation. In addition, we devise a temporally dependent interaction model to learn the complex social interactions among pedestrians from correlated historical trajectories. By resorting to a temporally dependent attention scheme and a progressive temporal fusion method, our interaction model can effectively reveal the interacting influence among pedestrians across temporal domains, and also capture the long-term dependencies of the historical trajectory. Experiments conducted on the ETH-UCY benchmark and the Stanford Drone dataset show that our method achieves much better results than existing algorithms. Particularly, our scheme exhibits superior performance in long-term pedestrian trajectory prediction.