Image denoising is a challenging task due to possible data bias and prediction variance. Existing approaches usually suffer from high computational cost. In this work, we propose an unsupervised image denoiser, dubbed as adaptIve Dual sElf-Attention Network (IDEA-Net), to handle these challenges. IDEA-Net benefits from a generatively learned image-wise dual self-attention region where the denoising process is enforced. Besides, IDEA-Net is not only robust to possible data bias but also helpful to reduce the prediction variance by applying a simplified encoder-decoder with Poisson dropout operations on a single noisy image merely. The proposed IDEA-Net demonstrated the outperformance on four benchmark datasets compared with other single-image-based learning and non-learning image denoisers. IDEA-Net also shows an appropriate choice to remove real-world noise in low-light and noisy scenes, which in turn, contribute to more accurate dark face detection. The source code is available at https://github.com/zhemingzuo/IDEA-Net.