Background: Although psychological distress is common among patients with chronic diseases, the degree of risk for developing psychological distress is not well-established. Our aim with this study is to determine the odds ratio for psychological distress in patients with cancer as compared to either 1) patients with chronic disease patients without cancer, or 2) healthy controls in a large representative sample of Chinese population. Methods: Using a multistage, stratified cluster sampling method, 21,101 subjects 18–79 years old were interviewed face-to-face in Jilin province, China. Their psychological status was assessed with the 12-item General Health Questionnaire (GHQ-12). A total score of ≥4 was used as the threshold for determining psychological distress. Results: The prevalence of psychological distress was 14.08% across the entire sample: 10.63% in healthy controls; 14.81% in patients with chronic diseases; and 20% in patients with cancer. Multiple logistic regression analysis indicated that, as compared to the health controls, both cancer (OR = 1.609, 95%CI = 1.245–2.081) and chronic disease patients (OR = 1.330, 95%CI = 1.189–1.478) were more likely to suffer from psychological distress. Moreover, cancer patients had a higher risk of psychological distress than patients with chronic diseases (OR = 1.295, 95%CI = 1.049–1.600; p = 0.016). In addition, the cancer group (2.68 ± 0.114) demonstrated a higher GHQ score than the patients with chronic diseases (2.30 ± 0.017) and healthy controls (1.98 ± 0.029) (both p < 0.001). Conclusion: Our results showed both a higher rate and greater degree of psychological distress in patients with cancer. This suggests a need for better psychological management in cancer patients to help alleviate their distress.