Purpose: Quality of life may predict survival. In addition to clinical variables, it may be influenced by psychological factors, some of which may be accessible for intervention. The primary objective of this study was to investigate the association of intrusive thoughts and the patients’ sense of coherence with pretreatment quality of life in patients with newly diagnosed rectal cancer. Methods: Patients were prospectively included in 16 hospitals in Sweden and Denmark. They answered an extensive questionnaire after receiving their treatment plan. Clinical data were retrieved from national quality registries for rectal cancer. Results: Of 1248 included patients, a total of 1085 were evaluable. Pretreatment global health-related and overall quality of life was lower in patients planned for palliative compared with curative treatment (median 53 vs. 80 on the EuroQoL visual analogue scale, p < 0.001 and odds ratio 0.56, 95% confidence interval 0.36–0.88, respectively). Quality of life was associated with intrusive thoughts (odds ratio 0.33, 95% confidence interval 0.24–0.45) and sense of coherence (odds ratio 0.44, 95% confidence interval 0.37–0.52) irrespective of the treatment plan. Conclusions: Pretreatment quality of life was influenced by the intent of treatment as well as by intrusive thoughts and the patients’ sense of coherence. Interventions could modify these psychological factors, and future studies should focus on initiatives to improve quality of life for this group of patients.