Background: During the COVID-19 pandemic, teleconsultations (TC) have been increasingly used in cancer care as an alternative to outpatient visits. We aimed to examine patient-related and cancer-specific characteristics associated with experiences with TC among patients with cancer during the COVID-19 pandemic. Material and methods: This population-based survey included patients with breast, lung, gastrointestinal, urological, and gynaecological cancers with appointments in the outpatient clinics, Department of Clinical Oncology and Palliative Care, Zealand University Hospital, Denmark in March and April 2020. Age- and sex-adjusted logistic regression analyses were used to study associations of sociodemographics, cancer and general health, anxiety, and health literacy with patients’ experiences of TC in regards to being comfortable with TC, confident that the doctor could provide information or assess symptoms/side effects and the perceived outcome of TC. Results: Of the 2119 patients with cancer receiving the electronic survey, 1160 (55%) participated. Two thirds of patients (68%) had consultations with a doctor changed to TC. Being male, aged 65–79 years, and having TC for test results were statistically significantly associated with more comfort, confidence, and perceived better outcome of TC. Having breast cancer, anxiety, low health literacy, or TC for a follow-up consultation were statistically significantly associated with less positive experiences with TC. Living alone, short education, disability pension, and comorbidity were statistically significantly associated with anxiety and low health literacy. Conclusions: Most patients reported positive experiences with TC, but in particular patients with anxiety and low health literacy, who were also the patients with fewest socioeconomic and health resources, felt less comfortable and confident with and were more likely to perceive the outcome negatively from this form of consultation. TC may be suitable for increasing integration into standard cancer care but it should be carefully planned to meet patients’ different information needs in order not to increase social inequality in cancer.