Background: This case control study aimed to investigate whether symptoms of sexual dysfunction are more common in males from infertile couples than in the general population and to explore whether symptoms of sexual dysfunction are associated to hypogonadism. Objectives: Participants were 165 subfertile men in infertile heterosexual relationships, 18–50 years of age, with sperm concentrations < 15 × 10⁶/mL. The controls were 199 men from a population-based group, matched for age. Material and methods: Logistic regression was applied in order to calculate odds ratios (ORs) for seven different symptoms of sexual dysfunction. In a multivariate model, we tested independent effects of infertility and primary as well as secondary hypogonadism. Results: Statistically significant association between subfertility and symptoms of sexual dysfunction was found for lack of ability to control ejaculation (OR 2.2, 95% CI: 1.2–4.2). For hypogonadism, statistical significance was seen both in relation to low sexual interest/desire for sex (OR 2.3, 95% CI: 1.0–5.5) and for being worried about the size or shape of the penis (OR 3.6, 95% CI: 1.3–9.5). These associations remained statistically significant in males with primary but not those with secondary hypogonadism. Discussion: Our study showed that men from infertile couples have an increased risk of symptoms of sexual dysfunction and this risk is linked to androgen deficiency. Conclusion: Assessment of reproductive hormone levels and sexual function should routinely be done in this group of males.