Background: Simple Congenital Heart Defects such as septal defects constitute a large proportion of Congenital Heart Defects. New research has demonstrated more co-morbidities than previously thought. In particular, co-morbidities involving neurocognitive, psychiatric, and social difficulties have been described. Neurocognitive and psychiatric morbidities affect social interaction. Social interaction is important in everyday social life (education, work life, family life). In this study, we investigated social interaction through self- and proxy-answered Social Responsiveness Scale 2 (SRS-2) in young adults with simple Congenital Heart Defects and compared their social interaction profile to healthy matched controls. Methods: We included a total of 80 patients with either atrial or ventricular septal defect (age 26.6 years) and 38 heart-healthy, age, sex, and ISCED educational matched controls (age: 25.3 years). A close relative proxy from each participant took part in the study as well. All participants answered the Social Responsiveness Scale 2 (SRS-2) (n = 225). Our primary and secondary outcomes were the SRS-2 Total score and the SRS-2 sub-scores. Results: In the Congenital Heart Defects group, 31.3% had a Total score above 60 compared to 7.9% in the control group (p = 0.005, RR = 3.96). The participants with a septal defect had a higher Total score (52.5 vs. 45.5, p = 0.004), a higher Social Cognition sub-score (55.0 vs. 47.0, p = 0.0004), and a higher Social Motivation sub-score (50.0 vs. 45.0, p = 0.003) than the heart-healthy participants. We found no difference between the two groups regarding the sub-scores of Social Awareness and Social Communication. A multiple linear regression model showed that the variable that explained most of the variation in Total Score was having a previously diagnosed psychiatric disorder. Conclusion: We found that young adults with atrial or ventricular septal defects have a fourfold increased risk of social interaction difficulties compared to heart-healthy peers. They have a social interaction profile, with difficulties in social cognition and social motivation, and preserved social awareness and social communication. Psychiatric morbidity explained most of the variation in social interaction problems. As social difficulties and psychiatric morbidities are intertwined, social interaction difficulties could be an indication of already underlying psychiatric morbidities or a risk factor for future psychiatric morbidity.