Twin-studies of social responsiveness have reported moderate to high heritabilities, but studies using parent–child data are lacking. Additionally, social impairments have been suggested as a vulnerability marker for schizophrenia and bipolar disorder, but the heritability of social responsiveness in this context is unknown. This study is part of the Danish High Risk and Resilience Study – VIA, comprising families with one parent with schizophrenia (n = 202) or bipolar disorder (n = 120) and population-based controls (PBC, n = 200). Social responsiveness was assessed with The Social Responsiveness Scale, Second Edition (SRS-2). Heritability was estimated from variance components, and a polygenic risk score (PRS) for autism spectrum disorder (ASD) was calculated to assess the genetic relationship between ASD and SRS-2. SRS-2 heritability was moderate to high and significantly different from zero in all groups when the children were rated by the primary caregiver. With teacher ratings, the heritability was lower and only significant in the full cohort and PBC. We found no significant association between SRS-2 and PRS for ASD. Our study confirms that social responsiveness is heritable, but that heritability estimates are affected by the child-respondent relation and familial risk of mental illness. This has implications for clinical practice and research using SRS-2 and provides insight on the familial transmission of mental illness.