Children with congenital heart defects (CHDs) have increased risk of cognitive disabilities for reasons not fully understood. Previous studies have indicated signs of disrupted fetal brain growth from mid-gestation measured with ultrasound and magnetic resonance imaging (MRI) and infants with CHDs have decreased brain volumes at birth. We measured the total and regional brain volumes of infants with and without CHDs using MRI to investigate, if certain areas of the brain are at particular risk of disrupted growth. MRI brain volumetry analyses were performed on 20 infants; 10 with- (postmenstrual age 39–54 weeks, mean 44 weeks + 5 days) and 10 without CHDs (postmenstrual age 39–52 weeks, mean 43 weeks + 5 days). In six infants with- and eight infants without CHDs grey and white matter were also differentiated. Infants with CHDs had smaller brains (48 ml smaller; 95% CI, 6.1–90; p = 0.03), cerebrums (37.8 ml smaller; 95% CI, 0.8–74.8; p = 0.04), and cerebral grey matter (25.8 ml smaller; 95% CI, 3.5–48; p = 0.03) than infants without CHD. Brain volume differences observed within weeks after birth in children with CHDs confirm that the brain impact, which increase the risk of cognitive disabilities, may begin during pregnancy.