Objectives: To estimate the association between fetal congenital heart defects (CHDs) and measures of brain size throughout pregnancy, from the end of the first trimester to birth. Study design: The cohort consisted of all fetuses scanned in Western Denmark in 2012 and 2013. Anthropometric measures in fetuses with isolated CHDs diagnosed within 12 months after birth were compared with those in the fetuses without CHDs. Z-scores standardized to gestational age were calculated for first trimester biparietal diameter, second trimester head circumference, fetal weight, birthweight, head circumference, and placental weight. Results: We obtained data from 63 349 pregnancies and identified 295 fetuses with isolated CHDs (major n = 145; minor n = 150). The first trimester mean biparietal diameter Z-scores were not different between those with and those without CHDs. The head circumference mean Z-score difference was −0.13 (95% CI, −0.24 to −0.01; P = .03) in the second trimester and −0.22 (95% CI, −0.35 to −0.09; P < .001) at birth. Fetuses with univentricular physiology or tetralogy of Fallot showed the most pronounced compromise in cerebral size. Conclusions: Our results suggest that the brain alterations inducing an increased risk of impaired neurodevelopment in children with CHDs begin during pregnancy. Although fetuses with univentricular physiology or tetralogy of Fallot exhibited the most pronounced compromise in cerebral size, we recommend neurodevelopmental follow-up for all children with CHDs.