The human sense of smell is closely associated with morphological differences of the fronto-limbic system, specifically the piriform cortex and medial orbitofrontal cortex (mOFC). Still it is unclear whether cortical volume in the core olfactory areas and connected brain regions are shaped differently in individuals who suffer from lifelong olfactory deprivation relative to healthy normosmic individuals. To address this question, we examined if regional variations in gray matter volume were associated with smell ability in seventeen individuals with isolated congenital olfactory impairment (COI) matched with sixteen normosmic controls. All subjects underwent whole-brain magnetic resonance imaging, and voxel-based morphometry was used to estimate regional variations in grey matter volume. The analyses showed that relative to controls, COI subjects had significantly larger grey matter volumes in left middle frontal gyrus and right superior frontal sulcus (SFS). COI subjects with severe olfactory impairment (anosmia) had reduced grey matter volume in the left mOFC and increased volume in right piriform cortex and SFS. Within the COI group olfactory ability, measured with the “Sniffin’ Sticks” test, was positively associated with larger grey matter volume in right posterior cingulate and parahippocampal cortices whereas the opposite relationship was observed in controls. Across COI subjects and controls, better olfactory detection threshold was associated with smaller volume in right piriform cortex, while olfactory identification was negatively associated with right SFS volume. Our findings suggest that lifelong olfactory deprivation trigger changes in the cortical volume of prefrontal and limbic brain regions previously linked to olfactory memory.