Visual deprivation from birth leads to reorganisation of the brain through cross-modal plasticity. Although there is a general agreement that the primary afferent visual pathways are altered in congenitally blind individuals, our knowledge about microstructural changes within the higher-order visual streams, and how this is affected by onset of blindness, remains scant. We used diffusion tensor imaging and tractography to investigate microstructural features in the dorsal (superior longitudinal fasciculus) and ventral (inferior longitudinal and inferior fronto-occipital fasciculi) visual pathways in 12 congenitally blind, 15 late blind and 15 normal sighted controls. We also studied six prematurely born individuals with normal vision to control for the effects of prematurity on brain connectivity. Our data revealed a reduction in fractional anisotropy in the ventral but not the dorsal visual stream for both congenitally and late blind individuals. Prematurely born individuals, with normal vision, did not differ from normal sighted controls, born at term. Our data suggest that although the visual streams are structurally developing without normal visual input from the eyes, blindness selectively affects the microstructure of the ventral visual stream regardless of the time of onset. We suggest that the decreased fractional anisotropy of the ventral stream in the two groups of blind subjects is the combined result of both degenerative and cross-modal compensatory processes, affecting normal white matter development.