There is a pressing need to identify markers of cognitive and neural decline in healthy late-midlife participants. We explored the relationship between cross-sectional structural brain-imaging derived phenotypes (IDPs) and cognitive ability, demographic, health and lifestyle factors (non-IDPs). Participants were recruited from the 1953 Danish Male Birth Cohort (N=193). Applying an extreme group design, members were selected in 2 groups based on cognitive change between IQ at age ~20y (IQ-20) and age ~57y (IQ-57). Subjects showing the highest (n=95) and lowest (n=98) change were selected (at age ~57) for assessments on multiple IDPs and non-IDPs. We investigated the relationship between 453 IDPs and 70 non-IDPs through pairwise correlation and multivariate canonical correlation analysis (CCA) models. Significant pairwise associations included positive associations between IQ-20 and gray-matter volume of the temporal pole. CCA identified a richer pattern - a single "positive-negative" mode of population co-variation coupling individual cross-subject variations in IDPs to an extensive range of non-IDP measures (r = 0.75, Pcorrected < 0.01). Specifically, this mode linked higher cognitive performance, positive early-life social factors, and mental health to a larger brain volume of several brain structures, overall volume, and microstructural properties of some white matter tracts. Interestingly, both statistical models identified IQ-20 and gray-matter volume of the temporal pole as important contributors to the inter-individual variation observed. The converging patterns provide novel insight into the importance of early adulthood intelligence as a significant marker of late-midlife neural decline and motivates additional study.