Impulsive aggression has commonly been associated with a dysfunction of the serotonin (5-HT) system: many, but not all, studies point to an inverse relationship between 5-HT and aggression. As cerebral 5-HT4 receptor (5-HT4R) binding has recently been recognized as a proxy for stable brain levels of 5-HT, we here test the hypothesis in healthy men and women that brain 5-HT levels, as indexed by cerebral 5-HT4R, are inversely correlated with trait aggression and impulsivity. Sixty-one individuals (47 men) underwent positron emission tomography scanning with the radioligand [(11)C]SB207145 for quantification of brain 5-HT4R binding. The Buss-Perry Aggression Questionnaire (BPAQ) and the Barratt Impulsiveness Scale were used for assessment of trait aggression and trait impulsivity. Among male subjects, there was a positive correlation between global 5-HT4R and BPAQ total score (P = 0.037) as well as BPAQ physical aggression (P = 0.025). No main effect of global 5-HT4R on trait aggression or impulsivity was found in the mixed gender sample, but there was evidence for sex interaction effects in the relationship between global 5-HT4R and BPAQ physical aggression. In conclusion we found that low cerebral 5-HT levels, as indexed by 5-HT4R binding were associated with high trait aggression in males, but not in females.