The identification of clinically relevant neuroimaging biomarkers in psychiatry is a research priority. Neuropsychological tasks and functional MRI (fMRI) are used, via FDA-approved assessments, in clinical decision-making in many neurology departments. However, currently, psychiatry lacks neuro-psychological/fMRI biomarkers that could help in diagnosis and treatment planning. In our opinion, this likely reflects task design choices commonly used with psychiatric patients that limit test re-test reliability (TRR). Clinical decision-making can only occur via tests with excellent TRR. Statistical analyses indicate that TRR is particularly compromised if: (1) there are relatively few trials per condition; and (2) contrast-based analyses are adopted. We suggest, on the basis of the simulation work, that machine learning techniques combined with increasing the number of trials (per condition) and limiting the reliance on contrast-based analyses, can increase TRR and thus allow the successful development of cognitive neuroscience-based biomarkers for psychiatry in the near future.