This study examines the association between psychological distress and two aggravating factors (childhood adversity and substance use) and two mitigating factors (social support and resilience) in a correctional sample of 943 men. Participants completed a questionnaire probing psychiatric distress using the DASS (Depression, Anxiety, and Stress Scale) and substance use behaviors using the ASSIST (Alcohol, Smoking, and Substance Involvement Screening Test). Adverse childhood experiences (ACEs) were measured incrementally and cumulatively. ACEs were reported by a majority (57.6%) of the sample. The number of ACEs and childhood emotional, physical, and sexual abuse experiences were strongly and consistently associated with prison-based psychological distress. The graded relationship found between ACE and psychological distress among incarcerated men supports the cumulative risk hypothesis. Prison-based substance use was positively associated with psychological distress, whereas resilience and, to a lesser extent, social support were negatively associated with distress. These findings can be used to inform the design of trauma-sensitive integrated interventions in correctional settings for men.