Adam did not impute his sin to us. God did! Evidently, according to Rom 5:12ff. God in eternity decreed that Adam would be the federal head of the human race. As such, all Adam's progeny participated in Adam's sin -- again, by God's eternal decree.
I quoted Rom 5:15, 18-19 to PT recently...