All sin is not equal. To lust after your neighbors wife is not as bad as actually fornicating with your neighbor's wife. Because in the latter you are destroying a family or perhaps even two families, hurting countless of people, in the first one you are hurting yourself and maybe your relationship with your wife.
James also speaks about certain sins being "unto death" and some "not unto death"
Just like in our penal code, murder is worse than incorrectly parking your car. Thats just reality.