Both the government and the thug on the corner will say "Give me your money." Neither really have a rightful claim to it. The reason we give our money to the government is not because we're following any Biblical command, it's ultimately just because we're more scared of the government than we are of the thug on the corner. If you tell the government "No!", then they'll come with guns, take your stuff and throw you in jail. But as soon as the government is no longer able to enforce it, then you'll see most everyone not paying anymore. The funny thing to me is that 501c3 tax exempt churches, loaded down with a bunch of debt, are always telling us "Jesus said to be sure to pay your taxes"! Hahahaha!!! Hypocrites!
But yes, I think you're right that I'm sure that the Romans were funding a bunch of bad stuff with tax money also. I don't think there really is a line, because I don't really think it's a real command. Jesus said that we're exempt from taxes but we pay "lest we offend them." In other words, "pay your taxes so you don't go to jail, because I've got work for you to do that you're not going to be able to do in jail."
I've thought about this same thing. I came to the conclusion that God wants us engaged with the world - even though it will invariably mean that we sin more. I don't think he wants us to run out to the mountains to live in isolation.