Oh it doesn't scare me, it can't... Solipsism is impossible, unless you don't believe in God then everything is possible and impossible at the same time (It's a huge mess)
See here is the thing, you have no clue that you are actually living, you have no reason to say there is right or wrong... Lets play this out logically...
Let's say I am born with a different chemical balance in my brain and I like to hurt people, and any time I hurt someone it makes me happy. Is it right or wrong that I do this?
Your only answer can be, it's wrong because I don't think you should hurt people, and the question is, how do we know evolution gave you the correct brain?
Or you answer, it's wrong because majority of people don't like to hurt people, and then the question is, how do you know I am not more evolved then everyone else?
You see if I only act the way I do because of the chemicals put into my brain by chance through natural selection, anything I do is not my fault, its evolution's fault for creating me this way and I have no control over what my brain does, only the illusion of free will.
With your pragmatic mind you have to understand that in your philosophy if I was to rape a 2 year old child, it is only because of the chemical reactions in my brain, nothing more, nothing less... You can try to attribute morality to the situation, however, when doing so you are only dancing to the chemicals in your brain. In your philosophy there is no morality, only appeal to majority, or appeal to self. In both cases you get logical fallacy's.
I digress but if you follow the argument further, the theory of no free will is a logical fallacy.