Unless you are totally misinterpreting what James said. James cannot be in conflict with Paul, since both were writing under the inspiration of the Holy Spirit. James was focused on the EVIDENCE or PROOF of justification by grace through faith and that is perfectly clear from what we read. If you claim you have saving faith and there is no outward evidence, then that was not saving faith.
However, Paul established the doctrine of justification in the sight of God, and by the direct action of God in response to faith. Read Romans 4. Then read Genesis 15 and AFTER that read Genesis 22.
And He [God] brought him [Abram] forth abroad, and said, Look now toward heaven, and tell the stars, if thou be able to number them: and He [God] said unto him [Abram], So shall thy seed be. And he [Abram] believed in the LORD; and he counted it to him for righteousness. (Gen 15:5,6)
What shall we say then that Abraham our father, as pertaining to the flesh, hath found? For if Abraham were justified by works, he hath whereof to glory; but not before God. For what saith the scripture? Abraham believed God, and it was counted unto him for righteousness. Now to him that worketh is the reward not reckoned of grace, but of debt. But to him that worketh not, but believeth on him that justifieth the ungodly, his faith is counted for righteousness. (Rom 4:1-5)
NOTICE:
1. Paul -- by divine inspiration -- refers to Abram as "Abraham".
2. Paul -- by divine inspiration -- clarifies "believed in the LORD" and states "believed God".
3. Paul categorically rules out works as the basis for justification.
4. Therefore the only proper conclusion is that it is faith alone that justifies the one who believes. That is justification in the sight of God by His grace alone.
5. Since James would never contradict Paul, he refers to justification in the sight of men. And he refers to what happened in Genesis 22, long after Genesis 15.
So now you do have to worry my friend. Since you are denying Gospel truth, and God is definitely not pleased with that.