David and Bathsheba's baby died, but David said by the Holy Spirit that "he would go to him, but he will never return to me." David had knowledge that the unfortunate infant that died was in Heaven.The Bible does not specifically answer that question, but a baby is incapable understanding the gospel. It is my belief that God is righteous and merciful and will make an exception for babies or people who can't mentally understand.
Jesus said this in John 9:41:
41Jesus said unto them, If ye were blind, ye should have no sin: but now ye say, We see; therefore your sin remaineth.
Maybe this is a clue to answering your question.
John the Baptist recognized the presence of Jesus while both were still being carried in the wombs of their mothers.