I see the offerings brought by Cain and Abel in Gen 4:3-4 as offerings in which God gave both Cain and Abel instruction -
Genesis 4:
3 And in process of time it came to pass, that Cain brought of the fruit of the ground an offering unto the LORD.
4 And Abel, he also brought of the firstlings of his flock and of the fat thereof. And the LORD had respect unto Abel and to his offering
5 But unto Cain and to his offering he had not respect. And Cain was very wroth, and his countenance fell.
The reason I believe God gave instruction to Cain and Abel about these offerings is because:
The offerings were made in process of time (Gen 4:3) which means "at the proper time". In order for there to be a "proper time", there had to be instruction as to what was the "proper time" to bring the offering. I do not believe these are sin offerings.
We know that Abel brought his offering through faith (Heb 11:4). Because faith comes by hearing and hearing by the Word of God (Rom 10:17), Abel must have heard the Word of God. Both Cain and Abel heard the Word of God, but only Abel's hearing resulted in faith. Cain rejected God's Word and had no faith. His offering was made through his own vanity.
When Cain saw that God had respect to Abel and to his offering, and God did not have respect to Cain and his offering, Cain became very wroth and his countenance fell (Gen 4:5). Cain thought he had followed the instruction God had given to him. However, we know he did not because if he had, God would have had respect to Cain and to his offering.
God knew that Cain was angry and tried to get Cain to repent. God told Cain that he would be accepted if he did well. In Gen 4:7, I believe the word "sin" actually should be "sin offering". The Hebrew word is chatta'ath and many times the word is translated "sin offering" (see Ex 29:14, 29:36, 30:10; Lev 4:3 (2nd usage), 4:8, 4:20, 4:21, 4:24, 4:25, and many more times).
Genesis 4:7 If thou doest well, shalt thou not be accepted? and if thou doest not well, chatta'ath lieth at the door. And unto thee shall be his desire, and thou shalt rule over him.
I believe the meaning of the verse is God is giving Cain instruction that there is a sin offering Cain can make. The sin offering is crouching at the door and Cain has dominion over the sin offering.
Cain became very angry over this and rather than humble himself before God, acknowledge his error, and repent, Cain focused his anger and bitterness on his brother, Abel, and killed him.
1 John 4:20 If a man say, I love God, and hateth his brother, he is a liar: for he that loveth not his brother whom he hath seen, how can he love God whom he hath not seen?
21 And this commandment have we from him, That he who loveth God love his brother also.
1 Corinthians 13:1 If I could speak all the languages of earth and of angels, but didn’t love others, I would only be a noisy gong or a clanging cymbal. 2 If I had the gift of prophecy, and if I understood all of God’s secret plans and possessed all knowledge, and if I had such faith that I could move mountains, but didn’t love others, I would be nothing.3 If I gave everything I have to the poor and even sacrificed my body, I could boast about it; but if I didn’t love others, I would have gained nothing.
Cain killed his brother. He did not love his brother. Out of these 2 brothers we can see one was righteous and had faith. If we have faith we therefore have love. Without love our faith means nothing. God said Abels offering was given with faith. So it must be true. Cain on the other hand was rejected. He was warned to do well or else sin was waiting at the door calling for him, desiring him. Cain gave in to sin.
James 1:12 God blesses those who patiently endure testing and temptation. Afterward they will receive the crown of life that God has promised to those who love him. 13 And remember, when you are being tempted, do not say, “God is tempting me.” God is never tempted to do wrong, and he never tempts anyone else. 14 Temptation comes from our own desires, which entice us and drag us away. 15 These desires give birth to sinful actions. And when sin is allowed to grow, it gives birth to death.
Matthew 15:16 “Don’t you understand yet?” Jesus asked. 17 “Anything you eat passes through the stomach and then goes into the sewer. 18 But the words you speak come from the heart—that’s what defiles you. 19 For from the heart come evil thoughts, murder, adultery, all sexual immorality, theft, lying, and slander.20 These are what defile you. Eating with unwashed hands will never defile you.”
The last passage Jesus explained its not the food we eat that defiles us, but rather what comes from the heart that defiles us. Murder and all evil lie waiting at the door for those who refuse to do well.. It all starts in the heart, starts with a thought. If we give in to sin the consequences lead to death. Spiritual death as well as physical death.
Cain and Abels story is a reflection of 2 brother who knew the truth, knew God yet one of whom was rejected. For the evil that lurked in Cains heart towards his brother. If the story was flipped and Abel had offered his fruits to the Lord with love and thanksgiving Im sure Abel would not have been rejected. God sees the heart in our actions.