If thou doest well, shalt thou not be accepted? and if thou doest not well, sin lieth at the door. And unto thee shall be his desire, and thou shalt rule over him.(KJV)
What should have Cain done? What should have he offered?
Heb 9:22 And almost all things are by the law purged with blood; and without shedding of blood is no remission.
Sacrifice must be an animal. until Jesus came of course. The lamb being a symbol of Christ. Able brought the acceptable offering for sin. but Cain relied rather on his own works. but our works never fly with God.
Cain would have offered the correct sacrifice as seen by these words.
Gen 4:3 And in process of time it came to pass, that Cain brought of the fruit of the ground an offering unto the LORD.
notice "in the process of time it came to pass" Denoting that Cain did not ordinarily bring fruit for an offering but in this case did. This tells us that Cain knew what was right and had been doing the right thing up to this point. this is also indicated in Gods words to Cain as you mentioned.
Gen 4:7 If thou doest well, shalt thou not be accepted? and if thou doest not well, sin lieth at the door. And unto thee shall be his desire, and thou shalt rule over him.
The indication is that Cain knows what is the right thing and what will be accepted. we know also what the right thing is and what will be accepted. because Able left an example.
Gen 4:4 And Abel, he also brought of the firstlings of his flock and of the fat thereof. And the LORD had respect unto Abel and to his offering:
Gen 4:5 But unto Cain and to his offering he had not respect. And Cain was very wroth, and his countenance fell.
God respected and accepted Able offering of His flock. but not Cains fruit offering. clearly the issue is over the offering. Cains attitude comes after the offering is rejected not before thus the attitude is not the issue initially.
There can be no doubt that on careful examination it is the offering that was wrong which followed with a wrong attitude etc. This shows that both Cain and Able understood what the sacrifice was about.
Rev 13:8 And all that dwell upon the earth shall worship him, whose names are not written in the book of life of the Lamb slain from the foundation of the world.
Blessings hope this helps.