Angels are not children of God, humans are the children of God. If you look in the genealogy it says "...and Seth the son of Adam, and Adam the son of God." Now there are righteous humans (God's people) and unrighteous humans (people that God has disowned because of their wicked hearts). Both these groups of people sin, but only the righteous are forgiven.
So it makes sense that the sons of God did come from Seth- because his genealogy leads to Christ. So when God's people, the righteous, started inter-marring with the unrighteous, that created a BIG problem- for God says "What fellowship can light have with darkness? How can the righteous be one with the unrighteous."
Sooo, since things were done physically in the Old Testament, God had to show this was a big problem in a physical way- therefore, the offspring got so big that normal sized people were literally the size of grasshoppers in comparison (glad I don't have to change their baby's diapers!) lol.