Who's standards are you measuring this by? Gods word is clear on this topic:
"I am writing these things to you, hoping to come to you before long; but in case I am delayed, I write so that you may know how one ought to conduct himself in the household of God, which is the church of the living God, the pillar and support of the truth." (1 Timothy 3:14-15)
The church is the "assembly" of God's people. Being part of the church provides spiritual nutrition. Being cut off is like a part of one's body being cut off. If you cut off your finger and do not have it re-attached, the cut off finger will die.
"Obey your leaders, and submit to them; for they keep watch over your souls, as those who will give an account. Let them do this with joy and not with grief, for this would be unprofitable for you." (Hebrews 13:17)\
A responsibility of church leaders is to keep watch over the souls of church members. These leaders will someday give an account for their work. There is another verse on that I cannot recall atm.
When we remove ourselves from the Body we hurt ourselves and others because its a cutting and a brokenness away from a living thing just like our own body needs eyes and ears (without others are forced to pick up the slack). The following passages are a worthwhile read for this topic - american standard version:
12 For as the body is one, and hath many members, and all the members of the body, being many, are one body; so also is Christ.
13 For in one Spirit were we all baptized into one body, whether Jews or Greeks, whether bond or free; and were all made to drink of one Spirit.
14 For the body is not one member, but many.
15 If the foot shall say, Because I am not the hand, I am not of the body; it is not therefore not of the body.
16 And if the ear shall say, Because I am not the eye, I am not of the body; it is not therefore not of the body.
17 If the whole body were an eye, where were the hearing? If the whole were hearing, where were the smelling?
18 But now hath God set the members each one of them in the body, even as it pleased him.
19 And if they were all one member, where were the body?
20 But now they are many members, but one body.
21 And the eye cannot say to the hand, I have no need of thee: or again the head to the feet, I have no need of you.
22 Nay, much rather, those members of the body which seem to be more feeble are necessary:
23 and those parts of the body, which we think to be less honorable, upon these we bestow more abundant honor; and our uncomely parts have more abundant comeliness;
24 whereas our comely parts have no need: but God tempered the body together, giving more abundant honor to that part which lacked;
25 that there should be no schism in the body; but that the members should have the same care one for another.
26 And whether one member suffereth, all the members suffer with it; or one member is honored, all the members rejoice with it.
27 Now ye are the body of Christ, and severally members thereof.
28 And God hath set some in the church, first apostles, secondly prophets, thirdly teachers, then miracles, then gifts of healings, helps, governments, divers kinds of tongues.
29 Are all apostles? are all prophets? are all teachers? are all workers of miracles?
30 have all gifts of healings? do all speak with tongues? do all interpret?
31 But desire earnestly the greater gifts. And moreover a most excellent way show I unto you.
I enjoy a good worship time (where instruments and voice together) while praising God. That is my favorite part of a gathering, as well as prayer time. I don't think your wrong if you are not in a church building every Sunday as long as you are following the word of God as I posted in the above verses and thoughts.