First, I want to say that I'm sorry that this has happened to you and your siblings. It doesn't matter if you were eleven or if you're an adult, sexual acts without consent IS harassment/rape. It was not your fault for someone else's actions, those who do such abuse and implant that into the victim's head do so because it's a way for them to have control and not to be ratted out.
That being said, I would have to assume that your brother is the one who is older than you, therefore had to know better about the actions than say if a kid committed them. Often times, kids who do these type of things are passing on what has been done to them. Does it excuse the act? Absolutely not. Have you told anyone about this or have your siblings done so? As suggested, it's likely that you need counselling if you have not. Things such as rape are a big deal and it impacts a person's life forever.