Not really a good point. Sure, on the surface it seems valid. But going deeper it's a flawed logic.
Extremists are a heavily armed, extremely violent group of trained people willing to kill, torture and even blow themselves up to forward their cause. Or to stop anyone who gets in their way. The moderates are likely made of of average people, probably not well fed or educated or trained, let alone well armed. Do not have the same violent natures required to fend off aggressive people who will make bombs out of their own bodies.
The moderates are not as mentally prepared for the risks of the fight such as fearing death, being injured, captured and tortured. The extremists are eager to face the risks, and also have less to fear from a group of moderates.. Mentality plays a big part in such things.
Put 10 people in a room, one has a gun. The majority of the time everyone will obey the one gunman, not mob him.
Many times crimes done out in the open are not interfered with because everyone thinks someone else will step in. So if no one else is stepping in, then they don't feel it's their place to either. Women have been raped openly in public as people walked by and no one tried to stop it. All because they felt it was someone else's responsibility.
So a little study on how humans actually do behave shoots down this little meme.