I think you underestimate the degree to which they are cowardly. They gain confidence from superior numbers, but one-on-one, or crowd vs crowd, a lot of them can be put off by speech or other non-violent means. I think that deep down a lot of them know what they are doing is wrong, even though they tell themselves otherwise.
Non-violent resistance has a long history and has often been effective. As an example, the fight for civil rights was not won because the progressives usually managed to beat up the racists. The progressives were largely non-violent. The racists were routinely violent, but they still lost.
It takes a lot of bravery to oppose someone who may try to hurt you. Some of us will get hurt. It shouldn’t be necessary, but if we want this to stop we will have to take some risks. (There’s nothing wrong with avoiding high-risk situations, but some risk is necessary.)
Personally, I am also okay with defending yourself if you are attacked. Non-violence is always better, but I doubt I could remain passive if faced with physical violence.
Leshoyadut