From what you have told me, you are well able to defend yourself in a fight. However, you would only cause harm to someone who meant harm to those you love. You are by no means harmless, but you are peaceful. One might say of you what Sir Ector said of Sir Lancelot.
"Thou wert the meekest man that ever ate in hall among ladies, and thou wert the sternest knight to thy mortal foe that ever put spear in the rest."
There are those in this world who like to think of themselves as peaceful because they are refuse to fight at all and disapprove of those who do. In reality, they are merely harmless, not peaceful.