If they are Christians, remember the confrontation steps in Matthew 18. First confront privately, then bring a witness, etc.
If they are unbelievers, you probably do need to distance yourself from them. Think of unbeliever confrontation as the opposite of believer confrontation - the first thing you have to do is tell the unbeliever that you aren't comfortable with what they are doing and that you need a break from the relationship. If it is specifically against the Scriptures, affirm your stance in Christ by telling the unbeliever what problems their bad behavior is causing you that impairs your witness for the Lord. Then you have to inform all of the mutual friends between you and the unbeliever that the relationship is cut off so you don't get back into the relationship that is dragging you down.
I may get contested over this, but in my experience dealing with a genuinely problematic unbeliever in your life is like doing Matthew 18 backwards. First you do the excommunication, then you bring the person on trial before everyone, then they might come to you with a witness that they have changed their ways, and then you finally have a private conversation after they stop being a hinderance to the work that you are doing for Christ. It's very interesting.
As for when to confront, for believers, it's simple: has the person sinned against you? Are they breaking God's laws and violating his commands? (Or are they just annoying/hurting you? If the latter, why are you annoyed/hurt? Sometimes the problem is inside of you, rather than the outside.) It helps me to be able to quote the Scripture I think they have violated - that reassures me that I have a real reason to confront them and I'm not just angry.
For unbelievers, it's when associating with them is impairing your reputation, your ability to carry the name of Christ. If they are slandering you, it's time to let go. Another reason to let go of an unbeliever is when they fail to respect your beliefs and try to tempt you to sin. The best witness to someone who wants to hurt you is the fact that God is more powerful than them. May the Lord give you the strength to push away those who need to go, and the insight to preserve those who can stay.
I'm praying for you. Sorry that this post probably isn't the best worded thing ever - I think I repeated myself a bit trying to find my way through my neural passageways, heh.
