Setting limits in a relationship can bring up unpleasant feelings, and leave the other party hurt. If it is something that is important to you, and will help you grow, and live in peace, you should not think twice. People who care about you and love you will remain in your life. As harsh as it sounds, at the end of the day you are not responsible for anyone’s feelings or happiness.