Should you be happy letting go of people who are a vortex of misery in your life?

Should family, no matter how toxic they are, be in your life because of your relationship with them? Some people are so emotionally dangerous to be around that unless you distance yourself, you will always suffer when they are around (and even when they aren't).