You aren't letting go of a person, or a place, or even a thing – you're letting go of your thoughts about it, or what it would be. And learning how to let go of your thoughts is the most powerful thing there is. It's how you get over fear, loss, panic, pain, judgment, and all the rest.
Everything exists in a constant cycle of gain and release. Find and lose. Use then purge. Everything. You aren't just letting go of one thing, and leaving your life an empty, bottomless abyss. You're just moving from one thing to the next, and if you quiet yourself enough, you'll be able to hear why.