"If this is not right for you right now, it's not right. And in the short term, you have to disappoint people," he says. "In the long term, it's going to create so much more opening for you to really actually pursue or go be with the person that you want to be with"