Don't Beat Yourself Up Having Emotions

The pedestal is the problem here. That is why you are so anxious. You both need to make an attempt to figure out what your ideal is, because two polyamerous people can be incompatible. Figure out if you're compatible in that way. Schedule some physical, actual days out, nights out, or nights in which you know for sure you always have this time with them. And last but not least, don't be so mean and hard on yourself about it. It's a perfectly normal reaction to feel scared to feel all o these things. Nothing that you've described is out of the norm when it comes to emotions.