Do We Need Math by Proving That You Could Do Without It if You Had To?

I understand that it's a useful way to answer the question, do we need math by proving that you could do without it if you had to? But I'm still not 100% convinced. To me, potentials are numbers and comparing is a relationship. So rather than those being a mathematical function that lives on top of that, it's just these are primitive properties of the space-time points. And so a lot of people who are concerned about believing in mathematical objects because after all, that's kind of spooky.