I think I'm starting to get an idea of the issue now. It's something that I've struggled with in programming as well, and that seems to be a common problem (one that seems to continue to exist because most people kinda ignore it)
In a way, seabear actually pointed out the crux of the matter. Do colors exist? Do words exist? I don't think it's really semantics. It's our brain trying to make sense of things, and being so good at it that most people forget it's doing it.
It's the disconnect between what is and what we experience of it. The way we mentally and automatically model things in our minds and then assign labels to them. It's a really cool and reliable system. The problem arises when we try to communicate with other people who have their own models of things and we forget that neither of us has the real thing in our minds, just a personal perception.
Is there such a thing as 'red'? In the real world, no. In our minds, yes. But no two labels representing 'red' are the same, and that's where the issues seem to arise. That's why people often can't agree on whether a specific shade of red is 'red' or 'purple' or 'bordeaux' or whatever.
Math, in a way, is the same thing. It's just even harder to grasp because unlike a normal language, it's pretty universal. But then you realise it's not, when you remember things like roman numerals, non-base-10 number system, the many different ways to calculate the same thing, etc.
Math is just a way of describing reality. And like all descriptions, it's imperfect, but it's workable. (More workable than most other languages, which leave a lot of margin for error). And we keep looking for better ways to describe reality in terms of math, which is where a lot of our progress comes from.
But I think this disconnect between reality and mathematics, where reality is leading and math is trying to describe it, is a crucial thing to keep in mind. In the same way that 'red' is an attempt to describe the bombardment of billions of unrelated photons hammering on your eye, math is merely a way to make sense of everything that is actually happening.
And if we forget that math is the descriptor (or View) of reality and that reality is the actual data (or Model) being described, we will start to ascribe properties to math like "It's real", and then we might not research alternatives anymore. I think a large part of human ingenuity and 'thinking outside the box' arises from people remembering that language is just a description of reality, and that sometimes you need to try a different language to describe reality and see if better solutions come up.
Philosophical? Not at all. Critically important, I'd say. The ability to look at a rock and describe it as "nut cracking tool" or "sharpening tool" was the first step towards intelligent thought. And that's basically just the ability to look at a part of reality, and to try different Views on it until you find one that helps you solve a problem at hand.
So is a rock a nutcracker, or a sharpening tool, or just a rock?
Or are they just different Views of the same Model and can we use different languages to solve different problems?
Is math real, or is it imagined? Or maybe it's just another language we use to try and make sense of reality, because that's what we do all day long, without even thinking about it?
...
Yeah I was kinda bored, sorry about that :P