I can't really proclaim innocence on this one, I'm very much a boob guy (shocker, right?), I'm attracted to them and I love them for more reasons than just feeding babies, so I can't really say "yeah, I don't get it, either," because I'm in that percentage of people who very much finds boobs appealing aesthetically and sexually, and I have to think that culture plays a huge part in it. Western society shields the boobs as much as possible. They're out there in the media, constantly, but always covered up, always kept tucked inside clothing of some kind, so it creates this taboo aspect of "well I'm not supposed to see what's beneath that cloth, so now I REALLY want to see what's beneath it," and that leads to the excitement of "well now I'm seeing what I'm not supposed to see, this is exciting!" Maybe that's a gross oversimplification of the situation, and really it all just boils down to male squeamishness from the olden days and a lack of gender equality allowing men to impose laws on the female body without truly recognizing that it's their own discomfort with the fact that boobs really are just boobs, sacks of flesh meant to feed babies and otherwise just another body part that women have to lug around until they have said baby(ies)? You're right, though, there are places in the world where boobs are just boobs, not sexualized in any way and not considered inappropriate to have them out just like men do, and it's no big deal because every woman has them out.
Whatever the case may be, we certainly live in a society where they're still taboo, still viewed as far more than just baby feeders, and whatever it is that does the damage to the fickle male mind, we can't stop wanting them (and that's another point about why they're probably overly idealized, men always want what they don't have, men don't have boobs, so men want boobs).
tl;dr: Men fucking suck.