The owner and I were pretty much in agreement about the Raleigh Militus (and the same frame as the Diamondback Podium), and absolutely amazing frame that can be had - complete - for less than the cost of just a frameset from the other big-name brands (listed in OP's list) and blows those other top-end frames off the road.
That is to say that there are exceptional frames out there, and a lot of the pricing, sadly, is strictly associated with marketing the exclusivity pricing of the brand and not necessarily associated with the product's purported exceptional qualities.
Here in lies the answer to the question.
1. The top manufacturers sell for what they do, because people will buy them.
2. They buy them because that is what is perceived as the piece or part to be on. The example of Raleigh and Diamondback are two great examples of exceptional bikes that will not gain a large market share, because the people buying bikes will always perceive them as inferior and not what they see as the majority of bikes on any given ride or race.
Peer pressure exist even in the local group you ride with. If enough guys in a group started buying the Diamondback, and really started pushing it, more and more of the group would start to agree that the DB is a great bike, but there will be that group that says," Yeah, its nice, but its not a ______" You can fill that blank in with any relevant high end frame out there.
My biggest concern with the price of bikes is that I always look at how much car I could buy for that price and that usually always puts me back on an allow frame with some good wheels, instead of a high end carbon frame with the same wheels.