Is it an old wives tale?
Find me one, just one, manufacturer of carbon rims/wheels that clealry states in print that it's quite allright to use whatever brake pad on alloy rims and their carbon product alike.
Yes, I know, there are brake pad manufacturers that produce pads they claim would work an both alloy and carbon rims. That does not mean you can use that very same pad on both carbon and alloy rims.
Even if you're prepared (which you're not, obviously) to thoroughly clean out a pad that's been used on an alloy rim, how are you going to be sure some piece of metal is still not embedded into that pad somewhere?
A single shard is all it takes to thorughly damage a carbon rim, regardless of its pedigree or price tag.
Those who think alloy rims don't lose pieces of metal should do a few rounds on a cyclo-cross bike. The european way if possible....
Being a snob is an expensive hobby.