If it's the same sort of finish as the B-pillar trim (and I believe it may be), the BE trim will scratch if you look at it crossly... it just may not be as obvious because it's not as large as surface area.
I pulled the mirror, B, and C-pillar trim off and wrapped it in satin black vinyl on my car. There's truly no hope for keeping it swirl free. Three of my last four cars have had gloss pillar trim. The finish just doesn't hold up.