I think it's too late to *start* worrying - it is already used widely as a descriptor with negative undertones.

It is also important to note the difference between Can be elite and Will be elite. Pretenders galore and those who renounce their claims for one reason or another.