The only way I can forsee you damaging the hair folicles at all is if you yank the piece off with a strong bond and the glue ends up pulling out your hair. But even then, if you have good hair growth, it should ultimately grow back.

I have not noticed any more hair loss from when I started wearing 6 years ago.

Whoever is giving you that advice has other motives, or is greatly misinformed.