If you pull off any tape residue from the lace without dissolving the glue, the likelihood is that you'll pull out the hairs as well from the base.
Most shedding of hairs is due to this.
Also, you won't always see the residue of glue left behind. It doesn't mean to say it isn't there.
Using adhesive remover is a wise precaution because 1% of residue glue remaining, equals 10% less adhesion on the next attachment.
Using soap afterwards to dry out the skin is ok, but you don't want flaky skin.