I've been wearing for a few months now and the only thing that has really changed is that I'm much more aware of where the other persons hands are, and I do find myself changing positions when I notice their hands going towards my head.
Tho I did ask one person who kept trying to run their hands thru my hair not to do it because I have some scars on my head which make it sensitive.. They said "ok then", and didn't touch my head again.

I'm sure I'm just being paranoid.. If properly attached, the odd brush of someone's hand on your head shouldn't matter.. But I'm still pretty new at wearing, and not psychologically ready to have someone's hands on my hair yet.