yes dont worry about wearing some hats. They wont get destroyed just because you wear a cap.
I wear sometime caps (mainly because if i dont want to care about my hairline on this day) and nothing happens. The only problem is, that if you take your cap off it could look a bit wiggy, because of course you cant style it beneath your cap.