It seems that might be something of a case of art imitating life, as apparently keeping Jack Sparrow’s hat on his head was quite difficult during Pirates of the Caribbean: The Curse of the Black Pearl.
Captain Jack Sparrow has a number of popular accessories, but few of them mean as much to him as his hat.
This resulted in the costume designer taking the unusual step of making the hat (eventually) not out of leather, but out of rubber.
He only takes it off willingly a few times, and when it does come off, there seems to be nothing he wants more than to get it back on.