This must be a problematic question.
When you go to the Lord in prayer, yes. Then put it back on if you must. If it is the 10 gallon variety, please consider those seated behind you.
We had a pastor who would put his cowboy hat on as soon as he steped down from the pulpit.