Although there is no extensive Talmudic discussion of the wearing of religious headgear, there are several places where the practice of covering one’s head is mentioned:
Kiddushin 31a: R’ Huna the son of R’ Yehoshua would not walk four cubits with his head uncovered. As an explanation for his practice he would say, “Hashem’s Presence is above my head.”
Shabbos 118b: R’ Huna the son of R’ Yehoshua said, “I am deserving of reward because I do not walk four cubits with my head uncovered.”
Shabbos 156b: The mother of R’ Nachman bar Yitzchak was was told that her son was destined to be a thief. In order to prevent this from happening, she would not let her son go with his head uncovered. She would say to him, “Cover your head in order that you should fear God, and pray that your evil inclination should not get the best of you.” R’ Nachman never understood why she said that. One day, he was studying under a date palm tree, and his head covering fell off. He saw the tree and was tempted to steal a cluster of dates.
Shabbos 77b: Why is a headscarf called a “sudra”? It is an abbreviation of the words “Sod Hashem Li’Rei’av” (“The secret of God is revealed to those who fear Him” [Psalms 25:14]).
For more sources, see Beiur HaGra, Orach Chaim § 8(2).
It is clear from these sources that covering one’s head is viewed as an expression of the fear of God. As the Aruch HaShulchan explains (Orach Chaim §2(10)):
A man’s head houses his intellect, which is the source of wisdom and fear. It should not be uncovered before Hashem, Whose Presence permeates the world, just like one should not stand in a holy place with his head uncovered. If one does not cover his head, his evil inclination can overcome him, even without him realizing.
In sum, there are two reasons for a man to cover his head. First, it is a way of showing humility before God. Second, it serves as a deterrent to sin.