Today, when we hear the term "men of the cloth," it almost always refers to members of the clergy, who can be distinguished by the special clothing they wear all the time, or merely during the performance of church services. Actually, the origin of the term was not specific to priests; special clothing worn only for a job meant a worker was a man of the cloth and bore no especial reference to clergymen. Anyone who wore a uniform while working, like a chef’s coat, or a servant’s livery, could be described with this term through the 16th century.
In the 17th century, language changed, as it frequently does, to make the term "men of the cloth" apply exclusively to members of the clergy. No longer was a servant or a page with a uniform included. Further, the priesthood in general may be called "the cloth."
Some also reference the collar as an essential part of the men of the cloth uniform. In fact, several ministers who were also poets wrote about the collar. The 17th century poet, George Herbert, used the collar as a metaphor for the restrictive but also inspirational nature of the ministry. The poem, “The Collar” refers several times to the word suit. He queries that even if he escaped the priesthood, “Shall I be still in suit?” Later in the poem Herbert states: “He that forbears / To suit and serve his need, / Deserves his load.” The collar and all clothing associated with the ministry becomes the symbol of service, which at once restricts and restores with the ending lines of Herbert’s poem:
But as I rav’d and grew more fierce and wilde,
At every word,
Me thoughts I heard one calling, Childe:
And I reply’d, My Lord.
Clearly, the collar at first antagonizes and then suits this man of the cloth. It’s likely Herbert would have described himself as one of the men of the cloth since he lived in the 17th century, when such usage became common.
Today you may still hear the term, but it is fast becoming an archaism. Most often, Catholic priests use it since no female priests exist.