Cadmus was born in New York and attended the National Academy of Design, ultimately leaving high school for full time studies there, followed by a year at the Art Students League in 1928. With fellow painter Jared French, he subsequently spent two years in Europe, learning and refining the egg tempera technique that became a Cadmus trademark. Returning to the United States, he found employment as a painter in the Public Works Of Art Project of the Works Progress Administration, and it is then that his 1934 painting The Fleet's In became the centerpiece of a public furor because of its sexually charged content and homoerotic overtones, fairly launching his career as a painter, printmaker, and photographer. Cadmus spent his later years in Weston, Connecticut, where he maintained a modest output of one or two paintings annually, and where he died just before his 95th birthday.