Richards started playing the xylophone at age six. In high school he performed with the Hartford Symphony Orchestra. He studied with Al Lepak at the Hartt School of Music in Hartford, graduating in 1952. After being drafted, he belonged to an Army band in Japan and played with Toshiko Akiyoshi. He cited Lionel Hampton as his first and biggest influence on vibraphone.
In 1954 Richards moved to New York City, where he played with Charles Mingus, Ed Shaughnessy, and Ed Thigpen while doing studio recordings for Perry Como, the Ray Charles Singers, and Mitchell Ayres. For about three years he was a member of a group led by George Shearing, then moved to Los Angeles and worked with Don Ellis and Paul Horn. He led his own band, the Microtonal Blues Band, and spent time with composer and inventor Harry Partch. As a sideman, he accompanied George Harrison on tour and recorded with Frank Sinatra, Frank Zappa, Doris Day, Judy Garland, Nelson Riddle, Steely Dan, and Sarah Vaughan.
Richards worked often as a studio musician for movies and television. His credits include playing bongos on the theme song for the television program Mission Impossible. Other television work included finger snaps for the Addams Family theme, and xylophone work for the opening theme of The Simpsons.
He led a band with Joe Porcaro, and he released a solo album, The Wonderful World of Percussion.
On June 25, 2019, The New York Times Magazine listed Emil Richards among hundreds of artists whose material was reportedly destroyed in the 2008 Universal fire.