Conniff was born in Attleboro, Massachusetts, and learned to play the trombone from his father. He studied music arranging from a course book.
After serving in the U.S. Army in World War II (where he worked under Walter Schumann), he joined the Artie Shaw big band and wrote many arrangements for him. After his stint with Shaw, he was then hired by Mitch Miller, then head of A&R at Columbia Records, as their home arranger, working with several artists including Rosemary Clooney, Marty Robbins, Frankie Laine, Johnny Mathis, Guy Mitchell and Johnnie Ray. He wrote a top 10 arrangement for Don Cherry's Band of Gold in 1955, a single that sold more than a million copies. Among the hit singles he backed with his orchestra (and eventually with a male chorus) were Yes Tonight Josephine and Just Walkin' in the Rain by Johnnie Ray; Chances Are and It's Not for Me to Say by Johnny Mathis; A White Sport Coat and The Hanging Tree by Marty Robbins; Moonlight Gambler by Frankie Laine; Up Above My Head, a duet by Frankie Laine and Johnnie Ray; and Pet Me, Poppa by Rosemary Clooney. He also backed up the albums Tony by Tony Bennett, Blue Swing by Eileen Rodgers, Swingin' for Two by Don Cherry, and half the tracks of The Big Beat by Johnnie Ray.