Jeff Williams was raised in Oberlin, Ohio where he began playing drums at age eight. Self-taught, with the aid of his parents’ collection of jazz recordings as a guide, he started playing gigs with local professionals at age fourteen. Jeff’s mother, Ann Williams, moved to New York to pursue a jazz singing career when he was nine, introducing him to a number of musicians who would be his early mentors, including drummers Jo Jones, Herb Lovelle, Elvin Jones, and Oliver Jackson. In 1967 Jeff made his recording debut with the psychedelic group Ant Trip Ceremony (Cicadelic).
In 1968 Jeff entered the Berklee College of Music and studied drumming with Alan Dawson. In 1971 he moved to New York and soon became acquainted with Dave Liebman and Richie Beirach, with whom he would form the group Lookout Farm. In 1972 Jeff joined Stan Getz for a two-year stay in groups that included Dave Holland on bass and pianists Hal Galper, Beirach, and Albert Daily. He appeared at the Half Note with Lee Konitz the same year, beginning an association that would continue for several decades. Jeff also recorded Friends (Oblivion) in 1972 with Marc Cohen (now Copland), John Abercrombie, and Clint Houston.