While attending high school in Chicago in the early '50s, Jarman took up the drums under the tutelage of the famous music teacher Walter Dyett. He switched to saxophone and clarinet while in the army. Upon his discharge in 1958, he returned to Chicago. There, he joined pianist Muhal Richard Abrams' Experimental Band (formed in 1961), alongside his future Art Ensemble compatriots Malachi Favors and Mitchell. Jarman played in a hard bop sextet with Mitchell, and in 1965, he became one of the first members of the Association for the Advancement of Creative Musicians.
Starting around 1967, Jarman was one of the first saxophonists to perform solo, a tactic also embraced by other members of the AACM, notably Anthony Braxton. Jarman led his own group from 1966-1968, which included bassist Charles Clark, drummer Thurman Barker, and pianist Christopher Gaddy, among others. Separate editions of that band recorded a pair of albums for Delmark: Song for... (1966) and As if it were the Seasons (1968). In 1967, Lester Bowie recorded Numbers 1 & 2 for Nessa; on 2, the four musicians who would become the Art Ensemble (Bowie, Mitchell, Favors, and Jarman) recorded together for the first time.