Gilmore grew up in Chicago and played clarinet from the age of 14.  He took up the tenor saxophone while serving in the United States Air Force from 1948-1952, then pursued a musical career, playing briefly with pianist Earl Hines before encountering Sun Ra in 1953.
For the next four decades, Gilmore recorded and performed almost exclusively with Sun Ra. This was puzzling to some, who noted Gilmore's talent, and thought he could be a major star like John Coltrane or Sonny Rollins. Coltrane, in fact, was impressed with Gilmore's playing, and took informal lessons from him in the late 1950's. Coltrane's epochal, proto-free jazz Chasin' the Trane was inspired partly by Gilmore's sound.
In 1957 he co-led with Clifford Jordan a Blue Note date that is regarded as a hard bop classic: Blowing In from Chicago. Horace Silver, Curly Russell, and Art Blakey provided the rhythm section. In the mid-1960s Gilmore toured with the Jazz Messengers and he participated in recording sessions with Paul Bley, Andrew Hill (Andrew! and Compulsion), Pete La Roca (Turkish Women at the Bath), McCoy Tyner (Today and Tomorrow) and a handful of others. In 1970 he co-led a recording with Jamaican trumpeter Dizzy Reece. His main focus throughout, however, remained with the Sun Ra Arkestra.
Gilmore's devotion to Sun Ra was due, in part, to the latter's use of harmony, which Gilmore considered both unique and a logical extension of bebop. Gilmore had stated that Sun Ra was more stretched out than Monk  and that I'm not gonna run across anybody who's moving as fast as Sun Ra ... So I just stay where I am.