Born in 1934 in Houston, Texas, Horace came from a musical family centered around his mother, Mary Malone Tapscott, who worked professionally as a singer and pianist. When Horace was nine, the family moved to Los Angeles. As a teenager in the late 1940's, Horace was surrounded by the music of Central Avenue: Art Tatum, Charlie Parker, Coleman Hawkins, Dexter Gordon, were among the many cats on the set. Around this time, Horace also began to take music lessons from teachers Dr. Samuel R. Browne and Lloyd Reese, whose other students included Eric Dolphy and Frank Morgan. Horace's musical studies included trombone in addition to piano.
In 1952, Horace graduated from Jefferson High, got married to Cecilia Payne and went into the Air Force. Horace played in an Air Force Band while he was stationed in Wyoming for his term of duty. After mustering out, he returned to Los Angeles where he worked around on various gigs until he joined the Lionel Hampton Big Band as a trombonist.
In 1959, Horace finally went with the Hampton Big Band to New York, where his friend Eric Dolphy introduced him to John Coltrane. A tough winter, a lack of gigs, and too many nights on the floor of a friend's art gallery finally sent Horace packing for sunny Southern California, where a life with wife and family awaited his return.
The sixties saw Horace emerge as a die-hard leader of the Avant Garde. Horace began to gain public notice playing with his own group, that included alto saxophonist Arthur Blythe, bassist David Bryant, and drummer Everett Brown II. Horace also appeared on records for the first time (see discography).