Melvin Rhyne was born in Indianapolis in 1936 and started playing the piano shortly thereafter. At 19 years old, Rhyne started playing piano with then-unknown tenor saxophonist Rahsaan Roland Kirk but quickly switched over to the instrument that would make him famous: the Hammond B3 organ. Rhyne's piano skills translated to the organ fluently and before long he was backing famous blues players like B.B. King and T-Bone Walker. In 1959 when he was asked to join fellow Indianapolis musician Wes Montgomery's newly formed trio.
Rhyne then moved to Wisconsin and largely kept to himself for the next two decades. In 1991, however, Rhyne returned to the jazz scene, playing on Herb Ellis' album Roll Call, Brian Lynch's At the Main Event, and his own comeback The Legend. Rhyne continued to be prolific in the years to come, releasing eight more solo albums on the Criss Cross jazz label.
In 2008 Rhyne teamed up with fellow Indianapolis jazz musician Rob Dixon to form the Dixon-Rhyne Project, a boundary-pushing jazz quartet that also includes Chicago guitarist Fareed Haque and drummer Kenny Phelps. The quartet released the album Reinvention in 2008 on Indianapolis jazz label Owl Studios. Rhyne's later career trio included drummer Kenny Washington and guitarist Peter Bernstein in the same organ, guitar, drum formation of the original Wes Montgomery Trio.