Barretto was born on 29 April 1929, in the Brooklyn borough of New York City, of Puerto Rican parents. Noted for his many years as a prominent Latin bandleader, his music career actually began as a studio performer on the conga for jazz recording sessions.
He was raised in the Latin ghettos of East Harlem and the Bronx, in an environment filled with music of Puerto Rico but with a love for the swing bands of Ellington, Basie and Goodman. He escaped the ghetto by joining the United States Army when he was 17 years old, but he did not escape the music.
Influenced by a record of Dizzy Gillespie, “Manteca”, with conguero Chano Pozo. He was hooked and he knew then that his calling was was to become a professional musician. Barretto sat in on jam sessions held at the Orlando, a GI jazz club in Munich, Germany. After military service in 1949, he returned to Harlem and taught himself how to play the drums.