The son of Hall founders Allan and Sandra Jaffe, Ben was raised in and around Preservation Hall. Jaffe marched in his first Carnival parade at the age of nine alongside his father and parrain (godfather) Harold Duke Dejan. When he was thirteen, the lease on the building that housed the Hall was up for renewal. His father sat the family down and asked Ben whether they should continue. By 1984, many of the original musicians who had played at the Hall in its early days had passed away, yet the soul of the Hall remained the same. The young Jaffe replied, Of course you continue!