Robert Brown was reputedly the illegitimate son of Frank Broonzy, who also fathered Big Bill Broonzy. Broonzy often joked that Sam was his half brother when they performed together.
Sam was raised in Arkansas, working on a farm. He moved to Memphis in the early '20s to play the blues. While in Memphis, he met Sleepy John Estes and Hammie Nixon and the trio played street corners, collecting tips from passer-bys. In 1932, Washboard Sam moved to Chicago. Initially he played for tips, but soon he began performing regularly with half brother Big Bill Broonzy. Within a few years, Sam was supporting Broonzy on the guitarist's Bluebird recordings. Soon, he was supporting a number of different musicians on their recording sessions, including pianist Memphis Slim and bassist Ransom Knowlin, and a handful of saxophone players, who all recorded for Bluebird.