Samuel Fraunces, might have immigrated from the West Indies, was often assumed to be Black due to his nickname, Black Sam. In several letters, George Washington refers to the tavernkeeper as "Mr Frauncis (als [sic]black Sam)." Philip Freneau's poem "Hugh Gains Life" recounts the cannonball that crashed through the roof of the tavern in August 1775 during the British bombardment of the harbor. One line alludes to Fraunces: "At first we supposed it was only a sham, 'till he drove a round ball through the roof of Black Sam."  Unfortunately no definitive portrait of Fraunces exists, nor do any personal accounts stating his racial background.

Fraunces listed himself as a free, white male and the head of a household in Manhattan's Dock Ward in the 1790 census. The household includes four free, white women—likely his wife and daughters. The census also notes one enslaved person. Entries from Dr. Peter Middleton's account books show prescriptions he gave to members of the Fraunces family. These include "Rhubarb Syrup" given to "Your Daughter" and "Venisection" (bloodletting) for "Your Negro.”