Descended from 17th Century New England pioneers, James Rossetter (1863-1921) was born in Hamilton County, Florida. Rossetter arrived in Eau Gallie in 1902 and became a leader in the local fishing industry, harvesting the many local waterways as a charter partner of the Indian River & Lake Worth Fish Company. Later, he would form his own wholesale fishing enterprise and would go on to develop a fish pound at the Bight of Canaveral, precursor of Port Canaveral. The James W. Rossetter House began as a small existing structure on this property, which Rossetter bought in 1904. To expand his newly purchased home, Rossetter bought the winter home of John Aspinwall, a wealthy New York industrialist, and moved it to its present location. The Aspinwall structure, built in 1890 and now the west wing of the house, was connected to the existing building with open-air breezeways. Many of the designs used in the construction of boats were employed n the building of the house, as evidenced by the whimsical wood patterns inside the home. The scale of the home in relation to the surrounding residential area reflects the early success of Rossetter's industrial endeavors.