James Whitall, a wealthy Quaker farmer and merchant, and his wife Ann Cooper built the main brick section in 1748. On April 16, 1777, the Pennsylvania militia commandeered the farm and built Fort Mercer in the northern apple orchard. Son Job Whitall noted in his diary on October 10, 1777 that the Americans "turned us out of our kitchens ye largest room upstairs and ye shop and took our hay to feed the horses." That day Colonel Christopher Greene had arrived to command the fort and used a portion of the house as military headquarters.
On October 22, 1777 when the attack began on Fort Mercer, family members fled to Woodbury; but Ann Whitall refused to abandon her home. When a cannonball broke through the north wall, according to legend, Ann calmly carried her spinning wheel to the cellar and continued to spin. The house, left intact, was used as a hospital after the battle. Ann remained to tend all the wounded Hessian and American soldiers.
The battle was a key victory and morale builder; however, news of a second siege caused the Americans to destroy the fort and leave the property on November 24, 1777.