In 1839, David Dunlop and his wife, Anna Mercer Minge, a niece of President William Henry Harrison, acquired the Ellerslie tract. Robert Young designed the castellated Gothic Revival mansion for Dunlop in 1856, and construction began the next year. Surrounded by elaborately landscaped grounds, Ellerslie was damaged during the Civil War by Union artillery on 9 May 1864. Later, Gen. P. G. T. Beauregard established his headquarters here, and a Confederate rest camp occupied the grounds. In 1910, Carneal and Johnston, of Richmond, remodeled Ellerslie in a fanciful version of the Bungaloid style, but the tower and the basic structure of the house are original.