In 1844, Lambert Spencer built a simple Greek Revival home detailed with Doric columns and acanthus leaves. In 1868, Mr. and Mrs. Lloyd Bowers enlarged and beautified the home. Mr. Bowers added two hexagonal wings and hired an itinerant painter to paint three ceiling frescoes. Mrs. Bowers, with the help of an English gardener, laid out a formal butterfly-shaped garden. The kitchen was a separate building, joined to the house by a covered porch. Other outbuildings included a two-story servant house, smokehouse, well, wash house, barn and cow shed. The house was placed on the National Register of Historic Places in 1972.
Lambert Spencer moved to Columbus from Talbot County, Maryland in 1828. He purchased twelve acres from William L. Wynn in 1844. Henson G. Estes later received the property from Spencer. In 1862, Lloyd Guyton Bowers, a cotton broker, traveled from Massachusetts to Macon, Georgia where he married Sarah Tabitha Bartlett. The Bowers soon moved to Columbus and purchased The Elms. The house remained in the Bowers family until 1966 and was then purchased by Allen M. Woodall, Jr. In 1999, Mrs. Maxwell C. Harden, daughter of a local builder, Thomas Watson Cooper, returned to Columbus and purchased The Elms for her home.