Once an Indian hunting ground called "Alafia," this area was surveyed by Samuel Hope in the 1840's and was successively known as "Alfiers," "The Neck," "Sapling Woods," and "Elfers" — a variation of the Indian name. Fishing, timber and citrus supported early residents. The nearby Samuel and Elizabeth Pinder Baker home was built in 1882. Elfers Post Office was established in 1909. The town was incorporated from 1925 to 1933.
The State Bank of Elfers failed, due to the collapse of the "Florida land boom," just as its building was completed in 1926. The packing house established by the Elfers Citrus Growers Association in 1920, operated until destroyed by fire in 1966. The Tampa and Tarpon Springs Land Company was a major promoter of the area in the 1900's. Leading citrus groves included St. Austell-Elkhorn, and Peter O. Knight's Sans Souci. Another local grower, Gerben DeVries, authored the Chasco Fiesta legend.