West Point Cemetery was Norfolk's first municipal cemetery for African Americans, after an 1827 ordinance provided for their interment in a section of Potter's Field just north of the borough limits. The section was set off exclusively for the burial of African Americans in 1852 and was named Calvary Cemetery in 1873. When a larger Calvary Cemetery was established several blocks to the east in 1877, the older cemetery resumed the name Potter's Field.
Potter's Field was named West Point Cemetery in 1885 at the suggestion of James E. Fuller, Norfolk's first black councilman. The name reflects the cemetery's location at the western end of Elmwood Cemetery. After a portion of West Point was set aside for Union veterans, the black community raised funds to erect a statue of a Union soldier on the site. The statue, completed in 1920, represents Norfolk native Sgt. William Carney, of Company C, 54th Massachusetts Colored Infantry Regiment, the first African American whose actions earned him the Congressional Medal of Honor.