The site of Port Conway is located five miles south on the Rappahannock River. Francis Conway laid out the town in 1783, and the next year the Virginia General Assembly passed an act establishing it. James Madison, Father of the Constitution and fourth president of the United States, was born at the Conway house on 16 March 1751. On 1 Sept, 1863, Brig. Gen. H. Judson Kilpatrick's Federal cavalry shelled two Union gunboats at Port Conway that had been captured by Confederates. John Wilkes Booth, the assassin of President Abraham Lincoln, fled across the river there on 24 Apr. 1865, Little remains of the town today.