When the 1875 Milam County Jailhouse grew too crowded in the 1890s, it was removed to make room for larger facilities. In March 1895, the Milam County Commissioners awarded a contract to the Pauly Jail Building and Manufacturing Company of St. Louis, Missouri, for the construction of a larger prison. The company furnished all supplies, including St. Louis pressed bricks. County Judge Sam Streetman, who later served on the Texas Supreme Court, approved the contract, although he had preferred the use of local building materials.
This structure, designed with Romanesque revival features and stone detailing above the windows, had three main floors and a "hanging tower" equipped with a trap door. The tower was never used for executions because most hangings took place outdoors. The first floor had ten rooms, three for storage and the remainder serving as a residence for the sheriff and his family. The second and third stories consisted of cell blocks for prisoners.
In 1975 a new county jail was constructed, and the Commissioners Court turned this facility over to the Milam County Historical Commission. After renovation, it was opened as a museum in 1977.