The site of the William Nelson House in Yorktown originally hosted a residence built around 1725 to 1730 by Thomas Scotch Tom Nelson, patriarch of the influential Nelson family of Virginia. His son William Nelson who served as acting colonial governor of Virginia from 1770 to 1771, owned the property and likely maintained a house on the site after his father's passing.

William Nelson’s house later passed into the hands of his sons, including Thomas Nelson Jr., a signer of the Declaration of Independence and brigadier general during the Siege of Yorktown. Thomas Nelson Jr inherited the family estate and lived nearby, though the site in question is specifically tied to William Nelson Sr. and his time as a prominent Yorktown merchant and politician.

The original William Nelson House was destroyed in the great Yorktown fire of 1814, which consumed many of the town's structures. While the main Nelson House built by Thomas Scotch Tom survived, the site belonging to William Nelson did not, leaving behind mostly archaeological remnants rather than an intact building.

Over the 19th and early 20th centuries the original house site remained largely vacant or redeveloped for other uses. The main Thomas Nelson House survived and became the primary focus of preservation, eventually acquired and restored by the National Park Service. Meanwhile the William Nelson site became a point of historical interest primarily through marker designations and archaeological investigation.

Today the William Nelson House site in Yorktown is interpreted as part of the larger Colonial National Historical Park. While no standing structure remains the location offers visitors visual context for understanding the Nelson family legacy connecting the political and commercial achievements of William Nelson with the later revolutionary heroics of his son Thomas Nelson Jr.

If you walk into town from the Yorktown Victory Monument, you’re in for a Colonial-era treat. The remaining houses on Main Street are from the Colonial period, and it’s an incredibly tidy walk back into time. On your right is an open field where once stood the home of William Nelson, one-time governor of Virginia, appointed by the King himself. On moonlit nights, British soldiers are seen racing across the yard, fleeing from the sounds of cannon, which the living cannot hear.