The Old Custom House in Yorktown was erected around 1720–1721 by Richard Ambler, who served as the customs collector for the Port of York River. Originally built as a brick storehouse, its architecture showcased the Flemish bond brickwork typical of early Colonial commercial buildings. It stood out as one of only two surviving colonial storehouses in Virginia, reflecting Yorktown’s importance as a deep-water port between Philadelphia and Charleston.

Through the 18th century, the Custom House handled the storage and taxation of incoming goods. Ambler and later his sons oversaw its operation for decades. Yorktown's growth as a bustling trade center intertwines with the building's history. Its subsequent sale to Thomas Wyld in 1778 marked a shift, as the house took on roles beyond customs duties, functioning temporarily as an inn and store.

During the Revolutionary War, British troops occupied the Custom House as a barracks. The building served as winter quarters for French forces following the British surrender in 1781. These military uses damaged the adjacent wooden residence and outbuildings. In 1783, Ambler successfully reclaimed ownership through legal challenges over financial arrangements, resuming civilian use of the property.

In the Civil War, the Old Custom House again played a strategic role, serving as General J. B. Magruder’s headquarters in 1862. The wooden passageway that linked the brick storehouse to Ambler’s house burned during this period. Postwar, the house found new purpose when Dr. Daniel McNorton, one of Virginia’s first African-American state senators, operated his medical office there, serving the local community.

By the early 20th century, ownership passed to Mrs. Adele Blow and later to the Daughters of the American Revolution’s Comte de Grasse Chapter in 1924. A restoration led by architect Duncan Lee was completed in 1930, adding dependencies, a garden wall, and period-accurate interior finishes. Since then, it has functioned as a museum and meeting house, and in 1999 it was listed on both the Virginia Landmarks Register and the National Register of Historic Places.

Yorktown is steeped in tragic wartime history, and many believe that the Old Custom House is haunted by restless spirits from its past. Visitors walking past the property often report hearing voices or footsteps when no one is present, adding to the region’s reputation for haunted encounters.

During ghost walking tours, guides frequently include the Old Custom House as a stop. Participants describe sensing cold spots near the house’s rooms or hallways, despite the warm summer air. Some claim that these chills coincide with fleeting whispers hinting at the presence of those who once occupied the space.

Though there are no well-documented violent events specifically tied to the Custom House, its military use during both the Revolutionary and Civil Wars makes it a likely candidate for paranormal residue. Guides recount that some night visitors feel an oppressive sadness, as though the building carries the grief of its long-ago occupants.

A few guests report more tangible experiences—objects shifting slightly or the faint sound of footsteps crossing the wooden floors above. While skeptics attribute these to natural causes, others believe they are echoes left by the spirits of soldiers, merchants, or patients who once walked the halls.