It first owner was John Potts, Jr., who built the house in 1795. The house was built simultaneously with its neighboring structure at 609 Oronoco Street, which became the Hallowell School (Benjamin Hallowell tutored Robert E. Lee as he prepared to enter West Point.) Potts was the Secretary of the Potomac Canal Company under George Washington, the company's president. Washington dined at the house. The second owner was William Henry Fitzhugh. The Marquis de Lafayette visited in 1824 during his visit to the United States.

The home was rented by Fitzhugh to his relative, Henry Lee III ("Light-Horse Harry"), in 1811, at a time when Alexandria was still part of the District of Columbia. After being beaten in the 1812 Baltimore riots, Lee left the country and moved to the Caribbean, leaving his wife Anne Hill Carter Lee to raise their children (including Robert E. Lee). The family lived at the home until 1816, and in 1820, the now-widowed Anne moved back into the home with her children. The house was the boyhood home of Robert E. Lee, who was born at the family's Stratford Hall plantation in Montross, Virginia, and lived at the Potts-Fitzhugh House until he left for West Point in 1825. Lee later became a Confederate general.

Notable later residents includes Royd Sayer, the head of the Bureau of Mines under Franklin D. Roosevelt, and Ada Hitchcock MacLeish, who helped create the United Nations with her husband Archibald MacLeish, a poet and Librarian of Congress.

The home is on the U.S. National Register of Historic Places and the Virginia Landmarks Register. It was added to both registers in 1979.

The home was operated as a museum from 1967 to 2000, when the Lee-Jackson Foundation, the nonprofit that operated the museum, sold the site to Mark Kington, a managing director at a venture capital firm, and his wife Ann. It then again became a residence. It was sold again in July 2020 for $4.7 million and was then offered for sale again in 2021.

The home is among the oldest extant homes in Alexandria; a handful of other structures are older, namely the Ramsay House (built 1695–1751) (today, the Alexandria Visitor Center), Carlyle House (1753), Murray-Dick-Fawcett House (1775), Benjamin Dulany House (1784-1785), Colonel Michael Swope House (1784-1786), Fairfax-Moore-Montague House (mid-1780s), and the Lee-Fendall House (1785).

Spirit of 4-year-old Boy

The presence of a four-year-old boy is most often heard in the downstairs hall, right underneath the second-floor landing. During the daylight hours, the residents of the building often heard the running of the small feet of a young child, accompanied by the delightful sounds of giggles coming from upstairs and up and down the staircase leading to the upper floors of the mansion. These occurrences happened 2 to 3 times a day, almost every day for 6 straight months. The occurrences tapered off after the 6 months, though they still happened sporadically, much to the family’s disappointment.

Little Boy Playing Pranks

This little fellow still enjoys playing jokes on the living. Ringing the doorbell, moving objects around, and dropping snowflakes on visitors. The residents experienced an awkward social moment one day when their new neighbors came to visit. All of a sudden, their lady guest found herself getting wet from a snowfall that began about a foot above her head as she sat in the living room. She finally got up to leave, gracefully excusing herself because she was all wet. On another occasion, a missing cigarette lighter came flying out of nowhere to the feet of one of the residents.

The Black Dog

The apparition of a black dog with short legs, a long nose, and short floppy ears has been seen trotting around the back yard. More recently, people have observed the apparition of a young boy playing with the ghostly dog.

Other activity

The apparitions of two little girls have been seen, and the eerie sound of string music has been heard coming from the upstairs.