In 1775, Robert Harper chose this location for his house because it was well above the flood line, and he had already been driven from three houses because of floods. The location also has a most spectacular view. Mr. Harper (and his wife) died before the building was completed.

The Harper House was used as a tavern for about 20 years after the death of Mr. Harper. It is very possible that Meriwether Lewis stayed here in 1803 when he obtained supplies for the Lewis and Clark Expedition. Both George Washington and Thomas Jefferson are known to have stayed in the Harper House. U.S. Supreme Court Justice Noah Swayne definitely stayed here, since he owned the house - he married Sarah Ann Wager, one of Robert Harper's heirs.

Over the years, the home has gotten a reputation for having at least one, and possibly two ghosts. A friend of Harper's named Hamilton is reported to have died in the home after an accidental fall, causing many to believe his spirit still lingers. Still, others believe that the main ghost is that of Rachel Harper, Robert's wife. Rachel allegedly died from complications after falling from a ladder. She is sometimes seen looking out from the uppermost window of the home, gazing down toward the garden. Local lore states that Robert may have buried treasure somewhere in the garden, and Rachel is looking after it.