Built in the mid 1800s, the home was originally built by Captain John Huling Geiger, who was Key West's first harbor pilot. Captain Geiger raised many children in the house, and planted beautiful tropical vegetation on the property. It was the beautiful plants which drew Audubon to it during his visit in 1832. Audubon took cuttings from the plants growing on the property, and used them as backgrounds in many works, including the White crowned Pigeon, which has the "Geiger tree" in the background.

At the beautiful Audubon House museum, repeated sightings have been reported of two Key West ghosts. Captain Geiger, a well known wrecker, and that of a young girl, possibly Hannah, who died here.

Captain Geiger has been witnessed on the balcony of his home looking out to sea, and he's also been seen on the grounds, perhaps guarding his buried fortune. A presence is also felt in the gallery, and footsteps are frequently heard on the stairs.

One of the most unsettling events on this property focuses on the portrait of Hannah, commissioned shortly after her death by her parents. The portrait possesses a certain sadness that many visitors to the museum found disturbing. So upsetting was the portrait, that eventually the oilogram was relocated to a more isolated corner of what was the Children's Room.

After the picture had been relocated, one of the manager's reported hearing the unmistakable sound of children talking and laughing.