Construction of the fort began in 1845 as part of a mid-19th century plan to defend the southeast coast through a series of forts after the War of 1812. Thompson Island, at the southwest tip of Key West, was selected as the site for the fort in 1822 and plans for the fort, drawn up by Simon Bernard and Joseph G. Totten, were approved in 1836. Two supporting batteries, Martello Towers, provided additional coverage, one of which exists today as the Martello Gallery-Key West Art and Historical Museum. The fort was named for United States President Zachary Taylor in November 1850, a few months after President Taylor's sudden death in office. The fort's foundation consists of oolitic limestone and New England granite. Its five-foot thick walls rose 50 feet above mean low water, and included two tiers of casemates plus a terreplein or barbette at the top. Three seaward curtains 495 feet between bastions, each containing 42 guns on three levels, were augmented by a land facing gorge. Troop barracks were built into this gorge with a capacity for 800 men. At either end of the barracks was a large gunpowder magazine while a Sally port was located in the center, connected to land by a 1200-foot causeway. Rainwater was collected in underground cisterns along the perimeter of the fort. Yellow fever epidemics and material shortages slowed construction of the fort, which continued throughout the 1850s. The Pensacola, Florida firm of Raiford and Abercrombie provided the bricks for Fort Zachary Taylor and Fort Jefferson, which was also under construction at the same time.

At the outset of the U.S. Civil War on 13 Jan. 1861, Union Captain John Milton Brannan, moved his 44 men of the First U.S. Artillery from Key West Barracks to Fort Taylor. His orders were to prevent the fort from falling into Confederate hands. The fort then became a key outpost to threaten blockade runners. Major William H. French arrived in April with his artillery unit.

In 1898, the fort was reduced down to the second floor and Battery Osceola was added to the south casemate. The battery consisted of two 12 inch artillery pieces. The Civil War ere pieces were used as fill, being buried within the new battery to save on materials. Battery Adair was added to the west casemate and included four 3-inch, 15-pounder Rapid Fire rifles.

The fort was heavily used again during the 1898 Spanish-American War, World Wars I and II, and the Cuban Missile Crisis.

Locals have reported a spirit called "Wendal" that is thought to haunt the entire fort. He has appeared in several photographs over the years. It is believed that he died of yellow fever during the Civil War.

It's not uncommon for military places to be haunted. The bloodshed of battle and other horrendous events often make room for ghostly encounters. Fort Zachary Taylor however was never attacked, but that doesn't mean the Fort escaped the severe taint of numerous deaths.

Yellow fever, tuberculosis, diphtheria and other dreaded diseases ravaged the people that lived and worked within it's walls, with thousands of people eventually dying over the years.

This massive toll, in such a concentrated area, has resulted in countless reported ghostly incidents by guests and workers at the old Key West Fort. Sighted Key West ghosts include soldiers mustering on the grounds, and people have reported hearing gunfire and whistles.

Even spookier are the haunted stories that revolve around the noon hour which is the time of day when hangings would occur. Unearthly screams from bodiless voices desperately begging for their lives have been reported, along with the audible sound of the hangman's trap door snapping open.

Unusual within the thick walls of the Fort are also unaccountable cold spots occurring in areas of the Fort where drafts are highly unlikely.