Fort Matanzas was built by the Spanish in 1742 to guard Matanzas Inlet, the southern mouth of the Matanzas River, which could be used as a rear entrance to the city of St. Augustine. Such an approach avoided St. Augustine's primary defense system, centered at Castillo de San Marcos. In 1740, Gov. James Oglethorpe of Georgia used the inlet to blockade St. Augustine and launch a thirty-nine-day siege. St. Augustine endured the siege, but the episode convinced the Spanish that protecting the inlet was necessary to the security of the town. Under Gov. Manuel de Montiano's orders, construction of the fort began that year and was completed in 1742. Engineer Pedro Ruiz de Olano, who had worked on additions to the Castillo de San Marcos, designed the fortified observation tower. Convicts, slaves, and troops from Cuba were used as labor to erect the structure, which was sited on present-day Rattlesnake Island and had a commanding position over Matanzas Inlet.

This historic fort is believed to be haunted and is one of the most well-known sites in St. Augustine. It has been said that the location is haunted by several soldier apparitions dressed in Spanish and French uniforms. After all, "Matanzas" is Spanish for "slaughter," as it was named for the 245 Huguenots who were killed here in 1565. Suspicious footsteps have also been reported here.