Forts are considerable investments and are quite powerful against most units. However, virtually all siege equipment has greater range than even upgraded forts, allowing a patient player to simply whittle down a fort's health safely out of range. This can be done both on land with traditional siege equipment and on sea with bombardment ships and, in the later ages, heavy ships.
In addition, Forts have no ability to attack air units whatsoever, making them especially vulnerable in the Industrial Age and above unless supported by some sort of air defense.
Have a Fort in your capital to reinforce its defense.
GarrisonFoot Archers into a Fort until it has 6 attack units. It can prove to be deadly against invading troops.
Never build a Fort stranded all by itself. Build Towers and Air Defenses (if the prerequisites are met) to add extra defense to it.