Ferniehirst Castle consists of an extended and altered towerhouse, which incorporates the cellars from the 16th century castle, with larger wings and extentions. A large conical-roofed stairturret is corbelled out above the first floor level, and bartizans, with shot-holes crowning the top of the tower.

The original entrance leads to a stair known as the 'Left-Handed Staircase', the story being that when Sir Andrew Kerr, who was himself left-handed, returned from Flodden in 1513 he had his followers trained to use their weapons with their left hands. This is said to be the origin of 'Corrie-fisted' or 'Kerr handed'. The basement is vaulted, and the hall has a 16th-century fireplace.

Ferniehirst was a property of the Kerrs and first built by Sir Thomas Kerr in 1476 on the remains of an earlier foundation, but was sacked by the English in 1523. It was recaptured with French help in 1549, and the leader of the English garrison was beheaded. Sir Thomas Kerr, protector of Mary, Queen of Scots invaded England in 1570, hoping to have her released, but all that resulted was an raid on Scotland, during which Ferniehirst was damaged. James VI destroyed the castle in 1593 because of help given by the family to the Earl of Bothwell. The castle was rebuilt about 1598. As late as 1767 the house was occupied and used by the Lord Lothian of that day but even then it was showing signs of delapidation. Between 1934 and 1984 it was leased by the Scottish Youth Hostel Association, except for during World War II when it served as a army billet.

Purchased recently by Lord Lothian, Ferniehirst Castle has been restored. A bothy wing has been added and built on original foundation stones to serve as the private apartment of Lord Lothian and his Lady. When he is present his personal flag is flown from the flagpole atop the castle. His flag is blue with the resplendant sun in gold in the center.

A 'Green Lady' is said to haunt the castle in some tales, and unusual occurrences were reported during its time as a youth hostel, although the story is refuted.

Collection of portraits. Turret library.