The Pomeroy family, originally from France, came to Devon at the time of the Norman Conquest. By 1268 they had founded a borough called Bridgetown on their land bordering the River Dart, opposite Totnes. Records show that Henry Pomeroy paid 10 marks to enclose a deer park at Berry Pomeroy in 1207 and that the Pomeroys occupied a manor house in the village of Berry Pomeroy from at least the thirteenth century.
Berry Pomeroy Castle was built within the deer park on a rocky promontory overlooking the Gatcombe Brook. It is surrounded on three sides by steep slopes, but was vulnerable to attack from the south where the ground rises to a high ridge. On this side the Pomeroys dug into the rock to form a defensive ditch. The ditch has since been infilled, but traces of it survive towards the east.
The precise date when the castle was constructed is not recorded. The first written reference appears in a document dating from 1496, when a third of the castle was assigned to the widow of Richard Pomeroy.
In 1547 Thomas Pomeroy, who had fallen into deep financial problems, was forced to sell the castle, the manor and other nearby properties to Edward Seymour.
Edward Seymour was one of the most powerful and wealthy men in England who, over the years, acquired many titles and offices, including Viscount Beauchamp (1536), Earl of Hertford (1537), and first Duke of Somerset (1547). During the first two years of Edward VI's reign he acted as High Steward of England, Treasurer of the Exchequer, and Earl Marshall and Protector of the Realm. But he also acquired political enemies who overthrew him as Protector in 1549 and eventually saw him beheaded in 1552.
In 1553 Edward's oldest surviving son by his first marriage, Sir Edward Seymour, acquired Berry Pomeroy, and soon after made it his home. In about 1560 he began to replace the Pomeroy domestic buildings with a new house built in a more fashionable style for the period. This was a tall, compact country house with three wings or ranges around an inner courtyard. This house was built entirely within the old castle defences, which were left standing, and occupied the eastern half of the site. Edward was made Sheriff of Devon in 1583, and in 1588 was appointed by Queen Elizabeth to raise troops to defend the Devon shores from the Spanish.
Edward died in 1593 and the estate passed to his eldest son, another Edward. He had already been Sheriff of Devon (c.1583) and was made Vice-Admiral of Cornwall in 1586. He was MP for Devon from 1590 until 1611, and again Sheriff in 1595 and 1605. He embarked on an ambitious building programme to enlarge and improve the house at Berry Pomeroy Castle. A new wing containing state rooms was built along the north west side of the site, extending beyond the limits of the old defences. This new range looked across the Gatcombe Valley instead of inwards, and contained a parlour, a great chamber and a great hall with a fashionable, classical style loggia (covered walkway) placed in front of its main entrance. At the west end was a tall kitchen with spacious lodgings above. The top floor was occupied by a long gallery which stretched for 207ft (63m).
Terraces were begun at both ends of this range and another was planned on its north west side, but these terraces were abandoned before completion. Another range of rooms was planned to extend over the infilled moat along the west side of the site, but this was never started.
The site was abandoned some time between 1688 and 1701, and the buildings were shortly afterwards stripped of valuable building materials. The Seymour family moved from their unfinished grand house to a more up-to-date country house which they had built on another of their properties at Maiden Bradley in Wiltshire