In 1438, John Brome, the Under Treasurer of England, bought the manor. It then passed to his son, Nicholas, who is thought to have built the East Range, which is the main entrance. Nicholas is also responsible for the extensive rebuilding of the nearby church of St Michael's, done as penance for killing the parish priest, a murder reputed to have taken place in the great house itself.
The house from this period was equipped with gun-ports, and possibly a drawbridge. When Nicholas Brome died in 1517, the house passed to his daughter, who married Sir Edmond Ferrers.
The house remained in the ownership of the Ferrers family until 1940 when it was purchased by Thomas Walker, a relative of the family who changed his name to Ferrers. His son, who inherited in 1970, sold the estate in 1980 to the National Trust.