The current manor house is the second Harlaxton Manor. The first was built on a different site during the fourteenth century and was used as a hunting lodge by John of Gaunt. By 1475, the de Ligne family purchased the manor. The original manor house was deserted after 1780 and, after Gregory inherited it, was torn down in 1857.
The current house was built by Gregory from 1837 to 1845 and helped usher in a renaissance of Elizabethan architecture. Upon Gregory's death, the manor passed to his cousin George Gregory and then in 1860 to a distant relative, John Sherwin. Upon the death of Sherwin's wife in 1892, it passed to his godson Thomas Sherwin Pearson, who owned the house but allowed it to fall into disrepair. Abandoned by 1935, the manor was purchased in 1937 by Violet van der Elst, the widow of a painter and a businesswoman, who developed the first brushless shaving cream. She restored the manor and arranged for it to be wired for electricity.
In 1943, the Royal Air Force used the manor to house a company of the 1st Battalion of the British Airborne Division. Five years later, Lady van der Elst was forced to sell the manor and it was purchased by The Society of Jesus who used it as a novitiate. They sold the manor, while retaining rights to some of the lands, to Stanford University in 1965. The University of Evansville acquired the manor in 1971. Immediately after the purchase of the property, the University of Evansville began renovating the entire facility starting with the most critical items first.