Dashwood's legacy is prominent in Kirtlington. He built the imposing manor house at Kirtlington Park, gave his name to the Dashwood Arms pub in the centre of the village, and is buried in the Dashwood family chapel at St. Mary the Virgin Church in Kirtlington.

Exactly what caused the people of Kirtlington to take against him is obscure. Perhaps it was his support for the various Enclosure Acts which deprived the common people of their historic grazing rights, or perhaps it is the influence of his notorious uncle Sir Francis Dashwood who founded the 'Hellfire Club' at West Wycombe.

Whatever the reason was, like other unpopular members of the nobility such as the Tanfield's of Burford or the Wilcote's of North Leigh, after his death Sir James Dashwood's ghost apparently caused the people of Kirtlington so much trouble that they eventually called upon members of the church to 'lay' his ghost.

This was apparently done with great difficultly in the village pond. According to the legend, if the pond ever dries up then Dashwood's ghost will return and resume terrorising the village!