Built in 1639, The Henry Whitfield House is the oldest remaining house in Connecticut. This Old Stone House was the home of Guilford's first minister, Henry Whitfield. In 1650, Henry and his family returned to England during the Counter Migrations. Henry died in 1657. The Whitfields later sold the Guilford house to the Thompson family, the property was used as a tenant farm until the 1770's.

From 1776 to 1900, the Griffing and Chittenden families (related through Marriage) owned the Whitfield House. Prominent Guilford citizens and successful business owners, they lived in the house at various times, rented it to tenant farmers, and possibly used the site for offices and warehouses. In the 1860s, they allowed St. George Catholic Parish to meet in the house until their church was built. The Chittendens heavily remodeled and expanded the building in 1868. Sarah Chittenden Clone sold the house to the State of Connecticut in 1900.