In the 1840s, the Boultons made several additions to the house. These changes included: the enlargement of the drawing room; the elimination of three bedrooms to create a large second-floor assembly room; and an addition on the west end to house an office. William Boulton, who was mayor of Toronto four times and a Member of Parliament, might have needed such a space. He also enlarged the front hall, installed a new central staircase and the painted glass window depicting the family crest. This crest is a visual play on the name Boulton, as it shows a barrel pierced by an arrow, or bolt, with a hogshead of wine or tun. The family motto – Dux vitae ratio or The guide to life is reason, rests below the crest, a reminder to the Boultons of their British ancestry.
After Harriet Boulton remarried to her second husband, Goldwin Smith, the couple added a large library to the west of the house and replaced the Georgian style staircase with an angular, Victorian style oak stair. They also replaced the wooden front porch with a stone version.
