Between 1955 to 1957 the original Fontaine Hall was built to house the scholastics, the faculty was moved to St. Peter's, and the infirmary was relocated to Tyngsboro, Massachusetts. The farmers were moved to St. Peter's. This left the Hermitage empty.
Bill Duncan from Pleasant Valley was hired to demolish the old building. He decided that the fastest way to demolish the building was to hit it with a demolition ball; therefore, he attached one of the furnaces to a crane and proceeded to try to knock down the building. All he succeeded in doing was to put holes in the walls thus making the building unsafe to enter.
On October 23, 1958, at about 8 a.m., a fire broke out in the building. This created great confusion both on campus and across Rte.9 at Western Printing. Men were quickly put on the roof of the Western Printing building to water down any sparks which came from the Hermitage fire. Flames soared a hundred feet into the air and the building burned to the ground. No attempt was made to extinguish the flames. So ended a building which had served the Marist Brothers for more than 50 years.