In exploring the history of this building, I uncovered a tale of two families, one most fortunate, and the other less so.
A different house on the lot at 31 South Main Street can be traced back to Isaac Fitts, a hatter, who petitioned for forty feet on the River bank in 1726, that he might set a dwelling thereon, which he accomplished in 1727. The house was purchased by Timothy Souther in 1794, and stayed in the Souther family until 1860. It was long known as the “Souther” house, and was taken down in 1917.
In 1928, the Dr. Joseph Manning house, also built in 1727 just a few doors down the street, was moved to this location so that an automobile dealership could be constructed across from the Old Town Hall. In 1928, Richard W. Davis sold the lot to Millard J. Patterson with the condition that he could maintain the right to enter the building and conduct repairs until it was removed from the lot to its present location.
Doctor Joseph Manning
In 1726, Dr. Joseph Manning built a fine early Georgian home on South Main Street opposite the intersection with Elm Street.
Joseph Manning was born in 1703 in Ipswich. He graduated at Harvard College in 1725 and returned to his native town where he served for more than 50 years as a physician, eminent and favorably known. Doctor Manning was the father of the legendary Dr. John Manning, whose home on North Main Street still stands.

From the History of the Manning Families, written in 1902, I read the following about Dr. Joseph Manning and his home on South Main Street:
“Dr. Manning owned the lot which is nearly opposite to the present town house (the Old Town Hall on South Main) and put up the square edifice still standing there. To make a substantial wall upon the riverside he needed large stones. In the river bed a mile or so down (the lower falls by the County Street Bridge) there were boulders in abundance. Selecting at low tide one of these he would put a chain about it and so mark its position as to be able to find it with no other light but the stars and moon. At night the ebbing tide would find the wily doctor with his boat anchored over the rock which would soon after be grappled to the little skiff. Then as the sea wave came the lifting and wafting force of the water was all that was needed to place the boulder in the very spot where he wished to have it. Small wonder that passersby on the following morning, seeing a large stone lying where no stone had been the night before and looking like a vast meteorite which had fallen from the sky, should turn their eyes askance as the young doctor passed, and almost fancy they detected a whiff of brimstone in the air.”

Dr. Joseph Manning died in 1784 at the age of 80. His tomb is at the Old North Burial Ground in Ipswich, is located in the book Memento Mori, page 171, and on the map C, #87. The inscription reads:
“Erected to the memory of Doc. Joseph Manning and Elizabeth, his amiable Partner in Life upwards of 46 years who died Jan. 30, 1779, in the 71st year of her age. He mourned her loss until the 8th of May, 1784, and then died in the 80th year of his Age. The toil of life and pangs of death are o’er And care and pain and sickness are no more. They both were Plain and unaffected in their Manners, steady and Resolute in their Conduct Humane,temperate, Just, and Bountiful.”
Timothy Souther

I don’t know if Dr. Manning knew Timothy Souther, an unfortunate young man who arrived in town in 1763, unwelcome and unwanted. In the 18th Century, towns were responsible for the poor people within them, and measures were sometimes taken to relieve the town of responsibility for residents who were unable to provide for themselves. Timothy Souther arrived with his wife in 1763, and was “warned out.” The town’s lack of hospitality did not serve him well, and in the book Memento Mori, a grave at the Old North Burial Ground at location D-41 tells us his sad story: “Here lies the remains of Mr. Timothy Souther who departed this life August 5th, 1766, in the 27th year of his age.” His widow, Sarah Morton Souther was only 23 years old. She married widower Paul Little of Newbury on August 30, 1772, and died in Windham, Maine on September 26, 1797.
Almost 40 years later in 1792, we read that another Timothy Souther, a native of Haverhill was also “warned out.” In the previous year he married Elizabeth Badger, daughter of Daniel Badger and Phoebe Lakeman, from an old Ipswich family. Timothy Souther was able to buy part of a small house near the Choate Bridge for his family, but things did not go well for him. A grave at the Old North Burial Ground for three-month-old Charles Souther, who died in 1799 shows his parents as Timothy Souther and Elizabeth Badger.
This Timothy Souter died in the West Indies at 36 years of age in 1804. By then he had sold “half of the half” he owned, but his wife Elizabeth Badger Souther continued living in the northwest corner of the house until her death on December 31, 1841 in Ipswich at age 74.
Their son, also named Timothy Souther, was born in Ipswich on April 7, 1800. He appears to have done much better, and at one time owned a home on Meeting House Green where the Kaede Bed and Breakfast is today. He involved himself in the affairs of the town, and in 1829 this Timothy Souther became the collector of customs for the district and inspector of the revenue for the port of Ipswich at the old Custom House. He was caught up in a payback scandal, and in 1842 Souther moved with his family of five sons and two daughters to Alton, Illinois, where he served as the postmaster of that city from 1846 to 1854.
A mystery unraveled
The old Souther house near the bridge, or at least part of it, stayed in the family until 1860, and was always known as the Souther house. Thomas Franklin Waters stated that the Souther house was torn down shortly before 1917. In 1928, the lot with Dr. Joseph Manning’s fine old home on it was sold by Richard W. Davis to Millard J. Patterson with the condition that Davis would still own the building and would have the right to maintain the foundation and eaves, enter the building and conduct repairs as long as the building remained on the lot (2814-20). By 1930 the house had been moved and a new automotive dealership had taken its place. That building now is home to AnnTiques.




SOURCES:
- Ipswich in the Massachusetts Bay Colony, Vol I, by Thomas Franklin Waters: (*Timothy Souther house) (*Joseph Manning house)
- Ipswich in the Massachusetts Bay Colony, Vol. II by Thomas Franklin Waters
- Encyclopedia of the History of St. Louis: A Compendium of History …, Volume 4
- Journal of the Senate
- Genealogical and Family History of the State of Maine
- Souther Family Association
- Vital Records of Essex County
- Ipswich Revisited by Bill Varrell
- History of Essex County