Parish records indicate that Episcopal services from the Book of Common Prayer have been held in Stony Brook since 1857. These services pre-date the present building by more than thirty years and were held in the Lower School House on Main street, which is still standing and is a private residence today.
Originally called St. James Chapel, the church was founded as a mission of St. James Church in St. James. Records of services were maintained at that church until 1952 when the name was changed to All Souls Episcopal Church and it became a mission of the Episcopal Diocese of Long Island.
The present church building was erected in 1889 from plans and under the direction of the noted architect Stanford White.
Two special points of interest are the stained glass window behind the altar and the Tracker pipe organ. The window, which was fabricated by the Mayer Company of New York City represents the ideal of Caritas (Charity) by depicting a wealthy woman giving a coin to a beggar.The window is dedicated in the memory of Jonas and Nancy Smith of the Smith family from whom Smithtown takes its name. It has recently been restored and given exterior protection, thanks to the generous donations from members of the All Souls parish.
The Tracker organ, which originally belonged to St. James Episcopal Church in St. James, was moved to All Souls in 1889. It had been built in 1855 by Henry Erban of New York City. Mr. Erban was one of the first to build organs of this type on this side of the Atlantic. The Tracker organ produces a very distinctive sound due to its unique construction. (Bach played on a Tracker.) It is one of the few remaining of its type in the area and is in the middle of an extended restoration project which will eventually include replacement of wind chests and leather bellows. The outer case, which was painted white many years ago, was originally natural grain wood and will be returned to that state when funds become available. Meanwhile this marvelous instrument is still used regularly at worship services.
In 2001 All Souls was placed on the National Historic Registry of historic sites.