Native Americans have inhabited the St. John River drainage area in what is Maine and New Brunswick for at least 3,000 years. With the advent of white settlement, reservations were formed in Quebec and New Brunswick, but none were formed in the Maine portion of the territory.

Caswell originally known as the Township of F.R. 1 and organized as Pleasant Ridge Plantation in 1878. The town was reorganized in 1879 and was named after the first settler, R.S. Caswell. The Township consisted of a six- mile square land plot and was considered a good piece of land by Caswell. The land was rich and fertile with plenty of timber for use in building homes. Charles Herd, one of the first settlers after Caswell moved in with his family and other settlers followed.

Between the years 1899 and 1926, there were five small schools in Caswell. In 1927, the Pleasant Ridge School got its start, grades 1 through 8 attended this three- room school. In 1928, St Mary's School was built and in 1974, a new school was built for all of the Caswell students and St Mary's School was closed. The building is still standing but is now privately owned. Children from grades 1 through 7 now attend classes in the new Caswell Elementary School. Students from grade 8 up are transported to Limestone High School.

In 1890, hops were grown for making beer. Major crops consisted of potatoes, hay, oats and buckwheat. Sheep, cattle, swine and poultry were also raised at each of the small farms in Caswell. Today, much of the cropland is used for growing potatoes and peas. Oats are still grown on some farms.

Shortly after Loring Air Force Base in Limestone, Maine was constructed, a radar station was built in Caswell on land which the government purchased around 1951. The highest number of men assigned to the station was 150.