After years of partnering with his father in the nursery, Milton also became involved in local government, serving five terms as DuPage County Clerk and working for the Internal Revenue Service. In this later part of his life, Milton moved to Wheaton which had become the county seat after the infamous records raid in 1867.
Milton was married to a Miss Jane Barber and they had three children, one who died in infancy and twins Lewis and Carrie. Carrie never married and worked for her father in the County Clerk’s office. Her brother Lewis, like his father and grandfather, also became a County Clerk.Milton’s brother, another Lewis, became involved in government work as well. He moved out to Denver, Colorado, was elected to the Senate there, and was influential in a number of legislative issues for the fledgling state. Apparently serving the public was an Ellsworth family trait.
Milton died in 1896 at age 67 of cystitis, an inflammation of the bladder. He is buried in the Naperville Cemetery, along with father Lewis and even the Colorado brother Lewis, near the family obelisk.