The Jefferson County Poor House (also known as Colonial Manor) was built on the 70-acre county poor farm that was established in 1857 and closed in 1920. It was designed by architect Brinton B. Davis to replace the original white painted frame buildings, which by the early 1900s had become rather dilapidated. At that time, the county wanted to relocate the poor farm, but the people of Jeffersontown, who had grown fond of the poor farm residents, insisted that the county allow the poor farm to stay where it was.

Shortly after the original construction of the two-story Colonial Manor building, a smaller one-story building of much the same design was erected further back on the property for the black poor farm residents. After the poor farm closed, this building sat empty a number of years before it was renovated in 1946 to house Sunshine Lodge, a temporary shelter for displaced children. It later became the Jeffersontown Police Department and was razed in 1998 to make way for a new gymnasium.

The poor farm housed black and white, young and old, poor and homeless residents of Jefferson County. Those who were able worked the farm, which extended to the edge of what is now Skyview Park. A reservoir for the farm sat on what is now the Public Works property at the end of Bluebird Lane; when the poor farm closed, some residents used the reservoir as a swimming pool.

In the 1940s, Colonial Manor served as the headquarters of the Jefferson County Parks and Recreation Board. In 1970, the building housed the Jeffersontown Branch of the Louisville Free Public Library, until the library received a new building in 1996. At that time, Colonial Manor was going to be razed, but after outcry from local citizens, the building was renovated and made into the Daniel Ruckriegel Senior Citizens Center.