The Red Brick House was built by Catherine Crawley Ward Bouldin Chambliss, a strong woman who was a prominent member of Nottoway County, Virginia society during the latter part of the nineteenth century.  Born in 1818, she was the daughter of Benjamin Ward, Jr. a wealthy local farmer, and Sarah Fitzgerald.  She grew up at Bellefield, "the dear old home place," and was educated at a female academy in Chesterfield County.  

In 1844, Catherine married Louis Contesse Bouldin, an attorney from a prominent  Charlotte County family who was 22 years her senior. Louis and Catherine lived with her parents for at least several years, after which they purchased Heathcourt, an elegant plantation northwest of where the Town of Crewe would eventually be built. Heathcourt was constructed around 1780 and was known for its high ceilings, large number of windows in each room, and unusual panelling in the entrance hall. According to local lore, Littleberry Royal, the builder, said that he became more of a gentleman whenever he tied his horse in the yard of Heathcourt. While overseeing the farming operations at Heathcourt, Louis served as Commonwealth Attorney for Nottoway County and was also a member of the Virginia state senate.  He and Catherine had four children, Sarah, Benjamin, Mary, and Anne. 

In a letter to his brother, Louis commented on life at Heathcourt.  Catherine was due to have a baby any day, and, consequently,he could not travel until the child was born.  Their oldest child, Sarah, was at school in Petersburg and their son, Benjamin, was at home and had not had any formal education.  The weather was very hot and dry.  The wheat looked good, but rain would be needed soon in order for the grains to develop properly.

Louis died in 1862 leaving Catherine with four children and a 1300 acre plantation to manage.  Life was difficult.  In a letter to one of Louis' relatives, she noted that she had added to her holdings by purchasing her childhood home, but that she was perennially short of cash because of high taxes and crop failures.  She also noted the deaths of two of her daughters, Anne and Mary, and said that she was living with her brothers because she could not bear to live alone.

By 1881, all four of Catherine's children had died, and she decided to remarry. Her second husband, Thomas E. Chambliss, a farmer from Brunswich County, was 10 years younger than she.  Perhaps she hoped that a younger man could help manage her property.  Unfortunately, they experienced serious financial difficulties and were forced to sell hundreds of acres of land.  It is not known what happened to Thomas Chambliss.  His name simply stops appearing in legal documents after 1887 when they sold 349 acres of land to N. M. Osborne, the local representative of the Iron Belt Land Development Company. 

The following year, Catherine began to re-purchase some of the land that had been sold, including a 35 acre parcel in the Town of Crewe, which she divided into lots, perhaps hoping to cash in on the local land speculation. 

Throughout this period, Catherine lived at her old home place, Bellefield.  In his diary, T. E. Watson noted that he visited her there in 1894.  In 1896, Catherine went to visit her niece, Sally Shore, and became ill and died while there. T. E. Watson attended her burial at Bellefield and wrote that the weather was "hot as Mexico."

After her death, her niece's husband claimed that her illness had been a tremendous financial burden and that Catherine had promised them her Red Brick House in the new town of Crewe.  Her will, however, stated that the house should go to another niece, Maria Fitzgerald, for her lifetime and that it should pass to the Presbyterian church afterwards for use as a manse.  

Like Catherine, Maria Fitzgerald never lived in the Red Brick House.  She "was afraid of her shadow" and continued to live with her niece Sarah Fitzgerald and her husband, T. B. Oliver, a local entrepreneur.  The Red Brick House was rented to various tenants until Maria's death when it was sold at auction and purchased by T. B. Oliver.