When Vanderbilt's house went up, most of the city's elite lived in unassuming homes such as J.P. Morgan's. The Houses produced a streetscape so monotonous that it provoked "Edith Wharton's ire. She wrote of "little, low studded rectangular New York with its universal chocolate-colored coating of the most hideous stone ever quarried," and bemoaned the lack of "towers, porticoes, fountains or perspectives."

Vanderbilt commissioned architect Richard Morris Hunt to create something that no doubt met Wharton's desire for something different.

With Vanderbilt's wife, Alva, directing the project, Hunt produced a French Renaissance-style house that came to be known as the "petit chateau." It quickly commanded not only local but also national attention. In 1885, the country's 75 top architects named it the third best building in the United States, after the United States Capitol and Trinity Church.

Within a few months of moving in, Alva hosted an extravagant costume ball that doubled as housewarming and social vindication. Mrs. Caroline Schermerhorn Astor, the doyenne of New York Society, had denied the Vanderbilt family invitations to her own affairs. This time, it was Alva who left Mrs. Astor off the list- but only after encouraging Astor's daughter to practice a dance for the party. In response, Astor paid her first house call to Mrs. Vanderbilt, which led to an invitation and social acceptance.

Just seven months later, the Vanderbilt family would further solidify their place at the center of elite society with the opening of the Metropolitan Opera house, which William's father, William H. Vanderbilt, had helped to establish.

The mansion went up for sale a few months after Vanderbilt's death in July 1920. It was demolished in 1926.

The site has been occupied over the years by two successive office towers.