Ahlefeldts baroque palace had two storeys and was quite delicately built in half timber. The castle had, like it has today, two wings towards Kongevejen. The central part was a large hall, which, unlike the side wings went through both floors. It meant that you could only pass through the castle on the ground floor if you needed to go from one residential wing to the other.
Sorgenfri Castle became the property of King Christian VI in 1730 and was the summer residence for his son, later King Frederik V from 1742. The castle was extensively renovated in 1743-44 by architect Laurids the Thurah, who also built Kavalerbygningen (???) and the stables (in two storeys) along the entrance road to the castle. These two buildings are larger than the castle itself, but because of the architect's deliberate choice of perspective, you do not notice it from the road.
When Frederik V soon after became king, he left the castle to his aunt, the widow Countess Sophie Caroline of Ostfrisland in 1747. She let Thurah demolish the castle to the foundations and rebuild it in brick as a one-storey house with a mansard roof and spire. It is this rococo palace from 1757 which still stands. After Sophie Caroline's death the castle was left to the 12-year-old Crown Prince Frederik in 1766, who sold it in 1769.
The following owners were Jean Henri Desmercières and then Henrik Bolten. After the latter's bankrupcy Prince Frederik bought Sorgenfri Castle back in 1789, and asked the architect Peter Meyn to restore it in the 1791-94. This added the classicist gables on the facades and the residential wing, the Ladies wing, was built south of the Castle's main building. Inside, the Thurah rococo decorations were removed, and Nicolai Abildgaard assisted in the new, neo-classical decoration.
When Crown Prince Frederik died in 1805, the castle was passed on to his son, later King Christian VIII, who used it as a summer residence. After his death in 1848 the Dowager Queen Caroline Amalie stayed at the palace every summer until her death in 1881.
Frederik VII had left Sorgenfri to the state in 1856, and after 1881 it was empty for years. But in 1898 it was left to Prince Christian (later King Christian X) for the summer. King Christian X and Queen Alexandrine continued to live at the castle during the summer, and the king often was seen riding in the neighborhood. Prince Knud (Canute) married Princess Caroline-Mathilde in 1933, they were living in Kavalerfløjen, until after Christian X died in 1947, when they moved into the castle itself. Since Princess Caroline-Mathilde died in 1995, the castle's main building has been empty.
A small forest and a garden surround the castle. The garden was created in the 18th century in the French style with symmetrical beds, topiary shrubs and ornamental vases. Crown Prince Frederik reorganized the garden into English style with winding paths and romantic garden furniture as well as a well, a cave and gazebos. The latter, the Norwegian House and the Swiss House, were designed by Nicolai Abildgaard.
Although the royal family has been linked to Lyngby through the last 250 years and has left its mark on the city's self-perception, the name Kongens Lyngby does not refer to the royal residents at Sorgenfri Castle. The name is much older and was first mentioned in 1348. At that time large parts of Northern Sealand belonged to the Catholic Church (represented by Roskilde Cathedral and administered from Hjortsholm), and the name Lyngby was attached to several places. Store Lyngby belonged to Arresø church. "Our" Lyngby, however, was a part of the Crown estate, that is the King's estate, and perhaps it was to distinguish the town from the other Lyngby'er that it was called Kongens Lyngby.
Much later, when the municipality of Lyngby (since 1909 Lyngby-Taarbæk) was created, the name Kongens Lyngby was used especially by conservative circles to describe the city as opposed to the municipality. From 2000, Kongens Lyngby is the official name of the city, and the zip code was changed to 2800 Kongens Lyngby.