The originally Gothic castle called Kámen ("Stone") stands above the village of the same name, about 7 km south of the town Pacov, 570 m a.s.l.

The first mention of the castle comes from 1316. In 1450, during the reign of King George of Poděbrady, Kámen was a place of the peaceful negotiation between Czech feudalists and the King. At that time the castle owners had often changed hands. At the beginning of the 16th century the family of Malovec acquired it and rebuilt it into a chateau in the third quarter of the 17th century. Between 1860 - 1870 Kámen was reconstructed in the English Romantic Gothic style. The last changes were finished in 1916.

The castle was closed among the walls from three sides. Only the southern short part has been preserved. The present entrance is the gate within a Neo-Gothic tower built in the middle of the 19th century.
The oldest part of the castle is the tower palace from about 1300. It adjoins the four-winged building with the small courtyard. In the south part there is a tower with Knight's Hall connected with the castle chapel.