The story of the Tagore family conveniently starts with Jayram. His father Panchanan had migrated to Calcutta from the district of Jessore now at Bangladesh and settled in Govindapur where the Fort William of Calcutta is presently located. Panchanan being of Priest class was called Thakur, later  transformed into Tagore because of his association with the foreigners.

The family of Jayram was displaced from their original home by the East India Co. which acquired the land at Govindapur for the shifting of Fort William. The Tagore family then shifted to Pathuriaghata, where they built up their family residence.

Out of the four descendents of Jayram, Nilmani Thakur secured a job under the East India Company and stayed at Cuttack for a long time. After taking retirement from his work Nilmani came to his home at Pathuriaghata but due to some misunderstanding he was compelled to leave the house and built a new home for his family. This new home was constructed at Jorasanko on a plot of land gifted by Baishnab Das Sett in 1784.

It may be mentioned that the Asiatic Society, the seat of Oriental learning was founded in the same year.